Dance Off | nCloth

Trying up nClothes functionality on rigged character. Its quite heavy on the simulation even when I tried to reduce the polygons, it still heavy. Did some adjustment to lighten it up. The adjustment that I done is not getting rid of the nCloth and Passive collider after I done and bake the animation. The problem I face is that when I removes the two things, the shirt mesh went back to its original scale that is too big for the rigged character. The only thing i can think why this happens is because i might have forgotten to freeze transformation for the shirt mesh. Need to be more careful next time and don't forget any important steps.