Now I know I know the collision and the cloth does collide with the mesh but in same frames of me animatiting it gets in the mesh which really bothers me . Am I rigging it the wrong way ? I rigged the cloth the same way I rigged the mesh with automatic weights so that it follows the movement of the body
I think you first should increase the Cloth Collision Quality. See How can I make sure that simulated cloth doesn't go through other objects?