I have a plane with subsurf modifier, And a circle to cut(knife project) with. The hole is cut, but it leaves mesh(artifacts) inside of it.
Creasing (Ctrl E) works on the outer edges only, leaving the artifacts.
Why is this happening?
The knife tool is creating ngons, which are faces with more than 4 vertices.
Ngons and tris do not work very well with the subdivision surface modifier. See When should N-Gons be used, and when shouldn't they?
When working with subserf, you pretty much always want to use quads (faces with four vertices) exclusively.
Instead of knife project, you could make a rough circle shape out of quads and then use the subserf to round it out:
Inset the plane (I).
Then in edge select mode (⎈ Ctrl↹ Tab), ⎈ Ctrl⎇ Alt⇧ Shift RMB on the edge ring created by the inset to deselect it.
Switch to vertex select mode (⎈ Ctrl↹ Tab) and delete the center vertex.
Make the inner loop circular with the To sphere operator (⎇ Alt⇧ ShiftS), or the circle operator of the bundled loop tools addon.
Extrude the shape to make it 3D (if you want), and add a constraining loop cut to make the subserf stay closer to the shape of the base geometry (⎈ CtrlR+E to make the new cut aligned to the existing edge loops, and F to pick between the edge loops you want to align to).
Make the outer edges of the cube/plane sharp with edge crease (⇧ ShiftE).