Is there any way i can have cleaner edges out of the cryptomatte image output ? The outline is just not clean enough.
I assume your screenshot shows what you would see in the image editor if you have rendered cryptomatte passes, and view them directly. In that case, they look absolutely horrible:
However, that's not how they are supposed to be used. The mattes have to be extracted from the created channels first using the
Cryptomatte node. You can find it under
Cryptomatte in the Compositor. Connect the nodes like in the screenshot below to extract the object mattes:
If you need to extract material mattes, connect the three material Cryptomatte outputs to the Cryptomatte node instead.
Next comes the tricky part: The Cryptomatte render passes contain all mattes at once using an internal ID. That ID needs to be set in the node. Since these are not simple 1, 2, 3, etc numbers, you have to pick the correct ID using the color picker. That is also the reason why in the screenshot above I have used two Image Editors, one showing the Composite, the other the Viewer node. I can see both images, the Pick one through the Viewer, and the Image through the Composite node.
Now, use the color picker to grab the ID from the bottom image (the one that shows the Pick image):
If you check what's in the Matte socket output, you will see that the created Matte is perfectly anti-aliased:
Of course, this anti-aliasing quality is dependent on the scene samples. But the matte quality always matches the image quality.
Two things I would add to what's been shared so far:
1. You will also see a difference in edge quality based upon Cycles Render Properties>Film>Pixel Filter settings. Filter "Type" and pixel "Width" will give you wildly different edges, ranging from super crunchy to sharp and smooth to blurry. Automatically, you should get mattes that match your renders.
2. A perfect matte will FAIL on edges if your subject and background are merged like in a photo, live video recording, or a 3D scene that is not done in separate subject / background layer passes. This foundational video by CG Cookie really opened my eyes to this and how to workaround it when you don't have separate render layers: Working with Alpha Edges for Compositing in Blender. Although the video is from 2013, way before Cryptomattes, the principles and techniques still apply and can be combined wonderfully with Cryptomattes. Basically, you have to use choke mattes and inpaint nodes to separate the subject from the background and decontaminate the edges from color blending. I hope this helps you!