Often when a mesh light is visible in the camera, the edges are not anti-aliased.
No matter how many samples I do, the lamp border always looks jagged and pixelated.
Is this a rendering error? How can I get around this issue?
And zoomed in 4x:
This is not a rendering artifact, everything is behaving exactly as designed.
First an explanation of why these edges appear aliased, but if all you want is to solve this issue and don't care about the background info, scroll down to the Solution.
Cycles works by randomly shooting rays from the camera into the scene. It does this many times per pixel, and gets the resulting color of the pixel from the average color of samples in that pixel.
Take for example three adjacent pixels, rendered with 20 samples:
The green shape is some surface, the white part is a light source, and the red dots are the sampled points.
If the white light source has an intensity of 1.0:
So because the final pixel color is the average color of the samples, what happens when we have a light source with an intensity greater than 1.0? Say, 4.0:
So if our light source is very bright (as it often has to be due to the natural quadratic falloff), the pixels on and immediately around the light will be very bright as well, since the average sample color is bright.
That's the theory, and here's proof:
A (zoomed) render of some rotated planes with varying intensity of Emission shaders:
The intensity starts from 0.5 and doubles until the second last plane (intensity of 16), then jumps to 1000.0 to show an exaggerated effect.
Below the horizontal line the intensity has been divided by 4 - so starting from 0.125, reaching 4.0 and jumping to 250.0.
Notice that the more the intensity increases, the more aliased it appears.
There's an easy way to solve this issue: Simply get the light source to appear as regular white (RGB: 1.0, 1.0, 1.0) to the camera, but still let it be brighter to non-camera rays so that it can light the scene properly.
To do this, use the Is Camera Ray output of the Light Path node as the Fac for a MixRGB node where two values are connected to the colors (the top input being the actual light intensity, and the bottom input being the intensity the camera will see):
Which results in:
But as you can see, because our light was colored it now looks a bit dull.
To fix this, we just need to use a similar light path trick for the color input to make the light appear white to the camera, but the original color to the scene:
click for full size
Note: The Light Color node in the screenshots is simply a node group containing some nodes which only allow light to be emitted from the front side of the mesh.
So there ya go! Nice anti-aliased light sources :)
The only down side of this method is if you are relying on those brighter-than-white values for compositing glare/glow - but usually the glare/glow itself will hide those nasty aliased edges.
As you note in your own answer, this happens because the light source is much brighter than the brightest white color representable in an 8-bit RGB image, so even pixels that only partially overlap the light source will be so bright that they get drawn as pure white.
The reason you don't see this happening in real-world photos is because the point spread function of a real-world camera is never a perfectly sharp point, and so some of the light from the bright source will spread into neighboring pixels as glare.
You can simulate this effect in Blender by adding a glare node to your image.1 For merely smoothing the edges of bright light sources, only a very small amount of glare should be needed; for example, try setting the mode to "fog glow" and the mix value to something like -0.9 or less.
1) I hope, at least. I'm not actually an experienced Blender user, so I'm just going by what the manual says, plus some general knowledge of optics. Please let me know if it doesn't work.
Use Blender's Color Management to get that aliasing back. It isn't gone until you save a render out to a low dynamic range format. Simply use a curve to reign in the peak values to restore anti-aliasing.