Every time I use glass material in Cycles render with diffuse or any other material with the help of mix shader, I always see 'fireflies' (small random illuminated pixels). I have already tried removing caustics after following BlenderGuru.
What helped me most times is increasing the size of the light source. At default, e.g. point lamps have a size of 0.1 units, so when calculating caustics blender just has rounding errors, I guess.
Of course, raising the size of your light source higher and higher will produce softer and softer shadows, so you have to play with the values.
Another guess: Don't lighten your scene with objects with emission materials alone – they will not produce enough light. Add standard point or area lights where necessary.