I tried rendering an object that has a mixed glass and diffuse shader and also a background image but after adding a holdout shader in image 2 to make the object transparent to see the background image the glass shader becomes darker.
That's because the way you have it set up, it's going to be transparent on the edges and solid in the middle. Switch the top and bottom slots, and it should fix it!
EDIT: The Holdout shader also only uses what you have set for your world background. It's going to show up as a basically a hole to show the background through, with no consideration for any other object or faces behind it. If you simply want transparency, use the Transparent node instead of Holdout.
(It's doing what you you want, this is just how it looks in preview; see how it looks in render, then switch the top and bottom slots to see if it looks better, as well as also hookup up your mix factor back into your original mix; it can hook up to more things at once!)
EDIT 10/19: By switch the slots, I mean the sections that say 'shader'. When you use a mix factor, it's telling Blender to make all the white one of the shaders and all the black the other shader. Any mix is a combination of the two. Fresnel as a mix mask looks black where it's facing the camera, and white where it's angled away, and a fairly short falloff between the two. This means if it looks funny, you might simply have it hooked up opposite from the way you want, where it's hooked into the little green dots that say 'shader'. To further this note, you don't have the same mix factor in your addition with the holdout. You just moved it to the second mix. For better results, plug the Fresnel Fac into the the gray dot that says Fac on both mix shaders.