I saw a response to the question "Render Glass like materials with Alpha Channel to composite in external app" that you can check a transparent glass option under film to make the background image visible through the object
Quote from linked answer:
Newer builds of Cycles (post 2.79, currently available from the buildbot) include a native functionality for this
The functionality is not included in 2.79, but in builds made after the official 2.79 release.
If you're feeling adventurous, those builds can be found at https://builder.blender.org/download/
Note! Using non-released builds may corrupt your .blend-files and may also cause them to not work with officially released builds.