My suggestion would be to set the objects that you don't want to appear in the shadow layer as a holdout. This would only be possible in Blender 2.8 and above and would make a transparent cut-out in the shadow where that geometry occludes the shadow.
Create a view for the Shadow.
Place the geometry in its own collection and the shadow catcher in its own ...
How to use the Temporal Denoiser
Step 01: Pack Denoising Data
Enable denoising data in View Layer > Passes > Denoising Data
Set file format to OpenEXR Multilayer (so Blender packs the denoising data)
Render your animation
Step 02: Denoising