You can see that the plaster walls are somewhat reflecting the floor (which is 45% glossy, 55% diffuse) and I neither changing the glossy setting in the Ray Visibility for the floor nor the walls fixes it. I've tried upping the roughness, doesn't work. Do you guys have a solution?
Reflectivity is controlled by surface roughness. The higher the roughness value, the less reflective the surface. Another thing to try is turn off reflective and refractive caustics in the render settings.
It appears this can be fixed by deselecting "Diffuse" in the Ray Visibility of the floor.