I have had the same with the denoiser when an area has had a high frequency noise. The denoiser can't know what pixels are fireflies and what are correctly calculated pixels. It just looks at the pixels and see a lot of white dots and think it’s the real thing. Same thing happens here where you have a high frequency texture and the denoiser considers an area that is too large into account when denoising.
Try setting the radius to something low, like 4 and the strength down to .2. Another alternative would be to deactivate denoising for diffuse direct to see if that helps. But I have never tried that.