It's not the material that's the issue here, it's that geometry nodes is merging the geometry of the mesh with another copy of the same geometry that has the default material (which is bright and fairly shiny) and that's what's causing the white reflection.

This shows what's happening - the above image shows the result when replacing the material (actually, all the materials) with a shadeless green, it ignores light and only displays green, and yet the white faces STILL show up with this node tree - because when the extruded mesh geometry was passed to two Set Material nodes, it was duplicated, and then when those two sets of geometry are merged, they keep the default material in whatever location was not selected. By displaying just one of the set material nodes (the Sides in this case) we can see that the faces are still set to the default material. I'm not actually certain why this only causes problems on the extruded faces (perhaps some sort of z-ordering issue), but that is the problem here.
It can be resolved by using both set material nodes in series, and not performing a join material node, like so: 
This sets the material in both places, but doesn't duplicate any geometry.