@moonboots I did already, it's there in my original question as a bolded edit at the very top and I upvoted your answer so that was the top response too.

For anyone else having a similar issue, sheen is causing the issue. I set my sheen and specularity to 0 and it bakes correctly now.