I'd like to turn my 500,000 poly model into low poly to make it game usuable. However I cannot bake due to reasons. How can I do this without losing noticable detail and destroying the uv map
If baking is of the table because of the 'reasons' you've mentioned, there is simply no way to reduce the poly-count AND retain all the relevant detail. Less information to store and process means... well, less information.