Every time I create new file this file keeps opening:
I would guess you've had that project open, deliberately, since installing the new Blender? If so, you've probably pressed Ctrl+U, which saves the opened file as your start-up scene. You'll need to delete all objects and set it up as much as you can remember like the normal start-up scene, then press Ctrl+U, and say "yes" when it asks you if you want to "Save Start-up Scene."
The issue is that a file was saved as startup file. To fix it you can load factory settings and save the file as the new startup file.
Alternatively you can delete the startup.blend file from the disk and let blender create a new one.
Depending on your operating system it will be stored in different places.
And read the manual page for the startup file.(https://docs.blender.org/manual/en/latest/data_system/files/startup_file.html)