I have been using Blender for a while now and I notice something that seem unusual at least to me. An obvious format is JPG because it's very common despite its flaws. EXR is pretty much the opposite JPG with greater non-destructive editing capabilities but quite rare in availability online. If I want to render a still image using the factory settings, PNG is automatically selected and have to be changed to another format manually. I'm aware that user defined settings can be saved as the default to resolve this question. Why Blender has chosen PNG as the default format while the others are not?
TL;DR: It's a well supported file type that is a pretty good mix of file size, detail preservation, and alpha support.
I'm not a developer, so I can't say for sure, but I think there are a number of good reasons why PNG is the default format. It's by no means perfect, there is no perfect image format, they all have some benefits and disadvantages.
One of the most likely reasons is that PNGs are widely used and well supported. You won't run into another program that doesn't support PNGs. Being a software that can be used with other programs, it would make sense for the default file type to be a widely accepted one.
PNGs also have better compression that JPEGs for the average blender render. JPEGs are more commonly used for photos with soft transitions, as the lossy compression is much less obvious. Once you add an sharp details though, such as edges or text, the compression becomes an issue. That is most likely why JPEG is not the default.
While EXR has a number of advantages that make it great for compositing and serious rendering, for the average blender user it's just not necessary. The downsides of EXR is the file size and the potential conflicts with other software or websites. I would guess that most blender users never even touch EXR, and thus the benefits would be lost, yes some disadvantages would be noticed.
Overall, I think PNG was chosen as its pretty good at everything. It has an alpha channel, lossless compression, decent color range, and relatively small file sizes. While you could find a file type better and several of these criteria, you won't find a well supported one that is better than PNG at everything.
I can't speak of the historical or epistemological reasons for the PNG format as default format, I figure is a mix of convenience and file size... blender is used for many different things...
But you are right in trying to avoid PNG. It does create unnecessary problems that can be avoided by using EXR. Keep your information as Scene Referred: Uncompressed, linear, and with proper associated Alpha channel (on top of the option to save other passes, like depth, undistorted).
To set EXR as the default save format:
Open blender, set the render output format to OpenEXR
Save the Startup File using Ctrl+U.
Next time you open blender it will default to saving as OpenEXR.
As well as the alpha channel (previously mentioned), PNG is lossless while JPEG is naturally lossy.
PNG is my natural default for saving images in anything. PNG is very common now - if you are interested you can research the history of PNG and how the humble GIF played a part in its origin.