The image format ".png" is not a good format to export, since it does not feature it's own alpha channel. It rather saves the transparency to pixels than to an alpha channel, which can make it hard to interpret the alpha the right way for some programs. It also uses compression, which means a loss of quality.
To my experience TARGA or OpenEXR are the best formats to use if you are going for a high quality.(large files)
The pixelation error does not necessarily have to result out of a wrong file format, it can also be the result of wrong image dimensions when rendering. An image or texture that covers more pixels in the render than in the original format is always going to be pixelated.
alex