TL;DR? --> read the bold parts only ;)
The question could also have been called: What is the difference between the settings "file format" in "output" and "Format" in "Encoding"? It is very close to this question and I think I got the answer from there.
But I am still left with my actual problem, which is this: from the VSE I want to render out an h.264 encoded video in a .mov container. as far as I see there are two options to achieve this:
1) select H.264 or MPEG in "output"--"file format to save the rendered image as" and then in "Encoding" select "Quicktime" as "Format" and "H.264" as "Codec".
(--> this will give you the option to select a target bitrate for the output file (nice!))
2) select "Qicktime" in "output"--"file format to save the rendered image as" and then H.264 in the new option "Video Codec" that pops up right below.
(--> this will give you a setting for the "Quality percentage" (awkward!))
When I compare similarly sized output files generated by 1) and 2), method 1) looks waaaaay better! (example: 2000 kb/s in 1) gives 4MB file vs. 25% in 2) 7MB file and still file 1) looks considerably better!)
The only problem with 1) is that the resulting .mov file is not fully compatible with OS X whereas file 2) is.
(Also, selecting "lossless output" in 1) generates a file that cannot be played with QT player in OS X (VLC will play it))
Why are 1) and 2) different (is it the VBR for only 1)?) and what should I do to get a high-quality, low-size fully QT-compatible file rendered out?