I have a flat plane I want to distribute some object on. The object should be standing straight up. There should be some slight variation in the Z rotation, while still standing straight.
Go to your particle object's original instance, enter edit mode and rotate it accordingly. Generally speaking if your object as no rotation or if you apply it's rotation, it should be "laying down" in the XY plane (don't ask me why I also find this extremely annoying and counter intuitive) for it to be "standing up straight" in the particle system.
Alternatively in your particle system in the Render Panel activate the Rotation option if you do not want to alter the objects shape and prefer to rotate it in object mode, and the adjust the particles rotation in object mode.
You can then use the options under the Properties Window > Particles > Rotation Phase Random or Initial Orientation Random to add variation to your particles.