From my comment
Bones/Joints are much better suited for movement which rotates around a pivot point, namely the eyelids and the jaw.
The Jaw will interpolate strangely between open and closed if it is just shifting between a shapekey. A bone would ensure a rotation similiar to the real world jaw bone.
Shapekeys are best suited for the eyebrows, mouthcorners and and other movements which rely more on muscle contractions instead of mechanical relations.
Edit: To understand the images illustrating this post, you must note, that I have used a custom weight paint range. Every vertex weight above 0.000001 looks similiar to the standard weight settings. However, a weight of 0.0 is set to white. I did this to be able to tell the difference between a very small weight (e.g. 0.00001) and actually no weights, which is not possible with the standard mapping of blue.
Weight Painting the Jaw
- Make sure all of your head geometry is associated with the head bone.
- Add the jaw bone, make sure that
- there is a vertex group named exactly like the jaw bone
- the jaw bone has deform enabled
- the jaw bone is part of the armature used by the the mesh (via the armature modifier)
- your geometry is not unnecessarily dense (it seemed you have applied the subsurf modifier, this makes the process very difficult. I would unsubsurf the model 1 step with the decimate modifier.)
- in weight paint mode, Auto-Normalize is checked
- Lock all groups except for the head and jaw vertex group.
- Roughly paint the jaw to the jaw bone.
- Now open the jaw with the bone and make sure the complete jaw is fully assigned to the jaw bone.
- Now choose the the mix brush and paint weights of approximately
0.5 on the corners of the mouth. Choose other quantized weight for other areas if needed. Roughly block out the correct shape. You can raise the mouth corners by adding weights to the head vertex group and you can lower them, by adding weights to the jaw group.
- Using the blur brush smooth the area. Then use the add brush with a very fine Strength to shift single vertices untill the desired shape is reached.
Image shows the head's weight group.