When building a physics simulation it's important to ensure it fits with 'real world' proportions and properties in order to achieve a realistic simulation.
Firstly, ensure your surface Normals are pointing in the right direction. You can view your normals in Edit mode (Tab), opening the Properties side-bar (N), scroll down to Mesh Display and enable 'Display face normals as lines'. This will display short blue lines at the centre of each face to indicate the face 'direction'. For physics collisions to correctly detect 'inside' and 'outside' you should ensure your normals are all pointing 'out'. If you need to adjust the normals there is an option to 'Flip Normals' (change the selected faces to point the other direction).
Ensure your objects are all at a 'realistic' scale in the real world - if you haven't set a specific scale on your scene (eg, Metres, Millimetres, etc.) then you can assume 1 Blender Unit (the units that your dimensions of objects are measured in) will represent 1 metre. Note that the scale is important for the physics simulation - the forces resultant from a falling 0.01m x 0.01m x 0.1m block (eg, similar size to a Jenga block) will be considerably different to a 1m x 1m x 10m wooden beam - even though the proportions are similar.
If your objects have had their 'scale' adjusted (using the Scale tool) you should ensure you 'Apply Scale' to have their new scale affect the simulation. This can be acheived by selecting the mesh in Object mode, pressing Ctrl+A and select Scale.
Finally, set the object's Mass to a realistic value for the behaviour of that object you desire. This is available in the Physics panel of the toolbar panel on the left-hand side (T) by clicking Calculate Mass. This allows you to select the material of that mesh to use in the physics simulation, such as Cork, Iron, Brass, Concrete, etc. - or set a Custom density if none of the preset ones are suitable.
Note that it can be beneficial to increase the Steps Per Second and Solver Iterations settings above their defaults (60 steps per second and 10 iterations) but care must be taken when increasing them too high as higher values result in shorter "timeslices" each time Blender evaluates a "step" in the simulation. Shorter timeslices result in much smaller forces and motions and the precision can result in rounding errors, adversely affecting the accuracy and realism of the simulation.