It's probably better to ask a question like this on a dedicated programmers Q&A site since people there are more likely to focus on these things. Anyways, here are a few considerations that I am aware of.
When textures are not powers of 2, you(or the software you use) will usually have to use special extensions made by the hardware/driver manufacturers if you want them to work consistently.
For instance, OES_texture_npot or ARB_texture_non_power_of_two.
Using special GPU extensions to increase a programs abilities can also reduce compatibility since not every manufacturer is guaranteed to support them all and older machines from the same manufacturer may not support them either. This means that the software may run fine on your machine but not on the computers of other people.
Here is an excerpt from the PowerVR Performance Recommendations guide. This is the sort of thing that you will see in various documents from other GPU manufactures as well but the specific details will change slightly between them(ATI, nVdia, PowerVR, Mali, etc..) and even between different generations of cards made by the same company.
Finally, a few additional points should be considered when using NPOT
textures: • POT textures should be favoured over NPOT textures for the
majority of use cases as this gives the best opportunity for the
hardware and driver to work optimally. • A 512x128 texture will
qualify as a POT texture, not an NPOT texture; rectangular POT
textures are fully supported. • 2D applications (such as a browser or
other application rendering UI elements where an NPOT texture is
displayed with a one-to-one texel to pixel mapping) should see little
performance loss from the use of NPOT textures other than possibly at
upload time. • To ensure that texture upload can be optimally
performed by the hardware, use textures where both dimensions are
multiples of 32 pixels. • The use of NPOT textures may cause a drop in
performance during 3D rendering. This can vary depending upon MIP-map
levels, size of the texture, the texture’s usage and the target
platform.
Basically, if you stick with powers of 2 then you are guaranteed multi-platform support and good performance, otherwise you may have problems when you try and work with a different machine other than the one you started on.
//------------------------------------------------------
I suppose as a metaphor, we could look at this as if textures and model data are like shipping containers used to transport goods on boats, trains and highways.
If all the containers are the same, then everyone involved knows what to expect, and what to do when they encounter a container that must be shipped.
All the cranes will be able to grab it, and the container will fit into the same slot as all the rest.
If the container is different then the rest of them then there will likely be problems since none of the equipment was designed to deal with this different format and none of the people involved were trained to deal with that situation.
Update:
Here something that was recently posted on the Blender Bug Tracker.
