Most of the time when I try to move a vertex or an edge, instead of translating that, it extrudes it, it creates another face or, vertiex or whatever.
I do not really get why it behaves like that. I hope I explained it in depth. I am sorry if this is a duplicate post but I could not find anything!
Paul Gonet solved your problem in the comments, but here is how you can spot the problem yourself, and a bit more detail for future reference.
You are not making a new vertex but rather moving only one vertex when there are two in the same place.
You can easily spot that problem, double vertices, by looking at the selection color of the edges. In your image below I highlighted in red the edge you should watch out for. It is not selected, that means that it is not connected to the vertex you do have selected (and thus there is a double vertex hiding there).
Whenever you select a vertex all the edges connected to it should be the selected color (the default is orange) around the vertex (the whole edge will not change color, the selected color fades out along the edge).
You can select the second vertex that is hiding there, change the shading mode to Wireframe in the 3D view header or by pressing Z. Now select one of the vertices that are there, then hold Shift and select the second.
You can also use border select B, circle select C, or lasso select (hold Ctrl and drag with the LMB 
held) to easily select double vertices.
To get rid of the problem you can use the Remove Doubles operator or Merge them. Both are accessible in the Tools tab of the tool shelf and many other ways.
Remove Doubles (W then R) will go through every vertex that is selected and see if there are any within its merge distance if there are, then it will merge them.
Merge (AltM) can be thought of as a manual version of the remove doubles command, but with much more control. For a scenario like you have where the two vertices are in the exact same place, any of the first three options will work fine (and do the same thing).
