OK - Imagine trying to stretch a rubber band - it's impossible to do it with only one finger, right? It doesn't matter how big or small you try to drag the top to, you still have to have a finger on the bottom.
So it is with electricity: In order for a voltage to be either a big number or a small one, it has to be relative to something else, and in this case our 'bottom finger' is the neutral. Just like a rubber band, the more tension (which also happens to be a term used for voltage sometimes) there is if you let go, the bigger the thwack you get on your thumb (normally accompanied by swearing and slight pain). If you stretch your rubber band up as far as it will go, keep the top finger in position but then move the bottom one up so it's flat again, you've moved your imaginary voltage up and up, but there's no actual thwack left as it's all at the same level. This principle (which I hope has made things clearer) we call Potential Difference (pd) and we use the unit of Volts to measure it.
And the reason that in the electrical sense we keep our bottom fingers stapled to the Earth is because it's the safest thing to do (and easiest and most prolific conductor that we have).