One key concept that might help in clarifying any misunderstandings is this:
- A voltage is the potential difference between two points in a circuit, it is not an absolute value of any physical characteristic at a single point in a circuit. Thus, there is no absolute potential involved, it is relative value, a difference.
How this applies:
The "ground" of the Arduino is the point (or PCB trace, to simplify the concept) relative to which, the potential on the "Vcc" trace of the Arduino is measured. Thus, when powered by the USB cable, the Arduino's "ground" is the measurement basis, and is incidentally at the same potential as the "ground" of the computer whose USB port is being used.
The "negative" of the battery is merely one of two points across the battery, between which the potential difference i.e. the voltage of the battery is measured. Relative to the rest of the (electrical) universe, the "negative" terminal of the battery is floating, i.e. has no specific relative value until this so-called negative terminal of the battery is connected to a circuit.
When you connect the negative terminal of your battery to the ground of the Arduino, you are providing a reference value for the battery, relative to the Arduino. Thus, only while such a connection is made, the "positive" terminal of the battery is at a potential of "battery voltage" compared to the Arduino's zero level or ground.
If, instead, you connected the battery "positive" to the Arduino ground line, then the other terminal of the battery would be at a negative value compared to the Arduino's zero level or ground.
The battery, or for that matter any arbitrary voltage, is relative to the reference you provide. It is a difference, from your defined base point in the circuit, not an absolute value, to reiterate the basic principle mentioned at the start here.
Best Answer
What Wouter van Ooijen said, but also the bottom halves of the two circuits are guaranteed to be at the same potential. You seem to be wondering why anyone would ever care about this.
To demonstrate why this important, go find a stereo or something with an audio plug coming out of it like you'd plug into an iPod or such. Turn up the volume just a bit, and touch just the tip of the connector. Hear that buzz? It's because your body is picking up all sorts of electrical fields from the electronics in your house and nature. These are at different potentials from the stereo, and it amplifies them as sound and you can hear it.
Now, touch all the parts on the plug. Squeeze so you get a good connection. The buzz is gone! Why? Because now you have a common ground. All those fields are still there, but now your body and the stereo are at the same potential, so there's no difference between you and the radio, so you don't hear anything. (Or not as much, anyway. The connection isn't infinitely conductive, so you will still hear something, if you turn the volume up more.)
So, your stereo amplifies the voltage difference between its idea of "ground" and the left and right channels on the connector. But if something (your body, an iPod, whatever) isn't connected to the stereo's ground, then the stereo will be amplifying the voltage output by the iPod and also the difference in ground between the iPod and the stereo, which is likely some crazy random noise and 60/50 Hz hum from powerlines. Giving the two a common ground reduces this difference to nearly zero, so that the two circuits can have a common reference, so they can agree what "4V" means.