You mentioned two possible issues: type of ground in use and direction of of current flow (source of the electrons).
There are four common types of ground:
- Earth - Literally, a copper rod shoved deep into the ground.
- Chassis - The common connection using the frame of something.
- Digital - The common connection for digital circuits.
- Analog - The common connection for analog circuits.
Realistically, all of these grounds are the same, but they are usually kept separate for the most part, only being connected together in a single place to avoid loops, and often using an inductor. It is called "ground" because of the original reference point being the actual ground, as in the a rod shoved into the earth. However, anything not connected to the earth (such as a battery powered device, or a vehicle) still employs a common node commonly called a ground.
When you use the negative terminal of the battery as a "ground" it is really just the common node (return) of the circuit. A circuit has to have a closed loop for current to flow, and using the negative terminal ensures that all of the components have an identical reference point.
Occasionally, separate portions of the circuit will be completely isolated, each with its own "ground," but again, this really just means the common node of that portion of the circuit.
As far as the "source of the electrons," that is a tricky matter to discuss. Typically, in the US at least, current is shown flowing from positive to negative. While this is not entirely accurate, that is the convention used for modern circuit design. Take a look at the answers to this question:
Does electricity go from negative to positive or vice versa?
Your voltage regulator is placed at the positive lead because that is the convention used to design the components - nothing too mystical about it. It is the positive voltage potential that is being regulated. The negative terminal is acting as the reference point (common node) in the circuit.
In the example of powering an LED from the 9V battery, it doesn't matter if the resistor was placed between the positive terminal and the LED, or the LED and the negative terminal. It is not "regulating the voltage" so much as restricting the current. What does matter is the orientation of the LED. The anode has to have a higher potential (voltage) than the cathode for current to flow, so the LED anode is connected towards the positive terminal while the cathode is connected towards the negative terminal.
The position of the resistor is insignificant, it just needs to be there because the current flowing through the LED at 9V would immediately burn it up. The LED more likely drops 2 or 3V. The remaining voltage is dropped across the resistor, and it is this value that determines the current through the LED according to Ohm's law: V = I*R.
I_LED = ( V_battery - V_LED ) / R, where V_battery - V_LED = V_resistor.
Best Answer
Assuming you're talking about round-type (cylindrical) batteries, such as D, AA and AAA, it's to ensure maximum contact with the flat end of the battery, which is the "negative" terminal as described in the ANSI standard. You'll commonly see leaf-spring contacts as well as coils. The side with the "nub" will automatically provide a solid contact if enough pressure is provided by the contact on the flat side, so no second spring is needed.
I can only assume that the flat and "nub" design that the ANSI standard describes for round-type batteries was chosen because it provides a clean contact mechanism as well as a clear indicator of polarity to consumers.