Electronic – Why does the voltage drop across an LED matter when determining an appropriate resistor value

There is no limit on the voltage, per se, that you use to power the circuit that drives the diode. The diode only cares about what the diode can see, and it can't see the voltage drop across the current limiting resistor.

That said, at some point what you're going to care about is the power dissipated across the resistor, which is \$ I^2R \$. If you want to keep the current to be constant in the case of growing required voltage drop, then R will eventually get big, and it will dissipate too much power. The power that run of the mill axial lead resistors can dissipate is 1/4 watt. For a 20mA current, that means to limit power across the resistor to 1/4 watt, you can't exceed 625 ohm, which means you can maximally drop 12.5 volts across it, and you're ceilinged out at a power supply of about 14.5V for a red LED. It's worse for small package SMD resistors, which are often 1/8 Watt or less. If you need more of a voltage drop, you would have to change to a higher power rated resistor, which can get physically big, as well as more expensive.

As to why the actual voltage across the LED doesn't change too dramatically given proper choice of current limiting resistor, one convenient way to look at this is with the "load line" technique. From http://i.stack.imgur.com/1cUKU.png, (Public domain image from Wikimedia):

The negative sloped line represents the resistor. If \$ V_D = 0 \$ there would be \$V_{DD}/R \$ of current through the resistor, and if \$ V_D = V_{DD} \$, then there is no current through the resistor (as there's no voltage drop across the resistor). The circuit "lives" at the equilibrium point where the resistor line and the diode curve intersect, as you MUST have the same current through the diode and the resistor. Note that changing R and \$ V_{DD} \$ less than dramatically won't move this point as much as you think it might in terms of the final voltage drop across the diode, because of how steep the diode curve gets.

One thing you need to remember is that voltage is relative. Voltage is a potential difference and it makes no sense to discuss voltage without a 'zero' reference.

In the case of your LED circuit there will be a voltage across the battery, a voltage across the LED, and a voltage across the resistor. If you add up all the voltages as you go around the loop, you get zero – up 9 at the battery, down 6 at the resistor, down 3 at the LED, the total is zero and you're back at the same point in the circuit. The LED only sees the difference in voltage between its two leads, as does the resistor. Since only the difference is important it makes no difference what order the parts are connected in.

As for current, it is only the same along a continuous path. Electrons are not created or destroyed (what goes in must come out). Since there is only one possible path for the electrons to take in your circuit, the current will be the same through all of the components. In a parallel circuit you see the opposite: all of the components have the same voltage, but the currents will be different.