I think, the answer is relatively simple.
Do you know the working principle of a "Schottky diode", which is based on a semiconductor-metal junction?
Now - what happens if you connect a voltmeter (or any other load) across the diode? You create two Schottky junctions which exactly compensate the diffusion voltage inside the pn diode. Thus, no voltage can be measured.
With other words: You cannot use the diffusion voltage to drive any current through an external load.
It's difficult to come up with an analogy because the usual analogies for electrical systems are fluid systems. A great thing about fluid systems is that the working fluid is also good at cooling things, and most people's practical experience with fluid systems involves rates of flow where heating is not very significant.
So let's try a different analogy: a string being pulled through the resistance of your fingers. Your fingers are the LED, and the voltage drop of the LED is analogous to the difference in tension of the string on either side of your fingers. Current is analogous to the rate at which the string is being pulled.
Will your fingers be damaged if the string is pulled too fast? Yes: we call it "rope burn". This will happen even if you adjust the resistance of your fingers to maintain a constant difference in tension on the rope regardless of its speed (analogous to the approximately constant voltage drop of the LED).
The reason is that the rate of work done, and thus, the heat generated, is the product of the force your fingers apply to the rope and the rate at which the rope is moving through your fingers. You can get a rope burn by squeezing too hard, or moving the string too fast.
"Rate of work" or "rate of energy" is called power. One way to define it, for mechanical systems, is the product of force (\$F\$) and velocity (\$v\$):
$$ P = Fv $$
Since power is a rate of energy it should be in units of energy per time. In SI units, thats joules per second, also known as the watt. So, however fast the rope is moving, and however much force your fingers are applying to it, you are doing work at the rate of some number of joules per second. This energy can't vanish: it becomes heat in the rope and your fingers. Once you exceed your body's ability to transfer heat away from your fingertips your skin gets too hot and you are burned.
The analogy for electrical systems is that power is the product of voltage and current:
$$ P = VI $$
\$V\$ is approximately constant for an LED, but if you increase \$I\$ enough, you generate heat faster than it can radiate to the ambient environment. The LED gets too hot and is damaged.
Best Answer
Before you drive a current through it, a semiconductor’s population of holes and electrons is (roughly) in balance between thermal excitation that supplies them, and recombination that removes them.
When you drive currents through the semiconductor, those processes are still working. Thermal excitation will continue to create holes, which the current can now fill to create light.