Perhaps it might be easier to start at the energy state.
Free electrons (those moving from one atom to another) are in the conduction band and holes (the lack of an electron in an orbit) are in the valence band (same link).
The conduction band is at a higher energy level than the valence band and that means that things move faster. More interestingly, for an electron to move from the conduction band to the valence band (and fill the hole) it must lose some energy.
From a more intuitive perspective, when a hole appears in a valence orbit, not all possible electrons will drop into it; quite a number will pass by until an electron that has (crucially) lost sufficient energy to move into a lower energy band will fill the hole.
When said electron left an orbit (creating a hole), it was because it had energy added perhaps by a collision or even just from heat (otherwise it could not take on a higher energy location in the conduction band). Only when it has used up that energy (by moving or maybe colliding with another object which can eject a photon - this means the electron has lost 1 photon worth of energy) can it lose that extra energy and drop into the valence band.
This is perhaps explained by a more detailed look at energy levels
The electron states in the band gap are localised, whereas the states which contribute to the bands are not.
The electron states in a solid are not simple, there's a lot of non-trivial quantum mechanics going on. The electron states in a free atom are localised around the atom - the electrons in those states can't leave the atom without a lot of energy, so they can't conduct anything. When you pack lots of atoms together, the surrounding electron states overlap and mix. Which states mix with each other is dictated by the energy: similar energies means more mixing. You end up with a new set of states which extend over the whole block of material. If the material has a periodic lattice, these electron states group together into bands.
Every state in a band has some velocity (called a Fermi velocity) associated with it, and an electron in that state can be thought of as moving through the material with that velocity. The Fermi velocity of electrons in the conduction band is very large, but because the electrons are all going in different directions, there is no net current. An applied electric field moves some electrons from states which were going in one direction, to states going in the other. In a metal, one of the bands is part full, so there are plenty of nearby states to move electrons into. In a semiconductor, there is a gap between a full band and an empty one so it's much harder to push electrons into the higher band.
When dopants are added, they don't form a nice periodic lattice and they are much more spread out than the silicon atoms that host them. This means that the electron states around the dopant can't mix with states from other dopants to form a band. Since the energy levels of the dopant states are different from the silicon states, they don't mix (much) with them either. Instead, the electron states are localised around the dopant, much like the states around the free atom. An electron in that state can't conduct in the way one in a band can. It either has to jump up into the band, or jump to another nearby dopant. The former happens in semiconductors, the later is known as incoherent transport, and appears in some other materials.
I'm not sure how well I've explained this, but if you don't get a clear answer here, you could try the physics stack exchange. This definitely feels more like condensed matter physics than electrical engineering!
Best Answer
It is not true that the valence band cannot contribute to conduction. That's just what happens in P-type semiconductors. The doping alters the band structure of the semiconductor so that there are "missing" electrons (holes) in the valence band. This allows other electrons to "move" from an atom to a nearby one without jumping into the conduction band: they fill a hole "near to them", leaving a hole "behind them". This mechanism is modeled by virtual charges (the holes) moving in the opposite direction. All this happens in the valence band, and this is (intuitively) the reason why the mobility of holes is less than that of electrons. Actual conduction is always due to moving electrons, but when conduction happens in the valence band all is more "difficult" (the energy of the moving electrons filling hole after hole is less than the energy they would have if they were in the conduction band). Bear in mind that this is only a qualitative explanation. This is something in the realm of quantum mechanics and solid-state physics, and the equations involved are rather nasty.
BTW, what you mention by "holes attract electrons from conduction band" is called recombination. In a P-type semiconductor there are very few electrons in conduction band, and they are due to thermal generation (the higher the temperature the higher the probability that a free electron-hole pair will be generated). So it is true that very few electrons in conduction band will contribute to current in P-type semiconductor (they are the thermally-generated minority carriers). But the bulk of the current is supported by holes "moving" in valence band, as I explained above.