I am having trouble understanding the three operation modes of a transistor.
When we talk about the modes of operation of a transistor, we're usually talking about cut-off, forward-active, and saturated operation.
The rest of your question seems to be about the different fundamental amplifier configurations, rather than the operating modes, so that's what I'll answer about.
Consider the CE mode. The collector and Emitter both are negative (for npn)
Check your diagrams again. For an NPN CE stage, the base and collector are both biased at higher potential than the emitter.
then how can we reverse bias them?
For a CE stage, the base-emitter junction should be forward biased; the base-collector junction is reverse biased. This is achieved by biasing the collector at a higher potential than the base. This is exactly what's shown in the diagram you posted.
Why is the common pin (the base , the emitter , and the collector respectively) grounded? what is the significance of grounding them?
They aren't necessarily grounded. They are connected to some potential that is equivalent to ground in the ac equivalent circuit. Particularly for common-base stages or PNP common-emitter stages, this is usually not the same as the circuit ground.
The significance is that this node is "common". A node that is used in common between the other nodes to define their potentials. The fact that the emitter is connected to the common node is why we call a common-emitter stage a common emitter stage.
How does a CE mode amplify more than a CB mode ?
A common emitter stage has voltage and current gain. A common base stage is essentially a unity gain current buffer. You need to study the common emitter stage to understand why it has voltage and current gain, and study the common base stage to understand why it is a unity gain buffer. Once you understand those two things, you'll understand why the one stage has more gain than the other.
The voltage divider rule between your two resistors does not work like you think because the base emitter junction of the BJT tends to go up to about 0.7V and then not go much higher whilst the current into the base can increase more and more. In other words the BE junction clamps the voltage level between the two resistors to about 0.7V.
When the R1 value is increased to a certain level the voltage at the BJT base lowers down below the 0.6 to 0.7V level and the transistor starts to shut off. At some point the voltage divider will begin to act like normal as the current into the base approaches zero.
ADDITIONAL INFORMATION
Since the OP is not yet quite getting it let me be specific with the examples that were posted. It is correct that at a voltage in range of 0.6 to 0.7V the transistor will begin to turn on.
Let's look at the 20K//1K case in the left picture. Assume for a moment that the transistor base is not connected to the two resistors. By the voltage divider equations the divider voltage is:
Vb = (Vsupply * R6)/(R5 + R6) = (12V * 1K)/(20K + 1K) = 0.571V
This voltage is less than the voltage needed to turn on a transistor so if you would reconnect the transistor base to the divider there will be virtually no current flowing into the base of the transistor and the voltage divider will remain near this 0.571V value.
Next step is to visualize what happens in the above equation when the R5 value is decreased. The divider voltage will increase slowly as the R5 value is decreased.
As R5 decreases more and more the Vb divider voltage will rise up to to the point where the transistor wants to begin turning on. That will be in the 0.6 to 0.7 voltage range. At this point the transistor base begins allowing some of the current from R5 to flow into the base of the transistor.
Be aware that transistors are current mode devices and are actually turned on when the current into the base starts to flow. Below the Vbe threshold the current is nearly zero. As the divider gets past the Vbe threshold the current into the base increases and the transistor starts to turn on.
Ok lets go back and decrease the value of R5 a little more. The lower resistance of R5 allows more current from the 12V supply to flow to R6 and the base of the transistor. The voltage across R5//R6 divider will no longer follow the above equation because the base of the transistor is placing a load on R5 and stealing current so that R6 does not get as much. The nature of the transistor base-emitter junction is that the current into the base can increase more and more whilst the voltage of the base will change only a little.
As I said before the base of the transistor begins to act like a clamp on the voltage divider not allowing the Vb to increase much above the 0.7V level as R5 is made increasingly smaller and smaller. Instead the base current increases to the point that the collector current starts to flow and the transistor eventually turns full on.
The amount of base current needed to turn the transistor full ON will depend on how much collector current is allowed to flow which is limited by components in the collector circuit. The relationship between the base current and the collector current is called the transistor gain or Beta. If the collector current is limited then the transistor will saturate to a Vce of near zero volts when the base current has reached a sufficient level.
It is possible to keep lowering the value of R5 more and more causing the base current to increase more. But beyond the level that caused saturation (Vce near zero) the Vb will only increase slightly and no additional collector current will flow because it has reached the level limited by the components in the collector circuit.
Best Answer
If you're familiar with double-subscript notation, you have your answer at hand.
For example, the base-emitter voltage \$v_{BE} \$ is positive when the base is more positive than the emitter, i.e., for \$v_{BE}\$, the \$ +\$ sign is at the base node.
Likewise, the emitter-base voltage \$v_{EB} \$ is positive when the emitter is more positive than the base, i.e., for \$v_{EB}\$, the \$ +\$ sign is at the emitter node.
With that in mind, for NPN transistors, the equations are written in terms of \$v_{BE}, v_{CB}, v_{CE}\$. By KVL, \$v_{BE} + v_{CB} = v_{CE}\$ so, if you know any two, you know the third.
Now, remembering the structure of NPN transistor, it is the case that the base-emitter junction is forward biased when \$v_{BE}\$ is positive and the base-collector junction is reverse biased when \$v_{CB}\$ is positive.
The cutoff region is formally defined as the condition that both junctions are reverse biased: \$v_{BE}< 0, v_{CB}>0\$
For PNP transistors, simply reverse the order of the subscripts and everything follows through.