How to regulate the voltage of a Li-ion battery with the least drop out voltage

You're looking for a power management IC.

Here's an example of one

http://www.linear.com/product/LTC3553-2

it includes a LiIon charger, linear and switching regulators to charge from USB and then switch back and forth between USB and the LiIon battery. Possibly not the part you want, but it's a start.

So, when the cell voltage is close to 4.2V the charging voltage will must be higher e.g. 4.5V, and this should not cause any damage to the cell. Is my understanding correct?

No. Your understanding is incorrect and your charger is suspect.
And/or your description is not quite complete and unambiguous.
For information on battery matters for most battery chemistries a good starting point is often the excellent site at Battery University.
NB: What I have written below is based both on experience and on input from a wide range of sources, including battery university.

Assume for following discussion a manufacturers spec of

• Maximum current = CCmax (usually 1C for LiIon but may be other for specific cells).
  Assume CCmax is 1C for the cell in question for convenience.
  Actual spec will be as per datasheet and is temperature dependant and also depends on how many charge/discharge cycles you wish to achieve before the battery turns to mush and/or is reduced to say 70% of original capacity.

• Maximum voltage of Vmax - usually 4.2V or less. Say 4.2V for now.
  As for current, the maximum Voltage applied will affect cell longevity (and capacity on a given charge). Charging at a terminal voltage of much above 4.2V will shorten you cell life, may lead to metallic lithium plating out and can lead to the exciting and equipment eating "vent with flame" battery meltdown phenomenon.

• Minimum current of Icv_min when charging at Vmax. This is the minimum that current should be allowed to fall to when charging in CV mode. When in CV mode, charging is terminated when current drops to this level. Icv_min is typically set at somewhere between 25% of Icc (early charge termination) and say 10% of Icc (maybe sometimes even 5% of Icc). The lower Icv_min is set the longer current trickles into the battery at Vmax in CV mode. Setting a low value of Icv_min adds slightly to the energy that can be stored in the battery on a given cyccle AND utterly tears the battery apart inside and shortens it life.

These two important points apply:

• The maximum voltage AT the battery (1 cell) under maximum constant current CCmax is Vmax = 4.2V in this case.

• BUT the maximum voltage AT the battery (1 cell) under ANY current is also Vmax.
  If the battery will not accept Imax when Vmax is applied then CC mode is no longer appropriate. Charging should be CV (or terminated if Icharge at Vmax is <= Icv_min - see below)

An important point here is where you measure what you call "the charging voltage".
This is properly measured at the cell electrodes as close to the cell internals as possible. In practice anywhere on the (usually) weld-attached tabs should be OK as at the max allowed current the voltage drop across the tabs should be minimal. As long as the voltage at the actual cell is <= Vmax then the voltage at other points in the charger may be > Vmax if the charger design requires it.

Consider: Apply a "true" constant current source to a discharged LiIon cell.
There will be lead resistance external to the cell so the voltage elsewhere to the system may be higher than at the battery terminals. Ignore that for now - comment on this at end.
For a discharged LiIon battery the terminal voltage will be somewhere around 3V and will slowly rise as CC is applied.

After about 40 to 50 minutes of charging a LiIon cell at 1C (= CCmax in this case) from fully discharged the TERMINAL voltage will reach 4.2V. This is where you stop applying CC and apply a CV of Vamx (= 4.2V in this case) at whatever current it takes to keep the voltage at 4.2V (up to a maximum of CCmax.)

The following paragraph may sound a little complex but it is important. It does make sense - read and understand if you care about the answer to the question that you asked.
It is a fallacy to think that you must apply a higher voltage at the cell to get it to accept CCmax when Vcell is at Vmax.
This IS true if the battery is fully charged or is charged above the point in the cycle where Vcell first reaches Vmax when charging at CCmax.
BUT that is because you are then trying then to do something which is outside the proper charging "envelope".
IF a LiIon cell will not accept CCmax when Vmax is applied it should be charged at not above Vmax until Ibattery falls to Icv_min.
If you apply Vmax and Ibattery is below Icv_min then the battery is fully charged and you should remove Vcharge. Leaving a battery connected indefinitely to a voltage source of Vmax when Icharge is less than Icv_min will damage the battery and reduce or greatly reduce its cycle life.

Charging voltage is removed when Icharge falls below Icv_min to prevent potentially irreversible electrochemical reactions and to prevent Lithium metal "plating out".
If Vmax is set at 4.15V then charge capacity is reduced noticeably but cycle life is extended.
If Vmax is set at 4.1V charge capacity is significantly reduced and cycle life is significantly extended.
The loss of capacity per cycle that occurs when Vmax is reduced leads to an overall INCREASE in total lifetime capacity as the extension in life cycles rises faster than the per cycle capacity falls. If you care mainly about highest capacity per charge set Vmax as high as allowed and accept low cycle life.
If you can tolerate say 80% to 90% of max possible capacity per cycle, set Vmax lower and get more overall energy storage before replacement.

The graph below from Battery University article How to Prolong Lithium-based Batteries shows what happens when Vmax is increased above 4.2V.

At the end are 3 tables from the same battery University page which show the effects on cycle life from varying various parameters (depth of discharge, temperature, Vmax)

Internal voltage versus terminal voltage:

There will be internal resistance in the cell so the "real" potential in the cell proper during charging at CC will be less than at the terminals. At CV the internal voltage will approach the external voltage as Icharge "tapers off".
IF you want to play 'fast and loose' with all manufacturers' specs and all advice given you can assume that you can 'allow' for this resistance and guestimate a true internal voltage which is lower than the terminal voltage. May the force be with you and with your battery, and may it live long and prosper - but, it probably won't.

Three excellent tables from Battery University showing how cycle life varies with various parameters.