Current flows into the battery from the charger regardless of the battery voltage measured since this voltage is artificially depressed due to the internal resistance of the battery. The battery voltage therefore is not an indicator of when the battery is fully charged.
The current flowing into the battery is measured via a current shunt resistor; when the current flowing into the battery falls below a programmed limit the charger cuts it off completely and only then is the battery considered fully charged.
(the numbers are given for LiCoO2 chemistry, for LiFePO4 they should be lower)
Short answer: yes, this is OK. I charge lithium cells with a bench supply set to 4.2 V and whatever C/10 figure is for the cell I'm charging. If you are not in a hurry, slow charging is better, even for cells claimed to be capable of withstanding higher charging currents.
However (quoting you):
charging at a constant voltage (say 4.2V) so long as the maximum
current is limited to a reasonable value for the cell
means you will have constant current charger till your cell is at ~95%. Up to this point the voltage across the battery will be less than 4.2V if you measure it. Only when your charger starts outputting 4.2V it will become constant voltage.
What you are about to build is CC/CV charger and this is the right thing to do. "Constant voltage only" charger will be set to 4.2 V with no current limiting and it will charge the lithium cell very slowly. You can check it youself, just construct var.voltage circuit and measure the current into (discharged) cell at 3.5, 3.7, 4.0, 4.2, and 4.5 V. Cheap Chinese chargers are constructed like that, they restrict the voltage to 4.2 V so the cell won't ignite after being charged but the consumer would have to wait longer. I once bought a portable emergency charger which could be emptied in a couple of hours then took 3 days to recharge its internal cells.
There are other precautions to observe while charging lithium, you can learn them from any modern charge controller IC datasheet (my favorite is Linear Tech., their literature is very high quality). If you don't implement these precautions in your design never leave it unattended while charging, otherwise it may ruin your morning one day.
Best Answer
If you use CC until Vcell is ~= 4.2V then.
Cell capacity will be about 80% of maximum
Charging time will be about 45 minutes.
Cell whole of life capacity stored and recovered will be signifixantly higher due to the gentler charging cycle.
4 hours should be very adequate for a full charge at I max = C.ie Imax =~ 3.2A for 3200 mAh cells. The CC stage will last about 45 minutes leaving 4h-45m = 3h-15m for the CV stage.
If you search SE EE for LiIon charging you will find this area is well covered.