(the numbers are given for LiCoO2 chemistry, for LiFePO4 they should be lower)
Short answer: yes, this is OK. I charge lithium cells with a bench supply set to 4.2 V and whatever C/10 figure is for the cell I'm charging. If you are not in a hurry, slow charging is better, even for cells claimed to be capable of withstanding higher charging currents.
However (quoting you):
charging at a constant voltage (say 4.2V) so long as the maximum
current is limited to a reasonable value for the cell
means you will have constant current charger till your cell is at ~95%. Up to this point the voltage across the battery will be less than 4.2V if you measure it. Only when your charger starts outputting 4.2V it will become constant voltage.
What you are about to build is CC/CV charger and this is the right thing to do. "Constant voltage only" charger will be set to 4.2 V with no current limiting and it will charge the lithium cell very slowly. You can check it youself, just construct var.voltage circuit and measure the current into (discharged) cell at 3.5, 3.7, 4.0, 4.2, and 4.5 V. Cheap Chinese chargers are constructed like that, they restrict the voltage to 4.2 V so the cell won't ignite after being charged but the consumer would have to wait longer. I once bought a portable emergency charger which could be emptied in a couple of hours then took 3 days to recharge its internal cells.
There are other precautions to observe while charging lithium, you can learn them from any modern charge controller IC datasheet (my favorite is Linear Tech., their literature is very high quality). If you don't implement these precautions in your design never leave it unattended while charging, otherwise it may ruin your morning one day.
"I know there is an IC or circuit inside my phone that will disconnect the charger once the battery is fully charged"
Yes. Which is why you're not getting the results you expect.
Before USB, some chargers were directly connected to the battery and the phone lacked intelligent battery management. You would see voltages in the 3-4.2V range.
After USB, the "charger" is just a power supply and will always supply 5V. Charge control and end-of-charge should be managed by an IC inside the phone. It should theoretically be safe to leave it plugged in forever.
Overheating is bad for electronics, but they should not be overheating in normal operation. However, your charger is 5.5V rather than 5V, and this may be causing the problem. Counterfiet chargers can have the wrong voltage.
Reccomendations about what level of charge to keep batteries at are for long term storage (about 50% or slightly lower). Life may be improved by not charging beyond 80%. But, and this is very important, the state of charge displayed on the phone may not be accurate and may already account for this.
Best Answer
You are taking an AC signal from the wall and then as you say, the transformer, recitifier and capacitors are bringing it down to a lower DC voltage. From there, most charging circuits are either linear regulator type chargers or switch mode power supply type regulators (both of those are advanced type links, but just there to see they both exist). They are basically normal DC/DC power converters with some kind of monitoring circuit inside. These two components fulfill the needs you list in number 1 and 2. To learn more about how they work, I'd check out Dave's explanation of linear and switch mode power supplies.
If you're looking to build your own eventually, you'll need to figure out how to do the monitoring with a micro or otherwise and then control some kind of DC/DC power converter. Good luck!