Li-Poly battery voltage changes a lot while discharging — from 4.2V to 2.7V. Most modern digital ICs (I'm currently thinking of Cortex-M3 microcontroller and an RF module) require input voltage levels of 2.7V-3.6V.
So, I think that connecting them to 4.2V power supply will most probably result in a dead IC. How this problem is solved in actual devices?
I can only think of two solutions: the first is a buck-boost converter, which should certainly work, but is really hard to build (at least for me; I never had success with self-made DC-DC converters, not to say the ICs for them are hard to obtain in local Russian stores); and a linear LDO regulator, which will still incur some dropout, maybe as low as 0.7V, but it means that the smallest battery voltage with which the circuit is usable is around 3.4V, and quite a big part of charge is then left unused.