A typical Li-Ion battery will be 3.7V nominal. This voltage corresponds to a state of almost full discharge. A fully charged Li-Ion battery will typically reach 4.2V.
It may be the case that the specific battery you're trying to use as a replacement has additional protective circuitry preventing it from delivering current to your recorder. All those extra terminals may be expecting to 'see' the mobile phone they're designed for.
Perhaps you have accidentally damaged the pcb of your device. To verify that, you can attach ~4V to the contacts of the recorder and see if it's broken or not.
If you're certain your recorder is not the source of the problem, you could try a different battery. Or, at your own risk, try to open up the li-ion battery and try to power the recorder directly from the li-ion cells hidden inside, omitting the protection circuitry (which I do not recommend).
The data sheet for the regulator suggests ... How important is this?
About as important as that your circuit work reliably.
Trying to second-guess datasheets is a bad idea. Unless the datasheet explains exactly what is going on and gives you guidance on different choices, the specifications are requirements, not options.
Only Microchip knows the limits of stability of the MCP1700. Their engineers have analyzed this over many cases of of current, headroom, output impedance, temperature, and other parameters. They have distilled the result of all this analysis down to a simple range of capacitance that need to be on the input and output for the device to reliably work. Why would you not follow that?
When you violate any specification in the datasheet, all remaining specifications become null and void. There is no longer any guarantee what the device will do. One or a few individual devices seeming to operate correctly at some limited combinations of current, dropout voltage, temperature, source impedance, output impedance, etc, is not useful evidence of anything.
Speaking in general, the input cap is to guarantee that the regulator sees some minimum impedance at certain frequencies. Ideally the input voltage has 0 impedance. Since that's not possible, they tell you the minimum input capacitance to put right by the regulator to guarantee the input impedance the design assumes.
The output capacitance is part of the overall feedback loop, so effects stability. The requirements vary considerably across regulators, especially LDOs like the MCP1700. Early LDOs were intended for tantalum capacitors on the output and actually relied on some minimum ESR (effective series resistance) of the cap. Others specify a range of capacitance, with both higher and lower being bad. One nice thing about the MCP1700 is that there is no minimum ESR requirement. You can connect a ceramic cap directly to its output. In fact, you need to.
Do what the datasheet says, else you're a test pilot.
Best Answer
So the goal here is to trick the phone into working without a battery? not to provide useful runtime.
I suspect you will have to do a few things.
First I suspect you will have to pre-charge the capacitor to a voltage that is reasonable for a lithium ion battery. Deeply discharged lithium ion batteries can be dangerous if later recharged, so lithium battery systems will generally have an under-voltage lockout.
Secondly the capacitor will have to be big enough to prevent over or under-voltage swings. Just how big may require some experimentation. IIRC the charge circuits in smartphones are often insufficient to cover peak power demand with the battery being expected to take up the slack.
Thirdly many phone battery packs will have some kind of temperature sensor in them. I suspect you may need to find a way to fake the response from said sensor (in many cases I suspect this will just be a resistor of appropriate value).