Electronic – Portable 5V power supply

I suggest taking a look at RC batteries used by RC planes, boats, cars and drones. They're the same thing as cell phone batteries only cheaper, smaller and lighter. And are available up to sizes that wouldn't fit even a laptop much less mobile phones (and down to sizes as small as you little finger).

Google "lipo battery" and the top 5 results will lead you to an online store that sells them. You can buy them from Amazon if you want if you don't mind paying typically 50% more than what hobby shops sell them for.

Additional Info:

RC batteries typically can source much more current compared to phone batteries. A typical park flying plane (wingspan of around 1m or between 36 and 42 inches) usually consumes around 20A of current. Though it's not uncommon to see people fly really fast planes that can easily slurp up to 40A. Big planes that need special flying fields to take off from consume anywhere from 50A up to more than 100A. Given this, I don't think the current ratings of these batteries matter much for your application as even the lowest rating will be overkill.

Current ratings for batteries are usually given as C-ratings. 1C is defined as the battery being drained in 1 hour. 2C means you can drain the battery in 30 minutes without excessively damaging it. Batteries are typically rated as 10C or more.

How to calculate maximum current source is simple: take the batteries' amp-hours (convert milliamp to amp) then multiply by the C rating. For example, a 2200mAH 20C battery can source 2.2A * 20 = 44A.

As for capacity. Again, it depends on the size of your aircraft/boat/car. The smallest battery you can get is around 120mAH and the largest is around 100000mAH. The most common RC battery is probably the 2200mAH 3-cell (12V) battery.

If you're going to be buying RC batteries, ask for SBEC (or S-BEC) as well. They're cheap switch-mode power supplies and are typically more than 90% efficient. Much cheaper than buying industrial grade DC/DC converters and easier than building your own buck converter. Most output 5V but some have jumpers allowing you to select between 5V and 6V output.

There might be some possible clever scheme using MOSFETs, but most likely the additional control circuitry will also have some leakage that will make it as inefficient as the simple scheme you planned.

I am sure the solution here is simply to find the appropriate components that, in the end, will lead to an acceptable compromise. For example:

Regarding the diode:

BAT54 has 0.5µA typ leakage at 25°C and 5V reverse voltage. The drop is higher than with more beefy schottkies though. About 0.2V at a 0.1mA. So if you keep using a 2.5V regulator, you may end up below the 2.4V min voltage required, so:

Regarding the voltage regulator:

You could use a 3.0V LDO with ultra low quiescent current to compensate for this higher diode drop. You can actually find some rather easily: TPS78330, for example, has 0.5µA Iq. NCP4681/NCP4684 claims 1µA typ. MCP1710 claims 20nA (!).

Just spend some more time on google/mouser/digikey and reading datasheets. What you want is achievable the way you planned it, with the right components.