Electronic – Control two batteries with one switch

The easiest solution (Initially suggested by @PeterJ) Would be to simply switch only the power connection:

^{simulate this circuit – Schematic created using CircuitLab}

Alternatively, you can probably get away with just using some schottky diodes:

^{simulate this circuit}

^{_{Wow, Circuitlab is a clumsy tool. It doesn't have entry/exit ports? Or Schottky Diodes? Really?}}

But if you're really loading your RPi or it's USB ports, the drop in the diodes could be large enough that you get brownouts, which would be a real pain in the ass to diagnose.

The best solution, I think, would be to put the diodes before the voltage regulator.
This would mean you would have to have a higher power supply voltage (>7V), but it would solve the problem with the diode V_f described above.

Smartphones chargers are constantly being upgraded to be more efficient. Many major mobile phone manufacturers are working to develop a rating system for charger energy efficiency in no-load mode. For example, Sony's new standard chargers (EP-300, EP-310 and EP-800,etc) have a no-load power consumption ≤30mW. A 2012 study at the Lawrence Berkeley National Laboratory found that an idle charger drew 0.26 watts on average; this figure goes up to 3.68 watts when a phone is attached and charging, and drops down to 2.24 watts when the phone is attached and fully charged. Overall, the cost is a handful of pounds over 12 months.

Here is the typical Standby Power Characteristics of the Texas Instruments 5.3V 2A Mobile Phone Charger Design (PMP4432): From this I conclude that it will be effecient to have two smartphone chargers connected to two separate wall points each controlled by their own switch.

^{simulate this circuit – Schematic created using CircuitLab}