Wilhelmsen and Chris were right - it was the circuit.
However, it wasn't resistors, external IO, etc. It was the LD1117 Voltage Regulator which I hadn't really considered before.
It turns out that pretty much all the 1A, 3.3v LDO voltage regulators I can find have a 5mA quiescent current. I've just soldered in a MIC5205 (which is rated for 150mA) and it now draws 0.12mA! \o/
First, to answer your question:
The actual sleep current of the Xbee Series 2 modules is dependent on voltage and temperature. Over the entire voltage range, it is less than 10µA. At the typical voltage, and 25°C, it is less than 1µA. Product-to-product variation makes it useless to specify the current any more precisely, because it will be different for each module. But it will definitely be under 1µA.
In other words, the actual sleep current is... well... what they say the actual sleep current is.
The thing to keep in mind is that this is not the sleep current you will get simply by entering power down mode. This is the lowest current it is possible to get in power down mode. Put another way, that doesn't imply that simply powering down the device will yield this current. It means if a bunch of other stuff is arranged just so, a bunch of conditions met, and all sorts of other little gotchyas taken care of, with not one single thing overlooked, you can get <1µA current draw in power down mode. Power down isn't a magical on/off switch (as much as we wish it was), it usually just shuts down the CPU core. You have to manually shut off other things before that, all the things that isn't effected by the SM = 1 condition and only when all of that is taken care of, do you then power down the core.
With that in mind, I'm going to go over some pitfalls that any one of, or some combination of, would yield such high sleep currents. In fact, it is especially important to avoid 'silver bullet' thinking in these situations. That's where you assume that there is a single mechanism behind the problem, and thus the problem has a single, 'silver bullet' solution. In reality, it is probably several different things working in combination.
GPIOh no
The Xbee Series 2 (and newer cousins 2B and 2C) modules IO pins operate independently of the CPU core. Pin sleep, simply put, doesn't actually effect the IO pins. The CPU core can change and configure them, but that's all. So those pins will simply continue doing whatever it was they were doing (being inputs, or driven how you last set them as outputs, etc.) regardless of if you're powered down the module or not. Pin sleep will not take care of anything GPIO related for you, you have to do that yourself.
You mention setting all unused pins to output... but with what state? You must set unused pins to output, default low to achieve the lowest sleep current. So double check that.
Also, turn off the pull-up resistors on all those unused GPIO pins, even though you're setting them as outputs, not inputs.
Once you've made extra sure you're doing the above, then we can move.
The battery always loses in a pin fight.
If you didn't have your outputs set low (even though they're unconnected), that could have contributed to some of the current draw. But I suspect you'll also find something not quite right with the only other thing we can examine: the connected pins. In other words, the lines running from the Xbee to your MCU.
You do not want to double up on pull-up/pull-down resistors. For every line between the Xbee and the MCU, make sure you've disabled the MCU's internal pull-up resistors. Regardless of their direction (input/output). Let the Xbee's pull-ups, and only those, do the work.
Also, make sure your MCU is behaving correctly in terms of what it is actually doing to those lines it has connected to the Xbee. Turning off the UART TX line is not behaving correctly. (Though, it didn't make things worse, obviously). Don't over think this: it's a UART port with CTS flow control. So do what you're supposed to do, and simply treat it like any other UART port with CTS flow control: respect the CTS, that's all. Don't forget the MCU's internal pull-ups, those should be turned off. Now, we want to make sure the pins are set at the right levels. The MCU's TX pin, when inactive, should be high, and since a sleeping Xbee asserts CTS high (meaning, 'don't talk to me!'), that means the MCU's TX pin should be set to high. Putting it in any sort of high Z state is not correct, don't disconnect it or whatever you're doing.
The MCU RX line will be driven high by the Xbee, so make sure the MCU's pull-up on this input is disabled. We don't need it, since it's being driven. And there will be leakage, as nothing is ever quite at the same potential, if there is that extra pull-up path.
Make sure your RSSI pin is being set to output low before sleep, and make sure your MCU has it's pin on the other end set to input low/no pull-up. If there is a pull up active, you'll be pulling to down and seeing extra sleep current draw.
Make sure any Xbee pins that are inputs are not being driven low by the MCU for some reason (and verify the behavior since you're putting the MCU to sleep as well. Make sure it isn't pulling something low unexpectedly. Silicon errata is a real thing).
And, finally...
- Mo measurement, mo problems
Sometimes, you might just be measuring wrong. How confident are you in the accuracy of the meter you're using? You're measuring via the voltage drop across a 10Ω resistor. Even in power-down mode, the Xbee will wake up to reset a few timers and what not every few seconds. There can be very short current spikes, and a 10Ω resistor might be dropping the voltage enough to trigger brownout circuitry or do other weird stuff. It would be a good idea to add a really nice and fat tantalum capacitor, at least 100µF, across the Xbee module's power inputs, but after the 10Ω sense resistor, to decouple the power more from the 10Ω resistor. This will also serve to filter out any current spikes, which can often cause erroneous readings on multimeters, since they expect steady DC. Again, this likely isn't the sole cause of your high sleep current (or at least, high sleep current reading), but it might also be contributing.
All of the above applies just as much to the more recent 2B or 2C series Xbees, so don't let the age of this question (almost 4 years old now!) fool you, my answer still applies even in 2018. I fear it might be too late to be much use to the question author, but I hope not.
Best Answer
The watchdog is only stopped on system reset. This way you can't accidentally turn it off.
If you want to use sleep modes and watchdog, you will have to set the watchdog the slowest clock, and wake-up regularly to restart the watchdog.
This is valid for almost all microcontrollers on the market that have an independently clocked watchdog.