Your calculations are correct, you would need a 2.38Ω resistor that can handle 10.5W of power.
Both resistors you found, however, only have a 10W rating. With the 2.5Ω at 5V, you would be producing exactly 10W; I don't recommend operating a power resistor at its maximum rating for any duration of time. Components likely exceed their tolerance to some degree, but there's always a margin of error, so you should avoid operating things at the max rating unless you have good reason to do so. In the case of the 2.4Ω resistor, you would actually be running it at 10.4W (\$2.08A * 5V\$), which exceeds its rating. Definitely don't do that.
Going back to the desired load: 5V at 2.1A, 10.5W. There aren't precisely any 2.38Ω power resistors, but if you double it you get 4.76Ω and 4.7 is a common value among resistors. If you put two 10W 4.7Ω resistors in parallel, you'll get 2.35Ω total resistance, and have 20W total power handling capability. Note that they are 5% tolerance, so actual resistance will vary.
Edit, per comments:
Note that power rating on resistors is the max amount of power that they can dissipate. How much power they need to dissipate depends on how much current they draw. Remember, current (measured in Amperes) is drawn (and determined by) the load, not the power supply*.
For two equal value resistors in parallel, resistance is halved but power handling is additive. So two 10W, 4.7Ω resistors in parallel have a resistance of 2.35Ω but a total power handling capability of 20W.
It's not so much a case of leading or lagging. On a non-PFC power supply, the circuit consists of a bridge rectifier, followed by a large bulk capacitor. The cap charges and droops between the AC line cycles. During a potentially large portion of the AC line cycle the bridge doesn't conduct because the cap voltage is still above the rectified AC line voltage. Right near the peak of the line the line voltage exceeds the cap voltage, and all of the current flows into the cap during that small conduction angle.
So it doesn't look strictly inductive or capacitive, but it does generate large line harmonics. That's really what "power factor correction" standards regulate. It's not really the power factor, but the harmonics. The huge peak currents compared to the average current draw is the issue for the utilities.
Best Answer
As MikeJ-UK mentioned, there are rheostats. I personally don't like them for the same reasons what Mike mentions. But there are two more options:
Build a a resistor box and use switches to change the load. Or if you're fancy you can use MOSFET's and a MCU to do the job as well. Careful selection of the resistor values could give you a very wide range of resistances with fairly fine control.
Buy an "electronic load". Just google the term (in quotes) and you'll come up with a lot of pages. Basically it's the opposite of a benchtop power supply. Most will work in 3 modes: Constant Current, Constant Voltage, and Constant Resistance. My employer has one that's rated for 30 KWatts! Of course they make smaller ones.