Solar energy from solar panels, when not being able to be consumed at the rate at which it's being produced (almost always the case), has to be either of the following:
- stored (usually in a battery) or
- fed back to the grid (e.g. a grid-tied system), or
- dumped into a dummy load (usually just beefy resistor packs) to convert the energy to heat
the energy has to go somewhere.
So it sounds like what you're interested in using the UPS as the battery bank, but your UPS system probably is designed internally to only charge its batteries from A/C power, where as your solar panels will give you D/C.
You could theoretically (but maybe not very practically) use an inverter to convert your D/C to A/C (if one such inverter existed with such voltage levels input/output specs, wouldn't really surprise me though), and then feed that A/C into the UPS (since the inverter would essentially just give you a normal wall outlet to use), but that would probably just be rather wasteful (e.g. not efficient due to the D/C->A/C in the inverter and then the A/C back to D/C within the UPS (and eventually you might want to go back to A/C for your appliances running off the UPS).
What you might try instead (although this is hardly trivial) is to disassemble (carefully) and try to tap into the battery banks of the UPS directly and figure out what voltage it works at (likely 12, 24, or maybe 48 volts DC). You could then theoretically step down your solar panel's output using some kind of custom (or off the shelf, it anyone knows of one) DC-DC converter, then you can feed that into a battery charger, preferably an MPPT type one. Those kinds would adapt the incoming power to what would be most suitable to charge the UPS batteries to do maximum power point tracking (what MPPT stands for) and reach upper-90% efficiency.
Or if possible (and i don't know if it is) re-arrange the solar panels such that they don't give you 320 volts (are you sure you don't mean watts?) but give you a lower voltage at a higher current rating. Otherwise you'll likely be hard-pressed to find a battery charger that accepts 320vdc to charge the batteries of your UPS.
A 6 Volt DC motor will function very well with a 5 Volt supply. A 9 Volt motor on the other hand will operate at around half the rated RPM at 5 Volts, and even slower with any appreciable load on the shaft. Therefore a 6 Volt motor is preferable.
Regarding solar panels, one would need to use multiple of the 5 Volt 500 mA panels in parallel, to preferably exceed the maximum stall current rating of the motor to be used. Stall current is typically much higher than the operating current. Thus, for a 6 Volt motor rated at say 1.1 Amperes maximum stall current, a minimum of 3 solar panels in parallel would be recommended, more if they are to be used in overcast conditions.
The question does not specify how much torque is expected from the motor, which is an important consideration for motor selection - speed ratings are typically at zero load or at some nominal load torque if this is explicitly stated.
Some examples of DC motors from eBay that will work on 5 Volts, and at prices well within the budget: 1, 2, 3.
For use with a propeller, a geared motor with lower RPM but higher torque would work better, for instance this one will deliver around 700 RPM with high torque, with a 5 Volt supply.
Similar motors can be found on many other sites - Radio Shack seems to be rather poorly stocked on such motors, so it would help to look further afield.
Edit to address updated question:
The 6 Volt motor does not seem to have current requirement specifications, but as stated above, it should work well enough with 2 of the 5 Volt panels in parallel.
To successfully supply the 9 Volt motor identified in the question, a series-parallel arrangement of 4 panels (2 x 2) will work - The panels will not be able to supply the full required current under load so the motor will slow down, but neither the panels nor the motor will suffer any harm.
Best Answer
The answer is simple:- you chose the wrong motor. For your application you need to turn a large prop, but not very fast. So you should be looking for a geared motor or an outrunner, with a rating of at least twice the power you will put actually into it (because you want to run it at close to peak efficiency, not maximum output power).
12V trolling motors typically draw about 500W, which would require 10 50W solar panels. Put the same power through a low Kv outrunner such as the Turnigy Aerodrive SK3 5055-280kv, and you should be good to go. The only other question is what size prop do you need to draw that power?