For a prototype, with careful use, this would still have problems. The LM317 does not tolerate reverse voltages, but you can protect it by putting a diode across it, allowing any reverse current to flow through the diode to the other side (just the capacitor in this case).
In no case should the external power and the USB be connected at the same time.
The USB port should be capable of supplying 500mA at 5V without a problem, so it would appear to meet your needs.
In a production circuit, if you can handle a little droop on USB voltage, use a diode from the USB connection to the circuit's 5V rail. If the USB is connected and no external adaptor is connected, then the USB will supply up to 5V (after the diode's voltage drop). If the external adaptor is connected and not USB, then the external adaptor will supply all the required current at 5V. If both are connected, the external adaptor will be at 5V, while the USB will be lower, thus the diode will protect the computer from possible issues, and the USB won't supply any current.
It's not the best design, but it's quick, simple, cheap, and will save others from damaging their USB ports. There are many ways to handle this problem more elegantly, many using mosfets to switch the best available power into the circuit. But for a simple prototype, this isn't a bad start.
Is it correct to assume that an ideal (theoretical) PSU for audio applications should produce a constant voltage regardless of load variations (i.e. it is a voltage source)?
Yes. An ideal power supply for any application should be an ideal voltage source, which has a constant voltage.
In practice, what are acceptable levels of supply voltage variation due to transients in an audio application (in percentage of Vs or mV)?
This is dependent on your application. You have to evaluate your desired noise/distortion, the power supply rejection of the audio components you are using, and the way the circuit is constructed. 0.1% power supply variation translates to a -60dB noise floor, which might be sufficient.
What is the correct approach to reduce these V swings? Should I place a capacitor on the INPUT or OUTPUT of the regulator? Is there a rule of thumb/calculation for the required capacitance? Are electrolytic capacitors OK, or should I use polyester or tantalum caps (i.e. something with a lower ESR)?
Probably both. You should have both bulk capacitance on the output and low-ESR decoupling caps in close proximity to all active chips (op-amps, ADCs, DACs, etc). And some more capacitance on the input certainly wouldn't hurt.
Typically, you might use large electrolytics for bulk capacitance, and low-ESR ceramics for faster decoupling. Again, how much you need depends on the magnitude and characteristics of the wiggles on the supply rail. Also, carefully read the datasheet of the regulator and make sure you are within its comfortable operating region: the regulator has a response speed and current limits, as well as input ripple rejection specs.
Best Answer
According to the datasheet (p10) the input capacitor is recommended (I would translate that as required) when the chip is not near the rectifier/filter-capacitor. You don't have a filter capacitor, and a 9V battery has a relatively high impedance, so I strongly suggest you include the 100nF input capacitor.
Again, according to the datasheet, the output capacitor is not needed but will improve the transient response. You motor won't care, so I'd leave that one out. (But it won't harm to leave it in.)