No you can't. You need an inductor, which 'reduces' your idea to the well-known concept of a switched regulator.
The problem with your idea is that there is some resistance in the path between your power source and the capacitor, and this resistance will dissipate the heat that would otheriwse be dissipated by the linear element (pass transistor or the like). Switching does not help. An indiductor does help, because it stores the energy (and lateron releases it) instead of dissipating it.
Just in case you wonder (as I did long long ago): reducing the resistance does not help. You can prove mathematically that as the resistance approaches 0 the problem stays the same. An no, you can not get it to be zero, which is just as well, because then the current would become infinte, which (at the very least) would give 'interesting' magnetic effects.
Using PWM to deliver a 'reduced' power or as an Digital-to-Analog converter works only when you either
smooth the PWM output with a filter (you can use a simple firts-order RC if dissipation and bandwith are no issues), or
the thing you deliver the PWM to does not need a smooth supply (motors, lamps, heaters, etc all fall in this category).
The input capacitance value is not critical. What is critical is that you have some way of preventing the input voltage from dipping during operation over a wide frequency band.
The connection between the power source(battery, generator, AC power, etc) and the regulator input will have some inductance L. In a linear regulator current passes from the input of the regulator, goes through a pass element (usually a BJT, MOSFET, or IGBT), and then flows to the load. The input and output current are typically about equal except for a small amount of extra input current used to run the regulator internal circuitry (reference, error amp, gate drive).
Suppose that the regulator input current increases at a rate of di/dt (due to the load current changing). Then without any input capacitance the input voltage to the regulator would dip by an amount V_dip = L * di/dt. Clearly if the voltage dips too much then the regulator will stop working and the output load voltage would drop.
The datasheet will usually recommend a minimum required capacitance on the input, but you can always use more. Ceramic capacitors tend to work well over a wide frequency band but have lower capacitance values. Electrolytic capacitors have larger values but work only at lower frequencies. Typically a combination of both types is used to get both high capacitance and wide frequency operation.
Linear regulators typically have an error amplifier and pass transistor inside of them. Both of those components have limited bandwidth. If the output current changes too rapidly then the regulator will not be able to adjust to the demand changes quickly enough and the output voltage may dip.
The amount of output dip at frequencies much higher than the regulator bandwidth is approximately dV = I/(2 * pi * f * C). For example if you had a regulator with a bandwidth of 100kHz, and you were running some digital electronics that drew 100mA spikes at 1MHz and had a 0.1uF output capacitor then the output ripple would be 100mA/(2*pi*1MHz * 0.1uF) = 15.9mV peak.
Typically you would try to pick a capacitance that leads to an acceptable ripple voltage (using the above equation) at the peak load current at the frequency corresponding to the regulators bandwidth given in the datasheet.
Another factor to consider for the output capacitor is stability. The error amplifier in a linear regulator typically uses feedback and can oscillate if too much or too little capacitance is used. Many linear regulators are stable with a wide range of output capacitances. The datasheet will often specify that the capacitance must be below or above a certain value for stable operation.
You cant really calculate how much capacitance is required for stability without knowing the characteristics of the error amplifier (phase and gain margin vs. frequency). Since the manufacturer often doesn't tell you that information you sort of have to take the manufacturer at their word on that one.
Best Answer
An interesting property of AC voltage is that it goes down to zero twice per cycle, which means if you use a transformer and a rectifier, you need a capacitor large enough after the rectifier to act as temporary energy storage during the parts of the period where the AC voltage is too low.
Basically, \$ dv/dt = i/C \$ so you take the max post-rectifier voltage and the min allowed voltage at the input of the regulator, which gives you max allowed voltage drop, divide by one half period, you get your max dv/dt, you know the max i, you get the minimum C value to use. Add a wide safety factor.
That's to keep the regulator stable if you feed it from a DC power supply.
Regulators can react badly to too much inductance in the supply (ie, long wires) so a local small value cap keeps this power supply impedance down. Use whatever ceramic cap of suitable voltage like 100nF or 1µF, value doesn't matter as long as it's enough, what matters is that it is placed close to the regulator.
Some regulators are conditionally stable depending on output cap value and ESR, so reading the datasheet is necessary. 7805 isn't picky though. A larger output cap will probably just give better transient response.