The inverting configuration is capable of gains less that 1, and can be used as a mixer. Here is a good primer.
http://chrisgammell.com/2008/08/02/how-does-an-op-amp-work-part-1/
I don't know exactly why (anyone feel free to chime in), but the fact that negative feedback is holding the negative input terminal at 0v means that node is a proper place to sum currents, making the mixer circuit viable (although inverting). Op amps are also cheap and come in packages with more than one, so you can usually just invert something again if it's "upside down"
There is a simple answer: The bandwidth for the closed-loop gain is determined by the frequency where the LOOP GAIN is 0 dB. In your example circuits the loop gain is not the same - hence, the bandwidth will not be the same. The circuit with the largest loop gain (non-inverter) has the largest bandwidth.
Explanation why the Loop Gain (LG) determines bandwidth:
The denominator of the closed-loop gain formula is
\$ D(s) = 1 - LG \$
From this, we can derive that "something" happens when \$LG=1\$ (0 dB).
At the corresponding frequency \$ \omega_{o} \$ we have a real pole (think of the behaviour of a first-order lowpass). And this pole gives the frequency where the 3dB-bandwidth is defined.
I should add that this is a simplified explanation; a detailed explanation involves the open-loop gain Aol and its frequency response:
\$ A_{CL} = \dfrac{H_{FW} \cdot A_{OL}}{1 - Hr \cdot A_{OL}} \$
with \$LG=Hr * A_{OL}\$ and forward factor \$H_{FW}\$.
We can see that for low frequencies (large \$LG\$) and negative feedback factor (\$Hr\$ negative) the "1" can be neglected and the gain is
\$A_{CL}= \dfrac{H_{FW}}{Hr} \$ = constant.
However, for large frequencies (\$A_{OL}\$ and \$LG\$ smaller) we cannot neglect the "1". When we reach the frequency \$ \omega_{o} \$ where \$ |LG|=1\$ the "1" starts to dominate for larger frequencies and we can neglect the loop gain LG.
In this case the numerator \$H_{FW} \cdot A_{OL}\$ determines mostly the frequency response (\$ A_{CL}= H_{FW} \cdot A_{OL}\$, approximately a first order lowpass).
Hence, the transition from the first region to the second region is at the cut-off frequency wo.
For inverter: \$H_{FW}=\dfrac{-R2}{R1+R2}\$
For non-inverter: \$H_{FW}=1\$.
Best Answer
There is a reason non-inverting 1st order high-pass filters are sometimes used and this is to preserve the dc level in the circuit. It's horses for courses - some designs will want to preserve the DC level, some won't care and some won't want to. It's not a black and white sort of thing.
When it comes to a 2nd order high-pass filter the non-inverting configuration (Sallen Key type) will usually be preferred and this doesn't preserve the dc level: -
Again horses for courses.