It is likely that the caps are there to compensate for attenuation at high frequencies in the probes (not shown on your diagram). It probably won't be anything to do with the inductance of the probe wires and connecting any wires to the input is of course allowable but doing this will not get the best result - a proper probe needs to be used that has a tuning cap inside it that at the correct position will balance against the caps on your diagram - it's a set-up thing you do when measuring signals on oscilloscopes and similar.
Neither is it a band-limiting filter - the resistor in parallel is in fact the main natural component for the signal to go thru - the cap, as I said earlier adds a bit of high frequency compensation.
A "parallel" band pass filter constructed from R,L and C has a centre frequency determined largely by the formula below: -
Fc = \$\dfrac{1}{2\pi\sqrt{LC}}\$
Given the following circuit: -
The impedance reaches a maximum at resonance and current I will only flow thru the resistor at resonance. Clearly, if R is big less current flows and if the frequency is moved away from Fc then the impedance drops rapidly. This type of circuit is used to let one frequency through (Fc) and rapidly attenuate frequencies that are not at resonance.
More typically the parallel RLC circuit looks like this (because it takes into account the biggest losses that tend to occur in the inductor): -
Now the frequency of resonance is slightly shifted from the previous formula to take into account R: -
Fc = \$\dfrac{1}{2\pi}\sqrt{\dfrac{1}{LC} - \dfrac{R^2}{L^2}}\$
The resistance in series with the coil (L) also reduces the Q of the circuit which makes the filter less peaky.
There is a lot more to these types of circuits and I'd take a look here - wiki page for RLC circuits (includes series RLC).
Best Answer
As pointed out in my comments the inverting input is a virtual 0V so the gain of this circuit is :
$$Gain = -\dfrac{Z_f}{Z_i}$$
Where \$ Z_i \$ is our input impedance \$ R_1 \$ in series with \$ C_1 \$
$$Z_i = R_1 + \dfrac{1}{j \cdot \omega \cdot C_1} = \dfrac{1+ j \cdot \omega \cdot C_1 \cdot R_1}{j \cdot \omega \cdot C_1}$$
And \$ Z_f \$is the feedback impedance, taking your second example \$ Z_f \$ is \$R_2\$ in parallel with \$ C_2 \$.
$$Z_f = \dfrac{R_2 \cdot \dfrac{1}{j \cdot \omega \cdot C_2}}{R_2+\dfrac{1}{j \cdot \omega \cdot C_2}}= \dfrac{R_2}{1+ j \cdot \omega \cdot C_2 \cdot R_2}$$
$$Gain = - \dfrac{Zf}{Zi} = - \dfrac{\dfrac{R_2}{1+ j \cdot \omega \cdot C_2 \cdot R_2}}{\dfrac{1+ j \cdot \omega \cdot C_1 \cdot R_1}{j \cdot \omega \cdot C_1}} = - \dfrac{j \cdot \omega \cdot C_1 \cdot R_2}{\left( 1+ j \cdot \omega \cdot C_1 \cdot R_1 \right) \cdot \left( 1+ j \cdot \omega \cdot C_2 \cdot R_2 \right)}$$
This gives you zero gain at DC raising until the first pole where it levels out then falling at the second pole.
The poles being when \$ \omega \cdot C_1 \cdot R_1 = 1 \$
and when \$ \omega \cdot C_2 \cdot R_2 = 1 \$
This answer is a lot more mathematically rigorous than the one by @DaveTweed but that doesn't make his answer any less correct.