If you use an "analogue switch" and feed it from the same supply as the op-amp, you shouldn't have a problem with output transients from the op-amp. This is a tried and tested way of controlling gain on an op-amp circuit.
You need to locate a "switch" with low enough on resistance so that it doesn't significantly affect the value of R2 it is placed in series with. You can use several combinations of switches and different values of R2 to create several different gains also.
This is a kind of silly question, but not in a bad way, it can be used as a teachable moment.
The issue you have isn't so much that resistors can be poorly matched, it is that you're assuming op-amps are ideal. That is, "the only source of error is from the resistors, so if I get rid of those I will reach nirvana".
Even if you get matched resistor made from un-obtainium you'll not achieve your (unstated - or rather unparameterized) goal.
In reality all op-amps have issues with input current balancing, input voltage offsets, frequency dependant behaviour etc. Your solution must account for all of these.
And the parameters of your solution space will dictate what can be ignored and what can't be corrected. Hint -> this means you need to give more details as to what is needed, BW, etc.
One classical technique is to use a chopper stabilized amplifier which switches back and forth and self corrects for op-amp and in some topologies, even the resistor network non-linearity.
Another technique is to use switched cap techniques.
Best Answer
If you are working at audio frequencies I would recommend using a monolithic Voltage Controlled Amplifier (VCA), with the analog control voltage coming from a single potentiometer, and perhaps a buffer driver for low impedance to drive the higher impedance gain control inputs of your 4 channels.
A single channel VCA is the SSM2018 from Analog Devices. The 4-channel SSM2154 is obsoleted. I am sure there are similar devices from other vendors.
If you are working with anything higher than audio frequencies then you should look at the vendors' selections of Variable Gain Amplifier (VGA) and Programmable Gain Amplifier (PGA) devices. However these are likely to be set by a 6-bit to 8-bit digital control and not via an analog control voltage that would easily be obtained from a potentiometer.
A simple approach is to use a JFET shunt VCA topology. This picture is courtesy of Elliott Sound Products (https://sound-au.com/articles/vca-techniques.html)
National Semiconductor AN32 from 1970 has a JFET as an attenuator at the input to an op-amp. In this configuration you would use a fixed gain to gain-up your output to a maximum and attenuate it down at the final stage using the control voltage. The image below would be the final stage, after your gain stage.