Are you digitizing a variable voltage produced by an analog potentiometer?
I have used gear like that and seen the jitter and flakiness in the values being read from the pots. In my experience it could be improved substantially by cleaning the potentiometers with compressed air and alcohol.
I suspect what the problem is that DC is naively being sent across the potentiometer, and then the divided voltage is sampled.
Audio designers know that significant DC currents flowing across potentiometers (which also carry audio) will add lots of scratchy, crackling noises when those potentiometers are moved (and maybe even when they aren't).
If I were designing a circuit which digitizes the position of an analog potentiometer (rather than the obvious approach of using a rotary or linear encoder), I would drop strictly an AC signal (from an oscillator) across the pot, with no DC component. This would then be rectified, peak-detected and digitized.
Another consideration is that cheap carbon pots are more noisy than ceramic, conductive plastic or wire wound units.
A very simple software technique to smooth values is exponentially-weighted smoothing. This is a beautiful trick which allows the smoothed value to depend on all prior values, without retaining a history of prior samples. (Familiar analogy: the charging of a capacitor is similarly exponential, and the current value depends on all the past currents. However, the capacitor has no memory, per se.) Looks like there is a Wikipedia page about it.
To implement exponential smoothing, simply keep track of the estimated value \$s\$ of the potentiometer (or whatever parameter \$x\$ you're sampling).
The estimator is initialized with the first sample:
$$s_0 = x_0$$
Therafter, when a new sample is obtained, we replace the estimator with a linear blend of the existing and new value, where \$\alpha\$ is the blending factor (called "smoothing factor" in this situation).
$$s_t = \alpha x_{t-1} + (1 - \alpha) s_{t-1}$$
This can be conveniently rewritten like this:
$$s_t = (s_{t-1} - s_{t-1})^{\rightarrow 0} + \alpha x_{t-1} + (1 - \alpha) s_{t-1}$$
$$s_t = s_{t-1} - s_{t-1} + s_{t-1} + \alpha x_{t-1} - \alpha s_{t-1}$$
$$s_t = s_{t-1} + \alpha x_{t-1} - \alpha s_{t-1}$$
$$s_t = s_{t-1} + \alpha (x_{t-1} - s_{t-1})$$
In other words, the new estimate is the old estimate, plus a fraction of the difference between the new sample value and the old estimate.
If all the quantities are integers and \$\alpha\$ is some power of two constant, this is very easy to implement in code. For instance, suppose we take \$\alpha\$ to be \$1/4\$, so that we blend 25% new value, 75% estimator:
/* C pseudo-code */
s += (sample - s)/4; /* optimizes to s += (sample - s) >> 2; */
/* s returned to caller as the value */
Exponentially-weighted smoothing implemented as one line: subtract, shift right, accumulate.
Best Answer
That's a tough problem, and it's probably impossible to solve reliably without exploiting the fact that you can, in fact, change the voltage across the potentiometer.
So, first of all, the 8-bit ADC is probably not the thing you'll want to use to determine the value of your resistance.
Discharge timer approach
Instead, you could implement a much higher-resolution ADC by using a timer/counter to determine the value of the potentiometer:
simulate this circuit – Schematic created using CircuitLab
So, after charging the capacitor, reconfigure the output pin to an input pin.
Problems here:
This is a very common approach – in fact, a lot of systems use this approach to measure a capacitance (in touch sensors), where the capacitor is adjustable but the resistor is fixed. It works.
Dithering for higher resolution
The problem that you have very finite resolution is one very well-understood mathematically. The solution is easy:
To illustrate:
So, in short, you intentionally noise up your observation, but then average the result to cancel out the noise and get an actual estimate for the underlying value.
The problem here is that adding "some" noise and averaging isn't enough in your case, because your observations \$X_k\$ are quantized, and the amount of noise you need to "cross" a step depends on the position of the actual voltage inside your interval.
You'd basically would want to write a probability-maximizing estimator (maximum likelihood, ML estimator) that given some distribution of observed values gives you the likeliest true value.
However, in the case of low-kurtosis symmmetric zero-mean noise, and because you "only" need to gain a few bits in resolution to get a good measurement: Your average of observed values will be pretty close to your actual value, if you just observe a couple hundred values – and that should really be no problem for a microcontroller with an ADC!
So, add noise to the high-side voltage of your potentiometer. That can happen through a random number generator in your microcontroller firmware used to control a DAC, but it really also is reasonable to just flip a pin high and low randomly, and couple that capacitively into your readout.
Really, get better components
The above are cheap and easy solutions, but honestly: get an optical encoder if you actually want to have an accurate six bit turn knob. Potentiometers really aren't known for their accuracy, so what you'd be measuring would probably really not have enough bits of useful information to begin with, unless it's very expensive – what good is it if you can simulate having a 20 bit ADC, when the actual uncertainty of potentiometer contact and temperature makes for 5% error, easily?