In a "purely digital" link where you set an output to "high" and an input the other end of a line is read as "high" then the probability error is purely to do with the SNR of the line. What is the probability that a HIGH can be interpreted as a LOW? By introducing a higher level protocol with error detection and correction you effectively negate most of the SNR errors and the question is now "What is the probability that the protocol cannot correct corrupted bits?"
So yes, the CODEC (or protocol) can be used (and is used) to negate the effects of SNR-induced signal corruption.
As for the second part...
If you assume 1 bit of information is transmitted per quantization level, and 1 bit is received per quantization level, then yes, increasing the quantization level will increase the number of bits sent at any one time. However, the SNR of the transmission medium will then have a greater effect on those now smaller quantization steps, so although you reduce the quantization noise, you now increase the SNR noise.
However, if you don't assume 1 bit per quantization level, but have multiple quantization levels per bit, then you can increase the number of quantization levels and keep the overall bitrate the same, but have more detail about each bit, so can make a better informed decision about what value that bit is.
For instance, you can think of a simple digital link with 2 states (HIGH and LOW) as a 1-bit quantized system. For simplicity we'll call it 1V for HIGH and 0V for low.
Now, you could then have it that anything received >= 0.5V is a HIGH and anything < 0.5V is a LOW. That's 1 bit quantization. 0.5V would be HIGH, but 0.499999999999V would be LOW. That's an infinitesimally small margin for noise.
However, increase the receiving quantization to 2 bits, say, would give you more detail. It would give you 4 voltage levels to consider - 0V, 0.33V, 0.66V and 1V.
You could now say that anything > 0.66V is a HIGH, and anything less than 0.33V is a LOW. You have now introduced a "noise margin". Anything that falls between those values is discarded as noise. The bitrate remains the same, but the overall SNR has fallen.
Then of course you can add a "schmitt trigger" to it (or software equivalent), whereby you toggle the value depending on a transition. When the input rises above 0.66V you see the value as HIGH, and keep it as HIGH. Only when it then drops down below 0.33V do you then switch it to LOW.
For systems where you have discrete voltage levels you could sample them at a higher resolution, and the line-induced noise would occupy the least significant bits of that sampled value. Discarding the noisy bits down to the resolution of the sent data can then reduce the noise in the system. Also taking multiple samples and averaging them, which in effect cancels the random noise out, (known as "oversampling") can reduce the noise as well.
None of those techniques affect the bitrate as such since you're not adding any extra information to the sent values.
Best Answer
The SNR is the ratio between the power of all the signal and the power of all the noise. If I'm not mistaken, the FFT in your diagram shows you the dB in volt, not in W. In the rest of this answer, every time I say dB, it's in voltage. So 0 dB = 1 volt, 6 dB ≃ 2 volt.
Let's assume you are sending a pure sine wave that looks like this: \$x(t) = A×\sin(\omega t)\$ volt.
The power of \$x(t)\$ will be its RMS value, squared. So you get \$\big(\frac{A}{\sqrt{2}}\big)^2=\frac{A^2}{2}\$
The power of the noise floor will be the sum of all the RMS values, squared. So you get \$\sum\frac{A_i^2}{2}\$ where \$A_i\$ is the amplitude of each individual frequency of the FFT (without a signal present, so the green graph in your question).
Going from dB in volt to linear volt numbers is simply \$10^{\frac{\text{dB}}{20}}\$
Going from dB in volt to linear power numbers is simply \$10^{\frac{\text{dB}}{10}}\$
The total power of the noise floor can be approximated, which I assume you want to do. So let's say you are doing a 1024 point FFT, you measure the noise floor and find out that it is about -40 dB, this is your green graph. Then let's say you do the red graph and measure your sine wave and find out that the maximum point is 6 dB.
The approximated floor noise is then:
$$P_{noise} = 1024×\frac{10^\frac{-40}{10}}{2}=0.0512 \text{ W}$$
The power of the signal is then:
$$P_{signal} = \frac{10^\frac{6}{10}}{2}=1.99 \text{ W} $$
The SNR is then:
$$\frac{P_{signal} }{P_{noise}}=\frac{1.99}{0.0512}≃38.87 = 10\log_{10}(38.87) \text{ dB}≃15.89 \text{ dB} $$
In case it's hard to absorb what I'm saying, here's the approximative equation simplified where I will mark \$\color{green}{\text{green}}\$ for your measurements according to your \$\color{green}{\text{green}}\$ graph, and \$\color{red}{\text{red}}\$ for your measurement according to your \$\color{red}{\text{red}}\$ graph.
$$ \begin{align} \text{SNR} &= \color{red}{\text{signal}}-\color{green}{\text{floor}}-10\log_{10}(\text{FFT})\text{ dB} \\ \text{SNR} &= 6-(-40)-10\log_{10}(\text{1024})\text{ dB}≃15.89 \text{ dB}\\ \end{align} $$
So no, your method is not 100% correct, but you were very close.