The derivation of the Shannon-Hartley theorem?

The difference between the two formulas arises from the fact that the Nyquist formula uses the number of encoding levels that was explicitly given (16 levels implies 4 bits/baud), while the Shannon formula is the theoretical maximum based on the SNR of the channel (40 dB implies about 6.64 bits/baud).

3000 Hz × 2 baud/cycle × 4 bits/baud = 24000 bits/sec

3000 Hz × 2 baud/cycle × 6.64 bits/baud = 39840 bits/sec

From a theoretical point of view, the maximum capacity of a channel affected by AWGN Noise (Additive Gaussian Gaussian Noise) is determined by the Shannon–Hartley theorem:

$$ C\leq log_2(1+\frac{S}{N_0B})$$

This means you can't put more than that information on a channel with a band (B= \$f_{MAX}-f_{MIN}\$) without making the communication unreliable.

Then we go on the modulations: every modulation has a particular spectrum efficiency and an erroneous bit probability. More levels you use (QPSK vs 16-QAM, p.e.), more bit for each symbol (= more efficiency) but more erroneous symbols (similar to the bit error rate, with a Gray code).
The spectrum is directly related to the shaping impulse used by the modulation. A very common one is the raised cosine impulse (cause it has no Inter-Symbol Interference), that decreases the efficiency of a factor \$ (1+ \alpha) \$

Again we go on codes, that could give a huge gain, especially using concatenated codes like Reed-Solomon + Viterbi, using Turbo codes or LDPC.

Every effort is done to approach the Shannon capacity limit.