This looks like homework, around here people are encouraged to learn from the experience of getting stuck.
You seem to be mixing up terms.
M = typically used to represent # of symbols and having zero ISI is a given, otherwise you've not chosen wisely. That is the point of constellation/trellis decoding system.
Your initial calculation should read 1 MBps = 125 KHz * 8 bits/symbol.
So the question is, why does a constellation of 20 symbols work when a constellation of 8 doesn't. Why isn't 20 a power of 2? Given that 20 fits into 5 bits what has happened to the other 12 symbols?
- there are some hints in here...
But your question is also lacking, you're not explaining how you are simulating the system, what is meant by fails to "finish the work" etc.
You will need to clearer, more concise and reread/understand the basic theory before asking a more detailed question, with information and partial analysis. Please come back and update the question.
The difference between the two formulas arises from the fact that the Nyquist formula uses the number of encoding levels that was explicitly given (16 levels implies 4 bits/baud), while the Shannon formula is the theoretical maximum based on the SNR of the channel (40 dB implies about 6.64 bits/baud).
3000 Hz × 2 baud/cycle × 4 bits/baud = 24000 bits/sec
3000 Hz × 2 baud/cycle × 6.64 bits/baud = 39840 bits/sec
Best Answer
There are actually three terms you want to know about
Bandwidth
Bandwidth is measured in Hz. It describes the frequency band that a communication channel is able to transmit with low loss.
Typically we talk about a 3-dB bandwidth, meaning the range of frequencies a channel can transmit with less than 3 dB of loss. For a baseband system, the bandwidth extends from 0 Hz to a frequency B which we call the bandwidth. For a modulated system if the carrier is at f0, then the transmission band would be from \$f_0 - B/2\$ to \$f_0 + B/2\$.
Also, outside of information theory, the term bandwidth may be used more broadly as a synonym for bit rate, or for data processing capacity, but when the units are Hz, we know we're talking about the analog bandwidth of a signal path of some kind.
Baud
You didn't ask about this, but its also important to keep this separated in your mind from the other two terms. Baud is the number of symbols transferred per second on the channel.
Bit rate
Bit rate indicates the amount of information transferred on a channel, and is measured in bits per second or bps. Bit rate is different from baud if more than one bit is transferred per symbol. For example, in a 4-level amplitude modulation scheme, each symbol can encode 2 bits of information. Alternately, for example when an error-correcting code is used, the bit rate can be less than the baud rate, as a larger number of symbols are used to convey a smaller number of bits of independent information.
The Shannon Theorem shows how bit rate is limited by bandwidth and the signal-to-noise ratio of the channel:
\$C = B\ \log_2(1 + \mathrm{SNR})\$
where C is the capacity (maximum bit rate of the channel), B is the bandwidth of the channel, and SNR is the signal to noise ratio.