What you are missing is that the pot is wired such that it produces a ratio of the input signal with the ratio varying from 0 to 1 accross the full sweep of the pot. This is why the 5 kΩ and 10 kΩ pots resulted in the same full volume.
The pot achieves this by being a resistor divider. It does not work by adding resistance in series with a signal. A resistor divider looks like this:
The output will be R2/(R1+R2) of the output. In the case of a pot, R1 and R2 are one continuous resistor with a mechanical wiper picking off OUT at some point along this resistor. The three pins of the pot are the two ends of this resistor and the wiper tap. Therefore, R2 will vary from 0 at no volume to (R1+R2) at maximum volume. Also, R1+R2 is always fixed, and is the resistance value specified for the pot. In your "5 kΩ" pot, for example, R1+R2 is 5 kΩ, which is the value of the physical resistor that the wiper slides over.
At half volume, for example with the 5 kΩ pot, R1 and R2 are each 2.5 kΩ. OUT is half of whatever signal is applied at IN. Note that since everything is ratiometric, you get the same answer whether the total pot resistance is 5 kΩ or 10 kΩ. This is why the volume levels didn't change.
The total pot resistance does matter in other ways to the driving circuit and whatever is using the signal at OUT. The 5 kΩ pot will require whatever is driving IN to provide twice the current than is necessary with the 10 kΩ pot. You don't know what exactly is driving IN and what its design constraints might have been, so it is best to replace the pot with one of the same value. It seems you got lucky in that whatever is driving IN can cope with the 5 kΩ load, but it could just as well have started clipping, otherwise distorting, or have the frequency balance different.
The crackling and the fact that you got sudden jumps in volume were due to the old pot being worn out. As pots age, dirt and oxidation accumulates on the surface of the resistor where the wiper slides over it. The resistor itself can also get worn down due to mechanical abrasion by the wiper. The wiper sometimes making good contact and sometimes not can sound like crackling, especially when the pot is being turned. Dead and worn out spots on the resistor can cause sudden jumps. These are all common failure modes of pots.
This is one area where construction quality makes a big difference. El-cheapo pots wear out a lot faster and may not be as well sealed against dirt or the materials are more prone to oxidation. If you want long-lived mechanical volume controls, you have to spend the money on good quality pots.
This is also one reason these things are done digitally nowadays. You can get a microcontroller to handle the audio stream digitally for less than the price of a top quality volume control. The digital multiplies inside the micro don't wear out, crackle, or drift over time.
@berto's answer and the comments on it refer to feedback, which normally results in a loud squeel which will continue until the amplfier gain is reduced.
Your problem appears to be something else. A possible cause could be a faulty automatic gain control or compressor/limiter in the amplifier.
Best Answer
I'm assuming you will use (or want to use) this headset exclusively with the cordless handset so it isn't a problem modifying it.
If you can get to the speaker terminals then try putting a 100 ohm resistor in series with each speaker. This should attenuate things a bit. If still too loud then try a 330 ohm resistor, Hey they're 3 pence each (at most) so experiment a bit.
If potentiometers are your ultimate aim then experimenting with the resistors will get you an approximate value for a pot ohm wise. Without the little experiment you might be needing a ten ohm pot or a 1k ohm pot.
You ought to consider soldering this modification.