Electronic – Fuse 125V/4A can be replaced with 250V/4A

The "speed", rupture capability, and voltage spec of a fuse are each separate traits that aren't necessarily related. They will be specified according to the nature of the application.

Many cheap multimeters (i.e. <50$us) will only have glass fuses, because they're cheaper. But 'high rupture' capability (the ability to break a circuit up to a rated voltage, when a high fault current is flowing) is important in a multimeter (and in a circuit breaker), especially in a DC scenario, because DC >40V can easily form an arc and continue the current flow, even though the fuse has blown.

For the "20A" fuse in your meter, a HRC type would be wise, and rated for the max V rating of your multimeter, and same speed as the original.

The design of a fuse conductor material gives rise to the PTC characteristic which causes a thermal runaway effect above rated current so that resistance rises rapidly with temp until it melts the conductor. The thin shape ensures that when this occurs, it explodes the material with sufficient gap that at least 1 mm air gap exists so that ~1kV spikes can be blocked on a 250Vac line.

It has no effect on low voltage circuits other than being relatively high resistance when conducting.

You also might want to consider getting an assortment of polyfuses.
- Why?
- Fuses do not provide great protection for low voltage circuits, because of the thermal mass of the transistor junction is much smaller than the fuse link so the fuse is too slow to protect semiconductors. It is effective for some low voltage applications such LiPo battery protection.

Thus 100mA fuses of this type rated for 250Vac are relatively useless for low voltage due to the maximum voltage drop possible when operating below the specified fusing limit.