As long as you don't select the 6, 9, or 12V settings on replacement power supply, the magic smoke won't escape. However, 4.5V may not be high enough to power up the device, in which case it either won't turn on or won't function properly, although it shouldn't cause any damage.
The replacement power supply is rated for a lower current rating, meaning that if the device draws more than 2.5A, it will brown out. This again shouldn't cause any damage. Even the cheap power supplies today have some basic short-circuit/over-current protection.
One thing to note: When you're measuring 5.16-5.18V, you're probably doing this with "no load" (i.e. when the power supply isn't connected to the device). Since the power supply is not perfect voltage source but a Thevenin equivalent, the voltage reading will drop when you connect it to the device.
Another thing to note: Most digital devices today that run off 5V feed that rail directly into a voltage regulator which drops it to 3.3V or lower. Those devices need the input voltage to be a little higher than the target voltage, so maybe 3.6-3.9V at a minimum. Bottom line, you should be fine with the 4.5V supply.
Holy cow, talk about an undersized heat sink.
Your 24 VAC transformer will produce about 32 volts at the main filter caps. Producing a 5 volt output from the LM317 means that you are dropping about 27 volts across it. A current of .14 amps says the LM317 is dissipating about 3.8 watts. The data sheet for your heatsink says that the temperature rise AT THE HEATSINK will be about 80 C, for a heatsink temperature of ~105 C. This is consistent with your measurement. If you look up the datasheet for the LM317, it specifies a thermal resistance (chip to case) of about 5 degrees per watt, so 4 watts on the chip will give it a 20 C rise over the case. Since the case is tied to the heatsink, your chip temperature is right about 125 C, which is the absolute maximum allowed. Dave Tweed is right about the regulator shutting down in order to keep from self-destructing.
If you could pull 1 amp, you would be dissipating 25 watts, and your heatsink temperature would be (roughly) 500 C. Plus another 125 C rise to the chip itself, and there is simply no way that's going to happen.
Best Answer
Since your multimeter reads significantly differently from non-zero with a battery input, your power supply is probably okay. That's a bit of a strange voltage reading- maybe 1/2 wave rectified with DC coupling.
You van try a capacitor in series with the input of the meter- maybe 10uF.