Electronic – Why is the red LED dimmer without a resistor than with a 100 Ohm resistor

I would wire up a Power N-FET, that could handle up to the full 5A and 12V across it, wire it in series between the LEDs' common-cathode and the GND, and pulse-width-modulate its base using the Arduino to control the brightness. That will give you the illusion of dimming, as brightness is a function of the average current experienced by the LED, and the physiology of the human eye will take care of the rest :).

Changing the power rating of a resistor may let it get less hot since higher power resistors are better at getting rid of heat. However, this does nothing about the actual dissipation.

A 270 Ω resistor with 100 mA thru it, for example, will dissipate 2.7 W regardless of the power rating of the resistor. The power rating only tells you whether the resistor will get damaged in the process. If it is a "2 W" resistor, it will get quite hot and possibly start smoking a little, but will probably survive for a while at least. If it is a 0805 1/8 W resistor it will vanish into a puff of smoke quite quickly. If it's a "5 W" resistor, it will just sit there getting reasonably hot but otherwise continue to function correctly indefintely, assuming nothing around it is preventing it from conducting heat away.

Your real problem seems to be a mismatch between the power supply voltage and what your LEDs actually want. If you show a schematic it will be possible to make specific recommendations. In general, it will help to string several LEDs in series so that their total voltage drop is a bit less than the supply voltage. You then chose a resistor that drops the difference at the desired current. That way the voltage accross the resistor is a small fraction of the total, which also means the power wasted in the resistor will be a small fraction of the total power.

You say your LEDs drop about 3.3 V when used at your desired current. That sounds plausible. 24V / 3.3V = 7.3, so you could string as much as 7 LEDs in series to use up most but not all of the available 24 V. However, that would total 3.3V * 7 = 23.1 V, which doesn't leave much for a resistor to regulate current. In this case it's probably better to put 6 LEDs in series. The nominal voltage of the string will then be 6 * 3.3V = 19.8V, which leaves 4.2 V accross the resistor. Let's say you want to run the LEDs at 100 mA. That will also be the current thru the resistor since the LEDs and the resistor are all in series. 4.2V / 100mA = 42Ω, which is the value of the resistor to cause the right current thru the LED string when 24 V is applied to the whole thing. In that case the resistor would dissipate 420 mW, so a "1 W" resistor would be good.

If you want 20 mA thru the LED string (like would be common with T1 3/4 LEDs), then just plug in different numbers. 4.2V / 20mA = 210Ω, which now dissipates only 84 mW. A 0805 resistor can handle that.

Added:

You now show you have 8 strings of 5 LEDs each with a 270 Ω resistor in the string. Since your LEDs are dropping 3.3 V each, the LEDs total 16.5 V leaving 7.5 V accross the resistor. Since the resistors are 270 Ω, that implies your current per string is 28 mA. That's a strange value. Did you really intend it to be 20 mA perhaps? The dissipation per resistor will then be 210 mW. That's too much for a common 0805, but would be fine for a "1/2 W" or larger, or even a "1/4 W" in theory although that's not leaving much margin.

If you want 20 mA thru each LED, arrange them in strips of 6 instead of 5 and use the values I calculated in my last example in the previous section.