Okay, well- we're talking about (6VDC-3.3VDC)* 50mA = 135mW, which should be just fine for an SO-8**. There are SOT-223 versions of the '1117 available if you want as well as larger SMT packages. It's a common LDO regulator- lots of sources, so a reasonable choice- maybe the SOT-223 is even more common.
** Thermal resistance junction-to-ambient is stated to be about about 55K/W, so you'd have only about 7°C rise. Barely warm. Note carefully that that's based on some standard layout, laminate and copper thickness which you'd have to dig for- but probably not the absolute minimum amount of copper! - datasheets do things like that).
It costs little to nothing to extend the four Vout connections under the package and maybe spread them out on either side of the chip, but it's not really necessary in this case.
Edit: Note that the SO8 (in ST's standard footprint) is better thermally, by 2:1, though either would work in this case. Also the '1117 is kind of a semi-LDO, there are much lower dropout parts available, but that is not a factor in this case- you have plenty of voltage, so there is no advantage in using a lower dropout part.
Thermal performance of SMT parts is heavily influenced by the footprint and copper thickness and other factors outside the chip, so one should be careful when running close to the limits, but this application is a slam-dunk (assuming nothing pathological like extremely hot environment or having to operate in a vacuum).
From the data you provide, this indeed seems like a bad design. I also get about 600 mW dissipation in R1 in the circuit you show.
The fact that the resistor is getting really hot is direct evidence that it is dissipating significant power for its size, but not necessarily too much. Resistors can run indefinitely without harm at temperatures that would burn your finger. A finger test doesn't really tell you whether it is dissipating just within the limit, or over it.
One possibility is that the circuit isn't as you show. Perhaps there is something else going on that isn't easily visible from the outside of the board. A good test would be to measure the actual voltage across the resistor. That together with the label on the resistor will give you a definitive answer to how much power it is dissipating.
Note that 0805 resistors are labeled with 3 or 4 digits. This is a floating point format with the last digit being the exponent of 10 and the previous digits the mantissa. A 5% 820 Ω resistor will be labeled "821", which means 82 x 101 = 820.
The power dissipated by a resistor is the square of the voltage across it divided by the resistance. In common units,
$$\mathrm{W}={\mathrm{V}^2\over\mathrm{\Omega}}$$
Therefore, the voltage that causes a particular dissipation is
$$\mathrm{V} = \sqrt{\mathrm{W}\cdot\mathrm{\Omega}}$$
At 125 mW, a 820 Ω resistor will have
$$V = \sqrt{125\mathrm{mW}\cdot820\mathrm{\Omega}} = 10.12\mathrm{V}$$
across it.
If the resistor is really 820 Ω, is really only good for 125 mW, and has more than 10 V on it, then yes, this is a flawed design. From the data you've given us, these premises seem to be true.
If it turns out the resistor really is overloaded, then probably what happened is that the unit was originally designed for a lower voltage. Somebody realized they were missing too much of the market by not supporting higher voltage. Whoever was supposed to check this in engineering either didn't, was generally incompetent, or just missed this one.
Of course why it is like this doesn't matter to you. You absolutely need to reject this system. Currently, it's just some other company putting a bad product in the field. If you incorporate that into your system, you are putting a bad product in the field, and own the resulting liability, and it will by your reputation that gets damaged.
While you definitely don't want to use this product (again, assuming things really are as you say), The device is very unlikely to catch on fire as a result. Such overloaded resistors will usually just burn out and fail open. There isn't enough flammable stuff around to cause a fire. However, the resistor could burn out and open before the fire fighters arrive, giving them wrong information as to where the fire is. That's the real danger of this system. Or, the system could latch the information until manually reset, so there are no symptoms during the first incident. However, now that channel is broken and won't respond to future fires in that zone. That's obviously really bad too.
Do the voltage measurement and point out your concern to the manufacturer. It might be worth hearing what they have to say, but it would have to be something really good for me to ever trust their products again. Remember that with electrical engineers, just like with any large group of people, there are really good ones at the top end, the decent-enough majority in the middle, and incompetents at the bottom. There are certainly incompetently designed products out there. You may have found one.
Best Answer
No, it means that each junction runs 18C hotter than the heatsink(PCB).
You need to know the heatsink's thermal resistance to air, and the total power you're dumping into it, and the air temperature, to calculate the heatsink temperature. Then add 18C to that, to get the chip temperature.
If the heatsink is just the PCB, its resistance will be high and its temperature will be high. It's difficult to calculate (as in, read a thermodynamics book first), maybe possible to simulate with professional design tools, but easiest to measure (bolt a power resistor and some thermocouples to it). But there's no point, you won't like the answers.
Practically this PCB isn't the heatsink, it just transfers the heat to the real heatsink (big lump of metal).
Now you need to know the real heatsink's thermal resistance to air, and the total power you're dumping into it, and the air temperature, to calculate the heatsink temperature. This thermal resistance will be in the heatsink's data, if you buy a commercial one. Otherwise, see above re: calculation, simulation, measurement.
But the PCB will run hotter than the heatsink : you also need to know its thermal resistance in transferring heat from chip through PCB to heatsink. Again, probably easiest to measure.
This is where you need to know the total power from all LEDs - if it's 1C/Watt, it will add 6.6C or 13.2C for 3 or 6 LEDs at 2.2W each.
Now, calculate the heatsink's temperature. Add the temperature due to the PCB. Then add 18C, and that's the chip temperature.