Electronic – Voltage regulator getting extremely hot

The current out of that 754410 is probably not enough to drive the boost converter. I would build the same circuit like so:

^{simulate this circuit – Schematic created using CircuitLab}

And 100 uF may not be enough. You may need 1000 uF! If so, you also want to add a few Ohms of resistance between boost-out and capacitor-plus, to avoid putting too much capacitive load on the boost converter all at once.

You've got a few misconceptions. But don't worry, many people get confused by these principles when they're first learning about electronics. I'll try to clear up the misconceptions and answer your specific questions at the same time:

1) A regulator that advertises 800mA means it can output up to 800mA without burning up. It does not mean the the regulator will automatically limit the current to 800mA. Nor does it mean it will always provide 800mA to any load. It is simply stating the maximum safe current the regulator is capable of. Under normal circumstances, your circuit load should be using far less than the maximum. Otherwise, you need a bigger regulator.

2) The LD1117 is a linear regulator. These types of regulators must have appropriate capacitance on the inputs and outputs. Otherwise, the output may oscillate and cause any number of problems downstream in your circuit. It will also cause the reading on your multimeter to be wrong without you knowing. The datasheet for your part will tell you how much capacitance and what type (ceramic, aluminum, etc) to use. Here's a screen capture of the recommended circuit from STMicroeletronic's LD1117:

3) From your fritzing diagram, it looks like you used the ammeter function on your multimeter incorrectly. Ammeters are designed to appear "invisible" in your circuit. That means the circuit will act as if the ammeter was just a piece of wire. You connected the 3.3V output of the regulator, through the ammeter, to ground like this:

^{simulate this circuit – Schematic created using CircuitLab}

Since the ammeter acts like a piece of wire, this schematic is equivalent to:

^{simulate this circuit}
See the problem? You've got the 3.3V output directly shorted to ground! The reason you're only seeing 3A is because the internal components of the regulator are overheating and acting like resistors. If the LD1117 was a little more beefy, it would be outputting a lot more than 3A. In fact, if the regulator were an ideal voltage source, your current would be infinite! The appropriate place to put the ammeter is here:

^{simulate this circuit}

4) It is not up to you, the circuit designer, to "limit the input current" to the load. The load (the Arduino) will draw as much current from the regulator as it needs to function at any given time. And that current draw will change as the Arduino does different tasks. In fact, if you attempt to implement a current limiter, it will likely cause a brown-out and the Arduino will reset or lock up.

It is unlikely that an Arduino will need more than 800mA to do anything, so that regulator should be fine. Under typical circumstances, a development board like an Arduino should only draw in the 10s or low 100s of mA.

So if the regulator is overheating, there are two possibilities:

a) The regulator you chose can't handle the current draw of the load. For example, if you were trying to power 50 LEDs that are rated at 20mA, your LD1117 would overheat trying to push the 1A of current. In this case, you would need to choose a regulator with a higher current capacity.

b) Something is shorting to ground or pulling more current then it should. You've experienced this by shorting the regulator output to ground through your multimeter's ammeter function. It could also happen with a misplaced wire, an unintentional solder bridge, or a bad component.

The problem you're having is obviously (b) above.