Electrical – THD calculation using microcontroller

Remember that there are voltage harmonics, because of the current harmonics. I mean, that voltage harmonics are "generated" by current harmonics. So, if the three lines impedances are not exactly the same, then the voltage drop generated by the 3rd harmonic current in each line will be different.

Have you tried to measure currents instead of voltages? You may go to phase voltage from line current (analizing the voltage drop)

*If I wasn´t clear I can write it in spanish (Im from Argentina and my eglish isn´t good enough sometimes)

You're confusing number of samples with resolution.

The 16-bit transform means each sample is 16-bits. In other words, the samples can be -32768 to 32767. The number of samples (N_WAVE) taken is completely irrelevant to the 'bitness' of the FFT. The 16-bits is referring to sample size, not the number of samples being taken. So the 1024 sample count and the resolution of those said samples have no relationship to each other, and 'where it came from' is, well, from the number of samples you wanted to take of a wave. This is determined by the nyquist frequency. You can just as readily use an N_WAVE of 256 if you wanted.

As for why your code isn't working, you're FFT is probably overflowing like crazy. The PIC doesn't magically change multiplication of 2 16-bit values into a 32 bit value like it does for turning 2 8-bit values into a 16-bit value. You need more than 16-bits of storage if you're multiplying 16-bit numbers. Even if your dynamic range is just 12 bits, you're scaling it, and if you didn't, well, anything except 0 multiplied by some of the values in your lookup table (which span the entire signed short value range) will overflow.

In the fix_fft function, use ints instead of shorts for the two arrays and m. This will force 32 bit multiplication and will resolve any overflow issues.

Honestly, this is probably not a question suited to electronics, so you might get a better answer in a programming related stackexchange. But I think this will help. Good luck!