It turned out there was an issue with the FFT calculation. The solution was to calculate it from scratch (no pre-built functions), and take the magnitude of the resulting FFT and shifting the spectra as needed.
Although you are in frequency domain you still should be able to get all parameters as you were in time domain. They are different domains but they both should represent the same thing. Time domain represent things in terms of amplitude in respect to time. Frequency domain represent things in terms of amplitude AND PHASE in respect to frequency values. Note that you should have both amplitude and phase in frequency domain, since in the time domain the phase can be represented in the same plot by a shift.
One way to represent these things in frequency domain is by dealing with complex numbers. Complex numbers can be viewed as vectors in a 2D space which have a length (as you said) and an angle. The length represents the output/input ratio and the angle represent the phase shift in comparison also to the input.
So, answering your question, you should calculate the H length to find your output/input ratio. To help you, imagine that:
\$e^{jw}=cos(w)+jsin(w)\$
In other words, its a complex number with always length of 1 and angle \$w\$
You can solve this by two methods:
-Vector method:
imagine that number 1 is \$Z=1+0.i\$ which is a vector to the right, with length 1 and angle \$0\$.
Imagine that \$e^{jw}\$ is a vector that I showed right above
Now add them. Then divive vectors 1 by the vector that you've found.
-Cartesian Coordinates:
represent all in terms of \$Z=a+jb\$ and also \$e^{jw}=cos(w)+jsin(w)\$
and imagine that you have:
\$\large Z = \frac{Z_1}{Z_2+Z_3}\$
and then find length of Z by:
\$|Z| = \sqrt{a^2+b^2}\$
Best Answer
In a comment you clarified that you're trying to find the modulation index of an amplitude modulated signal. I assume this means that the signal provided takes the form of your message signal, and that is the reason you're trying to find the maximum. If you're unable to predict the exact frequencies, amplitudes, and phases of your sinusoids, you should assume the worst case upper bound is \$ |A| + |B| \$.
If all of your signals are cosines and have zero phase shift, then if the amplitudes are all positive, obviously the maximum is the sum \$(t = 0)\$. Interestingly, if there are only two cosines with no phase shift, and if the amplitudes are not both positive, you can still assume the maximum is \$ |A| + |B| \$ if the ratio \$r = \frac{\omega_1}{\omega_2}\$ is not rational.