Best RF frequency for the application

In free space, it doesn't matter. The power per incident area of a propagating wave is inversely proportional to the square of the distance from the transmitter. This is true regardless of frequency.

Certain frequencies are reflected, refracted, absorbed, and scattered differently by different materials. There is no single monotonic relationship until you get to really high energies, like gamma rays and beyond. At these really high energies (high frequencies), the waves basically just blast thru any material in their way, with higher energies passing thru material with less attenutation. Up to below Xray frequencies, there is no single answer, and it depends on the material between the transmitter and receiver.

Diffraction effects can make low fequencies (long wavelengths) appear to bend around objects, but this actually occurs at all wavelengths. The "near" layer where diffraction effects occur scales with wavelength, so it appears to us at a fixed human scale that long wavelengths go "around" objects where short wavelengths don't, but that is due to our perception scale. On the scale of the earth, commercial AM radio frequencies around 1 MHz are low enough to diffract around the curvature of the earth to some extent making over the horizon AM reception possible. Commercial FM radio, being 100x shorter wavelength, exhibits this effect much less for the same size earth, so FM radio appears to us to be mostly occluded by the horizon.

To calculate this you can use the Shannon–Hartley theorem.

$$C = B \log_2 \left( 1+\frac{S}{N} \right) $$

where

C is the channel capacity in bits per second;
B is the bandwidth of the channel in hertz (passband bandwidth in case of a modulated signal);
S is the average received signal power over the bandwidth (in case of a modulated signal, often denoted C, i.e. modulated carrier), measured in watts (or volts squared);
N is the average noise or interference power over the bandwidth, measured in watts (or volts squared);
and S/N is the signal-to-noise ratio (SNR) or the carrier-to-noise ratio (CNR) of the communication signal to the Gaussian noise interference expressed as a linear power ratio (not as logarithmic decibels).

In a simple case, your maximum bandwidth will be the frequency you use. More likely it will be whatever frequency you modulate your signal at.

If your system were perfect, that would also be your channel capacity. But, it won't be perfect, ever. There will be noise and signal loss, more than you want.

You'll have to give more information about your system before you can get a reasonable first cut of your actual channel capacity.

To address your follow up question:

As an engineer, I like to draw straight lines and pretend they're close enough to the real world. For a situation like this, where a whole mess of small (and a few large) losses come into play, I would make a Fermi estimate.
I understand that's not easy. One of the biggest problems for aspiring engineers is they're not willing to just guess. This prevents them from moving forward, likely failing, and returning to make a better guess. Just get some paper and sit down in a place where you can make some noise and say "eehhuuhhhmm, fifty-six... ish?". Write that down. Now calculate it. Does it make any sense? Use whatever information you have; if your calculations show that you can set up your own radio station with some AA batteries then get a new piece of paper.
In the end the system you're trying to model is really complex; clearly, it depends on undulating ionospheric plasma and the time of day. You'll likely be able to guess your values within an order of magnitude, which might be enough information to get you to start building something.