I would consider a patch antenna. With a thick enough substrate you can cover the entire Wi-Fi 2.4 GHz band. It has a pretty decent front to back ratio (>10 dB) and an almost onmi hemispherical pattern.
While it is impossible to say for sure from you description, it doesn't sound like the antenna is the problem. If it were, you would have had a lot more problems in Tx than Rx (Tx is usually at least 50 dB above Rx). The fact that you are having problems in Rx makes me suspect some sort of coupling between your LNA and analog circuitry. When you put absorbers, you are lowering the signal level in the LNA (or further down the receive chain) and decrease the coupling.
What you need is a link budget. The specifications you supply say that the maximum transmit power is 25 dBm, and the receiver sensitivity is -112 dBm. This means that you can afford, at best:
$$ 25\:\mathrm{dBm} - (-112\:\mathrm{dBm}) = 137\:\mathrm{dB} \text{ of loss } $$
You will, of course, want to leave a healthy margin for robustness, but it gives you some bounds.
Loss calculation is greatly simplified for a balloon, since we can reasonably assume a clear line of sight. Atmospheric conditions (fog, rain, etc) can increase the loss, and you may have to compete against noise from other radios on the same frequency, but that's what the margin is for.
The most obvious loss is that due to the distance between the antennas. This is called the free space path loss, and we can calculate this loss, in decibels, as:
$$ 20 \log_{10}(d) + 20 \log_{10}(f) + 32.45 $$
Where:
- \$d\$ is the distance, in kilometers
- \$f\$ is the frequency, in megahertz
So for your specified distance of 30 km, and a frequency of 868 MHz:
$$ 20 \log_{10}(30) + 20 \log_{10}(868) + 32.45
= 120.76 \:\mathrm{dB}$$
This loss (121 dB) is less than the maximum loss based on the transmit power and sensitivity above (137 dB), so in theory, the link should work, even with an isotropic antenna.
In fact, you have a margin of \$137 - 121 = 16 \:\mathrm{dB}\$. Any gain that your antennas have is going to increase this margin. It doesn't matter if you add antenna gain at the receiver, transmitter, or both. Because of reciprocity, any gain in the system helps the same way. Additional margin may also allow the transmitters to operate at a lower power, which will increase your battery life, if that's a concern.
There's another source of loss that may not be obvious: polarization loss. Since the balloon is spinning about, you don't know what its orientation will be. Satellite communications have the same problem, and the canonical solution is circular polarization.
Since you don't need a lot of gain (and in fact, too much gain will make it hard to aim the antennas), a turnstile antenna may be a good option. It's circularly polarized and easy to construct. Adding a set of reflectors as in the first image from that Wikipedia article might not be a bad idea for the ground antenna, just for some extra margin:
This antenna could also be described as two crossed Yagi antennas on the same boom and fed in quadrature, so to calculate the geometry of the antenna elements you can use existing Yagi designs. If you research Yagi antennas for satellite communication you should find ample information.
Best Answer
A directional antenna is always preferable to omnidirectional. Both for transmitting and for receiving. But, as you say there may be factors that preclude using a directional antenna. For example, for portable (especially hand-held) use, a directional antenna may be to unwieldy, so a whip or "rubber duckie" antenna is typically used. Or if you don't know the direction of the communication, a directional antenna can be a liability.
If your mobile end knows what direction the fixed location end is, then you would benefit from at least using a directional antenna where you KNOW the direction.
A directional or omnidirectional antenna can be used with any type of antenna at the other end. Some VHF and UHF communication use polarization where the receiving antenna should be using the same polarization (vertical, horizontal) as the transmitting antenna