Electrical – Why ground plane is placed in helical antenna and how to calculate size of this ground plane

It would help to see a picture, but in general antennas built into modules do not use the ground plane as part of the antenna. For a ground plane to be useful as a reflector, the antenna needs to be perpendicular to the ground plane. Think of the ground plane as a mirror. With a half height antenna sticking above it at right angles, it looks like a whole antenna when viewed from above the ground plane due to the reflection.

Usually the built in antennas of integrated modules are self contained. Look at the datasheets and there should be a spec about where there must not be a ground plane under the module. Some modules are arranged such that the antenna is at one end past all the mounting points so that it can stick out from the board. In any case, follow the spec about how far to keep the ground plane and other metal from the antennas.

Other than keeping the ground plain away from the antennas, the ground plane is a good thing. It is surprising that you would try to do something like this with only a two layer board. After the cost of the GSM and Zigbee modules surely the extra cost of 4 layers is negligeable. Unless you're making at least 100k of these things, trying to get away with 2 layers sounds like penny wise and pound foolish. I would use 4 layers with layer 3 being a pervasive ground plane except for where the antennas are, which should be cutouts in the ground plane at the edge of the board. Don't make the cutouts holes in the middle. This allows for putting parts on the bottom side easily.

In the context of an RF antenna, 'ground plane' refers to any plane-like conductive thing near the antenna, which for, say, FM Radio transmission towers, is actually the ground. Like the kind with dirt that you walk on.

This is because conductive things generally reflect electromagnetic waves. That also means they block them from passing through to the other side. So to be explicitly clear, you may not have any copper at all on any layer or opposite side underneath the antenna. This is the 'copper free zone' in most app notes. The end points of the antenna must also be kept free of any conductive material for some distance. I'd recommend the entire width of the board be free of any conductive elements above the edge of the feeder's ground plane. Which fortunately is probably the case if this is a USB dongle.

Now, in the case of an inverting f antenna, it helps to understand how they actually work.

Current flowing into the actual antenna bit will be mirrored within the ground plane, and this generates a electric field. So the ground plane is really just the other pole of a dipole. The ground plane acts as an asymmetric pole and increases bandwidth, but for the most part, the current flow that matters happens entirely at the edge of the ground plane, parallel to the spine of the F. So, the height (dimension at a right angle to the spine of the F) of the ground plane is unimportant. It's fine if its the ground plane for the entire height of the board (except for the copper free zone around the antenna, of course).

This is good, because it makes our lives a little easier. The only dimension that really matters is the length/width of the ground plane, or parallel to the spine of the F. This is the length of the edge forming our other half of the dipole. And the answer is pretty simple:

As close to λ/4 wide as possible. This is always the distance of the L part of the F. So it will generally be slightly wider than the F's length spine-wise. If it is smaller, you will start losing bandwidth. If it is larger, you will get an increasingly multilobed radiation pattern. For a nice, omnidirectional pattern, quarter-wavelength width is best.

Note, since this is all about those edge currents, you can 'cheat' using some creative slotting of the ground plane along that edge.

Finally, remember that the plane is the other pole of a dipole. So you want it to be free of conductive things obstructing it's 'view' of free space. Ideally, you would make the ground plane be two planes, one on the top layer, and the other on the bottom layer, and stitched together with vias along the edge parallel to the F's spine. This is rarely practical however, so a good compromise is to have a plane on one of the outer-most layers (top or bottom) be the ground plane, and then a second ground plane on an internal layer, as close to the opposite end of the stackup. For a 4 layer board, this would be layers 2 and 4, or alternatively, 1 and 3. This way, on either side, there is as little crap in the way, with one side unobscured and the other with only the component layer. Stitch them together at the edge, of course.

Also, when in doubt, these kind of CAD-able antennas are prime candidates for antenna simulation software. Due to their relative simplicity, you can generally expect simulations to reflect (pun intended) the real performance of the antenna well enough for basic design performance checking. But, PIFAs are generally so simple as to not need even this.

I know this was probably a longer answer than you were looking for, but it's always helpful to understand what is really going on with antennas, as it can prevent overlooking would-be issues and avoiding them. Anyway, I hope this helped!