Copying an antenna design is hard, it won't work first time even if you keep all dimensions the same. Variations in the PCB dielectric coefficient will change the tuning.
Tuning the transmitter is different from tuning the antenna - the transmit frequency you're measuring is fixed by the crystal oscillator. The antenna is a filter that's independently tuned, and with your test equipment you can't easily sweep it, so you need to tune for best performance at your chosen frequency.
Fortunately its easy to tune the antenna, by cutting and/or extending it. This is the normal process when porting an antenna design to a new PCB, or even just changing PCB manufacturers or board suppliers.
Cutting process Try cutting the tip of the antenna shorter, 0.5 mm at a time. If the results get better, keep going, otherwise extend the track with some wire or copper tape, and start trimming again.
Measuring your results Set up a measurement so you can tell how you are doing, either use the two modules talking to each other, measure the RSSI or measure the range, which is much less accurate. Or measure the transmitted power with a spectrum analyser and a standard antenna. Keep the board under test close to the receive antenna. Rotate the board to get the maximum signal. Then you can start trimming, always measuring in the same position.
If you're going into production with the board, you'll need to control the dielectric. Specify the PCB material carefully, insist that it's from the same batch, etc. You can also keep some spare pads for tuning, zero ohm jumpers, a second capacitor pad in the tuning network, or some other way of adjusting the impedance on an existing board.
Have you read the following: Atmel AT02865 RF Layout with Microstrip ? It deals with exactly what you are doing, same chip and same balun.
One of the key parameter is the FR4 dielectric thickness between the RF ground plane and the microstrips. You have listed yours as 0.18mm (7 mils). In the Atmel app note, it uses an example of 0.25mm (10 mils) FR4 dielectric thickness. So the example is a little different, but is close enough that you can apply all the considerations while adjusting for the small dimensional differences accordingly.
Since the traces are sitting on top of a ground plane, you cannot use the coplanar waveguide model for the transmission lines. Using the microstrip model and dielectric thickness of 0.18mm, I get approx 0.3mm for 50-ohm line.
Also, you do want to isolate the microstrip lines from other elements of your design. For example, a quote directly from the Atmel app note page 10 -
For microstrip designs the copper
pour on layer-1 should be kept away from the transmission line. The underlying Layer-2 ground plane needs to be the
dominant ground reference to minimize variables. A keep-away distance of 4x the dielectric thickness will reduce the
parasitic effects of copper pour to less than 1%. In other words the gap between the microstrip transmission line and
copper pour on layer-1 should be 40 mils or more.
If you look at the balanced 100-ohm connection between the SAMR21 IC and the Balun, it ends up looking more like a "T" shape dipole antenna than a transmission line. I don't think there is anyway to make that better, so just make that connection as short as possible. Also for the same consideration of the preceding quote, I would probably take away that asymmetric ground pour.
Also, I would move the antenna to the left so the tail of antenna at the right is closer to the ground plane.
Best Answer
Your calculations check out for the given values, but keep in mind that the dielectric constant of FR-4 is not tightly controlled, and may vary between 4.35 and 4.7 between manufacturers [1]. Since your trace length is very short, this variation will not have a big effect (you can try the values in the calculator). For more demanding applications, special high-frequency PCB materials (for example: Rogers RO4000 [2]) are available, however they are much more expensive to produce.
It can be beneficial to disable the thermals around the GND-pin holes of the RF connector. By having a solid ground connection, you reduce the parasitic inductance in the return current path, which will improve your signal integrity.
If you use a coplanar waveguide, the copper pours below and on the sides of the conductor must be strongly referenced to each other. This means putting vias to 'stitch' the top and bottom planes together, along both sides of the conductor, to surround it with the ground connection. This is discussed in [3].
The recommended stitching distance between vias should be at most λ/4, with λ/10 as an optimum. For 2.4GHz this results in a via distance of maximum 3.12cm, with 1.25cm recommended. So, for longer trace lengths and higher frequencies stitching becomes more important than in this case with a very short trace length.
[1] https://en.wikipedia.org/wiki/FR-4 see: dielectric constant permittivity
[2] https://www.rogerscorp.com/documents/726/acs/RO4000-LaminatesData-sheet.pdf
[3] Choose the size of via for shielding and stitching