By my understanding at 5GHz, the input to the filer, S11 should be 0, with maybe a phase change,
You're right that the \$S_{11}\$ should be 0.
But a complex number with magnitude zero doesn't have a defined phase.
and also the reflection coefficient should be 0.
\$S_{11}\$ and the forward reflection coefficient are two names for the same thing, so this is correct (but redundant).
In the Smith chart below, at resonance, 5GHz, it shows S11 as being ( 1 + i0 ). This doesn't seem to make sense to me.
The smith chart is showing the input impedance (not the reflection coefficient) to be \$Z_0 (1 + 0 i)\$.
This should be what you expect. When the input is matched to the characteristic impedance \$Z_0\$ is when the reflection coefficient goes to 0.
Notice the marker is showing \$S_{11}\$ to have magnitude 0.004 and angle -89.762 degrees, which is very close to 0, as you expect.
So what is the meaning of the S11 plot on the smith chart?
The values along the curves on the Smith chart are the input impedance, not \$S_{11}\$.
To read \$S_{11}\$, you need to imagine an ordinary set of polar coordinates overlayed on top of the Smith chart, with the origin at the center of the chart and magnitude 1 corresponding to the outer edge of the chart.
If you want to read \$S_{11}\$ off the chart instead of \$Z_{in}\$ in ADS, you should use a polar plot instead of a Smith chart.
Or, is the Smith chart not appropriate for plotting S parameters? I figured that with S parameters being complex they would normally be plotted on the smith chart
The Smith chart is useful for plotting \$S_{11}\$ or \$S_{22}\$. It doesn't really tell you anything useful if you plot \$S_{21}\$ or \$S_{12}\$ on it.
The whole point of the Smith chart is to visualize the transformation between reflection coefficient and input impedance. You can use a ruler and protractor to plot measured reflection coefficient values on the chart, and then the curves on the chart will tell you the corresponding input impedance values.
But you need to be clear in your head what you are measuring. When you do the S-parameter simulation in ADS, it calculates the S-parameters at each frequency. The input impedance of the DUT can then be derived from the S-parameters, and that is what the Smith chart is helping you do. Assuming you don't want to just add an equation to calculate
$$Z_{in} = \frac{1+S_{11}}{1-S_{11}}Z_0.$$
What you're seeing is called "wrapped phase". The phase angle wraps around a circle, like a clock, and the plot only shows the angle as a value between -180 and 180 degrees. This makes the graph much more compact, especially if many multiples of 360 degrees would be shown. It could have chosen to show the angles between 0 and 360 degrees instead, or any other pair of numbers 360 degrees apart.
You can manipulate the data to unwrap the phase and see a linear plot in your graph by tracking when it crosses the -180 to 180 threshold and adding or subtracting 360 degrees as necessary.
But be careful with unwrapping, since all phase measurements are relative to some reference point. As an example, your chart shows 1GHz at about 125 degrees, and at approximately 1.4GHz it's about 125 - 360 = -235 degrees. But that's just relative to the first measurement of your graph. In this case, since you know you measured a transmission line, you can use the slope of the phase change to extrapolate down to DC and unwrap from there to get absolute phase angles for your plot. But in general, you should avoid extrapolating unless you have a model/equation for how the phase will behave outside of the graph.
See this page for more information:
https://en.wikipedia.org/wiki/Instantaneous_phase_and_frequency
Best Answer
The phase describes how much the signal is delayed in time from the input to the output.
Therefore the the S parameter can describe how much a signal is attenuated AND phase-shifted in time.
A positive phase means that the output signal is leading the input, while a negative phase results in a lagging (delayed) output signal.