$$
...=\frac{V_xR'_L}{-g_mR'_LI_xR'-I_xR'+V_x}\cdot\frac{\frac{1}{V_x}}{\frac{1}{V_x}}=...\\
$$
note that those pesky \$\frac{I_x}{V_x}=\frac{1}{R_{gd}}\$. You need to rearrange your equation and get to:
From your own source, and partly from my memory of Zero-Value Time Constant Analysis calculations, you seem to be confused about Dominant Poles, Non-Dominant Poles, and Bandwidth.
1) What you state as the Dominant pole calculation is actually the bandwidth calculation. (omega -3dB) = 1/(summation of all times constants)
2) Dominant poles are the pole who's time constant is at least one decade below the other poles. (Literally took this from the bottom of the linked page)
3) Non-Dominant poles are ALL other poles that are not dominant (per definition).
Taking these correction into account, simply calculate the three poles (three capacitors, three poles) using your short-circuit analysis (ZVTC, as I was introduced to calling it). If one of the pole fits the definition of a dominant pole, then the other two poles are the non-dominate poles. If no pole is dominant then there are three non-dominant poles.
In the context of the paper you linked on switching converters, it is the response time introduced by the filter that causes problems. Basically it's the transient response time: -
It is made more complex because the load can change and thus the damping ratio can change so it is difficult to know how long the output may take to get to within (say) 5% of its final settling point.
Because it is inside a feedback loop, instabilities can occur if not properly managed. When considered from the frequency domain, the RLC low pass filter can rapidly introduce a 180 degrees phase shift over a short span of frequencies from just below resonance to just above resonance: -
If you took the dotted-green curves as an example it appears to be roughly "butterworth" in shape and at 50% resonance it introduces a phase shift of about 35 degrees whereas at twice resonance this has shifted to about 145 degrees. This can easily cause instability if not properly managed.
In short, I believe they actually mean "time delay to reasonably settle" rather than the time constant associated with a simple RC network.
Best Answer
You are nearly done.
$$ R_{gd} = \frac{V_xR'_L}{g_mV_{gs}R'_L-I_xR'+V_x} $$
Use \$V_{gs}=-I_xR'\$:
$$ ...=\frac{V_xR'_L}{-g_mR'_LI_xR'-I_xR'+V_x}\cdot\frac{\frac{1}{V_x}}{\frac{1}{V_x}}=...\\ $$ note that those pesky \$\frac{I_x}{V_x}=\frac{1}{R_{gd}}\$. You need to rearrange your equation and get to:
$$ R_{gd}\left(1-\frac{R'}{R_{gd}}-\frac{gmR'_LR'}{R_{gd}}\right)=R'_L $$
Getting your book result is now straightforward.