There are two key frequencies you can quickly identify from the bode plot of the closed-loop system. The unity gain frequency is where the gain is 0dB i.e. neither any amplification nor any attenuation. The phase inversion frequency is where the phase is 180 degrees.
If, at the "phase inversion" frequency where the phase shift is 180 degrees, there's more than 0dB gain; then it's as though an op amp's inverting and non-inverting inputs were effectively swapped. Negative feedback at this frequency behaves like positive feedback, causing divergence instead of convergence. This makes just about any closed-loop system become unstable, regardless of whether it uses an op-amp or any other similar way of closing the loop.
In your example bode plots, the -180 degree phase shift occurs at about 5 rad/sec. And since there is more than 0dB of gain at that frequency, the system will tend to oscillate. And, the frequency where the gain drops to 0dB is just under 20 rad/sec, where the phase shift is about -245 (?) degrees. So both the gain margin and the phase margin are negative, and stability is not assured.
If the closed-loop gain was adjusted (without affecting phase response) such that the unity-gain frequency was 3 rad/sec, where the phase shift is -120 degrees, then such a system would have a comfortable 60 degrees of phase margin. This is a generally accepted design rule for most op-amp circuits.
So in terms of the Bode plot, phase margin is determined at the frequency where the gain is 0dB (unity gain): subtract the corresponding phase shift from 180 degrees.
Similarly, gain margin is determined at the frequency where the phase shift is 180 degrees (phase inversion): subtract the corresponding gain from 0dB.
There could be other conditions besides gain at 180 degrees phase shift, that might cause a system to become unstable; but for closed-loop systems built around any standard Op-Amp, gain at 180 degrees is typically the main cause of instability.
The general condition for stability is a bit more complicated, it involves tracing a contour on the complex frequency plane and comparing with poles and zeroes (Wikipedia Nyquist stability criterion http://en.wikipedia.org/wiki/Nyquist_stability_criterion ); I briefly learned it in school and never used it in 20+ years at my job. For engineering purposes we're interested in keeping the system stable, with some margin to guard against variations (like from one device to another device, or variation over temperature, or over time.) There's often enough uncertainty that a simplified heuristic like gain margin or phase margin is preferable to an exact, analytic mathematical proof like the Nyquist stability criterion. So the simplified heuristic is that as long as the 180 degree phase shift point is attenuated below unity gain, that is just barely sufficient to avoid an amplifier behaving as an oscillator.
On a side note: when you read the data sheets for commercially available op amps, some will be advertised as "unity-gain stable" and others will be advertised as "uncompensated". Many manufacturers offer both internally-compensated and uncompensated versions of the same basic op-amp. The internally-compensated version has its gain low-pass filtered, such that it can be operated in a unity-gain configuration with adequate phase margin. The uncompensated version has higher open-loop gain and can be operated with more bandwidth, but requires a minimum closed-loop gain (like 2V/V or 5V/V) for stable operation.
Best Answer
That's at DC where nobody really worries about phase margin because it's never going to be an issue. Look at a typical open-loop response of an op-amp: -
Picture source and other relevant information that could be useful.
At 1 kHz the open loop response might have dropped to 60 dB (G = 1000). At 1 MHz, the o/l gain is only 10 and this is an area where quite a few op-amp circuits have problems. Arctan of 10 (assuming a unity gain situation) is 84 degrees and consistent with the graph above. At 10 MHz the gain is unity and arctan of 1 is 45 degrees i.e. not a million miles off.
If you know the T.F. of the forward gain device and you know the feedback T.F. then certainly you can calculate phase margin by considering the loop broken with a signal being injected at the input and the output being taken from where the break is. But you have to respect impedances and loading when you open the loop and sometimes it can be difficult to realize.