There are two key frequencies you can quickly identify from the bode plot of the closed-loop system. The unity gain frequency is where the gain is 0dB i.e. neither any amplification nor any attenuation. The phase inversion frequency is where the phase is 180 degrees.
If, at the "phase inversion" frequency where the phase shift is 180 degrees, there's more than 0dB gain; then it's as though an op amp's inverting and non-inverting inputs were effectively swapped. Negative feedback at this frequency behaves like positive feedback, causing divergence instead of convergence. This makes just about any closed-loop system become unstable, regardless of whether it uses an op-amp or any other similar way of closing the loop.
In your example bode plots, the -180 degree phase shift occurs at about 5 rad/sec. And since there is more than 0dB of gain at that frequency, the system will tend to oscillate. And, the frequency where the gain drops to 0dB is just under 20 rad/sec, where the phase shift is about -245 (?) degrees. So both the gain margin and the phase margin are negative, and stability is not assured.
If the closed-loop gain was adjusted (without affecting phase response) such that the unity-gain frequency was 3 rad/sec, where the phase shift is -120 degrees, then such a system would have a comfortable 60 degrees of phase margin. This is a generally accepted design rule for most op-amp circuits.
So in terms of the Bode plot, phase margin is determined at the frequency where the gain is 0dB (unity gain): subtract the corresponding phase shift from 180 degrees.
Similarly, gain margin is determined at the frequency where the phase shift is 180 degrees (phase inversion): subtract the corresponding gain from 0dB.
There could be other conditions besides gain at 180 degrees phase shift, that might cause a system to become unstable; but for closed-loop systems built around any standard Op-Amp, gain at 180 degrees is typically the main cause of instability.
The general condition for stability is a bit more complicated, it involves tracing a contour on the complex frequency plane and comparing with poles and zeroes (Wikipedia Nyquist stability criterion http://en.wikipedia.org/wiki/Nyquist_stability_criterion ); I briefly learned it in school and never used it in 20+ years at my job. For engineering purposes we're interested in keeping the system stable, with some margin to guard against variations (like from one device to another device, or variation over temperature, or over time.) There's often enough uncertainty that a simplified heuristic like gain margin or phase margin is preferable to an exact, analytic mathematical proof like the Nyquist stability criterion. So the simplified heuristic is that as long as the 180 degree phase shift point is attenuated below unity gain, that is just barely sufficient to avoid an amplifier behaving as an oscillator.
On a side note: when you read the data sheets for commercially available op amps, some will be advertised as "unity-gain stable" and others will be advertised as "uncompensated". Many manufacturers offer both internally-compensated and uncompensated versions of the same basic op-amp. The internally-compensated version has its gain low-pass filtered, such that it can be operated in a unity-gain configuration with adequate phase margin. The uncompensated version has higher open-loop gain and can be operated with more bandwidth, but requires a minimum closed-loop gain (like 2V/V or 5V/V) for stable operation.
Gain and phase margin are usually applied to systems that are amplifiers of some sort with negative feedback around them. The more negative feedback, the tighter the system is controlled. However, you don't want to provide feedback in such a way that the system will oscillate. The gain and phase margin are two metrics to tell you how close the system is to oscillation (instability).
A system with over-unity gain will oscillate with positive feedback. Usually the intent is to stabilize a system by using negative feedback. However, if this is phase shifted by 180°, then it becomes positive feedback, and the system will oscillate. This can happen due to various characteristics of the system itself or what happens to the feedback signal.
Note the two criteria for oscillation: a gain greater than 1, and positive feedback. Since we are usually trying to provide negative feedback, we think of positive feedback as what happens when there is a 180° phase shift in the loop. This therefore gives us two metrics to decide how close to oscillation the system is. These are the phase shift at unity gain, and the gain at 180° phase shift. The first had better be below 180°, and the second had better be below 1. The extent they are less than 180° and less than 1 is how much room, or margin, there is. 180° minus the actual phase shift at unity gain is the phase margin, and 1 divided by the gain at 180° phase shift is the gain margin.
Since the main problem is usually that the overall phase and gain change as a function of frequency, loop gain and phase shift are often plotted as a function of Log(frequency). The gain curve is then basically a Bode plot. You have to examine the two curves carefully to see that the system stays away from the combination of characteristics that will make it oscillate. When this is the main point, something called a stability diagram shows you more directly how close the system is to instability and at what operating point. That closest approach to instability is called the stability margin.
Best Answer
There are plenty of compensator types you can think of when attempting to compensate a given plant. First off, you need the plant dynamic response. This is what is called the control-to-output transfer function. From this response, you can infer what compensation strategy is needed to fulfill your goals. Basically, at least in power electronics, there are three compensator types: type 1, type 2 and type 3. They can be built around an active amplifier like an op amp, an OTA and a shunt regulator (TL431) for instance. The below picture shows you what they can do in terms of dynamic response.
You select the type by knowing the amount of "phase boost" you need to meet the phase margin criteria. The phase boost is the amount of positive phase lead you need to compensate the lag incurred by the power stage at the selected crossover frequency \$f_c\$.
A type 1 is a simple integrator. It features a pole at the origin and lags the phase by 270°. There is no boost. A type 2 combines a pole at the origin and a pole-zero pair. By adjusting the distance between the pole and the zero, you adjust the boost up to 90° in theory. Finally, a type 3 adds another pole/zero pair to the original type 2 and lets you boost the phase up to 180°. I have a complete seminar on the subject of compensator that you can download here and a book you could consider for closing the loop is this one. Good luck with your project!