Some comments from my side:
1.) The stability check in the BODE diagram concerns the LOOP GAIN response only (because once you did mention "closed-loop system" in your text.)
2.) The shown system is "conditionally stable". That means: It is stable - regardless the properties at the frequency A. However, if you REDUCE the gain within the loop until the gain crosses the point A (the phase remains unchanged) the closed-loop system will be unstable.
Such conditional stable system should be avoided because a gain reduction can happen due to aging or other damping effects. Remember: Classical feedback systems with a continuos decreasing loop phase will become unstable (under closed-loop conditions) for rising gain values (beyond a certain limit) only.
As to your next question - the input signal Vi does not influence stability properties at all. Stability is determined by the loop components only. That is the reason, we investigate the loop gain only.
EDIT: Here is an explanation why the closed loop (your example) will be stable:
If a closed-loop system is unstable, this point of instability also must be "stable". That means - either we will have "stable" and continuous oscillations or the output is latched at one of the supply voltage rails. In both cases, this point of instability is fixed.
Now - what happens at the point A in your example? Here we have a rising phase which is identical to a NEGATIVE group delay at this point (group delay is defined as the negative phase slope). This is an indication for the unability of the closed-loop system to let the amplitudes rise (oscillations or latching at the supply rail). Rather, the system returns to a stable operating point.
A final information: The stability check investigates either (a) the -180deg line or (b) the -360 deg line. This depends on what you are investigating: (a) Either the simple product GH or (b) the loop gain LG which is LG=-GH.
There are two key frequencies you can quickly identify from the bode plot of the closed-loop system. The unity gain frequency is where the gain is 0dB i.e. neither any amplification nor any attenuation. The phase inversion frequency is where the phase is 180 degrees.
If, at the "phase inversion" frequency where the phase shift is 180 degrees, there's more than 0dB gain; then it's as though an op amp's inverting and non-inverting inputs were effectively swapped. Negative feedback at this frequency behaves like positive feedback, causing divergence instead of convergence. This makes just about any closed-loop system become unstable, regardless of whether it uses an op-amp or any other similar way of closing the loop.
In your example bode plots, the -180 degree phase shift occurs at about 5 rad/sec. And since there is more than 0dB of gain at that frequency, the system will tend to oscillate. And, the frequency where the gain drops to 0dB is just under 20 rad/sec, where the phase shift is about -245 (?) degrees. So both the gain margin and the phase margin are negative, and stability is not assured.
If the closed-loop gain was adjusted (without affecting phase response) such that the unity-gain frequency was 3 rad/sec, where the phase shift is -120 degrees, then such a system would have a comfortable 60 degrees of phase margin. This is a generally accepted design rule for most op-amp circuits.
So in terms of the Bode plot, phase margin is determined at the frequency where the gain is 0dB (unity gain): subtract the corresponding phase shift from 180 degrees.
Similarly, gain margin is determined at the frequency where the phase shift is 180 degrees (phase inversion): subtract the corresponding gain from 0dB.
There could be other conditions besides gain at 180 degrees phase shift, that might cause a system to become unstable; but for closed-loop systems built around any standard Op-Amp, gain at 180 degrees is typically the main cause of instability.
The general condition for stability is a bit more complicated, it involves tracing a contour on the complex frequency plane and comparing with poles and zeroes (Wikipedia Nyquist stability criterion http://en.wikipedia.org/wiki/Nyquist_stability_criterion ); I briefly learned it in school and never used it in 20+ years at my job. For engineering purposes we're interested in keeping the system stable, with some margin to guard against variations (like from one device to another device, or variation over temperature, or over time.) There's often enough uncertainty that a simplified heuristic like gain margin or phase margin is preferable to an exact, analytic mathematical proof like the Nyquist stability criterion. So the simplified heuristic is that as long as the 180 degree phase shift point is attenuated below unity gain, that is just barely sufficient to avoid an amplifier behaving as an oscillator.
On a side note: when you read the data sheets for commercially available op amps, some will be advertised as "unity-gain stable" and others will be advertised as "uncompensated". Many manufacturers offer both internally-compensated and uncompensated versions of the same basic op-amp. The internally-compensated version has its gain low-pass filtered, such that it can be operated in a unity-gain configuration with adequate phase margin. The uncompensated version has higher open-loop gain and can be operated with more bandwidth, but requires a minimum closed-loop gain (like 2V/V or 5V/V) for stable operation.
Best Answer
The author uses a different definition than I'm used to. According to wikipedia:
While your book describes it using:
So your book includes RHP poles, while Wikipedia's definition requires a stable system, so no RHP poles.
If RHP poles are allowed then it is relatively simple to find an example:
$$ F(s) = -\frac{2s\cdot (2s+3)}{(s-1)(s+5)} $$
The bode plot looks like this:
This shows a GM of about -3.67dB. So it technically violates the Bode criterion, but it is stable when closing the loop.
The closed-loop transfer function is:
$$ F_{cl}(s) = \frac{2s\cdot (2s + 3)}{3s^2 + 2s + 5} $$
which has only poles in the LHP. You can kind of see how I constructed it from the Nyquist plot:
In order to have a GM < 0, you want the transfer function to cross (not end/start at) the x-axis (real axis) left of \$s=-1\$. Given that I had a pole in the RHP, I want the curve to loop around \$-1\$ CCW once to make sure it is stable (\$Z = N + P, P = 1 \Rightarrow N = -1\$). So I just shifted and scaled the graph until \$s=-1\$ was in the right part of the \$\infty\$-figure.
This part has been added after the question in the comments
Yes it is possible. Take for example the following stable system:
$$ F(s) = \frac{1}{150}\frac{(s + 10)(s^2 + 6s + 4)(s^2 - 0.1s + 2)}{(s+1)^2} $$
It has 2 zero's in the RHP so it is a nonminimum phase system. It does not contain poles in the RHP so it is also stable.
The bode plot looks like this:
But the closed loop response is stable. The Nyquist diagram shows you how that is possible.
The two low-frequency LHP poles and LHP zero will make the curve start downwards. Then we can use multiple zero's in left and right half planes to make the curve go move up and down. I made sure to use two RHP zero's and not one to avoid that the curve would start on the left instead of on the right (I wanted to start at \$0^\circ\$ phase for this example).