Did you use the model of the MC1458 in your simulation, or a generic (ideal) opamp? In the latter case that may explain why it doesn't work. The MC1458 is not a rail-to-rail opamp, and can't work near the rails both for input and output. Especially with a low supply voltage the operating range of the opamp is very limited, and is situated around Vcc/2. The Schmitt-trigger thresholds and your input signal are way below that.
Still, the output signal looks rather strange. What's the scope's vertical resolution? Anyway, I'd try a rail-to-rail opamp first, that's something which has to be fixed anyway, and see what results that gives. BTW, you could also bias the inverting input to Vcc/2 via a resistor divider and couple the signal in via a capacitor, but you'll still have a problem with the output levels. If you use the divider, don't forget to recalculate the Schmitt-trigger resistors.
Note that the LMC6772 chip you are using is not an opamp; it's a comparator that has an open-drain output. Since it can't actively drive its output high, the Schmitt trigger circuit won't work as designed.
The datasheet says, "Refer to the LMC6762 datasheet for a push-pull
output stage version of this device."
However, look at how you have the optocoupler wired up: the anode of the LED goes to the output of the comparator, and the cathode goes to ground. This means that when the comparator output is "high", the LED is driven by the current coming through your resistor network, and when it is "low", the diode is shorted out. This also means that the "high" voltage at the output of the comparator is the forward voltage of the LED. (According to the datasheet, this is at most 1.8V)
Do NOT switch to the push-pull version of the chip unless you also add a current-limiting resistor in series with the LED.
Note also that the website you used to calculate your resistor values is assuming that the opamp in question has bipolar supplies, and that the output swings to -Vcc when the output is "low". Your opamp is being powered by a singled-ended supply, which makes the lower threshold 7.56V when the output is low.
So, the actual threshold voltages of your circuit are 7.92V when the output is at 1.8V and 7.56V when the output is at 0V.
The general equation for the the voltage at the junction of three resistors, each fed by a voltage source is:
$$V_{junction} = \frac{\frac{V_A}{R_A}+\frac{V_B}{R_B}+\frac{V_C}{R_C}}{\frac{1}{R_A}+\frac{1}{R_B}+\frac{1}{R_C}}$$
In your circuit, one of the three resistors is always connected to ground, so you can ignore that term in the numerator. If we plug in the voltages and resistors you used:
$$V_{junction} = \frac{\frac{12 V}{150 \Omega}+\frac{1.8 V}{470 \Omega}}{\frac{1}{150 \Omega}+\frac{1}{470 \Omega}+\frac{1}{560 \Omega}} = 7.92339 V$$
$$V_{junction} = \frac{\frac{12 V}{150 \Omega}+\frac{0 V}{470 \Omega}}{\frac{1}{150 \Omega}+\frac{1}{470 \Omega}+\frac{1}{560 \Omega}} = 7.56141 V$$
But you're right, these thresholds should work fine in your application, and they don't explain the problems you're seeing. I can only surmise that you're seeing some secondary effects arising from the fact that you're trying to operate a "micropower" comparator at some fairly high voltage and current levels.
Best Answer
Short:
Yes, it's a Schmitt trigger.
Yes, you have made a mistake.
Output low / high happens at Vin =~ -10V*
*I have slightly adjusted Vout range to make sums easier. Result is essentially the same. As shown the zener is being used in an unapproved manner.
Longer: Loosely a Schmit trigger can be considered to be a comparator with hysteresis or positive feedback such that the switching decision point alters when the circuit ir triggered to provide a "deadband". On that basis this IS a Schmitt trigger circuit.
Using a zener to clamp Vout as shown is a VERY bad idea.
The opamp is using all its output to try to drive the zener high. This is a non-approved way to do what is desired BUT OK for working through the example
Without using your notes.
Vout swings from about -0.8V to about 4.3V. For practical purposes we can work with 0V and 5V and get an OK answer to the main question.
Vref on the inverting input (OA-)is -5V. This does not change.
Vout is low = 0V when the non inverting input OA+ is below -5V and high = +5V when the non inverting input is above -5V.
When Vout = 0V then the trigger condition occurs when Vin = -10V, so OA+ = -5V.
When OA+ rises above -10V Vout rises to +5V.
So immediately OA+ rises to -2.5V (half way between Vin at -10V and Vout at +5V.
So the opamp is now ell away from the switching point.
To switch back OA+ needs to fall to -5V and to achieve this Vin will need to fall to -15V (as OA- is now +5 - (5 - - 15V) = 5 - 10 = -5V.)
SO the switching boundaries are
-10V rising = output low to high and -15V falling = output = high to low.