Opamp comparator output interfacing with digital circuit

A saw tooth generator circuit can be built using a constant current source to charge a capacitor in a linear ramp. A comparator is then set to monitor the ramp for its upper threshold. At the upper level the comparator then flips state and is used to discharge the capacitor quickly. If the comparator is setup with positive feedback so that it has both an upper and lower threshold then the comparator will flip back when the capacitor has discharged to the lower level. This would allow the cycle to repeat again. This would be a simple way to get a repeating sawtooth ramp. You may want to buffer the ramp with an opamp voltage follower depending on the type of load you want to use.

Note that it is not directly feasible to get a sawtooth waveform at the same frequency from a square wave or triangle wave. At least not without a lot of extra circuitry so it is generally best to generate your saw tooth separate from the square or triangle and then switch to the sawtooth when needed. Now that said it is entirely possible to setup a circuit that can be used to generate the square wave and triangle wave using the same set of comparators if you choose one function at a time.

Edit: Here is a simulation circuit showing how a sawtooth generator circuit could be built. This example with nominal values runs at about 1kHz.

It appears there's a bit of circuitry between the output of the dac and the input of the comparator on that board. Connect a four channel scope to the outputs and inputs directly on the chip and view the waveforms present. I suspect you'll find your answer.

You'll notice in the datasheet that the comparator jitter graph is performed at 40MHz, so it's capable of running at that speed. Chances are good you have low signal quality at VINP, an improperly set VINN threshold, or both. As the frequency goes up, the circuit impedance reduces the signal power, and it's having a hard time seeing it.