AD9850 can generate others waveform beside sine wave

This picture is crucial to understanding what is happening: -

Analog Devices use the example of a reconstructed 20MHz signal (F\$_{OUT}\$) on the diagram and as you can see, at 20MHz it's amplitude is a wee bit less than what it would be if it were much lower in frequency. By "much lower" I mean something like 1MHz and that would have been represented by a vertical line much closer to the Y axis. Here's a drawing showing just that: -

As you can see the out-of-band artifacts are much lower and the first artifact (at 99MHz) is much further away from the 1MHz fundamental frequency. This makes it: -

• Easier to filter-off the artifacts because the gap between 1MHz and 99MHz is massive compared to the original AD drawing where F was 20MHz and it's first artifact at 80MHz.
• The amplitude of the 1MHz sinewave is virtually unity whereas the amplitude of the 20MHz signal has fallen a couple of dB (or so) compared to the amplitude at 1MHz.

Filtering is therefore important; not only do you need to remove the high frequency artifacts in order to get a decent sinewave shape but, you have to compensate for the DAC being an imperfect device that progressively attenuates the wanted signal as you approach the reference clock frequency.

This is normal for all DACs that I'm aware of. A "perfect" DAC would prodice a single impulse of energy at each sample point with the energy level equating to the amplitude and this would give a flat pass-band and not the sinc envelope associated with standard DACs.

It appears there's a bit of circuitry between the output of the dac and the input of the comparator on that board. Connect a four channel scope to the outputs and inputs directly on the chip and view the waveforms present. I suspect you'll find your answer.

You'll notice in the datasheet that the comparator jitter graph is performed at 40MHz, so it's capable of running at that speed. Chances are good you have low signal quality at VINP, an improperly set VINN threshold, or both. As the frequency goes up, the circuit impedance reduces the signal power, and it's having a hard time seeing it.