Electrical – Unnatural AD9850 Output Attenuation

At the outset, use a crystal of at least 8 times the desired maximum output frequency, to provide a greater gap between desired cut-off and DDS-generated harmonics:

Using a spectrum analyzer will show you that the first big noise harmonic is at crystal frequency / 2. A minor harmonic is also showing up for me at 2x output frequency, but its amplitude is pretty close to the noise floor.

By increasing the crystal frequency to beyond 4x desired output, the first big harmonic which remains at crystal frequency / 2 provides a lot more headroom for the low-pass attenuation curve.

Thus, for a 10 MHz maximum output frequency, I would use either a 80 MHz or preferably a 160 MHz crystal.

After this is done, there are several ways of dealing with the specific requirement:

• Use LC filters instead of RC filters for an increase in filter attenuation (filter order) for each filter stage.
• Use a 4th or 8th order active low-pass filter consisting of several stages, to increase attenuation per decade. See this paper for some insights: "10 MHz Butterworth Filter Using the Operational Amplifier THS4001"
• Use a tightly tracking AGC (Automatic Gain Control) stage after the output, to keep the output frequency at a specific level regardless of filter stage attenuation - within meaningful limits, of course.

With a wideband Voltage Controlled Amplifier such as the Analog Devices AD600 / 602, two stages of AGC, both with a roll-off at 35 MHz (inherent to the IC), can be integrated, thus providing a stable output signal voltage for frequencies up to 35 MHz, while attenuating the 40 MHz or higher clock noise even further, beyond the preceding low pass filter stages. See figure 37 of the datasheet for an accurate AGC from DC to frequency range of interest:

Multi-stage active filtering coupled with an AGC seems to be the simplest way to achieve flat response up to the desired frequency. If you keep the filter stage corner frequencies sufficiently higher than the maximum desired output frequency, phase shift will also be minimal.

In addition to the normal attenuation at increasing frequencies (as @ChrisStratton and OP pointed out), there is the potential to unwittingly add a low-pass filter to the output by decoupling it. This is based on the fact that the trace (or wire) between the DDS output and the decoupling capacitor will have some resistance, which combines with the capacitor to form an RC filter. The cutoff frequency is defined as: $$ f_{c}=\frac{1}{2 \pi RC} $$ Taking an arbitrary trace resistance of \$60 m\Omega\$ (consistant with a ½ inch, 8 mil, 0.5 oz trace) with your 100nF capacitor, we get: $$ f_{c}=\frac{1}{2 \pi \cdot 0.06 \cdot 100n} \approx 26.5MHz $$ This will change based on the actual resistance, as well as the capacitance of the scope probe (as discussed by @ChrisStratton), but it shows that the capacitance could be having a very real effect on the output amplitude at higher frequencies.