AD9850 and other DDSs: How to stabilize output gain after low pass filtering

It would be useful to know more of the characteristics of the signal and what you need to do with it. You say it has a bandwidth of 3.5 Hz, but what is its upper and lower limit? In particular, do you care about DC? What impedance is the signal? What must the amplified signal drive? What output voltage and therefore gain? What is the overall purpose?

From the sparse information you supplied, it is hard to tell what you need the opamp to do. If the signal includes DC, then a low offset voltage will be necessary so as to not be large compared to the 5 mV input. If DC doesn't matter, then even this can be ignored as long as the offset doesn't cause the amplified signal to clip to either rail. If a lot of gain is needed, then capacitively coupling between multiple gain stages allows you to keep the AC gain but reset the offset to 0 each stage.

I would put a little low pass filtering before the first gain stage. It doesn't need to be tight. One or two passive poles rolling off around 20 Hz would be fine. It won't cut into the signal, but this keeps high frequency noise out of the system as early as possible so that it won't cause non-linear effects in the active circuit. If the result will ultimately go into a microcontroller and be processed there, then all you need is this loose low pass filtering on the input, then amplify to roughly fill the A/D range. Sample fast enough, like at 100 Hz (every 10ms, quite slow for a micro), to not alias given the loose input filter. Once in the micro, you can apply tighter and more accurate filtering, if that is needed. Again, we need more information.

Added:

You now say the signal will go into a microcontroller with a 0-5 V A/D range. A voltage gain of 100 looks to be about right then. Just about any opamp can handle that at this low frequency. Input offset voltage will be important, and it will be useful if the opamp can run off the same 5V supply as the PIC. A Microchip MCP603x with its 150µV input offset sounds like a good fit. As I said above, put a little low pass filtering in front of the opamp and run the output straight into the PIC A/D pin. I would still do some oversampling and additional low pass filtering in the PIC, which won't take much CPU at 100Hz or so sample rate.

Added 2:

Low pass filtering can be performed digitally by the algorithm:

FILT <-- FILT + FF(NEW - FILT)

FF is the "filter fraction" and controls the heaviness of the filter. For FF=0, the filter is infinitely heavy since its output never changes. For FF=1 the output simply follows the input with no filtering. Useful values are obviously in between. To keep the computation simple in a small micro, you usually chose FF so that it is 1 / 2^N. In that case the multiply by FF becomes a right shift by N bits.

For example, here is a plot of the response of two cascaded filters each with FF=1/4:

If you sample at 100 Hz then there are about 14 samples for every minimum necessary reading to support 3.5 Hz. From the step response you can see that this filter settles to about 92% within a 1/2 cycle of your 3.5 Hz maximum frequency. From the impulse response you can see that glitches will be attenuated by about 9.

Almost always when processing real world signals you want to oversample and then add a little digital low pass filtering. About the only exception I run into regularly is when the micro is doing pulse by pulse control of a switching power supply. In that case you need instantaneous readings as best you can manage and the speeds are high. For other things where the upper frequency is 1 kHz or less, digital low pass filtering is pretty much standard practise to attenuate noise.

Any waveform other than a sine at 12.5 MHz will be composed of the fundamental frequency 12.5 MHz, plus harmonics above that. For ideal waveforms with sharp edges, the harmonics extend to infinity.

It sounds like you have added a filter with a cutoff frequency around 12.5 MHz, so this is why your non-sinusoidal outputs don't look like they should -- your filter is stripping off the harmonics which give those waves their non-sinusoidal identity.

To illustrate, here are a few animations from Wikipedia which show various non-sinusoidal waveforms as harmonics are added/removed:

What your filter should be doing is stripping off the harmonics generated by the impulses at each sample output by your DAC. This is called an anti-aliasing filter, and ideally it's a brick-wall low-pass filter with a cutoff equal to half your sampling frequency. Given that true brick-wall filters aren't realizable, the cutoff is usually a bit below half the sampling frequency, and the roll-off is made as steep as feasible. If you can sample significantly more than twice the highest frequency of interest, then the filter design isn't as critical.

It also follows that if you want to synthesize a 12.5 MHz square (or triangle, or sawtooth...) wave, you need a DAC sampling fast enough to also synthesize the higher harmonics in that wave. How much higher depends on how closely you want to approach an ideal square/triangle/sawtooth wave.