Electronic – Digital potentiometer as INA129 gain

Any waveform other than a sine at 12.5 MHz will be composed of the fundamental frequency 12.5 MHz, plus harmonics above that. For ideal waveforms with sharp edges, the harmonics extend to infinity.

It sounds like you have added a filter with a cutoff frequency around 12.5 MHz, so this is why your non-sinusoidal outputs don't look like they should -- your filter is stripping off the harmonics which give those waves their non-sinusoidal identity.

To illustrate, here are a few animations from Wikipedia which show various non-sinusoidal waveforms as harmonics are added/removed:

What your filter should be doing is stripping off the harmonics generated by the impulses at each sample output by your DAC. This is called an anti-aliasing filter, and ideally it's a brick-wall low-pass filter with a cutoff equal to half your sampling frequency. Given that true brick-wall filters aren't realizable, the cutoff is usually a bit below half the sampling frequency, and the roll-off is made as steep as feasible. If you can sample significantly more than twice the highest frequency of interest, then the filter design isn't as critical.

It also follows that if you want to synthesize a 12.5 MHz square (or triangle, or sawtooth...) wave, you need a DAC sampling fast enough to also synthesize the higher harmonics in that wave. How much higher depends on how closely you want to approach an ideal square/triangle/sawtooth wave.

You could easily do a highly flexible DDS on an STM32F4 Discovery board for ~$15 using an internal DAC, DMA, a timer and a sine lookup table. People have blogged examples of similar things if you look for them.

In order to get the required frequency (the STM32F4xx DAC's can only do 300ksps at full swing, which equates to ~1V/\$\mu\$s) you will need to constrain the range of the DAC.

For a sinusoid, maximum slew rate (in V/s) is \$2 \pi \times f \times A\$. Inserting the values from the OP's requirements and assuming Vref of the DAC is 3.3V (which if I recall correctly is the case for an STM32F4Disco), a 3.3Vpp sine wave at 100kHz, max slew rate is \$2 \pi \times 10^5 \times 3.3 / 2 = 1036726\$V/s, or \$\approx1.04\$V/\$\mu\$s.

In order to comply with this constraint, the sine wave's amplitude has to be limited, from \$3.3\$V to \$3.3 \times 1 / 1.04 = 3.18\$V.

This now brings in the apparent issue of the 300ksps limit on the STM32F4 DAC. That limit is a furphy. I don't know what the actual physical upper limit is, but I suspect it's the APB1 bus speed. I do know (because I've done it) that you can write at least 2Msps to the DAC and as long as you respect the 1V/us slew rate it will behave predictably. So, you can do 16 samples per cycle for a 100kHz sine wave, a 1.6Msps update rate, as long as you limit the sine wave amplitude to 3.18V instead of 3.3V.

At some point you're probably going to have to deal with digital pots because the reality is that if you want a smooth, continuous output signal, particularly at low speed, then you're going to have to run your DDS through a filter, and that filter is going to have to be adjustable - you can't just put a 100kHz LPF on it and expect it to be useful at sub-1Hz frequencies.