Looking for possible cause for poor regulation in self designed transformer

You put the horse in front of the cart by presuming that galvanic isolation is the only solution to your problem. Do you even know what your problem is? Have you measured the noise on the input? How did you measure it? Broadband measurements are hard!

There is no such thing as a perfect galvanic isolation. If you're dealing with bad high frequency common-mode noise, the primary-to-secondary capacitance in an isolation transformer will happily transfer that noise to your isolated circuit. The rest of the circuit, and its output cabling, will happily convert the common mode into differential mode, corrupting your signals.

The first step is determining what kind of a noise you're dealing with. If you're using a switching wall-wart, common mode noise is a real deal and should be addressed first. Typically, a good mid-frequency common-mode choke placed on the 5V input, followed by a high-frequency common-mode choke, followed by differential mode low-pass filter(s), will work quite well. The isolation can be used to break a ground loop, but it won't magically help with noise, and might be completely unnecessary.

The general idea, shown below, is to:

1. Have common mode filtering first, as any asymmetric circuit will convert common mode into differential mode.

2. Have high-frequency attenuating elements first, before the low-frequency element's parasitics get excited by the high frequencies.

L1 can be a high-frequency common mode choke, say with attenuation peak around 25MHz, followed by the choke L2 with attenuation peak of a MHz or less. L3 is to get rid of excessive ripple, and can be a part with high series resistance to lower the Q of the circuit. FB1 is a SMD ferrite bead. C1 and C2 are "small", on the order of 1-470nF. C3 and C4 can be 1-47u, depending on what U1 needs for stability.

Other parts of your circuit certainly can pollute the 3.3V supply - it's impossible to ascertain how bad of an effect they have without knowing, well, what's there.

^{simulate this circuit – Schematic created using CircuitLab}

In an ATtinyXX there are two timers. Timer0 can be clocked at the same CPU frequency, by setting the corresponding bits (see below) in the TCCR0B (for timer 0).

At 8 MHz and 8 bit, the PWM will have a frequency of \$\frac{8\ MHz}{2^8} = 31.25\ kHz\$.

Timer 1, instead, can be also clocked by the PLL (up to 64 MHz), which multiplies the internal frequency by 8, yielding a maximum frequency of \$\frac{64\ MHz}{2^8}= 250\ kHz\$.

Here's the timer 0 control register B:

And here's the CS0x setting table:

Here's the timer 1 control register:

There you must set the correct clock prescaler value according to the table:

To enable PLL you must set bits PCKE and PLLE in PLLCSR.