Electronic – Does the noise voltage formula apply even if the impedance is not matched

Your problem is combining the voltage sources. This is incorrect, first because you can't add uncorrelated noise to each other, second because we don't even need to do worry about the other resistor's power generation for this problem.

Since we are only looking at the power that one resistor transfers to another, we look only at the voltage it generates and transfers to the other.

^{simulate this circuit – Schematic created using CircuitLab}

Now, we look at the voltage that would appear on the transferred resistor, which would be exactly half.

$$ V_\mathrm{transferred} = \frac{\sqrt{4k_BT \Delta FR}}{2} $$

Now with power: $$ P_\mathrm{transferred} = \frac{V^2}{R} $$ $$ P_\mathrm{transferred} = \frac{4k_BT \Delta FR}{4R} $$ $$ P_\mathrm{transferred} = k_BT \Delta F $$

Hope this helps!

I'm not sure specifically about the noise on your circuit, but here is a pretty extensive help guide for laying out TIA circuits:

http://www.linear.com/solutions/5633

I can't tell if you have voided the ground and power plane on your input trace from the photodiode. However, you might try the following. Stand the input cap and resistor on end (tombstone). Solder a very fine wire (40AWG perhaps) from one end that is airborne to the photodiode output pin. This will minimize the input capacitance, and thus give you the best high frequency response.

Another less drastic thing to try is to trim the pads of Rf and Cf to the smallest possible, then solder them on the board sideways. Parasitic input capacitance is your enemy at high frequencies, and both these ideas are aimed at minimizing it. Although expensive in mass production, it may give you some ideas to make it better performance.

Some other ideas - use 0402's instead of 0805's or 0603's. This will decrease input capacitance as well.

Another idea that was also in the LT literature was to run a ground trace between the pads of your input resistor. This brings the field strength to 0. I honestly don't have a good feel for how this helps, but they wrap some words around it in the link I gave above.

Good luck! You should post some screenshots of your frequency response and let us know what you did - what worked and what didn't.