Electronic – Transimpedance amplifier with diode feedback

I suspect that you have a significant AC signal (or noise) superimposed on your dc signal that you are unaware of. The cap would usually get rid of these AC effects. Basically as the AC signal/noise gets bigger it clamps against the power rails and this is where you see what appear to be non-linearities.

Try looking at the output using an oscilloscope to confirm this.

Here is my best guess:
The Inductance near the sumpoint has a large role in this.

By illuminating certain diodes, you can pull the voltage either positive or negative. See, you have 2 12V power supplies, with diodes in the middle, and each of them is connected to the ground.
As expected, this setup sums the intensity between the two. The comparator then does it's thing, and you get your result. However, there is no induction filter between Output and ground for your wires. I would make sure to add one (10-50kOhms), as currently it seems that your oscilloscope is the only drain.