I've been trying to do a very low light level project myself the past 2 days with photodiodes and phototransistors. This is for people like myself and the original poster who are pushing light detection without a photomultiplier to the limit (below 0.1 mW/cm^2).
I looked at the first receiver module and its minimum irradiance detection was 0.2 mW/m^2 which is about 10,000 times more (less capable) than what discrete photodiodes and phototransistors can do (maybe they meant cm^2 instead of m^2?). Neither are good for really low light levels according to "Art of Electronics" (1 uA per uW of light page 996), totally not capable of getting near what the human eye can do due to leakage current and noise. He describes using photomultipliers which may be required if your light levels are too low. However, in shining light through my fingers in a well-lit room, I am able to see what my eye can't detect on an oscilliscope (with either PhotoDiode or PhotoTransistor).
Assuming his 1 uA per uW is correct, here is an example: a 5 mm photodiodes and phototransistors have an area of 20 micro m^2. So 1 uW/m^2 (1/1000th of noon sunlight) would generate 20 uA (according to Art of Electr) . [[ 1/1000th of noon sunlight is 1 W/m^2 which is about twice as strong as a 20W incadescent light at 1 meter (6W light output into 12 m^2 surface area of a surrounding sphere). ]]
However, my 880nm phototransistor datasheet indicates 600 uA at 1W/m^2 (0.1 mW/cm^2), wich is 30 times more. This assumes all the light is within the active range of the diode's junction.
Sharp has a much better application note, but it seems to be lacking in explaining which design is best for which situations. Figure 13 is most applicable to what original poster and I need, and figure 10B is very interesting but I don't know what they mean by "improves response". http://physlab.lums.edu.pk/images/1/10/Photodiode_circuit.pdf
When used with an op amp, a phototransistor may not capabale of getting as good of a gain as a photodiode for very low light levels because it to uses a "cheap" method of getting its initial gain (transistor instead of op amp). I suspect a photodiode with a JFET op amp (very low input current) would ultimately provide a higher gain with less noise. In any event, the photodiode or phototransistor with the largest optical receiving area might have the best ability to detect low light levels, but that might also increase noise and leakage by a proportional amount and they are usually the underlying problem. So there is a limit to this type of light detection and ideally efficient phototransistors and photodiodes may ultimately be equally good when used with an op amp, but theoretically I suspect photodiode is a little better. It will be the op amp design that matters and I think figure 13 with a JFET op amp and a well-chosen shunt capacitor across the feedback will be best for phototransistor.
For the dual supply op amp, you can use a "lowish" valued resistor pair (two 1k for 10V Vcc to get 5 mA bias) to split the voltage to create a false ground for the +Vin.
I found R=1M for the feedback resistor much better than R=4.7M. Forrest Mimms in his simple opto book used a 10 M with a parallel 0.002uF and a solar cell instead of a phototransistor or photodiode for "extrememly" low light levels" (maybe a solar cell would be better for your application) It seems all PN junctions seem to operate as a solar cell to some extent, as I've read of using clear-cased small signal diodes to detect light. I'm using a regular 830 nm LED as my "photodiode".
The lens angle of whichever 5mm optical diode you use makes a big difference. +/-10 degrees is roughly 4 times more sensitive than +/-20 degrees....if the light source is coming in from less than +/-10 degrees. If the light source is a big area that is +/-20 degrees in front, then it doesn't matter.
I tested the two circuits below. I could detect 0.3V, 5 ms pulses on the phototransistor's Vo which means 0.3 uA which means 0.05 uW/cm^2 if my reading of the datasheet is correct and if it remained linear (big ifs) all the way down to 0.3uA. Maybe it was 5 uW/cm^2. If 0.05 uW/cm^2 is correct, then the off-the-shelf 830 LED was reading down to 0.5 uW/cm^2. I was shining 10 mW 830 nm light through 1 cm of tissue (my finger). I know that if the light levels I was working with were red, it would have been barely visible. The link below shows using 500 M ohm feedback with a photodiode, indicating much lower light levels. Notice the orientation of their photodiode, which is the same as my LED (backwards from most internet links). I got better results this way.
http://www.optics.arizona.edu/palmer/OPTI400/SuppDocs/pd_char.pdf
I'm assuming your circuit looks like
simulate this circuit – Schematic created using CircuitLab
and your expectation that "it should saturate" is exactly correct. It IS saturating. The op amp is trying to drive the output below zero, but it cannot since it's a single-supply amplifier.
If you're going to use a TIA you have to use a negative bias to get a positive output.
ETA: Oh yes, and just to make your day - you may well have killed your op amps. Maybe not, since designers have gotten pretty good at protecting chips against overload, but this is pretty extreme. Not as extreme as getting hit by lightning, but that kills most chips, too. You need to check them in a less stressful circuit if you want to reuse them.
Best Answer
The way you're checking your circuit is not appropriate. The dark current specified in the datasheet is for 20 V reverse voltage applied. However your circuit does not reverse bias the photodiode.
The datasheet shows typical dark current vs reverse bias:
With no reverse bias, you should expect 0 dark current, as in any other diode with 0 V applied.
You need to test your circuit with some light applied to the photodiode. An input signal on the order of 10 to 100 nW (depending on wavelength) should produce a measurable output signal. Be aware that your photodiode has a strongly directional response, and the input light must be entering from nearly "straight on" to be detected. This will limit your ability to detect diffuse reflections.