TL081 is not a particularly good choice for low DC voltages (it's better for AC coupled designs). Precision op-amps have offset voltages in the 10's of microvolts or lower, and there are even auto-zero op-amps that have negligible offset voltage, an drifts in the tens of nV/°C (at some significant cost in other characteristics). It's also quite noisy (25nV/\$ \sqrt{Hz}\$), but at least has a typical flicker noise corner frequency in the 1Hz range. Aside from high input Z, it has basically one really nice advantage, it's really cheap, and widely available, which is why I've actually designed such a 37-year-old op-amp into a new product recently.
A good general purpose (low noise, capable of handling +/-15V supplies, low distorion for AC signals) precision op-amp suitable for mV levels might be the OPA209A.
You can certainly null the offset voltage of your TL081 out using a trimpot as shown on the datasheet, but it won't stay that well nulled for long. A 10°C change will typically change the offset voltage by 100uV, and about one out of every two will be worse (there's no guarantee how much worse, but a guess would be most are better than +/-30uV/°C). An OPA209 is going to be roughly an order of magnitude better.
There are probably going to be better (and many worse) choices for any given application, all things considered. It's amazing what performance you can get for a a few dollars, so it's worth looking around rather than trying to make a silk purse of a sow's ear.
Just to give you an idea of the kind of (in) accuracy you could get, consider that the gain of the TL081 is only guaranteed to be >15,000, so a gain of 1000 amplifier could have a gain error in the 6% range even without the input offset error (which would be very temperature dependent, and has a -3dB corner of something like 20Hz. Cascading two \$\sqrt {1000}\$ gain amplifiers would help with that (null only the first one).
If the range is, say +/-5mV input, frequency is 0.001 to 1Hz and required accuracy 1% of FS + 50uV**, it might be typically *** okay in a lab environment with a light output load, if you null it after warm-up.
** Instrumentation type specification- it means the output could be as much as +/- 100mV from the ideal value with any input, so a 1mV input could give you 900mV or 1100mV.
*** "Typically" means that one chip might be okay, and the next might not meet the requirements. Guaranteed value is probably 10 times worse.
I'm afraid you've missed the point of the circuit. The second op amp does not need any diodes. You can adjust the offset with one adjustment of the second op amp.
There is one slight issue which you might not have considered, though. Unless R8 and R9 are exactly of a 1:2 ratio, the output will have slightly different gains for signals of different polarity. Since you appear to be looking at overall amplitude of an AC signal, this probably is not a concern.
Best Answer
There are a range of methods which can be used to provide offset voltage compensation.
The best method to use varies with the application circuit, but all either
apply a variable current to a circuit node
or vary the voltage of a node which a circuit element connects to.
The methods described below can easily be applied to your circuit by
Adding a divider and potentiometer at the ground end of your R2.
The ease of use of this method is improved by adding one two-resistor divider to the potentiometer voltage, as explained below.
Or a say 100 kohm resistor from the op-amp inverting input can be fed by a 10 kohm potentiometer connected to +/- 15 V. This injects a small current into the node which causes an offset voltage.
Current injection effectively occurs at a high impedance point and voltage adjustment at a low impedance point, but both methods are functionally equivalent. That is, injecting a current causes it to flow in related circuitry and causes a voltage change, and adjusting voltage causes current flows to alter.
To compensate for an offset voltage by injecting a current you can apply an adjustable voltage from a potentiometer via a high-value resistor to an appropriate circuit node. To adjust a "ground" voltage that a resistor connects to, you can connect it to a potentiometer which is able to vary either side of ground.
The diagram below shows one method. Here Rf would usually connect to ground.
If R1 is a short circuit and R2 an open circuit the whole change in potentiometer voltage is applied to the end of Rf. This causes two problems.
The equivalent resistance of Rf (equal to Rf/4) will add to Rf and cause gain errors. For a small error the potentiometer value would need to be small or Rf would need to be reduced by an equal amount.
For small offset voltage adjustments the adjustment of the potentiometer becomes difficult and most of the potentiometer range is not used...
Adding R1 and R2 overcomes both these problems.
R1 and R2 divide down changes in potentiometer voltage by the ratio R2/(R1+R2). If, for example, a +/- 15 mV change is required then the ratio of R1:R2 can be about 15 V:15 mV = 1000:1.
The effective resistance of the R1, R2 divider is R1 and R2 in parallel or about = R2 for large division ratios.
If the resistance of R2 is small relative to Rf then minimal errors are caused.
If Rf is, say, 10 kohm then a value of R2 = 10 ohm causes an error of 10/10,000 = 0.1%.
Maxim manage to say this in fewer words in the diagram below.
If R1 and R2 form a ~~ 1000:1 divider then R1 will be about 10 ohm x 1000 = 10 kohm.
Use of a, say, 50 kohm potentiometer will result in an equivalent resistance of about 12.5 kohm at the mid point and this can be used in place of R1.
The circuit becomes: R2 = 10 ohm, R1 = short circuit, potentiometer = 10 kohm linear.
The above circuit is taken from the useful Maxim Application note 803 - EPOT Applications: Offset Adjustment in Op-Amp Circuits which contains much other applicable information.
In his answer miceuz referred to NatSemi's AN-31 pages 6 & 7.
Not surprisngly, the circuits there apply the identical methods to what I describe above and to those in the Maxim app note, but the diagrams are more explanatory, so I've copied them here.