TL081 is not a particularly good choice for low DC voltages (it's better for AC coupled designs). Precision op-amps have offset voltages in the 10's of microvolts or lower, and there are even auto-zero op-amps that have negligible offset voltage, an drifts in the tens of nV/°C (at some significant cost in other characteristics). It's also quite noisy (25nV/\$ \sqrt{Hz}\$), but at least has a typical flicker noise corner frequency in the 1Hz range. Aside from high input Z, it has basically one really nice advantage, it's really cheap, and widely available, which is why I've actually designed such a 37-year-old op-amp into a new product recently.
A good general purpose (low noise, capable of handling +/-15V supplies, low distorion for AC signals) precision op-amp suitable for mV levels might be the OPA209A.
You can certainly null the offset voltage of your TL081 out using a trimpot as shown on the datasheet, but it won't stay that well nulled for long. A 10°C change will typically change the offset voltage by 100uV, and about one out of every two will be worse (there's no guarantee how much worse, but a guess would be most are better than +/-30uV/°C). An OPA209 is going to be roughly an order of magnitude better.
There are probably going to be better (and many worse) choices for any given application, all things considered. It's amazing what performance you can get for a a few dollars, so it's worth looking around rather than trying to make a silk purse of a sow's ear.
Just to give you an idea of the kind of (in) accuracy you could get, consider that the gain of the TL081 is only guaranteed to be >15,000, so a gain of 1000 amplifier could have a gain error in the 6% range even without the input offset error (which would be very temperature dependent, and has a -3dB corner of something like 20Hz. Cascading two \$\sqrt {1000}\$ gain amplifiers would help with that (null only the first one).
If the range is, say +/-5mV input, frequency is 0.001 to 1Hz and required accuracy 1% of FS + 50uV**, it might be typically *** okay in a lab environment with a light output load, if you null it after warm-up.
** Instrumentation type specification- it means the output could be as much as +/- 100mV from the ideal value with any input, so a 1mV input could give you 900mV or 1100mV.
*** "Typically" means that one chip might be okay, and the next might not meet the requirements. Guaranteed value is probably 10 times worse.
The LM324 has a maximum offset voltage of 9mV (worst case, over temperature), according to the datasheet.

With your circuit, with 0V in, you could have a current of 9mV/500m\$\Omega\$/9mV = 18mA below which your pot would not be able to set the current. So it's not a very good design if you need to set it to less than 18mA. It's luck of the draw- the next op-amp (even in the same package) could be 9mV in the opposite polarity, so you'd just move the pot.
Maximum temperature drift of the LM324 is not specified (it's not intended for precision applications, after all), but it might easily be +/-10uV/°C, so if the board changes by (say) 70°C as the MOSFET gets hot, the current will change by 0.7mV or 1.4mA, so you'd have to readjust the pot. Of course the highest power dissipation occurs at high output currents, so the change is relatively small (1.4mA out of 2A is < 0.1%). A 20°C change in ambient temperature means a change of perhaps (no guarantees) of 0.4mA, which is several percent of a 15mA current. If you only care about 5%, and currents above 20mA, probably just okay.
Another difference between a cheap amplifier and a good one is the gain. The LM324 can be as bad as 25,000 gain (and it changes with temperature). A precision op-amp will have a gain in the millions. The difference will show up in how well it compensates for load or line changes (not a big deal in this case).
The bias current of the LM324 can be as bad as 0.5uA (typical 20nA) and it changes with temperature so if you had a high resistance pot, you could see it change with temperature.
The noise of the LM324 is a fairly miserable 35nV/sqrt(Hz), and it has nasty crossover distortion, neither of which affects you much in this case.
A couple of things (other than being extremely cheap) that the LM324 has that a typical precision op-amp may not have- wide supply range (especially on the high end), though it may not do so well at very low supply voltages, and it's single supply (input common mode range includes the minus supply) which you absolutely require for your circuit.
So there are plenty of reasons to use a decent op-amp if it's required by the specifications. Or you can get clever with the circuit- increase the sense resistor to get good accuracy for low currents, but to get wide dynamic range, a good amplifier (and other techniques such as good resistors and good layout) may be worth it. For just hacking around and if your current range not huge (minimum to maximum), an LM324 is certainly acceptable. There's no point in using a $5 op-amp if a 1-cent one will do. On the other hand, there are some requirement for which the best ones are not good enough and one has to resort to discretes and other techniques.
By the way, your circuit may not be stable against oscillation. It can be fixed with some passive components, but loading op-amps with the equivalent of a large capacitance in series with a small resistance is inviting trouble.
Best Answer
Im curious why you are using aplification. In my opinion simple cell voltage measuerement could look just like this (i drawed it only for 3 cells):
This should give you acurracy probably below 0.5% + resistor tolerance, but you can compensate resistor tolerance in software.
This is how very acurrate cell measurement can be done
This is part of cell voltage measurement circuit from well designed 6 cell battery charger:
You can find full version of this schematic here
LM324 amplifiers were used in very good Turnigy Accucell 6 and some other RC battery chargers in cell balancer circuits.