Floating Point Comparison – Choosing an Epsilon When Comparing Floating Point Numbers Across Different Systems

floating pointunit testing

I am transcribing thousands of lines of computational code from MATLAB to C++. I don't fully understand the math myself, but I can run it with MATLAB, assume it's correct, and compare the results to my C++ code.

The issue is that the precision I get is very inconsistent. Sometimes MATLAB gives me four significant digits for a number on the order of 1e6, so I have to use an epsilon of 100. Other times, I will be using an epsilon of 1e-3, but then suddenly there will be one value that just different enough to require me to bump it up to 1e-2.

Are there any potential problems with adjusting the epsilon of my tests to a passing value? Is there a more reliable approach?

Best Answer

From the comments we exchanged, it seems you are not just unfamiliar with the maths, but also with basic numeric computing discipline.

First, for god's sake, don't automatically pick an epsilon to makes your tests "pass". If you fudge the epsilons until the error is below epsilon, your tests don't test anything at all and you may ignore really bad precision problems (perhaps even a wrong algorithm if it happens to produce similar results in your test cases).

Instead, pick a-priori reasonable epsilons and stick to them. If your get greater errors than you need or expect, that means you need to fix your code, not lower your expectations. Sadly, what expectations are "reasonable" depends on the application and the algorithm, but by default I'd expect, say, a relative error of less than 1e-12 if the floats are 64 bits.

Second, the keyword is relative error. Absolute error is, as you've seen, very sensitive to the magnitude of the values, and thus often useless. Moreover, due to how floating point works, there is a fixed relative error that you can't beat unless you happen to produce the exact same bit pattern (for 64 bit floats, this is about 1e-17). Thus, if numbers are large enough, you will find that your absolute error is either zero (unlikely) or quite large, even though the calculation may be only be off by one unit in the last place (ULP).

For this, you should also force MATLAB to output more digits (17 is the maximum number that makes sense for 64 bit floats). Oh, and relative error calculations can be quite tricky, as @CodeInChaos also points out, so you may want to rely on existing algorithms that handle edge cases better than the naive approaches, such as http://floating-point-gui.de/errors/comparison/

Related Topic