Example:
A certain test is run five times on the current build, to compare it to release 1.0, where the same test was also run five times.
Build 1.4
22 ms, 26 ms, 23 ms, 25 ms, 20 ms
Release 1.0
15 ms, 18 ms, 16 ms, 20 ms, 17 ms
Question
The requirement is that Build 1.4 is no worse than Release 1.0.
How would I test that? I've seen different methods used, including simple average comparison and statistical T-Tests (assuming normal distributions).
None of the tools, i.e. testing frameworks and CI systems, I've come across provide anything to help with calculating and passing/failing such tests. Why? Performance testing seems to be popular, so how are people approving or rejecting them?
Often it's way too impractical or even impossible to go in and manually determine acceptable ranges or distributions for every test.
Best Answer
It's popular in theory. In practice, I've seen only a handful of automated performance tests done and they were done haphazardly.
You define a pass/fail criteria. If you need the tests to be consistent, define some failure criteria and measure. If you need the tests to be at least X fast (where X is ideally some metric gathered by usability testing) then do that. If not X worse than last build (not recommended in automated tests, since it requires memory of past builds) then do that.
It is trivial to write up helper code to deal with measurements and/or statistics. But you still need to define the criteria, just like any other test. It's like asking to automatically define pass/fail criteria for business rules - computers do a poor job at that since all they know about business rules is what you tell them.