Unit Testing – Purpose of xfail Tests

testingunit testing

Some test frameworks, notably DejaGnu and py.test, have an "XFAIL" flag/status for a test which says that this test is expected to fail. If the test succeeds, it's marked as "XPASS" (unexpected pass).

Isn't this just a redundant overcomplication? It seems much easier and comprehensible to just use an assert that would expect an error — if the error happens, the test passes, if it doesn't, it fails.

gcc in particular is having so much trouble with this status it makes test output all but useless. I was ready to write this feature off as yet another misguided quirk of history — until I saw it in py.test.

What's the purpose of this feature? What can it do that SKIP and asserts cannot?

Best Answer

This feature allows you to write and commit a test separately from the code that will make it pass.
_{(It can be compared to double-entry bookkeeping in accounting that allows one to exchange goods and payments at separate moments while keeping a record of the outstanding debt in the meantime.)}

A test suite should always succeed. But what if we have tests for features that are not yet implemented, or tests for known bugs that we can't fix right now?

We write an xfail test.

The xfail tests encodes our assumptions. It is subject to source control, it must still compile, and we can execute it. This prevents our assumptions from falling out of date with the current state of the software. But during normal work with the test suite, these xfail tests do not cause the test suite to fail. This is similar to a skipped test, except that when an usually-skipped test is executed and fails, that would fail the complete test suite. Failure of an xfail test does not fail the test suite.

In test runners without xfail tests, we instead have to rely on the number of passing tests, or the percentage of passing tests. But this is extremely fragile. First, these counts are wrong if new tests are added – which should happen all the time. Second, these accumulated metrics fail to account for the case where one tests starts passing but another test starts failing.

In contrast, xfail tests ensure that passing tests keep passing, and will not hide accidental regressions. This works like a ratchet: our test suite can't slip back and introduce regressions, but can proceed forward by removing the xfail annotation from a test as soon as it passes.

Xfail tests as implemented in pytest are distinct from tests that are expected to produce an error. The xfail test should pass and is expected to pass in the future, but is expected to fail given the current state of the software. In contrast, tests that are expected to produce an error fail if there is no such error. Such error tests are e.g. useful to assert that input validation succeeds. Most test frameworks have special annotations to catch an expected exception.

Related Solutions

Unit-testing – What’s the best way to organize our unit tests

Generally, I tend to mirror the source tree for my unit tests. So, if I had src/lib/fubar, I would have a test/lib/fubar which would contains the unit tests for fubar.

However, what you seem to be describing are more functional tests. In that case, I would have a multi-dimensional table that enumerated all your possible conditions. Then, the ones that have no tests are either nonsensical or need a new test. You can, of course, then put them in sets of test suites.

Unit-testing – Unit Tests code duplication

Declare class TestArith as abstract. Then create two classes, TestIntegerArith and TestDoubleArith extending TestArith. In the TestArith create an abstract method getArithmetic, which returns an instance of Arithmetic. Then change the tests to get Arithmetic instance by calling getArithmetic.

So it would look like this:

public abstract class TestArith<T> {
  protected abstract Arithmetic<T> getArithmetic();

  @Test
  public void testAdd() {
      assertEquals(7, getArithmetic().add(term1, term2));
  }
}

Best Answer

Related Solutions

Unit-testing – What’s the best way to organize our unit tests

Unit-testing – Unit Tests code duplication

Related Topic