How to make the JUnit tests run in random order

junit

I have the classical structure for tests, I have a test suite of different suites like DatabaseTests, UnitTests etc. Sometimes those suites contains other suites like SlowDatabaseTests, FastDatabaseTests etc.

What I want is to randomize the running order of tests so I will make sure they are not dependent to each other. Randomization should be at every level, like suite should shuffle test class order, and test class should shuffle test method order.

If it is possible to do this in Eclipse that will be the best.

Best Answer

You do have a Sortable but I can't see how you would use it.

You could extend BlockJUnit4ClassRunner and have computeTestMethods() return a randomized copy of super.computeTestMethods(). Then use the @RunWith to set that as the runner to use.

e.g.

package com.stackoverflow.mlk;

import java.util.Collections;

import org.junit.runners.BlockJUnit4ClassRunner;
import org.junit.runners.model.InitializationError;

public class RandomBlockJUnit4ClassRunner extends BlockJUnit4ClassRunner {

    public RandomBlockJUnit4ClassRunner(Class<?> klass)
            throws InitializationError {
        super(klass);
    }

    protected java.util.List<org.junit.runners.model.FrameworkMethod> computeTestMethods() {
        java.util.List<org.junit.runners.model.FrameworkMethod> methods = super.computeTestMethods();
        Collections.shuffle(methods);
        return methods;
    }

}

Then

@RunWith(com.stackoverflow.mlk.RandomBlockJUnit4ClassRunner.class)
public class RandomOrder {
    @Test
    public void one() {
    }

    @Test
    public void two() {
    }

    @Test
    public void three() {
    }
}

Related Solutions

Java – How to assert that a certain exception is thrown in JUnit 4 tests

It depends on the JUnit version and what assert libraries you use.

For JUnit5 and 4.13 see answer https://stackoverflow.com/a/2935935/2986984
If you use assertJ or google-truth, see answer https://stackoverflow.com/a/41019785/2986984

The original answer for JUnit <= 4.12 was:

@Test(expected = IndexOutOfBoundsException.class)
public void testIndexOutOfBoundsException() {

    ArrayList emptyList = new ArrayList();
    Object o = emptyList.get(0);

}

Though answer https://stackoverflow.com/a/31826781/2986984 has more options for JUnit <= 4.12.

Reference :

JUnit Test-FAQ

Java – Conditionally ignoring tests in JUnit 4

The JUnit way is to do this at run-time is org.junit.Assume.

 @Before
 public void beforeMethod() {
     org.junit.Assume.assumeTrue(someCondition());
     // rest of setup.
 }

You can do it in a @Before method or in the test itself, but not in an @After method. If you do it in the test itself, your @Before method will get run. You can also do it within @BeforeClass to prevent class initialization.

An assumption failure causes the test to be ignored.

Edit: To compare with the @RunIf annotation from junit-ext, their sample code would look like this:

@Test
public void calculateTotalSalary() {
    assumeThat(Database.connect(), is(notNull()));
    //test code below.
}

Not to mention that it is much easier to capture and use the connection from the Database.connect() method this way.

Best Answer

Related Solutions

Java – How to assert that a certain exception is thrown in JUnit 4 tests

Java – Conditionally ignoring tests in JUnit 4

Related Topic