Junit: splitting integration test and Unit tests

integration-testingjunitjunit4testingunit testing

I've inherited a load of Junit test, but these tests (apart from most not working) are a mixture of actual unit test and integration tests (requiring external systems, db etc).

So I'm trying to think of a way to actually separate them out, so that I can run the unit test nice and quickly and the integration tests after that.

The options are..

Split them into separate directories.
Move to Junit4 (from v3) and annotate the classes to separate them.
Use a file naming convention to tell what a class is , i.e. AdapterATest and
AdapterAIntergrationTest.

3 has the issue that Eclipse has the option to "Run all tests in the selected project/package or folder". So it would make it very hard to just run the integration tests.

2: runs the risk that developers might start writing integration tests in unit test classes and it just gets messy.

1: Seems like the neatest solution, but my gut says there must be a better solution out there.

So that is my question, how do you lot break apart integration tests and proper unit tests?

Best Answer

You can split them very easily using JUnit categories and Maven.

This is shown very, very briefly below by splitting unit and integration tests.

Define A Marker Interface

The first step in grouping a test using categories is to create a marker interface.

This interface will be used to mark all of the tests that you want to be run as integration tests.

public interface IntegrationTest {}

Mark your test classes

Add the category annotation to the top of your test class. It takes the name of your new interface.

import org.junit.experimental.categories.Category;
@Category(IntegrationTest.class)
public class ExampleIntegrationTest{
  @Test
  public void longRunningServiceTest() throws Exception {
  }
}

Configure Maven Unit Tests

The beauty of this solution is that nothing really changes for the unit test side of things.

We simply add some configuration to the maven surefire plugin to make it to ignore any integration tests.

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-surefire-plugin</artifactId>
  <version>2.11</version>
  <dependencies>
   <dependency>
     <groupId>org.apache.maven.surefire</groupId>
     <artifactId>surefire-junit47</artifactId>
     <version>2.12</version>
   </dependency>
  </dependencies>
  <configuration>
    <includes>
      <include>**/*.class</include>
    </includes>
    <excludedGroups>com.test.annotation.type.IntegrationTest</excludedGroups>
  </configuration>
</plugin>

When you do a mvn clean test only your unmarked unit tests will run.

Configure Maven Integration Tests

Again the configuration for this is very simple.

To run only the integration tests, use this:

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-surefire-plugin</artifactId>
  <version>2.11</version>
  <dependencies>
   <dependency>
     <groupId>org.apache.maven.surefire</groupId>
     <artifactId>surefire-junit47</artifactId>
     <version>2.12</version>
   </dependency>
  </dependencies>
  <configuration>
    <groups>com.test.annotation.type.IntegrationTest</groups>
  </configuration>
</plugin>

If you wrap this in a profile with id IT, you can run only the fast tests using mvn clean install. To run just the integration/slow tests, use mvn clean install -P IT.

But most often, you will want to run the fast tests by default and all tests with -P IT. If that's the case, then you have to use a trick:

<profiles>
    <profile>
        <id>IT</id>
        <build>
            <plugins>
                <plugin>
                    <groupId>org.apache.maven.plugins</groupId>
                    <artifactId>maven-surefire-plugin</artifactId>
                    <configuration>
                        <excludedGroups>java.io.Serializable</excludedGroups> <!-- An empty element doesn't overwrite, so I'm using an interface here which no one will ever use -->
                    </configuration>
                </plugin>
            </plugins>
        </build>
    </profile>
</profiles>

As you can see, I'm excluding tests that are annotated with java.io.Serializable. This is necessary because the profile will inherit the default config of the Surefire plugin, so even if you say <excludedGroups/> or <excludedGroups></excludedGroups>, the value com.test.annotation.type.IntegrationTest will be used.

You also can't use none since it has to be an interface on the classpath (Maven will check this).

Notes:

The dependency to surefire-junit47 is only necessary when Maven doesn't switch to the JUnit 4 runner automatically. Using the groups or excludedGroups element should trigger the switch. See here.
Most of the code above was taken from the documentation for the Maven Failsafe plugin. See the section "Using JUnit Categories" on this page.
During my tests, I found that this even works when you use @RunWith() annotations to run suites or Spring-based tests.

Related Solutions

The difference between integration and unit tests

The key difference, to me, is that integration tests reveal if a feature is working or is broken, since they stress the code in a scenario close to reality. They invoke one or more software methods or features and test if they act as expected.

On the opposite, a Unit test testing a single method relies on the (often wrong) assumption that the rest of the software is correctly working, because it explicitly mocks every dependency.

Hence, when a unit test for a method implementing some feature is green, it does not mean the feature is working.

Say you have a method somewhere like this:

public SomeResults DoSomething(someInput) {
  var someResult = [Do your job with someInput];
  Log.TrackTheFactYouDidYourJob();
  return someResults;
}

DoSomething is very important to your customer: it's a feature, the only thing that matters. That's why you usually write a Cucumber specification asserting it: you wish to verify and communicate the feature is working or not.

Feature: To be able to do something
  In order to do something
  As someone
  I want the system to do this thing

Scenario: A sample one
  Given this situation
  When I do something
  Then what I get is what I was expecting for

No doubt: if the test passes, you can assert you are delivering a working feature. This is what you can call Business Value.

If you want to write a unit test for DoSomething you should pretend (using some mocks) that the rest of the classes and methods are working (that is: that, all dependencies the method is using are correctly working) and assert your method is working.

In practice, you do something like:

public SomeResults DoSomething(someInput) {
  var someResult = [Do your job with someInput];
  FakeAlwaysWorkingLog.TrackTheFactYouDidYourJob(); // Using a mock Log
  return someResults;
}

You can do this with Dependency Injection, or some Factory Method or any Mock Framework or just extending the class under test.

Suppose there's a bug in Log.DoSomething(). Fortunately, the Gherkin spec will find it and your end-to-end tests will fail.

The feature won't work, because Log is broken, not because [Do your job with someInput] is not doing its job. And, by the way, [Do your job with someInput] is the sole responsibility for that method.

Also, suppose Log is used in 100 other features, in 100 other methods of 100 other classes.

Yep, 100 features will fail. But, fortunately, 100 end-to-end tests are failing as well and revealing the problem. And, yes: they are telling the truth.

It's very useful information: I know I have a broken product. It's also very confusing information: it tells me nothing about where the problem is. It communicates me the symptom, not the root cause.

Yet, DoSomething's unit test is green, because it's using a fake Log, built to never break. And, yes: it's clearly lying. It's communicating a broken feature is working. How can it be useful?

(If DoSomething()'s unit test fails, be sure: [Do your job with someInput] has some bugs.)

Suppose this is a system with a broken class:

A single bug will break several features, and several integration tests will fail.

A single bug will break several features, and several integration tests will fail

On the other hand, the same bug will break just one unit test.

The same bug will break just one unit test

Now, compare the two scenarios.

The same bug will break just one unit test.

All your features using the broken Log are red
All your unit tests are green, only the unit test for Log is red

Actually, unit tests for all modules using a broken feature are green because, by using mocks, they removed dependencies. In other words, they run in an ideal, completely fictional world. And this is the only way to isolate bugs and seek them. Unit testing means mocking. If you aren't mocking, you aren't unit testing.

The difference

Integration tests tell what's not working. But they are of no use in guessing where the problem could be.

Unit tests are the sole tests that tell you where exactly the bug is. To draw this information, they must run the method in a mocked environment, where all other dependencies are supposed to correctly work.

That's why I think that your sentence "Or is it just a unit test that spans 2 classes" is somehow displaced. A unit test should never span 2 classes.

This reply is basically a summary of what I wrote here: Unit tests lie, that's why I love them.

Python – Where do the Python unit tests go

For a file module.py, the unit test should normally be called test_module.py, following Pythonic naming conventions.

There are several commonly accepted places to put test_module.py:

In the same directory as module.py.
In ../tests/test_module.py (at the same level as the code directory).
In tests/test_module.py (one level under the code directory).

I prefer #1 for its simplicity of finding the tests and importing them. Whatever build system you're using can easily be configured to run files starting with test_. Actually, the default unittest pattern used for test discovery is test*.py.