Continuous Integration – Importance of Running Unit Tests on a CI Server

You're not doing unit tests, but integration tests.

• A unit test is ensuring that a single component of a system works as expected.

• An integration test combines several components and verifies that, together, they still work normally.

True, you cannot normalize the data before you label it. But when you test data normalization, you should test it one a labeled data. Or data which was specifically not labeled, on purpose, to ensure that normalization successfully fails when the data is not labeled (or gracefully handles this situation).

In order to do that, you may use mocking. Mocking is the technique where you replace one part of a system by something which only provides fictive results. In your normalization tests, you may have a mock which provides labeled data, another mock which provides data which is labeled, but incorrectly, and another mock which doesn't label anything.

Those three mocks will serve you to unit test your normalization independently of the actual labeling process, without even carrying if it is already implemented or not, nor if it is working as expected.

This way, you can unit test the part which saves data and then develop it, then start to unit test labeling and implement it, and finally unit test normalization and write the corresponding code.

Once done, you will then use your actual integration tests to see if the whole system is working.

Example

I don't really understand the process from your question (is it domain-specific?), so I invent some easy to understand illustration. Let's imagine an app which loads weather data (temperature, precipitation risk, etc.), from several web services, compares it using specific rules, makes predicates from statistical data and stores the result in a database.

I can't compare data without loading it, and I can't make predicates from statistical data with no data whatsoever. But I want to implement statistical analysis first, because I'm unsure how to do it nor what my customer expects as a result, so I do this part first to increase the chance to finish the project on time.

For this, I can create a bunch of mocks based on the specification. Those mocks will provide me with the data I specify in the unit tests themselves. A real-world, untested app would look like this:

data = [];
foreach (provider in weatherProviders)
{
    data.push(provider.getTransformedData());
}

todayOverall = compareWeatherData(data); // We want to implement this first.
pastDaysOverall = loadHistoricalData();
weather = predictWeather(todayOverall, pastDaysOverall);

An app which uses mocks will look like that:

data = [];
foreach (provider in weatherProviders)
{
    data.push(provider.getTransformedData());
}

var dataAccess = new SQLiteDatabaseAccess();
todayOverall = compareWeatherData(new BasicCompare(), data);
pastDaysOverall = loadHistoricalData(new HistoricalLoader(dataAccess));
weather = predictWeather(new WeatherOracle(dataAccess), todayOverall, pastDaysOverall);

As you see, compareWeatherData takes an additional argument, which does the actual work, loadHistoricalData needs an object which actually loads the data from a data source, and which, itself, requires a data provider to be specified, etc. This have several benefits compared to the previous piece of code:

• If tomorrow, the customer tells me that my app has to support both SQLite and MySQL, I don't care: I'll do two providers instead of one, and specify the first or the second one depending on the application configuration. In the same way, if the weather oracle of the first version is really bad, I can simply work on a better oracle, then easily switch them.

• When unit testing the different parts, I can replace any of them by a mock. For example, I can feed WeatherOracle with FictionalDataMock: DataAccess instead of SQLiteDatabaseAccess: DataAccess, and in FictionalDataMock, specify the values manually.

When it finally comes to unit testing the todayOverall = compareWeatherData(new BasicCompare(), data); line, I have enough power to do it without even thinking about any other parts of the system:

/**
 * Ensures that the comparison handles correctly the case where one of the data providers
 * was on crack and supplied us with a temperature below Absolute Zero (-459.67°F).
 */
public test minimumTemperatureFailure()
{
    dataSet1 = new TransformedDataMock(sky.snow, fromFahrenheit(23));
    dataSet2 = new TransformedDataMock(sky.heavySnow, fromFahrenheit(-480)); // Too cold!

    data = [ dataSet1, dataSet2 ];

    actual = new BasicCompare().compare(data);

    // The temperature below Absolute Zero should be ignored.
    assert.areEqual(dataSet1.temperature, actual.averageTemperature);

    // Since the second data provider was doing drugs, other data from it shouldn't be
    // trusted neither.
    assert.areEqual(dataSet1.sky, actual.mostRealisticSky);
}

The answer is yes, you should write them and you should run them.

Your testing framework needs a category of "known failing tests" and you should mark these tests as falling into that category. How you do that depends on the framework.

Curiously, a failing test that suddenly passes can be just as interesting as a passing test that unexpectedly fails.