Continuous Integration – Should Tests Be Run Against src or dist?

Unit tests and integration tests have different purposes.

Unit tests verify the functionality of your code... that you get what you expect back from the method when you call it. Integration tests test how the code behaves when combined together as a system. You wouldn't expect unit tests to evaluate system behavior, nor would you expect integration tests to verify specific outputs of a particular method.

Unit tests, when done correctly, are easier to set up than integration tests. If you rely solely on integration tests, your testing is going to:

1. Be more difficult to write, overall,
2. Be more brittle, due to all of the required dependencies, and
3. Offer less code coverage.

Bottom line: Use integration testing to verify that the connections between objects are working properly, but lean on unit testing first to verify functional requirements.

All that said, you may not need unit testing for certain repository methods. Unit testing should not be done on trivial methods; if all you're doing is passing through a request for an object to ORM generated code and returning a result, you don't need to unit test that, in most cases; an integration test is adequate.

Unit tests are just one level of the hierarchy of automated tests. Unit tests exist to verify that the code you (the developer) have actually written behaves the way you thought it should when you wrote it.

There are two caveats inherent in unit testing. First, coverage is not exercise. You may execute every line of code in your codebase via one or more unit tests, but if you do not assert, somewhere, that code which needed to do something actually did it, then as long as you don't get an exception, the test passes with or without the code doing this key thing.

Second, unit tests by definition exercise small, isolated pieces of your code (units), making sure each piece behaves the way the developer thinks it should. They don't test that these units play nicely with each other (that's an "integration" test), nor do they assert that the code, at any level, behaves the way the client thinks it should (that's an "acceptance" and/or an "end-to-end" test).

This second problem is your main issue in the case in point. You have 100% unit test coverage, but no integration testing, which would prove that the little pieces are put together the right way to do the larger job you expect. You also seem to have no automated acceptance testing, which approaches the entire program from the top down from the perspective of an end user. These are the levels of testing, which can be automated, that will identify your "missing" code based on failure to satisfy acceptance criteria.