Unit Testing – Why Avoid Automatic Unit Tests?

communicationintegration-testsmanagementunit testing

I'm managing a team in .net.
They are writing unit tests, use them locally on a regular basis, and love it. However, they keep pushing to have the unit tests kept as a separate project and only want to run them on a communal build server once a week or so.

In contrast, our JS teams are running tests on the build server on every pull request.

I keep trying to explain to them the benefits of having the tests near the code, and running the tests often, but they keep bringing up fears of the builds being too slow, or what happens when the tests prevent an urgent build.

It's boggling my mind and I'm wondering if there is something I'm missing.

What are likely reasons for this kind of pushback? They are happy to write the tests and they tell me they have close to 100 tests already written on branches they refuse to merge into their main branches until "we come to a decision." ( I.e. I tell them it's ok to keep tests in an isolated project far away from the main code.)

The team was quick to adopt writing unit tests and they find them useful and continue to write more. I am now trying to get them to run the tests in the CI and that is where I'm hitting resistance.

Best Answer

Firstly, thank you to @Gnat for finding insight into the real issue here and giving me what I needed to know to get to the real issue.

We have agreed to isolate the unit tests and integration tests into their own projects to ensure that the added test code doesn't get compiled into the build.

We are also going to be running the tests in a non-blocking workflow until we are able to separate out the code into smaller repositories so that each build doesn't take 15-30 min.

And for now, we have agreed to not build the project on every Pull Request(PR), but instead to run the tests on a nightly build. This is what they really meant by the tests slowing things down... They actually meant doing a build on every PR. They do PRs dozens of times a day. Their main concern about tests blocking an urgent build, was really their concern that urgent builds often end up with dozens of PRs, when everyone is trying to merge their code at once, and they wanted to be able to just run the tests at the end of that process, and not require a separate build at every point along the way.

There wasn't one single answer which gave me all the information I needed, and most of the best answers came in comments, so I've created this answer to summarize up everything that made the most sense.

The important pieces of information I was missing were:

If the tests are too close to the code under tests, then its very time consuming to tell .net not to build the test files into the .dlls, and it's much easier to just keep them as separate projects.
The real issue was the build required to make the tests run. It wasn't the tests they didn't want to run, it was the build. They didn't want the build to prevent them from being able to merge code as frequently as they do. Locally, on their own computers, they are able to do a debug build and run the tests in under a minute, but on the CI servers, it takes much longer.

Related Solutions

Unit Testing – Ensuring Quality of Code

You should definitely take the same if not better care of your unit tests than your production code in terms of quality and readability. The unit tests are often the first thing you look at when trying to grasp what some piece of code does, and the reader should quickly understand what's at stake when looking at the test. Unit tests also tend to change a lot and will break a lot if their design is not robust, which kind of nullifies the benefits of having tests.

Violation of the Law of Demeter is definitely a problem in your unit tests for that reason as well as 2 others that come off my mind :

If your tests break the Law of Demeter in their Arrange or Act sections, it's probably a sign that your production code also does, since your unit tests are just another consumer of your code and will probably call and operate the class under test in the same way that any other production code would do.
If your tests break the Law of Demeter in their Assert sections (ie you verify the value of something that is deeply nested in the dependencies graph of the object under test), it might be that those are really integration tests rather than unit tests. In other words, if in TestA you assert that A.B.C.D equals something, it might be that you're actually trying to test D and A rather than just A.

By the way, when you say

I break very often the Law of Demeter, for faster writing and not using so many variables.

you should be aware that writing

var grab = myDependency.Grab;
var something = grab.Something;
var very = something.Very;

very.Deep();

is actually no better Demeter wise than

myDependency.Grab.Something.Very.Deep();

if that's what you meant.

C# – How do we make unit tests run fast

A possible solution would be to move the testing portion from the development machines to a continuous integration setup (Jenkins for example) using version control software of some flavor (git, svn, etc...).

When new code has to be written the given developer will create a branch for whatever they are doing in the repository. All work will be done in this branch and they can commit their changes to the branch at any time without messing up the main line of code.

When the given feature, bug fix, or whatever else they are working on has been completed that branch can be merged back into the trunk (or however you prefer to do it) where all unit tests are run. If a test fails the merge is rejected and the developer is notified so they can fix the errors.

You can also have your CI server run the unit tests on each feature branch as commits are made. This way the developer can make some changes, commit the code, and let the server run the tests in the background while they continue to work on additional changes or other projects.

A great guide to one way of doing such a setup can be found here (git specific but should work for other version control systems): http://nvie.com/posts/a-successful-git-branching-model/

Best Answer

Related Solutions

Unit Testing – Ensuring Quality of Code

C# – How do we make unit tests run fast

Related Topic