The key difference, to me, is that integration tests reveal if a feature is working or is broken, since they stress the code in a scenario close to reality. They invoke one or more software methods or features and test if they act as expected.
On the opposite, a Unit test testing a single method relies on the (often wrong) assumption that the rest of the software is correctly working, because it explicitly mocks every dependency.
Hence, when a unit test for a method implementing some feature is green, it does not mean the feature is working.
Say you have a method somewhere like this:
public SomeResults DoSomething(someInput) {
var someResult = [Do your job with someInput];
Log.TrackTheFactYouDidYourJob();
return someResults;
}
DoSomething
is very important to your customer: it's a feature, the only thing that matters. That's why you usually write a Cucumber specification asserting it: you wish to verify and communicate the feature is working or not.
Feature: To be able to do something
In order to do something
As someone
I want the system to do this thing
Scenario: A sample one
Given this situation
When I do something
Then what I get is what I was expecting for
No doubt: if the test passes, you can assert you are delivering a working feature. This is what you can call Business Value.
If you want to write a unit test for DoSomething
you should pretend (using some mocks) that the rest of the classes and methods are working (that is: that, all dependencies the method is using are correctly working) and assert your method is working.
In practice, you do something like:
public SomeResults DoSomething(someInput) {
var someResult = [Do your job with someInput];
FakeAlwaysWorkingLog.TrackTheFactYouDidYourJob(); // Using a mock Log
return someResults;
}
You can do this with Dependency Injection, or some Factory Method or any Mock Framework or just extending the class under test.
Suppose there's a bug in Log.DoSomething()
.
Fortunately, the Gherkin spec will find it and your end-to-end tests will fail.
The feature won't work, because Log
is broken, not because [Do your job with someInput]
is not doing its job. And, by the way, [Do your job with someInput]
is the sole responsibility for that method.
Also, suppose Log
is used in 100 other features, in 100 other methods of 100 other classes.
Yep, 100 features will fail. But, fortunately, 100 end-to-end tests are failing as well and revealing the problem. And, yes: they are telling the truth.
It's very useful information: I know I have a broken product. It's also very confusing information: it tells me nothing about where the problem is. It communicates me the symptom, not the root cause.
Yet, DoSomething
's unit test is green, because it's using a fake Log
, built to never break. And, yes: it's clearly lying. It's communicating a broken feature is working. How can it be useful?
(If DoSomething()
's unit test fails, be sure: [Do your job with someInput]
has some bugs.)
Suppose this is a system with a broken class:
![A system with a broken class](https://i.stack.imgur.com/611ZY.jpg)
A single bug will break several features, and several integration tests will fail.
![A single bug will break several features, and several integration tests will fail](https://i.stack.imgur.com/rKhSr.jpg)
On the other hand, the same bug will break just one unit test.
![The same bug will break just one unit test](https://i.stack.imgur.com/M64Vb.jpg)
Now, compare the two scenarios.
The same bug will break just one unit test.
- All your features using the broken
Log
are red
- All your unit tests are green, only the unit test for
Log
is red
Actually, unit tests for all modules using a broken feature are green because, by using mocks, they removed dependencies. In other words, they run in an ideal, completely fictional world. And this is the only way to isolate bugs and seek them. Unit testing means mocking. If you aren't mocking, you aren't unit testing.
The difference
Integration tests tell what's not working. But they are of no use in guessing where the problem could be.
Unit tests are the sole tests that tell you where exactly the bug is. To draw this information, they must run the method in a mocked environment, where all other dependencies are supposed to correctly work.
That's why I think that your sentence "Or is it just a unit test that spans 2 classes" is somehow displaced. A unit test should never span 2 classes.
This reply is basically a summary of what I wrote here: Unit tests lie, that's why I love them.
Best Answer
Specs and ScalaTest are both good tools with happy users, but they differ in several ways. You will probably want to pick one as your main testing tool in Scala, but need not give up the other because you can use pieces of both. If you like ScalaTest's
FeatureSpec
syntax and specs' Mockito syntax, for example, you can put both jar files in your classpath and use both at the same time. Here I'll try and capture the main design philosophy differences I've noticed between specs and ScalaTest.Probably the main philosophical difference between the tools is that specs is designed for Behavior-Driven Development (BDD), whereas ScalaTest is more general. ScalaTest provides traits that you can mix together to get the behavior you prefer in your test classes, including BDD, and you can also easily define your own behavior if you want something different.
ScalaTest supports BDD through its
Spec
,FeatureSpec
,WordSpec
,FlatSpec
, andGivenWhenThen
traits, and also has traits that you can mix in to get a nice matcher syntax. If you like "should", you mix in ShouldMatchers. If you like "must", you mix inMustMatchers
. But if you like BDD but don't like matcher syntax, you can just use one of ScalaTest's Spec traits without mixing in a matchers trait. Specs has a Specification class that you extend, and you must use the word "must" in your matcher expressions. A big philosophical difference that is evident here is that ScalaTest gives you a lot more choices. To make this space of choice easier to navigate, I provide a decision tree here:http://www.scalatest.org/quick_start
The matcher syntax is also different between ScalaTest and specs. In ScalaTest I tried to see how far I could go with operator notation, and ended up with matcher expressions that read very much like English sentences, with spaces between the words. Specs matcher syntax runs words together more with camel case.
Specs has more matchers than ScalaTest, and that I think reflects a difference in design attitude. I actually cut probably 2/3 of the matcher syntax I built and considered for release. I will add more matchers in future releases, but wanted to be sure I knew users actually wanted something before I added it. However ScalaTest's matchers includes a dynamic property matcher syntax takes up some of that slack. For example in Specs you can write on a
java.io.File
:This will invoke the
isDirectory
and make sure it is true. ScalaTest does not have any special matchers forjava.io.Files
currently, but in ScalaTest, you could just use a dynamic check like this:Anytime you pass a symbol in after
be
, it will use reflection to look for (in this case) a method or field nameddirectory
or a method namedisDirectory
. There's also a way to make this static, by defining aBePropertyMatcher
(which requires only 2 or 3 lines of code usually). So basically in ScalaTest I try to provide more functionality with less API.Another general design attitude difference between specs and ScalaTest involves implicit conversions. By default you get only one implicit conversion when you use ScalaTest, which is the one that puts the
===
operator on everything. (If you need to, you can "turn off" this implicit conversion with one line of code. The only reason you would need to do that is if you were trying to test something that has its own===
operator, and you get a conflict.) ScalaTest defines many other implicit conversions, but to use them you need to explicitly "invite" them into your code by mixing in a trait or doing an import. When you extend classSpecification
in specs I think you pretty much get dozens of implicit conversions by default. I'm not sure how much that will matter in practice, but I figure people will want to test code that uses their own implicits, and sometimes there may be a conflict between the test framework's implicits and those of the production code. When that happens I think it may be easier to work around the problem in ScalaTest than specs.Another difference in design attitude that I've noticed is comfort with operators. One goal I had was that any programmer looking at someone else's test code that uses ScalaTest would be able to guess what the meaning was without looking anything up in the ScalaTest documentation. I wanted ScalaTest client code to be drop dead obvious. One way that goal manifested itself is that ScalaTest is very conservative about operators. I only define five operators in ScalaTest:
===
, which means equals>
, which means greater than<
, less than>=
, greater than or equal<=
, less than or equal.That's it. So these things pretty much look like what mean. If you see in someone else's code:
My hope is that you won't need to run to the API documentation to guess what that
<=
means. By contrast, specs is much freer with operators. Nothing wrong with that, but it is a difference. Operators can make code more concise, but the tradeoff is you may have to run to the documentation when you find things like->-
,>>
,|
,|>
,!
, or^^^
(which all have special meanings in Specs) in your colleague's test code.One other philosophical difference is that I do try and make it just slightly easier in ScalaTest to use a functional style when you need to share a fixture, whereas Specs by default continues the tradition of the
setUp
andtearDown
approach popularized by JUnit, in which you reassign vars before each test. However if you want to test that way, it is also very easy in ScalaTest. You just need to mix in theBeforeAndAfter
trait.For more insight into ScalaTest, you can watch the "Get Higher with ScalaTest" presentation I gave at the 2009 Devoxx conference here:
http://parleys.com/play/514892260364bc17fc56bde3/chapter0/about