So here is what I learned:
There are some testing frameworks written in bash and for bash,
however...
It is not so much that Bash is not suitable for TDD (although some
other languages come to mind that are a better fit), but the
typical tasks that Bash is used for (Installation, System
configuration), that are hard to write tests for, and in
particularly hard to setup the test.
The poor data structure support in Bash makes it hard to separate
logic from side-effect, and indeed there is typically little logic
in Bash scripts. That makes it hard to break scripts into
testable chunks. There are some functions that can be tested, but
that is the exception, not the rule.
Function are a good thing (tm), but they can only go so far.
Nested functions can be even better, but they are also limited.
At the end of the day, with major effort some coverage can be
obtained, but it will test the less interesting part of the code,
and will keep the bulk of the testing as a good (or bad) old manual
testing.
Meta: I decided to answer (and accept) my own question, because I was unable to choose between Sinan Ünür's (voted up) and mouviciel's (voted up) answers that where equally useful and insightful. I want to note Stefano Borini's answer, that although not impressed me initially, I learned to appreciate it over time. Also his design patterns or best practices for shell scripts answer (voted up) referred above was useful.
The Quick Answer
One very important point to bring up is that there are two flavors of Behavior Driven Development. The two flavors are xBehave and xSpec.
xBehave BDD: SpecFlow
SpecFlow (very similar to cucumber from the Ruby stack) is excellent in facilitating xBehave BDD tests as Acceptance Criteria. It does not however provide a good way to write behavioral tests on a unit level. There are a few other xBehave testing frameworks, but SpecFlow has gotten a lot of traction.
xSpec BDD: NSpec
For behavior driven development on a unit level, I would recommend NSpec (inspired directly by RSpec for Ruby). You can accomplish BDD on a unit level by simply using NUnit or MSTest...but they kinda fall short (it's really hard to build up contexts incrementally). MSpec is also an option, there has been a lot of work put into it, but there are just somethings that are just simpilier in NSpec (you can build up context incrementally in MSpec, but it requires inheritance which can become complex).
The Long Answer
The two flavors of BDD primarily exist because of the orthogonal benefits they provide.
Pros and Cons of xBehave (GWT Syntax)
Pros
- helps facilitate a conversations with the business through a common dialect called (eg. Given ...., And Given ...., When ......, And When ..... , Then ...., And Then)
- the common dialect can then be mapped to executable code which proves to the business that you actually finished what you said you'd finish
- the dialect is constricting, so the business has to disambiguate requirements and make it fit into the sentences.
Cons
- While the xBehave approach is good for driving high level Acceptance Criteria, the cycles needed to map English to executable code via attributes makes it infeasible for driving out a domain at the unit level.
- Mapping the common dialect to tests is PAINFUL (ramp up on your regex). Each sentence the business creates must be mapped to an executable method via attributes.
- The common dialect must be tightly controlled so that managing the mapping doesn't get out of hand. Any time you change a sentence, you have to find method that directly relates to that sentence and fix the regex matching.
Pros and Cons of xSpec (Context/Specification)
Pros
- Allows the developer to build up context incrementally. A context can be set up for a test and some assertions can be performed against that context. You can then specify more context (building upon the context that already exists) and then specify more tests.
- No constricting language. Developers can be more expressive about how a certain part of a system behaves.
- No mapping needed between English and a common dialect (because there isn't one).
Cons
- Not as approachable by the business. Let's face it, the business don't like to disambiguate what they want. If we gave them a context based approach to BDD then the sentence would just read "Just make it work".
- Everything is in the code. The context documentation is intertwined within the code (that's why we don't have to worry about mapping english to code)
- Not as readable given a less restrictive verbiage.
Samples
The Bowling Kata is a pretty good example.
SpecFlow Sample
Here is what the specification would look like in SpecFlow (again, this is great as an acceptance test, because it communicates directly with the business):
Feature File
The feature file is the common dialect for the test.
Feature: Score Calculation
In order to know my performance
As a player
I want the system to calculate my total score
Scenario: Gutter game
Given a new bowling game
When all of my balls are landing in the gutter
Then my total score should be 0
Step Definition File
The step definition file is the actual execution of the test, this file includes the mappings for SpecFlow
[Binding]
public class BowlingSteps
{
private Game _game;
[Given(@"a new bowling game")]
public void GivenANewBowlingGame()
{
_game = new Game();
}
[When(@"all of my balls are landing in the gutter")]
public void WhenAllOfMyBallsAreLandingInTheGutter()
{
_game.Frames = "00000000000000000000";
}
[Then(@"my total score should be (\d+)")]
public void ThenMyTotalScoreShouldBe(int score)
{
Assert.AreEqual(0, _game.Score);
}
}
NSpec Sample, xSpec, Context/Specification
Here is a NSpec example of the same bowling kata:
class describe_BowlingGame : nspec
{
Game game;
void before_each()
{
game = new Game();
}
void when_all_my_balls_land_in_the_gutter()
{
before = () =>
{
game.Frames = "00000000000000000000";
};
it["should have a score of 0"] = () => game.Score.should_be(0);
}
}
So Yea...SpecFlow is cool, NSpec is cool
As you do more and more BDD, you'll find that both the xBehave and xSpec flavors of BDD are needed. xBehave is more suited for Acceptance Tests, xSpec is more suited for unit tests and domain driven design.
Relevant Links
Best Answer
While I agree generally with gishu's post, there are a couple of areas in which I disagree. In the IMHO section, he presents BDD specification as the user story specification developed by Rachel Davies et al: As a... I want... so that.
The BDD specification is Given... When... Then... as in
This is about conditions, actions, and expectations, and is core to BDD.
ATDD is, as gishu suggests, the practice of driving development through the use of acceptance test specifications, implemented as executable acceptance criteria. The specification, in the BDD form, is neither required nor "best practice." In practice, however, it is effective in focusing the thinking and language on how to verify that the work has been done satisfactorily and meets requirements.
Note that BDD is not particularly based on TDD. ATDD is loosely based on TDD in that it is testing that is done before development is done. Beyond that, it is not focused on developer work, but on the overall direction and validation of the project. ATDD meshes well with Story Mapping, in that it plays well during the discovery phase when higher level requirements are being written, and it's important to know "how will we know when it has been done properly?"