Command Line Arguments in Unit Tests – Best Practices

cgtestpatterns-and-practicesprogramming practicesunit testing

I am on C++ and using gtest as the main framework. Say I have a edge detection function I want to test that takes an image as an input and returns the edge detected image. I have 3 images ready to be checked.

Is it better to write it in a way that is self contained like

void EdgeCheck(const std::string& input, const std::string& ans) {
    cv::Mat in_img = cv::imread(input);
    cv::Mat ans_img = cv::imread(ans);
    // Do some checks here
}

TEST(Edge) {
    EdgeCheck(std::string("path to image 1"),std::string("path to ans image 1"));
    EdgeCheck(std::string("path to image 2"),std::string("path to ans image 2"));
    EdgeCheck(std::string("path to image 3"),std::string("path to ans image 3"));
}

int main(int argc, char** argv) {
    return UnitTest::RunAllTests();
}

so the test can be executed by simple ./unit-test or is it better to write it like

TEST(Edge) {
    cv::Mat in_img = cv::imread(argv[1]);
    cv::Mat ans_img = cv::imread(argv[2]);

    // Do some check here
}

int main(int argc, char** argv) {
    return UnitTest::RunAllTests();
}

but it has to be executed by 

./unit-test image1 ans_image1
./unit-test image2 ans_image2
./unit-test image3 ans_image3

From my understanding, the advantage of the first approach according to UnitTest++ command line arguments is "a well-written unit-test is self-contained, and parameterizing a test means the test is no longer just a test. It's now a function rather than a unit test."

However, the advantage of the second approach is it's more flexible in the sense that if I want to add image4 and image5, I don't need to recompile the code and it can be done on a bash script, but then it makes the test not self contained.

Which approach is considered a better practice?

Best Answer

This is all about how you want to manage your tests and integrate them into your whole environment. I assume you have the requirements to

  • add new tests easily
  • run all your tests at once, maybe as part of an test suite
  • make sure one failing test does not prevent the execution of other, independent tests
  • get a log afterwards which of your tests haved failed, and which not
  • run a subset of your tests or a single test to verify it does not fail any more after you applied a bugfix
  • make it easy for anyone else to run your tests without special knowledge

Looking at both of your two solutions I think making them parametrized by command line will give you more flexibility for switching between running them individually vs. "all at once", and it will make it easier to make sure a failing test does not influence other tests. Maybe it will also be easier to integrate them into a bigger test suite, if you have that requirement. However, this flexibility does not come for free. Since one needs to know the parameters, your command line program, when called in a wrong fashion, should give a descriptive message which kind of parameters it expects, and the test execution script should be integral part of your tests.

So as long as you treat the command line program together with the script as a self-contained component, and as long as you provide some integral documention, there is nothing wrong with the parametrized approach. If that is not "self-contained enough" for you for some weired reason, integrate things in one program, but live with the fact the other requirements I mentioned may need more effort to resolve.

Related Topic