6. Automatic Test Discovery

6.1. Forgot to Add Your Test?

When we write a new test we focus on the details about the test we are trying to write. And writing tests is no trivial matter so this might well take a lot of brain power.

So, it comes as no big surprise, that sometimes you write your test and then forget to add it to the suite. When we run it it appears that it passed on the first try! Although this should really make you suspicious, sometimes you get so happy that you just continue with churning out more tests and more code. It’s not until some (possibly looong) time later that you realize, after much headache and debugging, that the test did not actually pass. It was never even run!

There are practices to minimize the risk of this happening, such as always running the test as soon as you can set up the test. This way you will see it fail before trying to get it to pass.

But it is still a practice, something we, as humans, might fail to do at some point. Usually this happens when we are most stressed and in need of certainty.

6.2. The Solution - the 'cgreen-runner'

Cgreen gives you a tool to avoid not only the risk of this happening, but also the extra work and extra code. It is called the cgreen-runner.

The cgreen-runner should come with your Cgreen installation if your platform supports the technique that is required, which is 'programatic access to dynamic loading of libraries'. This means that a program can load an external library of code into memory and inspect it. Kind of self-inspection, or reflexion.

So all you have to do is to build a dynamically loadable library of all tests (and of course your objects under test and other necessary code). Then you can run the cgreen-runner and point it to the library. The runner will then load the library, enumerate all tests in it, and run every test.

It’s automatic, and there is nothing to forget.

6.3. Using the Runner

Assuming your tests are in first_test.c the typical command to build your library using gcc would be

$ gcc -shared -o first_test.so -fPIC first_test.c -lcgreen

The -fPIC means to generate 'position independent code' which is required if you want to load the library dynamically. To explicitly state this is required on many platforms.

How to build a dynamically loadable shared library might vary a lot depending on your platform. Can’t really help you there, sorry!

As soon as we have linked it we can run the tests using the cgreen-runner by just giving it the shared, dynamically loadable, object library as an argument:

$ cgreen-runner first_test.so
Running "first_tests" (2 tests)...
first_tests.c:12: Failure: Cgreen -> fails_this_test
	Expected [0 == 1] to [be true]

  "Cgreen": 1 pass, 1 failure in 42ms.
Completed "first_tests": 1 pass, 1 failure in 42ms.

More or less exactly the same output as when we ran our first test in the beginning of this quickstart tutorial. We can see that the top level of the tests will be named as the library it was discovered in, and the second level is the context for our System Under Test, in this case 'Cgreen'. We also see that the context is mentioned in the failure message, giving a fairly obvious Cgreen → fails_this_test.

Now we can actually delete the main function in our source code. We don’t need all this, since the runner will discover all tests automatically.

int main(int argc, char **argv) {
    TestSuite *suite = create_test_suite();
    add_test_with_context(suite, Cgreen, passes_this_test);
    add_test_with_context(suite, Cgreen, fails_this_test);
    return run_test_suite(suite, create_text_reporter());
}

It always feel good to delete code, right?

We can also select which test to run:

$ cgreen-runner first_test.so Cgreen:this_test_should_fail
Running "first_tests" (1 tests)...
first_tests.c:12: Failure: Cgreen -> fails_this_test
	Expected [0 == 1] to [be true]

  "Cgreen": 1 failure in 42ms.
Completed "first_tests": 1 failure in 42ms.

We recommend the BDD notation to discover tests, and you indicate which context the test we want to run is in. In this example it is Cgreen so the test should be refered to as Cgreen:this_test_should_fail.

If you don’t use the BDD notation there is actually a context anyway, it is called default.

6.4. Cgreen Runner Options

Once you get the build set up right for the cgreen-runner everything is fairly straight-forward. But you have a few options:

--xml <prefix>

Instead of messages on stdout with the TextReporter, write results into one XML-file per suite or context, compatible with Hudson/Jenkins CI. The filename(s) will be <prefix>-<suite>.xml

--suite <name>

Name the top level suite

--no-run

Don’t run the tests

--verbose

Show progress information and list discovered tests

--colours

Use colours (or colors) to emphasis result (requires ANSI-capable terminal)

--quiet

Be more quiet

The verbose option is particularly handy since it will give you the actual names of all tests discovered. So if you have long test names you can avoid mistyping them by copying and pasting from the output of cgreen-runner --verbose. It will also give the mangled name of the test which should make it easier to find in the debugger. Here’s an example:

Discovered Cgreen:fails_this_test (CgreenSpec__Cgreen__fails_this_test__)
Discovered Cgreen:passes_this_test (CgreenSpec__Cgreen__passes_this_test__)
Discovered 2 test(s)
Opening [first_tests.so] to only run one test: 'Cgreen:fails_this_test' ...
Running "first_tests" (1 tests)...
first_tests.c:12: Failure: Cgreen -> fails_this_test
	Expected [0 == 1] to [be true]

  "Cgreen": 1 failure in 42ms.
Completed "first_tests": 1 failure in 42ms.

6.5. Selecting Tests To Run

You can name a single test to be run by giving it as the last argument on the command line. The name should be in the format <SUT>:<test>. If not obvious you can get that name by using the --verbose command option which will show you all tests discovered and both there C/C++ and Cgreen names. Copying the Cgreen name from that output is an easy way to run only that particular test. When a single test is named it is run using run_single_test(). As described in Five Minutes Doing TDD with Cgreen this means that it is not protected by fork()-ing it to run in its own process.

The cgreen-runner supports selecting tests with limited pattern matching. Using an asterisk as a simple 'match many' symbol you can say things like

$ cgreen-runner <library> Cgreen:*
$ cgreen-runner <library> C*:*this*

6.6. Multiple Test Libraries

You can run tests in multiple libraries in one go by adding them to the cgreen-runner command:

$ cgreen-runner first_set.so second_set.so ...

6.7. Setup, Teardown and Custom Reporters

The cgreen-runner will only run setup and teardown functions if you use the BDD-ish style with BeforeEach() and AfterEach() as described above. The runner does not pickup setup() and teardown() added to suites, because it actually doesn’t run suites. It discovers all tests and runs them one by one. The macros required by the BDD-ish style ensures that the corresponding BeforeEach() and AfterEach() are run before and after each test.

The cgreen-runner will discover your tests in a shared library even if you don’t use the BDD-ish style. But it will not be able to find and run the setup() and/or teardown() attached to your suite(s). This will probably cause your tests to fail or crash.

In case you have non-BDD style tests without any setup() and/or teardown() you can still use the runner. The default suite/context where the tests live in this case is called default. But why don’t you convert your tests to BDD notation? This removes the risk of frustrating trouble-shooting when you added setup() and teardown() and can’t understand why they are not run…​

So, the runner encourages you to use the BDD notation. But since we recommend that you do anyway, that’s no extra problem if you are starting out from scratch. But see Changing Style for some easy tips on how to get you there if you already have non-BDD tests.

You can choose between the TextReporter, which we have been seeing so far, and the built-in JUnit/Ant compatible XML-reporter using the --xml option. But it is not currently possible to use custom reporters as outlined in Changing Cgreen Reporting with the runner.

If you require another custom reporter you need to resort to the standard, programatic, way of invoking your tests. For now…​

6.8. Skipping Tests

Sometimes you find that you need to temporarily remove a test, perhaps to do a refactoring when you have a failing test. Ignoring that test will allow you to do the refactoring while still in the green.

An old practice is then to comment it out. That is a slightly cumbersome. It is also hazardous habit as there is no indication of a missing test if you forget to uncomment it when you are done.

Cgreen offers a much better solution. You can just add an 'x' infront of the Ensure for the test and that test will be skipped.

...
xEnsure(Reader, ...) {
  ...
}
...

With this method, it is a one character change to temporarily ignore, and un-ignore, a test. It is also easily found using text searches through a complete source tree. Cgreen will also tally the skipped tests, so it is clearly visible that you have some skipped test when you run them.