1. Cgreen Quickstart Guide

1.1. What is Cgreen?

Cgreen is a unit tester for the C and C++ software developer, a test automation and software quality assurance tool for programmers and development teams. The tool is completely open source published under the ISC, OpenBSD, license.

Unit testing is a development practice popularised by the agile development community. It is characterised by writing many small tests alongside the normal code. Often the tests are written before the code they are testing, in a tight test-code-refactor loop. Done this way, the practice is known as Test Driven Development. Cgreen was designed specifically to support this style of development.

Unit tests are written in the same language as the code, in our case C or C++. This avoids the mental overhead of constantly switching language, and also allows you to use any application code in your tests.

Here are some of its features:

  • Fluent API resulting in very readable tests

  • Expressive and clear output using the default reporter

  • Fully functional mocks, both strict and loose

  • Each test runs in its own process for test suite robustness

  • Automatic discovery and running of tests using dynamic library inspection

  • Extensive and expressive constraints for many datatypes

  • BDD-flavoured test declarations with Before and After declarations

  • Extensible reporting mechanism

  • Fully composable test suites

  • A single test can be run in a single process for easier debugging

Cgreen also supports the classic xUnit-style assertions for easy porting from other frameworks.

Cgreen was initially developed to support C programming, but there is also some support for C++. It was initially a spinoff from a research project at Wordtracker and created by Marcus Baker. Significant additions by Matt Hargett and continuous nurturing by Thomas Nilefalk has made Cgreen what it is today.

1.2. Cgreen - Vanilla or Chocolate?

Test driven development (TDD) really catched on when the JUnit framework for Java spread to other langauges, giving us a family of xUnit tools. Cgreen was born in this wave and have many similarities to the xUnit family.

But TDD evolved over time and modern thinking and practice is more along the lines of BDD, an acronym for Behaviour Driven Development, made popular by people like Dan North and frameworks like JBehave, RSpec, Cucumber and Jasmine.

Cgreen follows this trend and has evolved to embrace a BDD-flavoured style of testing. Although the fundamental mechanisms in TDD and 'technical' BDD are much the same, the shift in focus by changing wording from 'tests' to 'behaviour specifications' is very significant.

This document will present Cgreen using the more modern and better BDD-style. In a later section you can have a peek at the classic TDD API.

1.3. Installing Cgreen

There are two ways to install Cgreen in your system.

1.3.1. Installing a package

At this point there are few supported pre-built packages available. For now you’ll probably have to build from source.

The first way is to use packages provided by the Cgreen Team. If your system uses a package manager ('apt' or 'port' and so on) there might be a prebuilt package that you can just install using your systems package manager.

If no Cgreen package is distributed for your system you can download a package from Cgreen GitHub project. Install it using the normal procedures for your system.

1.3.2. Installing from source

The second way is available for developers and advanced users. Basically this consists of fetching the sources of the project on GitHub, just click on "Download ZIP", and compiling them. To do this you need the CMake build system.

Once you have the CMake tool installed, the steps are:

$ unzip cgreen-master.zip
$ cd cgreen-master
$ make
$ make test
$ make install

The initial make command will configure the build process and create a separate build directory before going there and building using CMake. This is called an 'out of source build'. It compiles Cgreen from outside the sources directory. This helps the overall file organization and enables multi-target builds from the same sources by leaving the complete source tree untouched.

Experienced users may tweak the build configuration by going to the build subdirectory and use ccmake .. to modify the build configuration in that subtree.
The Makefile is just there for convenience, it just creates the build directory and invokes CMake there, so that you don’t have to. This means that experienced CMake users can just do as they normally do with a CMake-based project instead of invoking make.

The build process will create a library (on unix called libcgreen.so) which can be used in conjunction with the cgreen.h header file to compile and link your test code. The created library is installed in the system, by default in the /usr/local/lib/.

1.3.3. Your First Test

We will start demonstrating the use of Cgreen by writing some tests for Cgreen itself to confirm that everything is working as it should. Let’s start with a simple test module with no tests, called first_test.c…​

#include <cgreen/cgreen.h>

Describe(Cgreen);
BeforeEach(Cgreen) {}
AfterEach(Cgreen) {}

int main(int argc, char **argv) {
    TestSuite *suite = create_test_suite();
    return run_test_suite(suite, create_text_reporter());
}

This is very unexciting. It just creates an empty test suite and runs it. It’s usually easier to proceed in small steps, and this is the smallest one I could think of. The only complication is the cgreen.h header file and the mysterious looking "declarations" at the beginning of the file.

The BDD flavoured Cgreen notation calls for a System Under Test (SUT), or a 'context'. The declarations give a context to the tests and it also makes it more natural to talk about which module or class, the system under test, is actually responsible for the functionality we are expressing. In one way we are 'describing', or spec’ing, the functionality of the SUT. That’s what the Describe(); does. And for technical reasons (actually requirements of the C language), you must declare the BeforeEach() and AfterEach() functions even if they are empty. (You will get strange errors if you don’t!)

We are using the name "Cgreen" as the SUT in these first examples, as Cgreen itself is the object or class we want to test or describe.

I am assuming you have the Cgreen folder in the include search path to ensure compilation works, otherwise you’ll need to add that in the compilation command.

Then, building this test is, of course, trivial…​

$ gcc -c first_test.c
$ gcc first_test.o -lcgreen -o first_test
$ ./first_test

Invoking the executable should give…​

Running "main" (0 tests)...
Completed "main": No asserts.

All of the above rather assumes you are working in a Unix like environment, probably with 'gcc'. The code is pretty much standard C99, so any C compiler should work. Cgreen should compile on all systems that support the sys/msg.h messaging library. It has been tested on Linux, MacOSX, Cygwin and Windows.

So far we have tried compilation, and shown that the test suite actually runs. Let’s add a meaningless test or two so that you can see how it runs…​

#include <cgreen/cgreen.h>

Describe(Cgreen);
BeforeEach(Cgreen) {}
AfterEach(Cgreen) {}

Ensure(Cgreen, passes_this_test) {
    assert_that(1 == 1);
}

Ensure(Cgreen, fails_this_test) {
    assert_that(0 == 1);
}

int main(int argc, char **argv) {
    TestSuite *suite = create_test_suite();
    add_test_with_context(suite, Cgreen, passes_this_test);
    add_test_with_context(suite, Cgreen, fails_this_test);
    return run_test_suite(suite, create_text_reporter());
}

A test is denoted by the macro Ensure which takes an optional context (Cgreen) and a, hopefully descriptive, testname (passes_this_test). You add the test to your suite using add_test_with_context().

On compiling and running, we now get the output…​

Running "main" (2 tests)...
first_tests.c:12: Failure: fails_this_test
	Expected [0 == 1] to [be true]

  "main": 1 pass, 1 failure in 42ms.
Completed "main": 1 pass, 1 failure in 42ms.

The TextReporter, created by the call to create_text_reporter(), is the easiest way to output the test results. It prints the failures as intelligent and expressive text messages on your console.

Of course "0" would never equal "1", but this shows that Cgreen presents the value you expect ([be true]) and the expression that you want to assert ([0 == 1]). We can also see a handy short form for asserting boolean expressions (assert_that(0 == 1);).

1.4. Five Minutes Doing TDD with Cgreen

For a more realistic example we need something to test. We’ll pretend that we are writing a function to split the words of a sentence in place. It would do this by replacing any spaces with string terminators and returns the number of conversions plus one. Here is an example of what we have in mind…​

char *sentence = strdup("Just the first test");
word_count = split_words(sentence);

The variable sentence should now point at "Just\0the\0first\0test". Not an obviously useful function, but we’ll be using it for something more practical later.

This time around we’ll add a little more structure to our tests. Rather than having the test as a stand alone program, we’ll separate the runner from the test cases. That way, multiple test suites of test cases can be included in the main() runner file. This makes it less work to add more tests later.

Here is the, so far empty, test case in words_test.c…​

#include <cgreen/cgreen.h>
#include <cgreen/mocks.h>

#include "words.h"
#include <string.h>

Describe(Words);
BeforeEach(Words) {}
AfterEach(Words) {}

TestSuite *words_tests() {
    TestSuite *suite = create_test_suite();
    return suite;
}

Here is the all_tests.c test runner…​

#include <cgreen/cgreen.h>

TestSuite *words_tests();

int main(int argc, char **argv) {
    TestSuite *suite = create_test_suite();
    add_suite(suite, words_tests());
    if (argc > 1) {
        return run_single_test(suite, argv[1], create_text_reporter());
    }
    return run_test_suite(suite, create_text_reporter());
}

Cgreen has two ways of running tests. The default is to run all tests in their own protected processes. This is what happens if you invoke run_test_suite(). All tests are then completely independent since they run in separate processes, preventing a single run-away test from bringing the whole program down with it. It also ensures that one test cannot leave any state to the next, thus forcing you to setup the prerequisites for each test correctly and clearly.

Building this scaffolding…​

$ gcc -c words_test.c
$ gcc -c all_tests.c
$ gcc words_test.o all_tests.o -lcgreen -o all_tests

…​and executing the result gives the familiar…​

Running "main" (0 tests)...
  "words_tests": No asserts.
Completed "main": No asserts.

Note that we get an extra level of output here, we have both main and words_tests. That’s because all_tests.c adds the words test suite to its own (named main since it was created in the function main()). All this scaffolding is pure overhead, but from now on adding tests will be a lot easier.

Here is a first test for split_words() in words_test.c…​

#include <cgreen/cgreen.h>

#include "words.h"
#include <string.h>

Describe(Words);
BeforeEach(Words) {}
AfterEach(Words) {}

Ensure(Words, returns_word_count) {
    char *sentence = strdup("Birds of a feather");
    int word_count = split_words(sentence);
    assert_that(word_count, is_equal_to(4));
    free(sentence);
}

TestSuite *words_tests() {
    TestSuite *suite = create_test_suite();
    add_test_with_context(suite, Words, returns_word_count);
    return suite;
}

The assert_that() macro takes two parameters, the value to assert and a constraint. The constraints comes in various forms. In this case we use the probably most common, is_equal_to(). With the default TextReporter the message is sent to STDOUT.

To get this to compile we need to create the words.h header file…​

int split_words(char *sentence);

…​and to get the code to link we need a stub function in words.c…​

int split_words(char *sentence) {
    return 0;
}

A full build later…​

$ gcc -c all_tests.c
$ gcc -c words_test.c
$ gcc -c words.c
$ gcc all_tests.o words_test.o words.o -lcgreen -o all_tests
$ ./all_tests

…​and we get the more useful response…​

Running "main" (1 tests)...
words_tests.c:13: Failure: words_tests -> returns_word_count
	Expected [word_count] to [equal] [4]
		actual value:			[0]
		expected value:			[4]

  "words_tests": 1 failure in 42ms.
Completed "main": 1 failure in 42ms.

The breadcrumb trail following the "Failure" text is the nesting of the tests. It goes from the test suites, which can be nested in each other, through the test function, and finally to the message from the assertion. In the language of Cgreen, a "failure" is a mismatched assertion, or constraint, and an "exception" occurs when a test fails to complete for any reason, e.g. a segmentation fault.

We could get this to pass just by returning the value 4. Doing TDD in really small steps, you would actually do this, but we’re not teaching TDD here. Instead we’ll go straight to the core of the implementation…​

#include <string.h>

int split_words(char *sentence) {
  int i, count = 1;
  for (i = 0; i < strlen(sentence); i++) {
    if (sentence[i] == ' ') {
      count++;
    }
  }
  return count;
}

Running it gives…​

Running "main" (1 tests)...
  "words_tests": 1 pass in 42ms.
Completed "main": 1 pass in 42ms.

There is actually a hidden problem here, but our tests still passed so we’ll pretend we didn’t notice.

So it’s time to add another test. We want to confirm that the string is broken into separate words…​

...
Ensure(Words, returns_word_count) {
    ...
}

Ensure(Words, converts_spaces_to_zeroes) {
    char *sentence = strdup("Birds of a feather");
    split_words(sentence);
    int comparison = memcmp("Birds\0of\0a\0feather", sentence, strlen(sentence));
    assert_that(comparison, is_equal_to(0));
    free(sentence);
}

Sure enough, we get a failure…​

Running "main" (2 tests)...
words_tests.c:21: Failure: words_tests -> converts_spaces_to_zeroes
	Expected [comparison] to [equal] [0]
		actual value:			[-8192]
		expected value:			[0]

  "words_tests": 1 pass, 1 failure in 42ms.
Completed "main": 1 pass, 1 failure in 42ms.

Not surprising given that we haven’t written the code yet.

The fix…​

#include <string.h>

int split_words(char *sentence) {
  int i, count = 1;
  for (i = 0; i < strlen(sentence); i++) {
    if (sentence[i] == ' ') {
      sentence[i] = '\0';
      count++;
    }
  }
  return count;
}

…​reveals our previous hack…​

Running "main" (2 tests)...
words_tests.c:13: Failure: words_tests -> returns_word_count
	Expected [word_count] to [equal] [4]
		actual value:			[2]
		expected value:			[4]

  "words_tests": 1 pass, 1 failure in 42ms.
Completed "main": 1 pass, 1 failure in 42ms.

Our earlier test now fails, because we have affected the strlen() call in our loop. Moving the length calculation out of the loop…​

int split_words(char *sentence) {
  int i, count = 1, length = strlen(sentence);
  for (i = 0; i < length; i++) {
    ...
  }
  return count;
}

…​restores order…​

Running "main" (2 tests)...
  "words_tests": 2 passes in 42ms.
Completed "main": 2 passes in 42ms.

It’s nice to keep the code under control while we are actually writing it, rather than debugging later when things are more complicated.

That was pretty straight forward. Let’s do something more interesting.

1.5. What are Mock Functions?

The next example is a more realistic extension of our previous attempts. As in real life we first implement something basic and then we go for the functionality that we need. In this case a function that invokes a callback for each word found in a sentence. Something like…​

void act_on_word(const char *word, void *memo) { ... }
words("This is a sentence", &act_on_word, &memo);

Here the memo pointer is just some accumulated data that the act_on_word() callback might work with. Other people will write the act_on_word() function and probably many other functions like it. The callback is actually a flex point, and not of interest right now.

The function under test is the words() function and we want to make sure it walks the sentence correctly, dispatching individual words as it goes. So what calls are made are very important. How to test this?

Let’s start with a one word sentence. In this case we would expect the callback to be invoked once with the only word, right? Here is the test for that…​

#include <cgreen/cgreen.h>
#include <cgreen/mocks.h>
...
void mocked_callback(const char *word, void *memo) {
    mock(word, memo);
}

Ensure(Words, invokes_callback_once_for_single_word_sentence) {
    expect(mocked_callback,
           when(word, is_equal_to_string("Word")), when(memo, is_null));
    words("Word", &mocked_callback, NULL);
}

TestSuite *words_tests() {
    TestSuite *suite = create_test_suite();
    ...
    add_test_with_context(suite, Words, invokes_callback_once_for_single_word_sentence);
    return suite;
}

What is the funny looking mock() function?

A mock is basically a programmable object. In C objects are limited to functions, so this is a mock function. The macro mock() compares the incoming parameters with any expected values and dispatches messages to the test suite if there is a mismatch. It also returns any values that have been preprogrammed in the test.

The test is invokes_callback_once_for_single_word_sentence(). It programs the mock function using the expect() macro. It expects a single call, and that single call should use the parameters "Word" and NULL. If they don’t match, we will get a test failure.

So when the code under test (our words() function) calls the injected mocked_callback() it in turn will call mock() with the actual parameters.

Of course, we don’t add the mock callback to the test suite, it’s not a test.

For a successful compile and link, the words.h file must now look like…​

int split_words(char *sentence);
void words(const char *sentence, void (*callback)(const char *, void *), void *memo);

…​and the words.c file should have the stub…​

void words(const char *sentence, void (*callback)(const char *, void *), void *memo) {
}

This gives us the expected failing test…​

Running "main" (3 tests)...
words_tests.c:33: Failure: words_tests -> invokes_callback_once_for_single_word_sentence
	Expected call was not made to mocked function [mocked_callback]

  "words_tests": 2 passes, 1 failure in 42ms.
Completed "main": 2 passes, 1 failure in 42ms.

Cgreen reports that the callback was never invoked. We can easily get the test to pass by filling out the implementation with…​

void words(const char *sentence, void (*callback)(const char *, void *), void *memo) {
  (*callback)(sentence, memo);
}

That is, we just invoke it once with the whole string. This is a temporary measure to get us moving. For now everything should pass, although it doesn’t drive much functionality yet.

Running "main" (3 tests)...
  "words_tests": 4 passes in 42ms.
Completed "main": 4 passes in 42ms.

That was all pretty conventional, but let’s tackle the trickier case of actually splitting the sentence. Here is the test function we will add to words_test.c…​

Ensure(Words, invokes_callback_for_each_word_in_a_phrase) {
    expect(mocked_callback, when(word, is_equal_to_string("Birds")));
    expect(mocked_callback, when(word, is_equal_to_string("of")));
    expect(mocked_callback, when(word, is_equal_to_string("a")));
    expect(mocked_callback, when(word, is_equal_to_string("feather")));
    words("Birds of a feather", &mocked_callback, NULL);
}

Each call is expected in sequence. Any failures, or left-over or extra calls, and we get failures. We can see all this when we run the tests…​

Running "main" (4 tests)...
words_tests.c:38: Failure: words_tests -> invokes_callback_for_each_word_in_a_phrase
	Expected [[word] parameter in [mocked_callback]] to [equal string] ["Birds"]
		actual value:			["Birds of a feather"]
		expected to equal:		["Birds"]

words_tests.c:39: Failure: words_tests -> invokes_callback_for_each_word_in_a_phrase
	Expected call was not made to mocked function [mocked_callback]

words_tests.c:40: Failure: words_tests -> invokes_callback_for_each_word_in_a_phrase
	Expected call was not made to mocked function [mocked_callback]

words_tests.c:41: Failure: words_tests -> invokes_callback_for_each_word_in_a_phrase
	Expected call was not made to mocked function [mocked_callback]

  "words_tests": 4 passes, 4 failures in 42ms.
Completed "main": 4 passes, 4 failures in 42ms.

The first failure tells the story. Our little words() function called the mock callback with the entire sentence. This makes sense, because that was the hack we did to get to the next test.

Although not relevant to this guide, I cannot resist getting these tests to pass. Besides, we get to use the function we created earlier…​

void words(const char *sentence, void (*callback)(const char *, void *), void *memo) {
  char *words = strdup(sentence);
  int word_count = split_words(words);
  char *word = words;
  while (word_count-- > 0) {
    (*callback)(word, memo);
    word = word + strlen(word) + 1;
  }
  free(words);
}

And with some work we are rewarded with…​

Running "main" (4 tests)...
  "words_tests": 8 passes in 42ms.
Completed "main": 8 passes in 42ms.

More work than I like to admit as it took me three goes to get this right. I firstly forgot the + 1 added on to strlen(), then forgot to swap sentence for word in the (*callback)() call, and finally third time lucky. Of course running the tests each time made these mistakes very obvious. It’s taken me far longer to write these paragraphs than it has to write the code.