Cgreen : Unit Tests, Stubbing and Mocking for C and C++

Cgreen is a unit testing framework for the C and C++ software developer, a test automation and software quality assurance tool for programmers and development teams. The tool is completely open source published under the ISC, OpenBSD, license.

Unit testing is a development practice popularised by the agile development community. It is characterised by writing many small tests alongside the normal code. Often the tests are written before the code they are testing, in a tight test-code-refactor loop. Done this way, the practice is known as Test Driven Development. Cgreen was designed specifically to support this style of development.

Unit tests are written in the same language as the code, in our case C or C++. This avoids the mental overhead of constantly switching language, and also allows you to use any application code in your tests.

Here are some of its features:

Cgreen also supports the classic xUnit-style assertions for easy porting from other frameworks.

Cgreen was initially developed to support C programming, but there is also support for C++. It was initially a spinoff from a research project at Wordtracker and created by Marcus Baker. Significant additions by Matt Hargett and continuous nurturing by Thomas Nilefalk has made Cgreen what it is today.

Test driven development (TDD) really catched on when the JUnit framework for Java spread to other langauges, giving us a family of xUnit tools. Cgreen was born in this wave and have many similarities to the xUnit family.

But TDD evolved over time and modern thinking and practice is more along the lines of BDD, an acronym for Behaviour Driven Development, made popular by people like Dan North and frameworks like JBehave, RSpec, Cucumber and Jasmine.

Cgreen follows this trend and has evolved to embrace a BDD-flavoured style of testing. Although the fundamental mechanisms in TDD and 'technical' BDD are much the same, the shift in focus by changing wording from 'tests' to 'behaviour specifications' is very significant.

This document will present Cgreen using the more modern and better BDD-inspired style. In a later section you can have a peek at the classic xUnit-family TDD API, but you should consider that as outdated.

There are two ways to install Cgreen in your system.

For a more realistic example we need something to test. We’ll pretend that we are writing a function to split the words of a sentence in place. It would do this by replacing any spaces with string terminators and returns the number of conversions plus one. Here is an example of what we have in mind…​

The variable sentence should now point at "Just\0the\0first\0test". Not an obviously useful function, but we’ll be using it for something more practical later.

This time around we’ll add a little more structure to our tests. Rather than having the test as a stand alone program, we’ll separate the runner from the test cases. That way, multiple test suites of test cases can be included in the main() runner file. This makes it less work to add more tests later.

Here is the, so far empty, test case in words_test.c…​

Here is the all_tests.c test runner…​

Cgreen has two ways of running tests. The default is to run all tests in their own protected processes. This is what happens if you invoke run_test_suite(). All tests are then completely independent since they run in separate processes, preventing a single run-away test from bringing the whole program down with it. It also ensures that one test cannot leave any state to the next, thus forcing you to setup the prerequisites for each test correctly and clearly.

Building this scaffolding…​

…​and executing the result gives the familiar…​

Note that we get an extra level of output here, we have both main and words_tests. That’s because all_tests.c adds the words test suite to its own (named main since it was created in the function main()). All this scaffolding is pure overhead, but from now on adding tests will be a lot easier.

Here is a first test for split_words() in words_test.c…​

The assert_that() macro takes two parameters, the value to assert and a constraint. The constraints comes in various forms. In this case we use the probably most common, is_equal_to(). With the default TextReporter the message is sent to STDOUT.

To get this to compile we need to create the words.h header file…​

…​and to get the code to link we need a stub function in words.c…​

A full build later…​

…​and we get the more useful response…​

The breadcrumb trail following the "Failure" text is the nesting of the tests. It goes from the test suites, which can be nested in each other, through the test function, and finally to the message from the assertion. In the language of Cgreen, a "failure" is a mismatched assertion, or constraint, and an "exception" occurs when a test fails to complete for any reason, e.g. a segmentation fault.

We could get this to pass just by returning the value 4. Doing TDD in really small steps, you would actually do this, but we’re not teaching TDD here. Instead we’ll go straight to the core of the implementation…​

Running it gives…​

There is actually a hidden problem here, but our tests still passed so we’ll pretend we didn’t notice.

So it’s time to add another test. We want to confirm that the string is broken into separate words…​

Sure enough, we get a failure…​

Not surprising given that we haven’t written the code yet.

The fix…​

…​reveals our previous hack…​

Our earlier test now fails, because we have affected the strlen() call in our loop. Moving the length calculation out of the loop…​

…​restores order…​

It’s nice to keep the code under control while we are actually writing it, rather than debugging later when things are more complicated.

That was pretty straight forward. Let’s do something more interesting.

The next example is a more realistic extension of our previous attempts. As in real life we first implement something basic and then we go for the functionality that we need. In this case a function that invokes a callback for each word found in a sentence. Something like…​

Here the memo pointer is just some accumulated data that the act_on_word() callback might work with. Other people will write the act_on_word() function and probably many other functions like it. The callback is actually a flex point, and not of interest right now.

The function under test is the words() function and we want to make sure it walks the sentence correctly, dispatching individual words as it goes. So what calls are made are very important. How do we go about to test this?

Let’s start with a one word sentence. In this case we would expect the callback to be invoked once with the only word, right? Here is the test for that…​

What is the funny looking mock() function?

A mock is basically a programmable object. In C objects are limited to functions, so this is a mock function. The macro mock() compares the incoming parameters with any expected values and dispatches messages to the test suite if there is a mismatch. It also returns any values that have been preprogrammed in the test.

The test is invokes_callback_once_for_single_word_sentence(). It programs the mock function using the expect() macro. It expects a single call, and that single call should use the parameters "Word" and NULL. If they don’t match, we will get a test failure.

So when the code under test (our words() function) calls the injected mocked_callback() it in turn will call mock() with the actual parameters.

Of course, we don’t add the mock callback to the test suite, it’s not a test.

For a successful compile and link, the words.h file must now look like…​

…​and the words.c file should have the stub…​

This gives us the expected failing test…​

Cgreen reports that the callback was never invoked. We can easily get the test to pass by filling out the implementation with…​

That is, we just invoke it once with the whole string. This is a temporary measure to get us moving. For now everything should pass, although it doesn’t drive much functionality yet.

That was all pretty conventional, but let’s tackle the trickier case of actually splitting the sentence. Here is the test function we will add to words_test.c…​

Each call is expected in sequence. Any failures, or left-over or extra calls, and we get failures. We can see all this when we run the tests…​

The first failure tells the story. Our little words() function called the mock callback with the entire sentence. This makes sense, because that was the hack we did to get to the next test.

Although not relevant to this guide, I cannot resist getting these tests to pass. Besides, we get to use the function we created earlier…​

And with some work we are rewarded with…​

More work than I like to admit as it took me three goes to get this right. I firstly forgot the + 1 added on to strlen(), then forgot to swap sentence for word in the (*callback)() call, and finally third time lucky. Of course running the tests each time made these mistakes very obvious. It’s taken me far longer to write these paragraphs than it has to write the code.

Cgreen tests are like C, or C++, functions with no parameters and no return value. To signal that they actually are tests we mark them with the Ensure macro. Here’s an example…​

The Ensure macro takes two arguments (in the BDD style) where the first is the System Under Test (SUT) which must be declared with the Describe macro.

The second argument is the test name and can be anything you want as long as it fullfills the rules for an identifier in C and C++. A typical way to choose the named of the tests is what we see here, reading the declaration of the test makes sense since it is almost plain english, "Ensure strlen returns five for 'hello'". No problem understanding what we aim to test, or in TDD lingo, test drive. And it can be viewed as an example from a description of what strlen should be able to do. In a way, extracting all the Ensure:s from your test might give you all the documentation you’ll need.

The call to assert_that() is the primary part of an assertion, which is complemented with a constraint, in this case is_equal_to(), as a parameter. This makes a very fluent interface to the asserts, that actually reads like English. The general format is then

Assertions send messages to Cgreen, which in turn outputs the results.

Here are the standard constraints…​

The boolean assertion macros accept an int value. The equality assertions accept anything that can be cast to intptr_t and simply perform an == operation. The string comparisons are slightly different in that they use the <string.h> library function strcmp(). If you use is_equal_to() with char * pointers then it is the value of the pointers themselves that has to be the same, i.e. the pointers have to point at the same string for the test to pass.

The constraints above should be used as the second argument to one of the assertion functions:

When you use CGreen with C++ there is one extra assertion available:

So far we have encouraged the modern BDD style. It has merits that we really want you to benefit from. But you might come across Cgreen test in another style, more like the standard TDD style, which is more inline with previous thinking and might be more similar to other frameworks.

The only difference, in principle, is the use of the SUT or 'context'. In the BDD style you have it, in the TDD style you don’t.

If you use the older pure-TDD style you skip the Describe macro, the BeforeEach and AfterEach functions. You don’t need a SUT in the Ensure() macro or when you add the test to the suite.

The tests are only run by running a test suite in some form. (But see also Using the Runner.) We can create and run one especially for this test like so…​

In case you have spotted that the reference to returns_five_for_hello should have an ampersand in front of it, add_test_with_context() is actually a macro. The & is added automatically. Further more, the Ensure()-macro actually mangles the tests name, so it is not actually a function name. (This might also make them a bit difficult to find in the debugger…​.)

To run the test suite, we call run_test_suite() on it. So we can just write…​

The results of assertions are ultimately delivered as passes and failures to a collection of callbacks defined in a TestReporter structure. There is a predefined TestReporter in Cgreen called the TextReporter that delivers messages in plain text like we have already seen.

The return value of run_test_suite() is a standard C library/Unix exit code that can be returned directly by the main() function.

The complete test code now looks like…​

Compiling and running gives…​

We can see that the outer test suite is called our_tests since it was in our_tests() we created the test suite. There are no messages shown unless there are failures. So, let’s break our test to see it…​

…​we’ll get the helpful message…​

Cgreen starts every message with the location of the test failure so that the usual error message identifying tools (like Emacs’s next-error) will work out of the box.

Once we have a basic test scaffold up, it’s pretty easy to add more tests. Adding a test of strlen() with an empty string for example…​

And so on.

It’s common for test suites to have a lot of duplicate code, especially when setting up similar tests. Take this database code for example…​

We have already factored out the duplicate code into its own functions create_schema() and drop_schema(), so things are not so bad. At least not yet. But what happens when we get dozens of tests? For a test subject as complicated as a database ActiveRecord, having dozens of tests is very likely.

We can get Cgreen to do some of the work for us by calling these methods before and after each test in the test suite.

Here is the new version…​

With this new arrangement Cgreen runs the create_schema() function before each test, and the drop_schema() function after each test. This saves some repetitive typing and reduces the chance of accidents. It also makes the tests more focused.

The reason we try so hard to strip everything out of the test functions is the fact that the test suite acts as documentation. In our person.h example we can easily see that Person has some kind of name property, and that this value must be unique. For the tests to act like a readable specification we have to remove as much mechanical clutter as we can.

In this particular case there are more lines that we could move from the tests to BeforeEach():

Of course that would require an extra variable, and it might make the tests less clear. And as we add more tests, it might turn out to not be common to all tests. This is a typical judgement call that you often get to make with BeforeEach() and AfterEach().

A couple of details. There is only one BeforeEach() and one AfterEach() allowed in each TestSuite. Also, the AfterEach() function might not be run if the test crashes, causing some test interference. This brings us nicely onto the next section…​

Consider this test method…​

Crashes are not something you would normally want to have in a test run. Not least because it will stop you receiving the very test output you need to tackle the problem.

To prevent segmentation faults and other problems bringing down the test suites, Cgreen runs every test in its own process.

Just before calling the BeforeEach() (or setup) function, Cgreen fork():s. The main process waits for the test to complete normally or die. This includes calling the AfterEach()(or teardown) function, if any. If the test process dies, an exception is reported and the main test process carries on with the next test.

For example…​

When built and run, this gives…​

The normal thing to do in this situation is to fire up the debugger. Unfortunately, the constant fork():ing of Cgreen can be one extra complication too many when debugging. It’s enough of a problem to find the bug.

To get around this, and also to allow the running of one test at a time, Cgreen has the run_single_test() function. The signatures of the two run methods are…​

The extra parameter of run_single_test(), the test string, is the name of the test to select. This could be any test, even in nested test suites (see below). Here is how we would use it to debug our crashing test…​

When run in this way, Cgreen will not fork(). But see the section on Debugging Cgreen tests.

The following is a typical session:

Which shows exactly where the problem is.

This deals with the case where your code throws an exception like segmentation fault, but what about a process that fails to complete by getting stuck in a loop?

Well, Cgreen will wait forever too. But, using the C signal handlers, we can place a time limit on the process by sending it an interrupt. To save us writing this ourselves, Cgreen includes the die_in() function to help us out.

Here is an example of time limiting a test…​

When executed, the code will slow for a second, and then finish with…​

Note that you see the test results as they come in. Cgreen streams the results as they happen, making it easier to figure out where the test suite has problems.

Of course, if you want to set a general time limit on all your tests, then you can add a die_in() to a BeforeEach() (or setup()) function. Cgreen will then apply the limit to each of the tests in that context, of course.

Another possibility is the use of an environment variable named CGREEN_TIMEOUT_PER_TEST which, if set to a number will apply that timeout to every test run. This will apply to all tests in the same run.

Cgreen protects itself from being torn down by an exception in a test by fork()-ing each test into a separate process. A catastrophic error will then only affect the child process for that specific test and Cgreen can catch it, rather than crashing too. It can then report the exception and continue with the next test.

The TestSuite is a composite structure. This means test suites can be added to test suites, building a tree structure that will be executed in order.

Let’s combine the strlen() tests with the Person tests above. Firstly we need to remove the main() functions. E.g…​

Then we can write a small runner with a new main() function…​

It’s usually easier to place the TestSuite prototypes directly in the runner source, rather than have lot’s of header files. This is the same reasoning that let us drop the prototypes for the test functions in the actual test scripts. We can get away with this, because the tests are more about documentation than encapsulation.

As we saw above, we can run a single test using the run_single_test() function, and we’d like to be able to do that from the command line. So we added a simple if block to take the test name as an optional argument. The entire test suite will be searched for the named test. This trick also saves us a recompile when we debug.

When you use the BDD notation you can only have a single test subject (which is actually equivalent of a suite) in a single file because you can only have one Describe() macro in each file. But using this strategy you can create composite suites that takes all your tests and run them in one go.

What happens to setup and teardown functions in a TestSuite that contains other TestSuite:s?

Well firstly, Cgreen does not fork() when running a suite. It leaves it up to the child suite to fork() the individual tests. This means that a setup and teardown will run in the main process. They will be run once for each child suite.

We can use this to speed up our Person tests above. Remember we were creating a new connection and closing it again in the fixtures. This means opening and closing a lot of connections. At the slight risk of some test interference, we could reuse the connection accross tests…​

The trick here is creating a test suite as a wrapper whose sole purpose is to wrap the main test suite in the fixture. This is our 'fixture' pointer. This code is a little confusing, because we have two sets of fixtures in the same test script.

We have the MySQL connection fixture. This will run open_connection() and close_connection() just once at the beginning and end of the person tests. This is because the suite pointer is the only member of fixture.

We also have the schema fixture, the create_schema() and drop_schema(), which is run before and after every test. Those are still attached to the inner suite.

In the real world we would probably place the connection fixture in its own file…​

This allows the reuse of common fixtures across projects.

How would we test the following code…​?

This is a fairly generic stream filter that turns the incoming characters into C string paragraphs. Each call creates one paragraph, returning a pointer to it or returning NULL if there is no paragraph. The paragraph has memory allocated to it and the stream is advanced ready for the next call. That’s quite a bit of functionality, and there are plenty of nasty boundary conditions. I really want this code tested before I deploy it.

The problem is the stream dependency. We could use a real stream, but that will cause all sorts of headaches. It makes the test of our paragraph formatter dependent on a working stream. It means we have to write the stream first, bottom up coding rather than top down. It means we will have to simulate stream failures - not easy. It will also mean setting up external resources. This is more work, will run slower, and could lead to spurious test failures.

By contrast, we could write a simulation of the stream for each test, called a "server stub".

For example, when the stream is empty nothing should happen. We hopefully get NULL from read_paragraph when the stream is exhausted. That is, it just returns a steady stream of `EOF`s.

Fortunately, this function takes the stream as a parameter. This is called dependency injection and is a very important concept. Thanks to this we can write a small function, a stub, with the same signature, that simulates a real stream, and inject that instead of a real stream, which the production code probably does.

Our simulation is easy here, because our fake stream returns only one value. Things are harder when the function result changes from call to call as a real stream would. Simulating this would mean messing around with static variables and counters that are reset for each test. And of course, we will be writing quite a few stubs. Often a different one for each test. That’s a lot of clutter.

Cgreen can handle this clutter for us by letting us write a single programmable function for all our tests.

We can redo our example by creating a stream_stub() function. We can call it anything we want, and since I thought we wanted to have a stubbed stream…​

Hardly longer that our trivial server stub above, it is just a macro to generate a return value, but we can reuse this in test after test. Let’s see how.

For our simple example above we just tell it to always return EOF…​

Let’s see if our production code actually works…​

So far, so good. On to the next test.

If we want to test a one character line, we have to send the terminating EOF or "\n" as well as the single character. Otherwise our code will loop forever, giving an infinite line of that character.

Here is how we can do this…​

Unlike the always_expect() instruction, expect() sets up an expectation of a single call and specifying will_return() sets the single return value for just that call. It acts like a record and playback model. Successive expectations map out the return sequence that will be given back once the test proper starts.

We’ll add this test to the suite and run it…​

Oops. Our code under test doesn’t work. Already we need a fix…​

around a bit everything is fine:

So, how do the Cgreen stubs work? Each expect() describes one call to the stub and when a call to will_return() is included, the return values will be collected and returned in order as the expected calls arrive.

The mock() macro captures the parameter names, their values and the func property (the name of the stub function). Cgreen can then use these to look up entries in the return list, and also to generate more helpful messages.

We can now crank out our tests quite quickly…​

I’ve been a bit naughty. As each test runs in its own process, I haven’t bothered to free the pointers to the paragraphs. I’ve just let the operating system do it. Purists may want to add the extra clean up code.

I’ve also used always_expect() for the last instruction. Without this, if the stub is given an instruction it does not expect, it will throw a test failure. This is overly restrictive, as our read_paragraph() function could quite legitimately call the stream after it had run off of the end. OK, that would be odd behaviour, but that’s not what we are testing here. If we were, it would be placed in a test of its own. The always_expect() call tells Cgreen to keep going after the first three letters, allowing extra calls.

As we build more and more tests, they start to look like a specification of the wanted behaviour…​

…​and just for luck…​

This time we must not use always_return(). We want to leave the stream where it is, ready for the next call to read_paragraph(). If we call the stream beyond the line ending, we want to fail.

Oops, that was a little too fast. Turns out we are failing anyway…​

Clearly we are passing through the line ending. Another fix later…​

And we are passing again…​

There are no limits to the number of stubbed methods within a test, only that two stubs cannot have the same name. The following will cause problems…​

You could program the same stub to return values for the two streams, but that would make a very brittle test. Since we’d be making it heavily dependent on the exact internal behaviour that we are trying to test, or test drive, it will break as soon as we change that implementation. The test will also become very much harder to read and understand. And we really don’t want that.

So, it will be necessary to have two stubs to make this test behave, but that’s not a problem…​

We now have a way of writing fast, clear tests with no external dependencies. The information flow is still one way though, from stub to the code under test. When our code calls complex procedures, we won’t want to pick apart the effects to infer what happened. That’s too much like detective work. And why should we? We just want to know that we dispatched the correct information down the line.

Things get more interesting when we think of the traffic going the other way, from code to stub. This gets us into the same territory as mock objects.

To swap the traffic flow, we’ll look at an outgoing example instead. Here is the prewritten production code…​

This is the start of a formatter utility. Later filters will probably break the paragaphs up into justified text, but right now that is all abstracted behind the void write(void *, char *) interface. Our current interests are: does it loop through the paragraphs, and does it crash?

We could test correct paragraph formation by writing a stub that collects the paragraphs into a struct. We could then pick apart that struct and test each piece with assertions. This approach is extremely clumsy in C. The language is just not suited to building and tearing down complex edifices, never mind navigating them with assertions. We would badly clutter our tests.

Instead we’ll test the output as soon as possible, right in the called function…​

By placing the assertions into the mocked function, we keep the tests minimal. The catch with this method is that we are back to writing individual functions for each test. We have the same problem as we had with hand coded stubs.

Again, Cgreen has a way to automate this. Here is the rewritten test…​

Where are the assertions?

Unlike our earlier stub, reader() can now check its parameters. In object oriented circles, an object that checks its parameters as well as simulating behaviour is called a mock object. By analogy reader() is a mock function, or mock callback.

Using the expect() macro, we have set up the expectation that writer() will be called just once. That call must have the string "a" for the paragraph parameter. If the actual value of that parameter does not match, the mock function will issue a failure straight to the test suite. This is what saves us writing a lot of assertions.

It’s about time we actually ran our test…​

Confident that a single character works, we can further specify the behaviour. Firstly an input sequence…​

A more intelligent programmer than me would place all these calls in a loop.

Next, checking an output sequence…​

Again we can se that the expect() calls follow a record and playback model. Each one tests a successive call. This sequence confirms that we get "a", "b" and "c" in order.

So, why the 5 passes? Each expect() with a constrait is actually an assert. It asserts that the call specified is actually made with the parameters given and in the specified order. In this case all the expected calls were made.

Then we’ll make sure the correct stream pointers are passed to the correct functions. This is a more realistic parameter check…​

And finally we’ll specify that the writer is not called if there is no paragraph.

This last test is our undoing…​

Obviously blank lines are still being dispatched to the writer(). Once this is pointed out, the fix is obvious…​

Tests with never_expect() can be very effective at uncovering subtle bugs.

All done.

Using mocks is a very handy way to isolate a unit by catching and controlling calls to external units. Depending on your style of coding two schools of thinking have emerged. And of course Cgreen supports both!

Using expect() and mock() is a very powerful way to isolate your code under test from its dependencies. But it is not always easy to follow what happens, and when.

Here are some important things to remember when working with Cgreen mocks.

After a while you are bound to get tests with calls to expect(). You might even have common patterns in multiple tests. So your urge to refactor starts to set in. And that is good, go with it, we have tests to rely on.

But there are a lot of things going on behind the scenes when you use Cgreen, often with the help of some serious macro-magic, so special care needs to be taken when refactoring tests that have expect() in them.

When specifying behavior of mocks there are three parts. First, how often the specified behaviour or expectation will be executed:

You can specify constraints and behaviours for each expectation (except for never_expect() naturally). A constraint places restrictions on the parameters (and will tell you if the expected restriction was not met), and a behaviour specifies what the mock should do if the parameter constraints are met.

A parameter constraint is defined using the when(parameter, constraint) macro. It takes two parameters:

There is a multitude of constraints available (actually, exactly the same as for the assertions we saw earlier):

For the double valued constraints you can set the number of significant digits to consider a match with a call to significant_figures_for_assert_double_are(int figures). The section on how to work with doubles has a more detailed discussion of the algorithm used for comparing floating point numbers.

Then there are a couple of ways to return results from the mocks. They all provide ways to return various types of values through mock(). In your mocked function you can then simply return that value, or manipulate it as necessary.

You can combine the expectations for a mock() in various ways:

If multiple when() are specified they all need to be fullfilled. You can of course only have one for each of the parameters of your mock function.

You can also have multiple will_set_contents_of_output_parameter() in an expectation, one for each reference parameter, but naturally only one will_return().

To ensure that a specific call happens n times the macro times(number_times_called) can be passed as a constraint to a specific call:

This feature only works for expect().

When you have multiple constraints in an expect the order in which they are executed is not always exactly then order in which they where given.

First all constraints are inspected for validity, such as if the parameter name given cannot be found, but primarily to see if the parameters, if any, matche the actual parameters in the call.

Then all read-only constraints are processed, followed by constraints that set contents.

Finally all side effect constraints are executed.

The expections still need to respect the order of calling, so if we call the function mocker_file_writer with the following pattern:

The expectation code should look like the following

TBD. Hint: this involves using will_set_contents_of_output_parameter().

If the function we are mocking returns structs by value, then our mock function need to do that too. To do this we must use specific return macro, will_return_by_value(). Below is some example code using an imaginary struct typedef’ed as Struct and a corresponding function, retrieve_struct(), which we want to mock.

The underlying mechanism of this is that in the test we create the struct that we want to return. The macro will_return_by_value() then copies that to a dynamically allocated area, saving it so that a pointer to that area can be returned by mock().

The mock function will then look like this:

This would cause a memory leak since the area allocated by the return_by_value() macro is not deallocated. And in many scenarious this might not be a big problem, and you could make do with that simple version.

We might want to be sure, e.g. if we want to use valgrind on when unittesting to catch leaks early. Then we don’t want our unittests to pollute the actual leakage analysis.

In that case the mock function needs to free up the area that was automatically allocated by will_return_by_value(). The pointer returned by mock() will point to that area. So, here’s a better, although slightly more complicated, version:

Modern C standards allows function parameters to be structs by value. Since our mock() only can handle scalar values this presents a bit of a conundrum.

And we also can not compare a non-scalar value with any of the is_equal_to…​() constraint macros in the expect() call. Also remember that the C language does not allow comparing non-scalar values using ==.

There are a couple of ways to handle this and which one to select depends on what you want to do.

TBD. Hint: this involves using will_capture_parameter().

We are not talking about test doubles here, but about values of C/C++ double type (a.k.a. double float.)

Cgreen is designed to make it easy and natural to write assertions and expectations. Many functions can be used for multiple data types, e.g. is_equal_to() applies to all integer type values, actually including pointers.

But the C language has its quirks. One of them is the fact that it is impossible to inspect the datatypes of values during run-time. This has e.g. forced the introduction of is_equal_to_string() to enable string comparisons.

The examples in this guide uses the C langauge to shows how to use CGreen. You can also use CGreen with C++.

All you have to do is

There is also one extra feature when you use C++, the assert_throws function.

Since BDD talks about behaviour, there has to be something that we can talk about as having that wanted behaviour. This is usually called the SUT, the System Under Test. The "system" might be whatever we are testing, such as a C module ("MUT"), class ("CUT"), object ("OUT"), function ("FUT") or method ("MUT"). We will stick with SUT in this document. To use Cgreen in BDD-ish mode you must define a name for it.

Cgreen supports C++ and there you naturally have the objects and also the Class Under Test. But in plain C you will have to think about what is actually the "class" under test. E.g. in sort_test.c you might see

In this example you can clearly see what difference the BDD-ish style makes when it comes to naming. Convention, and natural language, dictates that typical names for what TDD would call tests, now starts with 'can' or 'finds' or other verbs, which makes the specification so much easier to read.

Yes, I wrote 'specification'. Because that is how BDD views what TDD basically calls a test suite. The suite specifies the behaviour of a `class´. (That’s why some BDD frameworks draw on 'spec', like RSpec.)

The complete specification of the behaviour of a SUT might become long and require various forms of setup. When using TDD style you would probably break this up into multiple suites having their own setup() and teardown().

With BDD-ish style we could consider a suite as a behaviour specification for our SUT 'in a particular context'. E.g.

The 'context' would then be shopping_basket_for_returning_customer, with the SUT being the shopping basket 'class'.

So 'context', 'system under test' and 'suite' are mostly interchangable concepts in Cgreen lingo. It’s a named group of 'tests' that share the same BeforeEach and AfterEach and lives in the same source file.

When we write a new test we focus on the details about the test we are trying to write. And writing tests is no trivial matter so this might well take a lot of brain power.

So, it comes as no big surprise, that sometimes you write your test and then forget to add it to the suite. When we run it it appears that it passed on the first try! Although this should really make you suspicious, sometimes you get so happy that you just continue with churning out more tests and more code. It’s not until some (possibly looong) time later that you realize, after much headache and debugging, that the test did not actually pass. It was never even run!

There are practices to minimize the risk of this happening, such as always running the test as soon as you can set up the test. This way you will see it fail before trying to get it to pass.

But it is still a practice, something we, as humans, might fail to do at some point. Usually this happens when we are most stressed and in need of certainty.

Cgreen gives you a tool to avoid not only the risk of this happening, but also the extra work and extra code. It is called the cgreen-runner.

The cgreen-runner should come with your Cgreen installation if your platform supports the technique that is required, which is 'programatic access to dynamic loading of libraries'. This means that a program can load an external library of code into memory and inspect it. Kind of self-inspection, or reflexion.

So all you have to do is to build a dynamically loadable library of all tests (and of course your objects under test and other necessary code). Then you can run the cgreen-runner and point it to the library. The runner will then load the library, enumerate all tests in it, and run every test.

It’s automatic, and there is nothing to forget.

Assuming your tests are in first_test.c the typical command to build your library using gcc would be

The -fPIC means to generate 'position independent code' which is required if you want to load the library dynamically. To explicitly state this is required on many platforms.

How to build a dynamically loadable shared library might vary a lot depending on your platform. Can’t really help you there, sorry!

As soon as we have linked it we can run the tests using the cgreen-runner by just giving it the shared, dynamically loadable, object library as an argument:

More or less exactly the same output as when we ran our first test in the beginning of this quickstart tutorial. We can see that the top level of the tests will be named as the library it was discovered in, and the second level is the context for our System Under Test, in this case 'Cgreen'. We also see that the context is mentioned in the failure message, giving a fairly obvious Cgreen → fails_this_test.

Now we can actually delete the main function in our source code. We don’t need all this, since the runner will discover all tests automatically.

It always feel good to delete code, right?

We can also select which test to run:

We recommend the BDD notation to discover tests, and you indicate which context the test we want to run is in. In this example it is Cgreen so the test should be refered to as Cgreen:this_test_should_fail.

If you don’t use the BDD notation there is actually a context anyway, it is called default.

Once you get the build set up right for the cgreen-runner everything is fairly straight-forward. But you have a few options:

The verbose option is particularly handy since it will give you the actual names of all tests discovered. So if you have long test names you can avoid mistyping them by copying and pasting from the output of cgreen-runner --verbose. It will also give the mangled name of the test which should make it easier to find in the debugger. Here’s an example:

You can name a single test to be run by giving it as the last argument on the command line. The name should be in the format <SUT>:<test>. If not obvious you can get that name by using the --verbose command option which will show you all tests discovered and both there C/C++ and Cgreen names. Copying the Cgreen name from that output is an easy way to run only that particular test. When a single test is named it is run using run_single_test(). As described in Five Minutes Doing TDD with Cgreen this means that it is not protected by fork()-ing it to run in its own process.

The cgreen-runner supports selecting tests with limited pattern matching. Using an asterisk as a simple 'match many' symbol you can say things like

You can run tests in multiple libraries in one go by adding them to the cgreen-runner command:

The cgreen-runner will only run setup and teardown functions if you use the BDD-ish style with BeforeEach() and AfterEach() as described above. The runner does not pickup setup() and teardown() added to suites, because it actually doesn’t run suites. It discovers all tests and runs them one by one. The macros required by the BDD-ish style ensures that the corresponding BeforeEach() and AfterEach() are run before and after each test.

In case you have non-BDD style tests without any setup() and/or teardown() you can still use the runner. The default suite/context where the tests live in this case is called default. But why don’t you convert your tests to BDD notation? This removes the risk of frustrating trouble-shooting when you added setup() and teardown() and can’t understand why they are not run…​

So, the runner encourages you to use the BDD notation. But since we recommend that you do anyway, that’s no extra problem if you are starting out from scratch. But see Changing Style for some easy tips on how to get you there if you already have non-BDD tests.

You can choose between the TextReporter, which we have been seeing so far, and the built-in JUnit/Ant compatible XML-reporter using the --xml option. But it is not currently possible to use custom reporters as outlined in Changing Cgreen Reporting with the runner.

If you require another custom reporter you need to resort to the standard, programatic, way of invoking your tests. For now…​

Sometimes you find that you need to temporarily remove a test, perhaps to do a refactoring when you have a failing test. Ignoring that test will allow you to do the refactoring while still in the green.

An old practice is then to comment it out. That is a slightly cumbersome. It is also hazardous habit as there is no indication of a missing test if you forget to uncomment it when you are done.

Cgreen offers a much better solution. You can just add an 'x' infront of the Ensure for the test and that test will be skipped.

With this method, it is a one character change to temporarily ignore, and un-ignore, a test. It is also easily found using text searches through a complete source tree. Cgreen will also tally the skipped tests, so it is clearly visible that you have some skipped test when you run them.

In every test suite so far, we have run the tests with this line…​

We can change the reporting mechanism just by changing this call to create another reporter.

Cgreen has the following built-in reporters that you can choose from when your code runs the test suite.

If you write a runner function like in most examples above, you can just substitute which runner to create. If you use the cgreen-runner (Automatic Test Discovery) to dynamically find all your tests you can force it to use the XML-reporter with the -x <prefix> option.

Although Cgreen has a number of options, there are times when you’d like a different output from the reporter, the CUTE and CDash reporters are examples that grew out of such a need.

Perhaps your Continuous Integration server want the result in a different format, or you just don’t like the text reporter…​

Writing your own reporter is supported. And we’ll go through how that can be done using an XML-reporter as an example.

Here is the code for create_text_reporter()…​

The TestReporter structure contains function pointers that control the reporting. When called from create_reporter() constructor, these pointers are set up with functions that display nothing. The text reporter code replaces these with something more dramatic, and then returns a pointer to this new object. Thus the create_text_reporter() function effectively extends the object from create_reporter().

The text reporter only outputs content at the start of the first test, at the end of the test run to display the results, when a failure occurs, and when a test fails to complete. A quick look at the text_reporter.c file in Cgreen reveals that the overrides just output a message and chain to the versions in reporter.h.

To change the reporting mechanism ourselves, we just have to know a little about the methods in the TestReporter structure.

The Cgreen TestReporter is a pseudo class that looks something like…​

The first block are the methods that can be overridden:

The second block is simply resources and book keeping that the reporter can use to liven up the messages…​

The breadcrumb pointer is different and needs a little explanation. Basically it is a stack, analogous to the breadcrumb trail you see on websites. Everytime a start() handler is invoked, the name is placed in this stack. When a finish() message handler is invoked, a name is popped off.

There are a bunch of utility functions in cgreen/breadcrumb.h that can read the state of this stack. Most useful are get_current_from_breadcrumb() which takes the breadcrumb pointer and returns the current test name, and get_breadcrumb_depth() which gives the current depth of the stack. A depth of zero means that the test run has finished.

If you need to traverse all the names in the breadcrumb, then you can call walk_breadcrumb(). Here is the full signature…​

The void (*walker)(const char *, void *) is a callback that will be passed the name of the test suite for each level of nesting.

It is also passed the memo pointer that was passed to the walk_breadcrumb() call. You can use this pointer for anything you want, as all Cgreen does is pass it from call to call. This is so aggregate information can be kept track of whilst still being reentrant.

The last parts of the TestReporter structure are…​

Let’s make things real with an example. Suppose we want to send the output from Cgreen in XML format, say for storing in a repository or for sending across the network.

Suppose also that we have come up with the following format…​

In other words a simple nesting of tests with only failures encoded. The absence of "fail" XML node is a pass.

Here is a test script, test_as_xml.c that we can use to construct the above output…​

We can’t use the auto-discovering cgreen-runner (see Automatic Test Discovery) here since we need to ensure that the nested suites are reported as a nested xml structure. And we’re not actually writing real tests, just something that we can use to drive our new reporter.

The text reporter is used just to confirm that everything is working. So far it is.

Our first move is to switch the reporter from text, to our not yet written XML version…​

We’ll start the ball rolling with the xml_reporter.h header file…​

…​and the simplest possible reporter in xml_reporter.c.

One that outputs nothing.

Yep, nothing.

Let’s add the outer XML tags first, so that we can see Cgreen navigating the test suite…​

Although chaining to the underlying reporter_start_*() and reporter_finish_*() functions is optional, I want to make use of some of the facilities later.

Our output meanwhile, is making its first tentative steps…​

We don’t require an XML node for passing tests, so the show_fail() function is all we need…​

We have to use vprintf() to handle the variable argument list passed to us. This will probably mean including the stdarg.h header as well as stdio.h.

This gets us pretty close to what we want…​

For completeness we should add a tag for a test that doesn’t complete. We’ll output this as a failure, although we don’t bother with the location this time…​

All that’s left then is the XML declaration and the thorny issue of indenting. Although the indenting is not strictly necessary, it would make the output a lot more readable.

Given that the test depth is kept track of for us with the breadcrumb object in the TestReporter structure, indentation will actually be quite simple. We’ll add an indent() function that outputs the correct number of tabs…​

The get_breadcrumb_depth() function just gives the current test depth as recorded in the reporters breadcrumb (from cgreen/breadcrumb.h). As that is just the number of tabs to output, the implementation is trivial.

We can then use this function in the rest of the code. Here is the complete listing…​

And finally the desired output…​

Job done.

Possible other reporter customizations include reporters that write to syslog, talk to IDE plug-ins, paint pretty printed documents or just return a boolean for monitoring purposes.

Sometimes the built-in constraints that Cgreen provide are not sufficient. With Cgreen it is possible to create custom constraints, although you will be depending on some internal structures if you do so.

Here’s how to implement a simple example custom constraint that asserts that the value is bigger than 5. We’ll implement this using a static constraint since it does not take any parameter.

First we need the actual compare function:

And then the static constraint structure, for which we’ll need some of Cgreen's internal functions:

This implementation can use a statically declared Constraint structure that is prefilled since it does not need to store the value to be checked. This static custom constraint can then be used directly in the assert like this:

To create a custom constraint that takes an input parameter, we need to add a function that creates a constraint structure that correctly saves the value to be checked, and, for convenience, a macro. This time we need to dig into how Cgreen stores expected values and we’ll also make use of Cgreen's utility function string_dup().

This gives a custom constraint that can be used in the assert in the same way as Cgreen's built-in constraints:

The last, and definitely more complex, example is a constraint that takes two structures and compares fields in them. The constraint will, given a structure representing a piece and another structure representing a box, check if the piece can fit inside the box using a size field.

Assuming two "application" structures with size fields:

We want to be able to write a test like this:

To implement the can_fit_in_box constraint we first need a comparer function:

And this time we can’t rely on Cgreen's checker and message generating function test_want() which we used in the previous examples. So we also need a custom function that calls the comparison and formats a possible error message:

Finally we’ll use both of those in the constraint creating function and add the convenience macro:

Are you starting out with Cgreen on a largish legacy system? And there are loads and loads of functions to mock to get a unit under test?

You could try the cgreen-mocker that is supplied as a contributed part of the Cgreen source distribution.

It is a Python program that parses C language header files and tries to create a corresponding .mock file where each function declaration is replaced with a call to mock().

So given a header file containing lines like

cgreen-mocker will, given that there are no errors, print something like this on the screen:

Of course, you would pipe this output to a file.

To use cgreen-mocker you need Python, and the following packages:

These can easily be installed with:

Sometimes you might get cryptic and strange error messages from the compiler. Since Cgreen uses some C/C++ macro magic this can happen and the error messages might not be straight forward to interpret.

Here are some examples, but the exact messages differ between compilers and versions.

Cgreen attempts to handle primitive type comparisons with a single constraint, is_equal_to(). This means that it must store the actual and expected values in a form that will accomodate all possible values that primitive types might take, typically an intptr_t.

This might sometimes cause unexpected comparisons since all actual values will be cast to match intptr_t, which is a signed value. E.g.

On a system which considers char to be signed this will cause the following Cgreen assertion error:

This is caused by the C rules forcing an implicit cast of the signed char to intptr_t by sign-extension. This might not be what you expected. The correct solution, by any standard, is to cast the actual value to unsigned char which will then be interpreted correctly. And the test passes.

In order to reveal what really happens you might want to see the actual and expected values in hex. This can easily be done with the is_equal_to_hex().

This might make the mistake easier to spot:

Cgreen is compatible with coverage tools, in particular gcov/lcov. So generating coverage data for your application should be straight forward.

This is what you need to do (using gcc or clang):

Your coverage data will be available in coverage/index.html.

If the output from your Cgreen based tests appear garbled or duplicated, this can be caused by the way Cgreen terminates its test-running child process. In many unix-like environments the termination of a child process should be done with _exit(). However, this interfers severily with the ability to collect coverage data. As this is important for many of us, Cgreen instead terminates its child process with the much cruder exit() (note: no underscore).

Under rare circumstances this might have the unwanted effect of output becoming garbled and/or duplicated.

If this happens you can change that behaviour using an environment variable CGREEN_CHILD_EXIT_WITH__EXIT (note: two underscores). If set, Cgreen will terminate its test-running child process with the more POSIX-compliant _exit(). But as mentioned before, this is, at least at this point in time, incompatible with collecting coverage data.

So, it’s coverage or POSIX-correct child exits and guaranteed output consistency. You can’t have both…​

None.

First official non-beta release.