8. Changing Cgreen Reporting

8.1. Replacing the Reporter

In every test suite so far, we have run the tests with this line…​

return run_test_suite(our_tests(), create_text_reporter());

We can change the reporting mechanism just by changing this call to create another reporter.

8.2. Built-in Reporters

Cgreen has the following built-in reporters that you can choose from when your code runs the test suite.

Reporter Purpose Signature Note

Text

Human readable, with clear messages

create_text_reporter(void)

XML

ANT/Jenkins compatible

create_xml_reporter(const char *file_prefix)

file_prefix is the prefix of the XML files generated.

CUTE

CUTE Eclipse-plugin (http://cute-test.org) compatible output

create_cute_reporter(void)

CDash

CMake (http://cmake.org) dashboard

create_cdash_reporter(CDashInfo *info)

info is a structure defined in cdash_reporter.h

If you write a runner function like in most examples above, you can just substitute which runner to create. If you use the cgreen-runner (Automatic Test Discovery) to dynamically find all your tests you can force it to use the XML-reporter with the -x <prefix> option.

Currently cgreen-runner only supports the test and XML built-in reporters.

8.3. Rolling Our Own

Althoug Cgreen has a number of options, there are times when you’d like a different output from the reporter, the CUTE and CDash reporters are examples that grew out of such a need.

Perhaps your Continuous Integration server want the result in a different format, or you just don’t like the text reporter…​

Writing your own reporter is supported. And we’ll go through how that can be done using an XML-reporter as an example.

Cgreen already has an XML-reporter compatible with ANT/Jenkins, see Built-in Reporters.

Here is the code for create_text_reporter()…​

TestReporter *create_text_reporter(void) {
    TestReporter *reporter = create_reporter();
    if (reporter == NULL) {
        return NULL;
    }
    reporter->start_suite = &text_reporter_start_suite;
    reporter->start_test = &text_reporter_start_test;
    reporter->show_fail = &show_fail;
    reporter->show_skip = &show_skip;
    reporter->show_incomplete = &show_incomplete;
    reporter->finish_test = &text_reporter_finish_test;
    reporter->finish_suite = &text_reporter_finish;
    return reporter;
}

The TestReporter structure contains function pointers that control the reporting. When called from create_reporter() constructor, these pointers are set up with functions that display nothing. The text reporter code replaces these with something more dramatic, and then returns a pointer to this new object. Thus the create_text_reporter() function effectively extends the object from create_reporter().

The text reporter only outputs content at the start of the first test, at the end of the test run to display the results, when a failure occurs, and when a test fails to complete. A quick look at the text_reporter.c file in Cgreen reveals that the overrides just output a message and chain to the versions in reporter.h.

To change the reporting mechanism ourselves, we just have to know a little about the methods in the TestReporter structure.

8.4. The TestReporter Structure

The Cgreen TestReporter is a pseudo class that looks something like…​

typedef struct _TestReporter TestReporter;
struct _TestReporter {
    void (*destroy)(TestReporter *reporter);
    void (*start_suite)(TestReporter *reporter, const char *name, const int count);
    void (*start_test)(TestReporter *reporter, const char *name);
    void (*show_pass)(TestReporter *reporter, const char *file, int line,
                                   const char *message, va_list arguments);
    void (*show_skip)(TestReporter *reporter, const char *file, int line);
    void (*show_fail)(TestReporter *reporter, const char *file, int line,
                                   const char *message, va_list arguments);
    void (*show_incomplete)(TestReporter *reporter, const char *file, int line,
                                   const char *message, va_list arguments);
    void (*assert_true)(TestReporter *reporter, const char *file, int line, int result,
                                   const char * message, ...);
    void (*finish_test)(TestReporter *reporter, const char *file, int line);
    void (*finish_suite)(TestReporter *reporter, const char *file, int line);
    int passes;
    int failures;
    int exceptions;
    void *breadcrumb;
    int ipc;
    void *memo;
    void *options;
};

The first block are the methods that can be overridden:

void (*destroy)(TestReporter *reporter)

This is the destructor for the default structure. If this is overridden, then the overriding function must call destroy_reporter(TestReporter *reporter) to finish the clean up.

void (*start_suite)(TestReporter *reporter, const char *name, const int count)

This is the first of the callbacks. At the start of each test suite Cgreen will call this method on the reporter with the name of the suite being entered and the number of tests in that suite. The default version keeps track of the stack of tests in the breadcrumb pointer of TestReporter. If you make use of the breadcrumb functions, as the defaults do, then you will need to call reporter_start_suite() to keep the book-keeping in sync.

void (*start_test)(TestReporter *reporter, const char *name)

At the start of each test Cgreen will call this method on the reporter with the name of the test being entered. Again, the default version keeps track of the stack of tests in the breadcrumb pointer of TestReporter. If you make use of the breadcrumb functions, as the defaults do, then you will need to call reporter_start_test() to keep the book-keeping in sync.

void (*show_pass)(TestReporter *reporter, const char *file, int line, const char *message, va_list arguments)

This method is initially empty as most reporters see little point in reporting passing tests (but you might do), so there is no need to chain the call to any other function. Besides the pointer to the reporter structure, Cgreen also passes the file name of the test, the line number of failed assertion, the message to show and any additional parameters to substitute into the message. The message comes in as printf() style format string, and so the variable argument list should match the substitutions.

void (*show_fail)(TestReporter *reporter, const char *file, int line, const char *message, va_list arguments)

The partner of show_pass(), and the one you’ll likely overload first.

void (*show_skip)(TestReporter *reporter, const char *file, int line)

This method will be called when a skipped test is encountered, see Skipping Tests.

void (*show_incomplete)(TestReporter *reporter, const char *file, int line, const char *message, va_list arguments)

When a test fails to complete, this is the handler that is called. As it’s an unexpected outcome, no message is received, but we do get the name of the test. The text reporter combines this with the breadcrumb to produce the exception report.

void (*assert_true)(TestReporter *reporter, const char *file, int line, int result, const char * message, …​)

This is not normally overridden and is really internal. It is the raw entry point for the test messages from the test suite. By default it dispatches the call to either show_pass() or show_fail().

void (*finish_test)(TestReporter *reporter, const char *file, int line)

The counterpart to the (*start_test)() call. It is called on leaving the test. It needs to be chained to the reporter_finish() to keep track of the breadcrumb book keeping.

void (*finish_suite)(TestReporter *reporter, const char *file, int line)

The counterpart to the (*start_suite)() call called on leaving the test suite, and similar to the (*finish_test)() if your reporter needs a handle on that event too. The default text reporter chains both this and (*finish_test)() to the same function where it figures out if it is the end of the top level suite. If so, it prints the familiar summary of passes and fails.

The show_fail() and show_pass() functions are called from the child process, i.e. the isolated process that is fork():ed to run a single test case. All others, notably start_…​(), finish_…​(), show_incomplete() and show_skip() are run in the main (parent) process. This fact might be important since the processes do not share memory. Information is passed from the child to the parent using messaging performed within the show_…​() functions.

The second block is simply resources and book keeping that the reporter can use to liven up the messages…​

passes

The number of passes so far.
skips

The number of tests that has been skipped by the xEnsure mechanism (see Skipping Tests)
failures

The number of failures generated so far.
exceptions

The number of test functions that have failed to complete so far.
breadcrumb

This is a pointer to the list of test names in the stack.

The breadcrumb pointer is different and needs a little explanation. Basically it is a stack, analogous to the breadcrumb trail you see on websites. Everytime a start() handler is invoked, the name is placed in this stack. When a finish() message handler is invoked, a name is popped off.

There are a bunch of utility functions in cgreen/breadcrumb.h that can read the state of this stack. Most useful are get_current_from_breadcrumb() which takes the breadcrumb pointer and returns the current test name, and get_breadcrumb_depth() which gives the current depth of the stack. A depth of zero means that the test run has finished.

If you need to traverse all the names in the breadcrumb, then you can call walk_breadcrumb(). Here is the full signature…​

void walk_breadcrumb(Breadcrumb *breadcrumb, void (*walker)(const char *, void *), void *memo);

The void (*walker)(const char *, void *) is a callback that will be passed the name of the test suite for each level of nesting.

It is also passed the memo pointer that was passed to the walk_breadcrumb() call. You can use this pointer for anything you want, as all Cgreen does is pass it from call to call. This is so aggregate information can be kept track of whilst still being reentrant.

The last parts of the TestReporter structure are…​

ipc

This is an internal structure for handling the messaging between reporter and test suite. You shouldn’t touch this.
memo

By contrast, this is a spare pointer for your own expansion.
options

A pointer to a reporter specific structure that can be used to set options. E.g. the textreporter defines the structure TextReporterOptions which can be used by calling code to define the use of colors when printing passes and failures. You set it with set_reporter_options(*void).

8.5. An Example XML Reporter

Let’s make things real with an example. Suppose we want to send the output from Cgreen in XML format, say for storing in a repository or for sending across the network.

The cgreen-runner already has an XML-reporter that you can use if you need to produce Jenkins/ANT compatible XML output. See Cgreen Runner Options.

Suppose also that we have come up with the following format…​

<?xml?>
<suite name="Top Level">
    <suite name="A Group">
        <test name="a_test_that_passes">
        </test>
        <test name="a_test_that_fails">
            <fail>
                <message>A failure</message>
                <location file="test_as_xml.c" line="8"/>
            </fail>
        </test>
    </suite>
</suite>

In other words a simple nesting of tests with only failures encoded. The absence of "fail" XML node is a pass.

Here is a test script, test_as_xml.c that we can use to construct the above output…​

#include <cgreen/cgreen.h>

Describe(XML_reporter);
BeforeEach(XML_reporter) {}
AfterEach(XML_reporter) {}

Ensure(XML_reporter, reports_a_test_that_passes) {
    assert_that(1 == 1);
}

Ensure(XML_reporter, reports_a_test_that_fails) {
    fail_test("A failure");
}

TestSuite *create_test_group() {
    TestSuite *suite = create_named_test_suite("A Group");
    add_test_with_context(suite, XML_reporter, reports_a_test_that_passes);
    add_test_with_context(suite, XML_reporter, reports_a_test_that_fails);
    return suite;
}

int main(int argc, char **argv) {
    TestSuite *suite = create_named_test_suite("Top Level");
    add_suite(suite, create_test_group());
    return run_test_suite(suite, create_text_reporter());
}

We can’t use the auto-discovering cgreen-runner (see Automatic Test Discovery) here since we need to ensure that the nested suites are reported as a nested xml structure. And we’re not actually writing real tests, just something that we can use to drive our new reporter.

The text reporter is used just to confirm that everything is working. So far it is.

Running "Top Level" (2 tests)...
test_as_xml.c:12: Failure: A Group -> reports_a_test_that_fails
	A failure

  "A Group": 1 pass, 1 failure in 42ms.
Completed "Top Level": 1 pass, 1 failure in 42ms.

Our first move is to switch the reporter from text, to our not yet written XML version…​

#include "xml_reporter.h"
...

int main(int argc, char **argv) {
    TestSuite *suite = create_named_test_suite("Top Level");
    add_suite(suite, create_test_group());
    return run_test_suite(suite, create_xml_reporter());
}

We’ll start the ball rolling with the xml_reporter.h header file…​

#ifndef _XML_REPORTER_HEADER_
#define _XML_REPORTER_HEADER_

#include <cgreen/reporter.h>

TestReporter *create_xml_reporter();

#endif

…​and the simplest possible reporter in xml_reporter.c.

#include <cgreen/reporter.h>

#include "xml_reporter.h"

TestReporter *create_xml_reporter() {
    TestReporter *reporter = create_reporter();
    return reporter;
}

One that outputs nothing.

$ gcc -c test_as_xml.c
$ gcc -c xml_reporter.c
$ gcc xml_reporter.o test_as_xml.o -lcgreen -o xml
$ ./xml

Yep, nothing.

Let’s add the outer XML tags first, so that we can see Cgreen navigating the test suite…​

#include <cgreen/reporter.h>
#include <cgreen/breadcrumb.h>

#include <stdio.h>
#include "xml_reporter.h"


static void xml_reporter_start_suite(TestReporter *reporter, const char *name, int count) {
    printf("<suite name=\"%s\">\n", name);
    reporter_start_suite(reporter, name, count);
}

static void xml_reporter_start_test(TestReporter *reporter, const char *name) {
    printf("<test name=\"%s\">\n", name);
    reporter_start_test(reporter, name);
}

static void xml_reporter_finish_test(TestReporter *reporter, const char *filename, int line, const char *message) {
    reporter_finish_test(reporter, filename, line, message);
    printf("</test>\n");
}

static void xml_reporter_finish_suite(TestReporter *reporter, const char *filename, int line) {
    reporter_finish_suite(reporter, filename, line);
    printf("</suite>\n");
}

TestReporter *create_xml_reporter() {
    TestReporter *reporter = create_reporter();
    reporter->start_suite = &xml_reporter_start_suite;
    reporter->start_test = &xml_reporter_start_test;
    reporter->finish_test = &xml_reporter_finish_test;
    reporter->finish_suite = &xml_reporter_finish_suite;
    return reporter;
}

Although chaining to the underlying reporter_start_*() and reporter_finish_*() functions is optional, I want to make use of some of the facilities later.

Our output meanwhile, is making its first tentative steps…​

<suite name="Top Level">
<suite name="A Group">
<test name="reports_a_test_that_passes">
</test>
<test name="reports_a_test_that_fails">
</test>
</suite>
</suite>

We don’t require an XML node for passing tests, so the show_fail() function is all we need…​

...

static void xml_show_fail(TestReporter *reporter, const char *file, int line, const char *message, va_list arguments) {
    printf("<fail>\n");
    printf("\t<message>");
    vprintf(message, arguments);
    printf("</message>\n");
    printf("\t<location file=\"%s\" line=\"%d\"/>\n", file, line);
    printf("</fail>\n");
...

TestReporter *create_xml_reporter() {
    TestReporter *reporter = create_reporter();
    reporter->start_suite = &xml_reporter_start_suite;
    reporter->start_test = &xml_reporter_start_test;
    reporter->show_fail = &xml_show_fail;
    reporter->finish_test = &xml_reporter_finish_test;
    reporter->finish_suite = &xml_reporter_finish_suite;
    return reporter;
}

We have to use vprintf() to handle the variable argument list passed to us. This will probably mean including the stdarg.h header as well as stdio.h.

This gets us pretty close to what we want…​

<suite name="Top Level">
<suite name="A Group">
<test name="reports_a_test_that_passes">
</test>
<test name="reports_a_test_that_fails">
<fail>
	<message>A failure</message>
	<location file="test_as_xml.c" line="15"/>
</fail>
</test>
</suite>
</suite>

For completeness we should add a tag for a test that doesn’t complete. We’ll output this as a failure, although we don’t bother with the location this time…​

static void xml_show_incomplete(TestReporter *reporter, const char *file, int line, const char *message, va_list arguments) {
    printf("<fail>\n");
    printf("\t<message>Failed to complete</message>\n");
    printf("</fail>\n");
}
...
TestReporter *create_xml_reporter() {
    TestReporter *reporter = create_reporter();
    reporter->start_suite = &xml_reporter_start_suite;
    reporter->start_test = &xml_reporter_start_test;
    reporter->show_fail = &xml_show_fail;
    reporter->show_incomplete = &xml_show_incomplete;
    reporter->finish_test = &xml_reporter_finish_test;
    reporter->finish_suite = &xml_reporter_finish_suite;
    return reporter;
}

All that’s left then is the XML declaration and the thorny issue of indenting. Although the indenting is not strictly necessary, it would make the output a lot more readable.

Given that the test depth is kept track of for us with the breadcrumb object in the TestReporter structure, indentation will actually be quite simple. We’ll add an indent() function that outputs the correct number of tabs…​

static void indent(TestReporter *reporter) {
    int depth = get_breadcrumb_depth((CgreenBreadcrumb *)reporter->breadcrumb);
    while (depth-- > 0) {
        printf("\t");
    }
}

The get_breadcrumb_depth() function just gives the current test depth as recorded in the reporters breadcrumb (from cgreen/breadcrumb.h). As that is just the number of tabs to output, the implementation is trivial.

We can then use this function in the rest of the code. Here is the complete listing…​

#include <cgreen/reporter.h>
#include <cgreen/breadcrumb.h>

#include <stdio.h>
#include "xml_reporter.h"

static void indent(TestReporter *reporter) {
    int depth = get_breadcrumb_depth((CgreenBreadcrumb *)reporter->breadcrumb);
    while (depth-- > 0) {
        printf("\t");
    }
}

static void xml_reporter_start_suite(TestReporter *reporter, const char *name, int count) {
    if (get_breadcrumb_depth((CgreenBreadcrumb *)reporter->breadcrumb) == 0) {
        printf("<?xml?>\n");
    }
    indent(reporter);
    printf("<suite name=\"%s\">\n", name);
    reporter_start_suite(reporter, name, count);
}

static void xml_reporter_start_test(TestReporter *reporter, const char *name) {
    indent(reporter);
    printf("<test name=\"%s\">\n", name);
    reporter_start_test(reporter, name);
}

static void xml_show_fail(TestReporter *reporter, const char *file, int line, const char *message, va_list arguments) {
    indent(reporter);
    printf("<fail>\n");
    indent(reporter);
    printf("\t<message>");
    vprintf(message, arguments);
    printf("</message>\n");
    indent(reporter);
    printf("\t<location file=\"%s\" line=\"%d\"/>\n", file, line);
    indent(reporter);
    printf("</fail>\n");
}

static void xml_show_incomplete(TestReporter *reporter, const char *file, int line, const char *message, va_list arguments) {
    indent(reporter);
    printf("<fail>\n");
    indent(reporter);
    printf("\t<message>Failed to complete</message>\n");
    indent(reporter);
    printf("</fail>\n");
}


static void xml_reporter_finish_test(TestReporter *reporter, const char *filename, int line, const char *message) {
    reporter_finish_test(reporter, filename, line, message);
    indent(reporter);
    printf("</test>\n");
}

static void xml_reporter_finish_suite(TestReporter *reporter, const char *filename, int line) {
    reporter_finish_suite(reporter, filename, line);
    indent(reporter);
    printf("</suite>\n");
}

TestReporter *create_xml_reporter() {
    TestReporter *reporter = create_reporter();
    reporter->start_suite = &xml_reporter_start_suite;
    reporter->start_test = &xml_reporter_start_test;
    reporter->show_fail = &xml_show_fail;
    reporter->show_incomplete = &xml_show_incomplete;
    reporter->finish_test = &xml_reporter_finish_test;
    reporter->finish_suite = &xml_reporter_finish_suite;
    return reporter;
}

And finally the desired output…​

<?xml?>
<suite name="Top Level">
	<suite name="A Group">
		<test name="reports_a_test_that_passes">
		</test>
		<test name="reports_a_test_that_fails">
			<fail>
				<message>A failure</message>
				<location file="test_as_xml.c" line="15"/>
			</fail>
		</test>
	</suite>
</suite>

Job done.

Possible other reporter customizations include reporters that write to syslog, talk to IDE plug-ins, paint pretty printed documents or just return a boolean for monitoring purposes.