8. Changing Cgreen Reporting
8.1. Replacing the Reporter
In every test suite so far, we have run the tests with this line…
return run_test_suite(our_tests(), create_text_reporter());
We can change the reporting mechanism just by changing this call to create another reporter.
8.2. Built-in Reporters
Cgreen has the following built-in reporters that you can choose from when your code runs the test suite.
Reporter | Purpose | Signature | Note |
---|---|---|---|
Text |
Human readable, with clear messages |
|
|
XML |
ANT/Jenkins compatible |
|
|
CUTE |
CUTE Eclipse-plugin (http://cute-test.org) compatible output |
|
|
CDash |
CMake (http://cmake.org) dashboard |
|
|
If you write a runner function like in most examples above, you can
just substitute which runner to create. If you use the cgreen-runner
(Automatic Test Discovery) to dynamically find all your tests you can force
it to use the XML-reporter with the -x <prefix>
option.
Currently cgreen-runner only supports the test and XML
built-in reporters.
|
8.3. Rolling Our Own
Althoug Cgreen has a number of options, there are times when you’d like a different output from the reporter, the CUTE and CDash reporters are examples that grew out of such a need.
Perhaps your Continuous Integration server want the result in a different format, or you just don’t like the text reporter…
Writing your own reporter is supported. And we’ll go through how that can be done using an XML-reporter as an example.
Cgreen already has an XML-reporter compatible with ANT/Jenkins, see Built-in Reporters. |
Here is the code for create_text_reporter()
…
TestReporter *create_text_reporter(void) {
TestReporter *reporter = create_reporter();
if (reporter == NULL) {
return NULL;
}
reporter->start_suite = &text_reporter_start_suite;
reporter->start_test = &text_reporter_start_test;
reporter->show_fail = &show_fail;
reporter->show_skip = &show_skip;
reporter->show_incomplete = &show_incomplete;
reporter->finish_test = &text_reporter_finish_test;
reporter->finish_suite = &text_reporter_finish;
return reporter;
}
The TestReporter
structure contains function pointers that control
the reporting. When called from create_reporter()
constructor,
these pointers are set up with functions that display nothing. The
text reporter code replaces these with something more dramatic, and
then returns a pointer to this new object. Thus the
create_text_reporter()
function effectively extends the object from
create_reporter()
.
The text reporter only outputs content at the start of the first test,
at the end of the test run to display the results, when a failure
occurs, and when a test fails to complete. A quick look at the
text_reporter.c
file in Cgreen reveals that the overrides just
output a message and chain to the versions in reporter.h
.
To change the reporting mechanism ourselves, we just have to know a little
about the methods in the TestReporter
structure.
8.4. The TestReporter Structure
The Cgreen TestReporter
is a pseudo class that looks
something like…
typedef struct _TestReporter TestReporter;
struct _TestReporter {
void (*destroy)(TestReporter *reporter);
void (*start_suite)(TestReporter *reporter, const char *name, const int count);
void (*start_test)(TestReporter *reporter, const char *name);
void (*show_pass)(TestReporter *reporter, const char *file, int line,
const char *message, va_list arguments);
void (*show_skip)(TestReporter *reporter, const char *file, int line);
void (*show_fail)(TestReporter *reporter, const char *file, int line,
const char *message, va_list arguments);
void (*show_incomplete)(TestReporter *reporter, const char *file, int line,
const char *message, va_list arguments);
void (*assert_true)(TestReporter *reporter, const char *file, int line, int result,
const char * message, ...);
void (*finish_test)(TestReporter *reporter, const char *file, int line);
void (*finish_suite)(TestReporter *reporter, const char *file, int line);
int passes;
int failures;
int exceptions;
void *breadcrumb;
int ipc;
void *memo;
void *options;
};
The first block are the methods that can be overridden:
void (*destroy)(TestReporter *reporter)
-
This is the destructor for the default structure. If this is overridden, then the overriding function must call
destroy_reporter(TestReporter *reporter)
to finish the clean up. void (*start_suite)(TestReporter *reporter, const char *name, const int count)
-
This is the first of the callbacks. At the start of each test suite Cgreen will call this method on the reporter with the name of the suite being entered and the number of tests in that suite. The default version keeps track of the stack of tests in the
breadcrumb
pointer ofTestReporter
. If you make use of the breadcrumb functions, as the defaults do, then you will need to callreporter_start_suite()
to keep the book-keeping in sync. void (*start_test)(TestReporter *reporter, const char *name)
-
At the start of each test Cgreen will call this method on the reporter with the name of the test being entered. Again, the default version keeps track of the stack of tests in the
breadcrumb
pointer ofTestReporter
. If you make use of the breadcrumb functions, as the defaults do, then you will need to callreporter_start_test()
to keep the book-keeping in sync. void (*show_pass)(TestReporter *reporter, const char *file, int line, const char *message, va_list arguments)
-
This method is initially empty as most reporters see little point in reporting passing tests (but you might do), so there is no need to chain the call to any other function. Besides the pointer to the reporter structure, Cgreen also passes the file name of the test, the line number of failed assertion, the message to show and any additional parameters to substitute into the message. The message comes in as
printf()
style format string, and so the variable argument list should match the substitutions. void (*show_fail)(TestReporter *reporter, const char *file, int line, const char *message, va_list arguments)
-
The partner of
show_pass()
, and the one you’ll likely overload first. void (*show_skip)(TestReporter *reporter, const char *file, int line)
-
This method will be called when a skipped test is encountered, see Skipping Tests.
void (*show_incomplete)(TestReporter *reporter, const char *file, int line, const char *message, va_list arguments)
-
When a test fails to complete, this is the handler that is called. As it’s an unexpected outcome, no message is received, but we do get the name of the test. The text reporter combines this with the breadcrumb to produce the exception report.
void (*assert_true)(TestReporter *reporter, const char *file, int line, int result, const char * message, …)
-
This is not normally overridden and is really internal. It is the raw entry point for the test messages from the test suite. By default it dispatches the call to either
show_pass()
orshow_fail()
. void (*finish_test)(TestReporter *reporter, const char *file, int line)
-
The counterpart to the
(*start_test)()
call. It is called on leaving the test. It needs to be chained to thereporter_finish()
to keep track of the breadcrumb book keeping. void (*finish_suite)(TestReporter *reporter, const char *file, int line)
-
The counterpart to the
(*start_suite)()
call called on leaving the test suite, and similar to the(*finish_test)()
if your reporter needs a handle on that event too. The default text reporter chains both this and(*finish_test)()
to the same function where it figures out if it is the end of the top level suite. If so, it prints the familiar summary of passes and fails.
The show_fail() and show_pass() functions are called from
the child process, i.e. the isolated process that is fork() :ed to
run a single test case. All others, notably start_…() ,
finish_…() , show_incomplete() and show_skip() are run in the
main (parent) process. This fact might be important since the
processes do not share memory. Information is passed from the child to
the parent using messaging performed within the show_…()
functions.
|
The second block is simply resources and book keeping that the reporter can use to liven up the messages…
passes
|
The number of passes so far. |
skips
|
The number of tests that has been skipped by the |
failures
|
The number of failures generated so far. |
exceptions
|
The number of test functions that have failed to complete so far. |
breadcrumb
|
This is a pointer to the list of test names in the stack. |
The breadcrumb
pointer is different and needs a little explanation.
Basically it is a stack, analogous to the breadcrumb trail you see on
websites. Everytime a start()
handler is invoked, the name is
placed in this stack. When a finish()
message handler is invoked, a
name is popped off.
There are a bunch of utility functions in cgreen/breadcrumb.h
that
can read the state of this stack. Most useful are
get_current_from_breadcrumb()
which takes the breadcrumb pointer and
returns the current test name, and get_breadcrumb_depth()
which gives
the current depth of the stack. A depth of zero means that the test
run has finished.
If you need to traverse all the names in the breadcrumb, then you can
call walk_breadcrumb()
. Here is the full signature…
void walk_breadcrumb(Breadcrumb *breadcrumb, void (*walker)(const char *, void *), void *memo);
The void (*walker)(const char *, void *)
is a callback that will be
passed the name of the test suite for each level of nesting.
It is also passed the memo
pointer that was passed to the
walk_breadcrumb()
call. You can use this pointer for anything you
want, as all Cgreen does is pass it from call to call. This is so
aggregate information can be kept track of whilst still being
reentrant.
The last parts of the TestReporter
structure are…
ipc
|
This is an internal structure for handling the messaging between reporter and test suite. You shouldn’t touch this. |
memo
|
By contrast, this is a spare pointer for your own expansion. |
options
|
A pointer to a reporter specific structure that can be
used to set options. E.g. the textreporter defines the structure
|
8.5. An Example XML Reporter
Let’s make things real with an example. Suppose we want to send the output from Cgreen in XML format, say for storing in a repository or for sending across the network.
The cgreen-runner already has an XML-reporter that you can
use if you need to produce Jenkins/ANT compatible XML output.
See Cgreen Runner Options.
|
Suppose also that we have come up with the following format…
<?xml?>
<suite name="Top Level">
<suite name="A Group">
<test name="a_test_that_passes">
</test>
<test name="a_test_that_fails">
<fail>
<message>A failure</message>
<location file="test_as_xml.c" line="8"/>
</fail>
</test>
</suite>
</suite>
In other words a simple nesting of tests with only failures encoded. The absence of "fail" XML node is a pass.
Here is a test script, test_as_xml.c
that we can use to construct the
above output…
#include <cgreen/cgreen.h>
Describe(XML_reporter);
BeforeEach(XML_reporter) {}
AfterEach(XML_reporter) {}
Ensure(XML_reporter, reports_a_test_that_passes) {
assert_that(1 == 1);
}
Ensure(XML_reporter, reports_a_test_that_fails) {
fail_test("A failure");
}
TestSuite *create_test_group() {
TestSuite *suite = create_named_test_suite("A Group");
add_test_with_context(suite, XML_reporter, reports_a_test_that_passes);
add_test_with_context(suite, XML_reporter, reports_a_test_that_fails);
return suite;
}
int main(int argc, char **argv) {
TestSuite *suite = create_named_test_suite("Top Level");
add_suite(suite, create_test_group());
return run_test_suite(suite, create_text_reporter());
}
We can’t use the auto-discovering cgreen-runner
(see
Automatic Test Discovery) here since we need to ensure that the nested
suites are reported as a nested xml structure. And we’re not actually
writing real tests, just something that we can use to drive our new
reporter.
The text reporter is used just to confirm that everything is working. So far it is.
Running "Top Level" (2 tests)... test_as_xml.c:12: Failure: A Group -> reports_a_test_that_fails A failure "A Group": 1 pass, 1 failure in 42ms. Completed "Top Level": 1 pass, 1 failure in 42ms.
Our first move is to switch the reporter from text, to our not yet written XML version…
#include "xml_reporter.h"
...
int main(int argc, char **argv) {
TestSuite *suite = create_named_test_suite("Top Level");
add_suite(suite, create_test_group());
return run_test_suite(suite, create_xml_reporter());
}
We’ll start the ball rolling with the xml_reporter.h
header file…
#ifndef _XML_REPORTER_HEADER_
#define _XML_REPORTER_HEADER_
#include <cgreen/reporter.h>
TestReporter *create_xml_reporter();
#endif
…and the simplest possible reporter in xml_reporter.c
.
#include <cgreen/reporter.h>
#include "xml_reporter.h"
TestReporter *create_xml_reporter() {
TestReporter *reporter = create_reporter();
return reporter;
}
One that outputs nothing.
$ gcc -c test_as_xml.c $ gcc -c xml_reporter.c $ gcc xml_reporter.o test_as_xml.o -lcgreen -o xml $ ./xml
Yep, nothing.
Let’s add the outer XML tags first, so that we can see Cgreen navigating the test suite…
#include <cgreen/reporter.h>
#include <cgreen/breadcrumb.h>
#include <stdio.h>
#include "xml_reporter.h"
static void xml_reporter_start_suite(TestReporter *reporter, const char *name, int count) {
printf("<suite name=\"%s\">\n", name);
reporter_start_suite(reporter, name, count);
}
static void xml_reporter_start_test(TestReporter *reporter, const char *name) {
printf("<test name=\"%s\">\n", name);
reporter_start_test(reporter, name);
}
static void xml_reporter_finish_test(TestReporter *reporter, const char *filename, int line, const char *message) {
reporter_finish_test(reporter, filename, line, message);
printf("</test>\n");
}
static void xml_reporter_finish_suite(TestReporter *reporter, const char *filename, int line) {
reporter_finish_suite(reporter, filename, line);
printf("</suite>\n");
}
TestReporter *create_xml_reporter() {
TestReporter *reporter = create_reporter();
reporter->start_suite = &xml_reporter_start_suite;
reporter->start_test = &xml_reporter_start_test;
reporter->finish_test = &xml_reporter_finish_test;
reporter->finish_suite = &xml_reporter_finish_suite;
return reporter;
}
Although chaining to the underlying reporter_start_*()
and reporter_finish_*()
functions is optional, I want to
make use of some of the facilities later.
Our output meanwhile, is making its first tentative steps…
<suite name="Top Level">
<suite name="A Group">
<test name="reports_a_test_that_passes">
</test>
<test name="reports_a_test_that_fails">
</test>
</suite>
</suite>
We don’t require an XML node for passing tests, so the show_fail()
function is all we need…
...
static void xml_show_fail(TestReporter *reporter, const char *file, int line, const char *message, va_list arguments) {
printf("<fail>\n");
printf("\t<message>");
vprintf(message, arguments);
printf("</message>\n");
printf("\t<location file=\"%s\" line=\"%d\"/>\n", file, line);
printf("</fail>\n");
...
TestReporter *create_xml_reporter() {
TestReporter *reporter = create_reporter();
reporter->start_suite = &xml_reporter_start_suite;
reporter->start_test = &xml_reporter_start_test;
reporter->show_fail = &xml_show_fail;
reporter->finish_test = &xml_reporter_finish_test;
reporter->finish_suite = &xml_reporter_finish_suite;
return reporter;
}
We have to use vprintf()
to handle the variable argument list passed
to us. This will probably mean including the stdarg.h
header as
well as stdio.h
.
This gets us pretty close to what we want…
<suite name="Top Level">
<suite name="A Group">
<test name="reports_a_test_that_passes">
</test>
<test name="reports_a_test_that_fails">
<fail>
<message>A failure</message>
<location file="test_as_xml.c" line="15"/>
</fail>
</test>
</suite>
</suite>
For completeness we should add a tag for a test that doesn’t complete. We’ll output this as a failure, although we don’t bother with the location this time…
static void xml_show_incomplete(TestReporter *reporter, const char *file, int line, const char *message, va_list arguments) {
printf("<fail>\n");
printf("\t<message>Failed to complete</message>\n");
printf("</fail>\n");
}
...
TestReporter *create_xml_reporter() {
TestReporter *reporter = create_reporter();
reporter->start_suite = &xml_reporter_start_suite;
reporter->start_test = &xml_reporter_start_test;
reporter->show_fail = &xml_show_fail;
reporter->show_incomplete = &xml_show_incomplete;
reporter->finish_test = &xml_reporter_finish_test;
reporter->finish_suite = &xml_reporter_finish_suite;
return reporter;
}
All that’s left then is the XML declaration and the thorny issue of indenting. Although the indenting is not strictly necessary, it would make the output a lot more readable.
Given that the test depth is kept track of for us with the
breadcrumb
object in the TestReporter
structure, indentation will
actually be quite simple. We’ll add an indent()
function that
outputs the correct number of tabs…
static void indent(TestReporter *reporter) {
int depth = get_breadcrumb_depth((CgreenBreadcrumb *)reporter->breadcrumb);
while (depth-- > 0) {
printf("\t");
}
}
The get_breadcrumb_depth()
function just gives the current test
depth as recorded in the reporters breadcrumb (from
cgreen/breadcrumb.h
). As that is just the number of tabs to output,
the implementation is trivial.
We can then use this function in the rest of the code. Here is the complete listing…
#include <cgreen/reporter.h>
#include <cgreen/breadcrumb.h>
#include <stdio.h>
#include "xml_reporter.h"
static void indent(TestReporter *reporter) {
int depth = get_breadcrumb_depth((CgreenBreadcrumb *)reporter->breadcrumb);
while (depth-- > 0) {
printf("\t");
}
}
static void xml_reporter_start_suite(TestReporter *reporter, const char *name, int count) {
if (get_breadcrumb_depth((CgreenBreadcrumb *)reporter->breadcrumb) == 0) {
printf("<?xml?>\n");
}
indent(reporter);
printf("<suite name=\"%s\">\n", name);
reporter_start_suite(reporter, name, count);
}
static void xml_reporter_start_test(TestReporter *reporter, const char *name) {
indent(reporter);
printf("<test name=\"%s\">\n", name);
reporter_start_test(reporter, name);
}
static void xml_show_fail(TestReporter *reporter, const char *file, int line, const char *message, va_list arguments) {
indent(reporter);
printf("<fail>\n");
indent(reporter);
printf("\t<message>");
vprintf(message, arguments);
printf("</message>\n");
indent(reporter);
printf("\t<location file=\"%s\" line=\"%d\"/>\n", file, line);
indent(reporter);
printf("</fail>\n");
}
static void xml_show_incomplete(TestReporter *reporter, const char *file, int line, const char *message, va_list arguments) {
indent(reporter);
printf("<fail>\n");
indent(reporter);
printf("\t<message>Failed to complete</message>\n");
indent(reporter);
printf("</fail>\n");
}
static void xml_reporter_finish_test(TestReporter *reporter, const char *filename, int line, const char *message) {
reporter_finish_test(reporter, filename, line, message);
indent(reporter);
printf("</test>\n");
}
static void xml_reporter_finish_suite(TestReporter *reporter, const char *filename, int line) {
reporter_finish_suite(reporter, filename, line);
indent(reporter);
printf("</suite>\n");
}
TestReporter *create_xml_reporter() {
TestReporter *reporter = create_reporter();
reporter->start_suite = &xml_reporter_start_suite;
reporter->start_test = &xml_reporter_start_test;
reporter->show_fail = &xml_show_fail;
reporter->show_incomplete = &xml_show_incomplete;
reporter->finish_test = &xml_reporter_finish_test;
reporter->finish_suite = &xml_reporter_finish_suite;
return reporter;
}
And finally the desired output…
<?xml?>
<suite name="Top Level">
<suite name="A Group">
<test name="reports_a_test_that_passes">
</test>
<test name="reports_a_test_that_fails">
<fail>
<message>A failure</message>
<location file="test_as_xml.c" line="15"/>
</fail>
</test>
</suite>
</suite>
Job done.
Possible other reporter customizations include reporters that write to
syslog
, talk to IDE plug-ins, paint pretty printed documents or just
return a boolean for monitoring purposes.