Sunday, May 15, 2011

Reporting only Failed Test in GoogleTest

GoogleTest with GoogleMock (gtest and gmock) is an outstanding platform for testing C++ code.  At work, we use gtest and gmock for unit test as well as integration style test.  On one of out projects, we are up to over 600 unit tests.  This is great for code coverage, but can generate very verbose output.  When adding new tests, and a failure occurs, the explanation of the failed will scroll by to quickly to read and may be lost if your console buffer is not large enough.  It would be really nice to only output a summary of the executed tests and any failed tests. 
The default output printer in gtest is an object named PrettyUnitTestResultPrinter that extends the TestEventListener class.  TestEventListener objects can hook into the test framework and handle various events as tests are executed. You can learn about implementing TestEventListener classes here.  The documentation even shows you how to implement a minimalist printer, but without the functionality of the default printer.  The built in PrettyUnitTestResultPrinter does a nice job of reporting test results.  The results are even colored green for success and red for failure.  Being an open source project, you could just copy the implementation of PrettyUnitTestResultPrinter and modify it to your needs.  I did not want to take this approach as it would be messy to extract this class and all of its dependencies and cause more maintenance. 
Instead I created a TestEventListenerProxy class that implements all of the methods in TestEventListener and forwards the calls to a TestEventListener object.  This class will take ownership of the pointer passed in to the constructor (deleting it in the destructor).  Thus if I can get a pointer to a TestEventListener object, I can wrap it and override methods.  See the code below.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23




class TestEventListenerProxy : public TestEventListener 
{
public:
explicit TestEventListenerProxy(TestEventListener* event_listener);
virtual ~TestEventListenerProxy();

virtual void OnTestProgramStart(const UnitTest& unit_test);
virtual void OnTestIterationStart(const UnitTest& unit_test, int iteration);
virtual void OnEnvironmentsSetUpStart(const UnitTest& unit_test);
virtual void OnEnvironmentsSetUpEnd(const UnitTest& unit_test);
virtual void OnTestCaseStart(const TestCase& test_case);
virtual void OnTestStart(const TestInfo& test_info);
virtual void OnTestPartResult(const TestPartResult& result);
virtual void OnTestEnd(const TestInfo& test_info);
virtual void OnTestCaseEnd(const TestCase& test_case);
virtual void OnEnvironmentsTearDownStart(const UnitTest& unit_test);
virtual void OnEnvironmentsTearDownEnd(const UnitTest& unit_test);
virtual void OnTestIterationEnd(const UnitTest& unit_test, int iteration);
virtual void OnTestProgramEnd(const UnitTest& unit_test);

protected:
TestEventListener* listener;
};

Then from this class I created the CaseSummaryAndFailurePrinter class.  This class will override some of the TestEventListener methods with empty implementations to effectively remove the printing functionality of those methods.  Then, in the OnTestEnd method, the default printer is only called in the event that the test failed.  This class will then only print the overall summaries, case summaries and failed tests.


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17




class CaseSummaryAndFailurePrinter : public TestEventListenerProxy
{
public:
explicit CaseSummaryAndFailurePrinter(TestEventListener* default_printer)
    : TestEventListenerProxy(default_printer)
{
}

virtual void OnEnvironmentsTearDownStart(const UnitTest& /*unit_test*/) { }
virtual void OnEnvironmentsSetUpStart(const UnitTest& /*unit_test*/) { }
virtual void OnTestStart(const TestInfo& /*test_info*/) { }

virtual void OnTestEnd(const TestInfo& test_info) {
    if (test_info.result()->Failed())
        listener->OnTestEnd(test_info);
    }
};


I use the CaseSummaryAndFailurePrinter for integration style tests.  These tests may take a while to run and having the case summaries printed allows you to see progress.  Test cases are basically the tests defined in one C++ class.

Finally, for unit tests I use the SummaryAndFailurePrinter class.  This class extends CaseSummaryAndFailurePrinter and overrides the OnTestCaseStart and OnTestCaseEnd methods to remove the printing of the test case information.  Thus for my 600+ unit test I see only the summary of the number of tests that were executed, and any that may have failed.  This removes the verbosity of the default printer in gtest while still getting its functionality when needed.



1
2
3
4
5
6
7
8
9
10
11




class SummaryAndFailurePrinter : public CaseSummaryAndFailurePrinter
{
public:
explicit SummaryAndFailurePrinter(TestEventListener* default_printer)
    : CaseSummaryAndFailurePrinter(default_printer)
{
}

virtual void OnTestCaseStart(const TestCase& /*test_case*/) { }
virtual void OnTestCaseEnd(const TestCase& /*test_case*/) { }
};


The only thing left is to replaces the default printer in gtest with out wrapper.  This can be accomplished in your main method with the following code:



1
2
3
4
5
6
7
8
9
10




testing::InitGoogleTest(&argc, argv);
testing::InitGoogleMock(&argc, argv);

testing::TestEventListeners& listeners =
testing::UnitTest::GetInstance()->listeners();

auto default_printer = listeners.Release(listeners.default_result_printer());
listeners.Append(new SummaryAndFailurePrinter(default_printer));

return RUN_ALL_TESTS();


As shown above, we can tell the TestEventListeners container to release ownership of the default printer.  We can then wrap this default printer with a SummaryAndFailerPrinter and add that object to the TestEventListeners container.

The more I use gtest and gmock the more impressed I am.  This is by far the best C++ testing framework that I have ever seen.  Great job google people!

Saturday, May 14, 2011

Going Agile…

I haven’t posted in a while because I started a new job.  The new job is an agile shop with a firm belief in software craftsmanship.  I had never worked on an agile team before so it was all new to me.  We use Test Driven Development (when possible), pair programming, and have a 2 week iteration cycle.  I say when possible about TDD because working in a C++ project with an approximately 3 minute compile time does not allow you to do TDD the way it was meant to be done.  The new gig is mostly a .NET shop but, as mentioned, they still have some C++ applications out there.  The interesting thing is that not many of the team members had any real experience with C++.  It is a youngish team that mostly started with C# and .NET, with a few that have some Java experience.  C++ frustrates these people so much, it is kind of funny to sit back and watch.  I get the feeling that the C++ code is a major reason that I was pulled in to this team.

I have been busy learning the new code base.  Now about 3 iterations in, I believe that I am adding a lot of value to the team.  The one major omission with this job is that they do not use, and don’t plan on using, Python.  I will miss using Python at work.  Any new software is to be done in C# on the .NET platform.  Not that there is any real dependency on Microsoft, this is just team policy. A well respected contractor tried to turn the team on to Ruby, but so far nothing has come of it.

The team has been reading “Clean Code” by Robert C. Martin.  There is a weekly book club meeting where we discuss a chapter of the book.  This is part of the software craftsmanship initiative.  I think this is a great idea and it helps to improve the quality of all of the software developers.  We will be reading a TDD book after we finish the clean code book.

I hope to get back to writing one post a month.  This was my initial goal for this blog.  Time flies when your coding!