Friday, November 07, 2008

cdart, cdart2 or cdash? no, cdash public!

Having a way to monitor the quality of your project is a nice thing. Uffe, my former boss from Motorola, will happily agree on this.

CMake comes with CTest and you can use it to run and collect the results of your unit-tests. Even more, you can use CTest to run gcov and valgrind.

Another nice feature of CTest is its ability to store in a central location these results, though this is also one of its problem: it's a bit more difficult (aka, not obvious) how to not publish your results and still get code coverage!

Also difficult is setting a server where to publish your data. Based on the CMake wiki, I have assumed (wrongly), that I need to set up a CDart server. Then I've found a tutorial about using a CDart2 server that looked even simpler. But just when I was going to start installing CDart2, I've come upon an e-mail from CDart2 mailing list that stated that both CDart and CDart2 are obsoleted and I should use CDash instead.

Lucky me I was too lazy to install any of the CDart of CDart2 servers.

Even better, you don't have to install CDash either: just register at and create a project (or more) there. Once your project is created, you can download an auto-generated CTestConfig.cmake file that can be dropped into your top source dir. Once that is done, you can run ctest and publish the results (from the build directory, of course) to CDashPublic.

Unfortunately, you still need a couple of tweaks:
  • none of the default CMake build types supports the flags needed by gcov. I've solved this by creating my own Maintainer mode:
    set(CMAKE_CXX_FLAGS_MAINTAINER "-g -O0 -Wall -W -Wshadow -Wunused-variable -Wunused-parameter -Wunused-function -Wunused -Wno-system-headers -Wno-deprecated -Woverloaded-virtual -Wwrite-strings -fprofile-arcs -ftest-coverage" CACHE STRI
    "Flags used by the C++ compiler during maintainer builds."
    set(CMAKE_C_FLAGS_MAINTAINER "-g -O0 -Wall -pedantic -W -fprofile-arcs -ftest-coverage" CACHE STRING
    "Flags used by the C compiler during maintainer builds."
    "-Wl,--warn-unresolved-symbols,--warn-once -fprofile-arcs -ftest-coverage" CACHE STRING
    "Flags used for linking binaries during maintainer builds."
    "-Wl,--warn-unresolved-symbols,--warn-once -fprofile-arcs -ftest-coverage" CACHE STRING
    "Flags used by the shared libraries linker during maintainer builds."
  • activate the maintainer mode (without this, you'll not get gcov data):
    mkdir build
    cd build
    cmake -DCMAKE_BUILD_TYPE=Maintainer ..
  • publish the data (you need an explicit call to ExperimentalMemCheck to get valgrind)
    ctest -D Experimental -D ExperimentalMemCheck
Not perfect, but not that bad either and now I can publish and analyze my metrics :-)

Wednesday, November 05, 2008

new lib of choice: log4cxx

During the last couple of weeks I've been evaluating various logging libs. The finalists were rlog-1.4 and log4cxx-0.10.0.

Inspired by the Panthios performance comparison, I've concocted my own performance test to decide what should I use: rlog or log4cxx. The test was also good to better understand the two libs and see what's the complexity to achieve similar results with them.

Instead of the tests from Pantheios, I've used these:
  • Scenario 0 No Logging (used to get a reference for the other tests)
  • Scenario 1 A single string.
  • Scenario 2 Several string instances
  • Scenario 3 Several numeric type instances.
  • Scenario 4 A custom type `Person`.
  • Scenario 5 A composite scenario, based on the previous scenarios.
  • Scenario 6 Multiple log channels, instantiated by a large number of objects, with messages only on a couple of channels (this should indicate the impact of having a new log channel defined in each class)
The tests were run logging to a file descriptor. To distinguish between lib performance and disk performance, the tests were run once logging to /dev/null and once to a real file.

Each test was also repeated for 2 types of logging formats: ttcc (this uses the TTCC log class from Log4cxx) and rlog (this uses a custom format logger from Log4cxx that outputs a text similar with that generated by rlog).

Though rlog was always almost twice as fast compared with log4cxx, in the end I'll go with log4cxx because:
  • support for wide chars (not needed now, but nice to have)
  • text formatting is much more flexible and C++ style
  • more storage formats
  • I liked more the way I've been able to add logging at class level (this is personal, I know, and somebody might get a simpler and nicer way to add rlog to a class than I was able to)
Hopefully, log4cxx will be fast enough for my isolib and I will not regret this decision later :)

Here are the benchmark results: