SubText cruises with CruiseControl.NET

Introduction

The Subtext development team has grown a lot in the last 9 months, so we needed something to better manage the build and testing process. We decided to adopt a Continuous Integration process (Referred to as CI later in this document) to help manage our build and testing process moving forward. Fortunately one of our team members had experience in this area and time to set it up in a VMWare virtual machine.

Due to the Open Source nature of this project we decided to use CruiseControl.NET, an open source Continuous Integration tool.

Why continuously integrate

Once a development team starts to grow beyond a one or two person operation, Continuous Integration starts to be come an attractive choice. At some point in the project timeline, the code written by the various developers on their individual desktops need to be merged into a single codebase. Ideally, the code should compile, have all the needed features, and the features implemented should work correctly.

Keep in mind that this isn’t necessarily an issue of branching and merging. This is an issue in concurrency. Even without source control branching, problems can occur. Suppose Bob and Alice are both developers working on the same codebase. They both get latest from the source control at the same time and start making changes over the course of a week. At the end of the week, they each complete their features build on their local machines and their code passes all the unit tests. Now they both check in their changes and go home for the day.

Unfortunately, earlier in the week Bob made a change to a class that Alice had been relying on and the build as it stands in the source control system is broken, yet it worked on each individual workstation. At this point Alice and Bob need to spend some time working together to integrate their respective changes. As the project goes on and the number of team members grows, each integration step may involve more and more developers thus taking longer and longer.

Ideally Alice and Bob should have been integrating once a day. Or at the very least testing the integration.

CI helps ameliorate the problems reported above. Each time someone checks in code, a complete build and run of automated tests (as well as any other build actions desired) is performed. Everyone in the team receives immediate feedback when the build is broken along with who broke the build and why it is broken. Actions to fix the build can be started immediately, typically by the person who broke the build.

With this approach when everyone completes his or her tasks, the product is ready for deployment, and a lengthy end of project code integration process is no longer needed. Of course there may still be integration issues between subsystems and external systems that cannot be controlled by the CI process. But at least the internal integration issues between developers on the same codebase should be complete.

If you are interested in CI best practices (and in an explanation better than this one) consult the CI Manifesto by Martin Fowler.

The tools behind the CI process

As said above, we are using CruiseControl.NET (referred to as CCNET), incubated by the same guys that developed the original CruiseControl for Java, ThoughtWorks, where Martin Fowler is employed.

But CCNET is just the tip of the iceberg, it's just the orchestrator of the continuous build and testing process. The other tools used for the CI process are:

Subversion, to get the code from SVN repository
NAnt, to manage the overall build and testing process
.NET SDK 1.1, to compile all the projects
MbUnit, to run the tests that all the wise SubText developers write for all important features
NCover, to instrument the automated test assembly, and generate a report with the lines tested
NCoverExplorer, to generate an human readable summary of the code coverage report
FxCop, to test the final assemblies against all sort of code quality rules
Vil, to generate a code metrics report

The Integration process

What does the CI process do?

Mainly it builds, tests the project and then it produces several reports.

Let's see in detail the steps performed by our CI process:

CCNET constantly monitors the SVN repository, and when a new check-in is discovered, it pulls it down to the local working folder
NAnt starts the build process performing a clean up of all projects artifact folders (bin and obj)
A Debug build is performed
The Unit testing project is built
Then comes the unit testing, which is performed inside the NCover instrumentation process
NCoverExplorer read the NCover report, and generates a summary for the code coverage
At this point everything that could have caused a broken build has completed, so let's make a Release build
FxCop performs all its rules analysis
and Vil computes the code metrics
the last step in the build is zipping the release build, and copy it to the build archive folder
now it's the time for the post build task: all reports are merged
and build statistics are gathered

It's all about communication

If nobody knows that a build is broken, nobody will never fix it.

So another important part of CI is communication, and CCNET allows many types of communications:
first there is the web dashboard, that contains all the build's reports and logs, then it can send an email to all the registered developers after each build.
But a really cool tool shipped with CCNET is CCTray:
this is a tiny little app that sits on the Windows traybar and, continuously monitoring the build server, notifies the developer of any build completed, and of its status.
It can notify the user by sound, with a balloon, or even driving an X10 device, like a lava lamp or normal lamps.

Performance measurements

Since we at SubText are crazy, we are not using the last released version of CCNET, but we decided to use the latest integration of CCNET. Why?
Because the 1.1 version added a lot of new features. But the most important in my opinion is the possibility to have the statistics for each project: we can have any number reported listed on a table and, later, drawn on a graph.

We are currently collecting test runs and failed, FxCop errors and warnings, and code lines and coverage %.