Build Pattern: Green-lit build

green traffic light

by admin on February 25, 2010

Continuous Integration should be a highway, not a parking lot. But that’s what happens sometimes when developers end up competing for limited Continuous Integration capacity. Developers working on critical and time-senstive work like production bugfixes can struggle to get their builds serviced promptly; they can be fighting a tide of checkins from their colleagues working on new functionality. Also, functional tests can jam up the available build agents while short builds queue up. This delays the feedback to the developers that they have integrated their code properly.

What to do? Dedicate some capacity to those builds that could actually take priority. How you implement depends on your continuous integration system. On CruiseControl (the original one), I made a seperate server for this. I’m implementing the same thing on Team City at the moment and I’ve added an environment variable to the build agents that I want to reserve for fast builds. Any build that is longer than 15 minutes is told not to use it if the variable exists.

One might argue that I’m not making best use of the CI server by doing this. That doesn’t matter. People are more expensive than Continuous Integration servers; let’s optimise the system for them.

(image from Ted Percival)

Share with the group:
  • Digg
  • del.icio.us
  • Facebook
  • DZone
  • LinkedIn
  • Slashdot
  • StumbleUpon

Related posts:

  1. Build Pattern: Facade Let’s face it: Build scripts can age as badly...
  2. Continuous Integration in the cloud: good idea? Continuous Integration can be tricky to provision. It’s IO...
  3. Continuous Integration is an attitude Question: How do we prevent integration pain? Answer: Continuous...
  4. Justifying Continuous Integration Expenditure Banos commented on my last post: So why, oh...

Related posts brought to you by Yet Another Related Posts Plugin.

Tagged as: patterns

  • admin
    EJ: I needed to reserve some capacity, and allow the projects to build simultaneously. The Ant builds had a tendancy to do things outside the sandbox, and I didn't fancy trying the distributed CruiseControl. It worked out all right in the end. One server was barely ever touched, it just built trunk, over and over again. From memory there were 3 separate CruiseControl servers. It would have been better to use Team City or Hudson but there were political issues.

    Mike: thank you - your comment made my afternoon!

    Banos: your comment turned into a new post - thanks :)
  • Banos
    So why, oh why, oh why is it so difficult to get an additional server? Has anyone come up with a formula to produce some numbers for the bean counters to justify this already?

    I propose this is an endemic problem that the guys on the ground give up fighting to resolve because there is no budget for more servers.

    I might try passing this point around. Thanks for the write up.
  • MikeNereson
    People are more expensive than Continuous Integration servers; let’s optimise the system for them.


    You're an insightful man. Thanks.
  • EJC
    I should have fully qualify that thought with -

    If you can't add more executors (build threads) and can't add more nodes (machines that run the build threads).

    In the old CC land, it was one executor per machine in your cluster (unless you wanted a headache). In Hudson (and likely in TeamCity), you can have multiple executors per node. If you're playing with maven, make sure you use private repositories and make sure you clean them out regularly (to avoid collisions and maxing disk space).
  • EJC
    One thing to consider - not sure how this would work with TeamCity, but in Hudson, you can label the "nodes" in your cluster. Why not reserve a node for only the fastest running builds and/or high priority builds? So you could adjust the label on the node to say, "ok, for the next hour, this node is reserved for high priority builds".

    I've always wished someone would build this concept into a CI server so I didn't have to...

    Additionally, in Hudson (not sure again about TeamCity) you can kill a running build (not something you could easily do in a clustered cruise control setup). Can you just punt say, a project branch build for your higher priority build?

    While I can't put my finger on it, having two production CI servers has a "code smell" of its own.
blog comments powered by Disqus

Previous post: Links for 2010-02-24

Next post: Justifying Continuous Integration Expenditure