The guardian.co.uk Build Radiator - a guest post by Michael Brunton-Spall
•
Michael Brunton-Spall and I both worked on the Guardian 'R2' project - in fact, Michael is still there. Last year, we worked together a couple of times on that project. In this guest post he discusses the thing I'm most pleased that we worked on together - an Extreme Feedback device known at the Guardian as the Build Radiator.
Q: Who are you?
Hi there. I'm Michael: developer, Guardianista and build radiator fan.
Q: So what is a build radiator?
A build radiator is anything that is able to radiate information about the current status of the build to all the team members. Examples might include lava lamps, strings of fairy lights or as in our case, a computer with a large display attached to it.
Q: Surely developers know what's going on with their builds?
Developers are the only people to have ccTray, whereas a big TV on the wall is visible to everybody on the project. CCtray also provides limited information: is the build Red or Green? Our TV is able to display a lot more analog information.
Q: Okay. Why would other people in the team want to know about the Cruise build?
Of course business owners want to see whether the build is green, building or broken. If they see it's broken a lot, they get feedback that something is wrong with procedures. The people preparing to launch the software want to see what the latest build is doing. But our system does more than that: QAs want to see which builds are deployed to which environments. Everybody should be interested in the live status of the production system, and so on.
Q: I think I get it. What does your radiator display?
We have a number of modules, and the display rotates between different modules. Firstly it shows the always shows the status of different builds, both trunk and release builds.
It also displays:
- a live view of the guardian.co.uk front page,
- a list of what software build version is deployed to which QA environment
- a 'message of the day' pulled from a Wiki page that anyone can edit
- a graph of build duration from CruiseControl for the last few days
- current status of our live servers
- a log of the last 5 things said in our project IRC room.
Q: So how did you go about writing this radiator?
Firstly Julian, one of our ThoughtWorkers at the time, wrote a spike. For those of you unfamiliar with XP lingo, that is an ugly hacked together proof of concept (Julian's words, not mine!)
Once the concept was approved we scheduled a pair from our project to develop the code. We spoke with the product owners. For a radiator, the product owners were the lead business analyst, the technical architect and the team leads. We got ideas from each stakeholder as to what they wished could be shown if we had an infinite amount of time. We then got all the stakeholders to agree a priority order for the tasks.
The requirements of the project were to have a display that: updated fairly often; was visible from across the room and radiated that information out to people, so was visible in their peripheral vision. We also had some specific information requests, things that people wanted to know, examples included a customisable message, a list of currently deployed versions and the build status indicators themselves.
[Julian: We had a request for live cricket scores, which was sadly denied]
Q: Isn't that a lot of work for non-production code?
You might think so, but actually, we spent only about 2 or 3 days of development time on the first iteration. When the amazingly successful results could be seen, the team managers could see that the benefits were worth it.
Think about it: we have approximately 80 people on our team. If we save them 15 minutes a day each in finding information, then we made back approximately 20 man-hours of development in the first day of use alone, if we even use the conservative figure of saving the developers only 5 minutes each a day, we still save 6 hours 40 minutes of man-hours a day of running.
Q: What did you implement it in?
We decided to write it in Ruby. I can't remember the exact reason why, I know that we wanted a fast to develop and deploy language, and I think we knew that with a limited amount of time, we had to get as much value as possible. This was my first Ruby on Rails project, and I found Ruby to be very easy to pickup and run with.
Q: Yes, but will it scale?
Why do you need it to scale? People talk about the scalability of Rails, but in terms of a build radiator, it needs to run on a single machine, 24 hours a day. We don't have several million visitors to its website every day. If we did, we may have to rethink the way that we did it. To be honest, our code would probably scale to some degree. We used the rails methodology of decoupling the views from the models (mostly), so for many modules the number of viewers doesn't matter. Some models get information from our servers on request, which would overload our backend servers (especially cruise control, the poor thing) easily. But we can cache that if it ever becomes a problem.
Q: Isn't Rails a web app framework? What made the image on the TV?
That's an interesting question. We got a PC, connected to the TV, running Ubuntu linux. It runs the Rails server locally, but also runs a fullscreen version of Firefox. We figured that it was the easiest way of displaying the data we needed without writing our own fullscreen application.
Q: What happened after the first iteration?
Well, because we arranged to get the hardware before we started, the developers in the room noticed what we had started, and started coming up with ideas of their own. We had a large list of possible ideas, which we put on hold for a month or two. The stakeholders decided that a second iteration was needed after a few issues that we could have spotted if we had implemented some of the ideas. So we went back to work, and added a graph of current build times, a basic burndown chart and the facility to display some page rendering times of the guardian.co.uk site.
At this point we needed for the first time to be able to store data. Previously the
app was stateless: it queried the data it needed as it needed it. So we had to add the Rails database and model work a bit more. We also got some help from a rails developer who refactored the code into the models from our overweight controllers (our bad!)
Q: What did the team think? Was it a success?
Unimaginably so. The first message was updated within 10 minutes of going live to let people know that a development environment was going down for an upgrade. These days developers don't question the radiator, it's just part of the furniture. If it was gone, many of our processes would not work. In daily standups, people often refer to the current build, or deployed version as per the radiator. I see developers turning to look at the radiator all the time, and the facility to see the live status of the servers is very helpful in ensuring that developers notice when the site goes down. We've had requests by developers, QA's, business analysts and our systems operations team to add things to the radiator.
Q: What are your favourite bits of your build radiator code? As a development shop we develop in Java, using Spring, Hibernate and Velocity as core technologies, we don't use ruby or rails for anything inhouse. We are lucky to have ThoughtWorks as a consultancy, so we have a few Ruby programmers around, but most of our guardian.co.uk programmers don't do Ruby. It would have made more sense to write it in Java, with Spring, and we could have used it as an opportunity to try out some features of the new Spring release, or to try JSP instead of our present view language. However, I think that the time taken to get the project started would have been at least a week instead of two days. Whether the technical leads would have gone for that amount of time without seeing the benefit? Who can say? Michael, thank you. It was a pleasure.
I think the decision to display the build status at all times, and the other page in a rotating iframe beside the build status. The build status is used so much, and is so important that it must be on every page. Using an iframe to show the other module has meant that the list of the modules is simply a list of urls, mostly relative. But that has meant that adding a new module to display can be added by either adding a new rails module, or a remote URL, allowing us to tie in with other development efforts such as the page rendering stats, or a server status perl script that our operations team has.
Q: What is your least favourite bit of your build radiator code?