spending money on continuous integration hardware

Banos commented on my last post:

So why, oh why, oh why is it so difficult to get an additional server? Has anyone come up with a formula to produce some numbers for the bean counters to justify this already?

I propose this is an endemic problem that the guys on the ground give up fighting to resolve because there is no budget for more servers.

Here’s a start: by measuring all the time that developers waste waiting for Continuous Integration. Paul Julius writes about the cup of coffee metric. If you can measure that time, there’s your formula:

hours wasted per dev * number of developers * average hourly rate * 22 working days a month (on average) = monthly cost of not having the right hardware.

A local example: 30 minutes a day, for six developers, average cost £50 gives £3,300 a month. That’s not even a high rate for developers, or much time spent. Of course, your organisation may be unable to budget any more on infrastructure. At the same time as it pays developers to watch the Continuous Integration machine. That’s a different problem than managers understanding the true cost of something.

Thanks for the interesting comment, Banos. Has anyone tried this kind of approach and won? Tell us!

(photo via See-Ming Lee)

Share with the group:

Continuous Integration is an attitude Question: How do we prevent integration pain? Answer: Continuous...
Continuous Integration: GitHub announce CI Joe Thought the world had enough Continuous Integration servers? Guess...
Outsourcing Continuous Integration Outsourcing Continuous Integration isn’t a new idea, but we’re...
Continuous Integration Quick Reference Continuous Integration is frequently written about. That’s all good....
18 Continuous Integration Questions Top .NET dog Roy Osherove just blogged some good...

Related posts brought to you by Yet Another Related Posts Plugin.

Tagged as: managing upward

EJC

Sorry - I want to clarify one point - I don't think releng needs to be hands on with the set up of the bare machine or patching the OS etc., but I do think they should be autonomous when it comes to managing the software they care about.

Releng (here at least) isn't versed in byzantine (but necessary) security requirements, they're versed in codeline management, CI best practices (and the tools to support them), etc.

EJC

@Banos - Spot on!

We us Solaris machines racked/stored in the local datacenter. Releng is NOT hands on with the racking/powering/os patching/etc. In fact, it's only recently we've been added to the list of users who vet what changes are to be installed as part of any OS patching cycle.

We're not even allowed to manage our own packages or install them.

So yeah, while it's cheap to buy the hardware (machine or drive or ram or cpu) upgrades, it's the squishy costs associated with bringing said upgrade into a build cluster.

As far as dev waiting for CI, originally, one build here took 3+ hours on the build machine and 45 min locally. I worked with a Sr. dev guy here and we lead a charge to refactor the code and modularize it to death. Now, fewer things cycle and the build times are as little as 7 minutes for dev locally (and generally shorter) and 10 - 12 min on our build machines (including a rm -rf and a force sync). Times can creep a bit higher on Mondays we have a hudson job to remove local private repositories, but that doesn't add much time.

Maybe if you find dev is sitting idle (or find yourself branching libA and appB ALL the time), think about shrinking your codebase into more bite size pieces and refactoring. That process helped us a TON.

Banos

That was a bit of a tongue in cheek comment originally, but it is valid.

There are other problems around agile in the enterprise. Its never just the cost of a server but all hell that comes along with it. The drive to virtualisation and cloud computing is still in its infancy in many cases and the early efforts may not be well implemented which compounds problems with progress, and hence my comment.

I can feel a bumpy road ahead before we reach the highway...

Adam Leggett

This is why I firmly believe that, in spite of the opinion expressed in this previous post, the cloud presents a great business case for using CI at scale. Using computing resource on-demand, only when you need it, at a $ price per minute, resonates extremely well with those who pay the CI bills. Our approach takes that even further by rolling it all up into a flat monthly fee - and bean counters really like that type of cost model IMHO.

Douglas Squirrel

Last autumn we made up a careful case showing all the developer time wasted waiting for builds. Despite tough economic times, our case was convincing enough to get us additional CI kit - not fancy VM servers, but still enough to help us substantially drop the build time. Maybe the Doctor will ask me to do a guest post with more info.

Justifying Continuous Integration Expenditure

Meta

Categories

Justifying Continuous Integration Expenditure

Blogroll

Meta

Categories