Releng (here at least) isn’t versed in byzantine (but necessary) security requirements, they’re versed in codeline management, CI best practices (and the tools to support them), etc.

We us Solaris machines racked/stored in the local datacenter. Releng is NOT hands on with the racking/powering/os patching/etc. In fact, it’s only recently we’ve been added to the list of users who vet what changes are to be installed as part of any OS patching cycle.

We’re not even allowed to manage our own packages or install them.

So yeah, while it’s cheap to buy the hardware (machine or drive or ram or cpu) upgrades, it’s the squishy costs associated with bringing said upgrade into a build cluster.

As far as dev waiting for CI, originally, one build here took 3+ hours on the build machine and 45 min locally. I worked with a Sr. dev guy here and we lead a charge to refactor the code and modularize it to death. Now, fewer things cycle and the build times are as little as 7 minutes for dev locally (and generally shorter) and 10 – 12 min on our build machines (including a rm -rf and a force sync). Times can creep a bit higher on Mondays we have a hudson job to remove local private repositories, but that doesn’t add much time.

Maybe if you find dev is sitting idle (or find yourself branching libA and appB ALL the time), think about shrinking your codebase into more bite size pieces and refactoring. That process helped us a TON.

There are other problems around agile in the enterprise. Its never just the cost of a server but all hell that comes along with it. The drive to virtualisation and cloud computing is still in its infancy in many cases and the early efforts may not be well implemented which compounds problems with progress, and hence my comment.

I can feel a bumpy road ahead before we reach the highway…