Frankly as a Web ops person I’ve had a lot more luck getting developers to want to collaborate than our other Infrastructure teams. Heck, recently I got the Infrastructure director to agree that we need an email list with all the various teams on it that people need to send change notices to so that everyone knows what’s going on in the environment – and most of the teams refuse to do it. “Why should I need to tell anyone what I’m doing? Why do you need to know? And I might hear about changes I don’t care about! Never!”

@ej – glad you got outta town okay!

This post is long overdue, as the idea struck me when dicussing with Lefred while preparing his Fosdem talk on Maintaining too big tables

I got triggered finishing this post by Mr BuidlDoctor

Fred has been struggling with a typical DevOps problem…

We went whole hog and faced pitchforks and torches (I’ll admit to carrying some of them as well).

In hindsight, adopting a smoother transition would have been better. I look at things like this now with an eye toward evolution, NOT revolution.

You might want to add our Parabuild to your list. Parabuild now provides Continuous Integration for Git.

Releng (here at least) isn’t versed in byzantine (but necessary) security requirements, they’re versed in codeline management, CI best practices (and the tools to support them), etc.

We us Solaris machines racked/stored in the local datacenter. Releng is NOT hands on with the racking/powering/os patching/etc. In fact, it’s only recently we’ve been added to the list of users who vet what changes are to be installed as part of any OS patching cycle.

We’re not even allowed to manage our own packages or install them.

So yeah, while it’s cheap to buy the hardware (machine or drive or ram or cpu) upgrades, it’s the squishy costs associated with bringing said upgrade into a build cluster.

As far as dev waiting for CI, originally, one build here took 3+ hours on the build machine and 45 min locally. I worked with a Sr. dev guy here and we lead a charge to refactor the code and modularize it to death. Now, fewer things cycle and the build times are as little as 7 minutes for dev locally (and generally shorter) and 10 – 12 min on our build machines (including a rm -rf and a force sync). Times can creep a bit higher on Mondays we have a hudson job to remove local private repositories, but that doesn’t add much time.

Maybe if you find dev is sitting idle (or find yourself branching libA and appB ALL the time), think about shrinking your codebase into more bite size pieces and refactoring. That process helped us a TON.