I recently gave a talk at the Web 2.0 Expo giving an overview of how (and why) we build automated infrastructure.

I’ll be giving a more technical version of his talk (titled “Building an Automated Infrastructure”) at the Velocity Conference, June 23-24 down in Burlingame, CA.

Velocity is going to be the best place to send your Operations people this year: they’ll be able to get all the best advice from some of the best operations folks on the planet, all in one place. (There is a discount code at the end of the slide deck!)

Hope to see you there!

In John Willis’ response to Puppet, iLike and Infrastructure 2.0, he poses the question:

Maybe I should ask this one more time… and why are people paying in excess of 3 to 5 million a year for this kind of software?

I think the answer revolves around two things.

People don’t realize that manual systems administration is a problem until it’s too late.

By “people” I mean everyone involved… many excellent systems administrators just don’t see the need for it. (“Why would I spend two months building out a configuration management infrastructure when I could just spend a week building my current systems by hand?”)

The result of this is that automation comes in too late in the game, after you’ve already got a couple of hundred (or couple of thousand) systems being managed by hand, huge teams of systems administrators, and entire support structures built around doing it by hand. If you’re a growing startup, this winds up impacting your agility and stability as a platform, and you maybe start looking for tools like Puppet to help out. You talk to companies like mine to help, or at least talking to Luke’s own Reductive Labs to help get you running quickly.

If you are the Fortune 1000, it’s another thing altogether. You start to look for a way out, and someone tells you that a software package and 3 to 5 million in licensing and consulting will get you to the end game. In this case, it’s hard to be nimble enough to realize that Open Source provides a better path here.

You have to see it to believe it.

If you ask anyone working in technology whether they want the ability to rebuild the entire infrastructure, from bare metal, in an afternoon the answer is yes.

What follows, though, is “but our infrastructure can’t work that way because of X”. Or “your systems must not be very complex.” As of today, if we took a poll, the majority of enterprise grade IT environments don’t have this kind of functionality. They’ve never even seen it. So they believe you when they say it, but they don’t have any confidence that it will really work out that way for them.

When you realize you must have it, and you already believe that it’s impressively difficult to do, having someone say it will only work with a 3-5 million investment and a fleet of consultants actually affirms what you already believe.

Your own experience tells you it is that hard, and it ought to require all of that investment. You want to believe it.

Lets hope that the growing movement of practical, real, automated infrastructures being built on Open Source software starts to change that perception.

John Willis, one of the founders of Gulf Breeze Software (an IBM Tivoli consulting house,) met up with Luke Kanies and interviewed him about Puppet.

In addition to lots of insightful commentary on how Puppet is constructed, and a nice compare/contrast with how Tivoli is built (and you would be hard pressed to talk to someone who knows more about how Tivoli is built than John Willis, I expect,) there is also a section about iLike and HJK:

Uncomfortable with his recent celebrity at conferences, Luke told me that he has difficulty measuring his successes because he has his head so deep in the development and services of Puppet. One of his better success stories is with iLike.com, a website that allows users to download and share music. When iLike created one of the first Facebook applications, it grew from about ½ million users to over 6 million in a week. Luke, being the entrepreneur that he is, asked how iLike planned to manage that growth. He discovered that a services company in Seattle was managing iLike.com’s infrastructure build out using Puppet. In fact, one of the owners of that company told Luke that he makes a healthy living installing Puppet. Luke admitted that he felt feel pretty good to know that other people can make a living from his product.

Well, Luke, we feel pretty good about the fact that we make a living with your product too. :)

Puppet enables us to get a huge jump-start on building automated, scaleable, easy to manage infrastructures for our clients. Using puppet, we:

1. Automate as much of the routine systems administration tasks as possible.
2. Get 10 minute unattended build times from bare metal, most of which is data transfer. Puppet takes it the rest of the way, getting the machines ready to have applications deployed on them. It’s down to two and a half minutes for Xen.
3. Bootstrap our clients production environments while building their development environment. I can’t stress how cool this really is. Because we are expressing the infrastructure at a higher level, when it comes time to deploy your production systems, it’s really a non-event. We just roll out the Puppet Master and an Operating System auto-install environment, and it’s finished.
4. Cross-pollinate between clients with similar architectures. We work with several different shops using Ruby on Rails, all of whom have very similar infrastructure needs. By using Puppet in all of them, when we solve a problem for one client, we’ve effectively solved it for the others. I love being able to tell a client that we solved a problem for them, and all it’s going to cost is the time it takes for us to add the recipe.

Puppet, today, is a tool that is good enough to handle the vast majority of issues encountered in building scalable infrastructures. Even the places where it falls short are almost always just a matter of it being less elegant than it could be, and the entire community is working on making those parts better.

Thanks, Luke.

Apt is a wonderful tool, but it does have its quirks. One of those is that it likes to ask you interactive questions during package installation. This makes total sense when a human is doing apt-get install foobar, but it causes all sorts of consternation when you want to automatically install packages. (You do this all the time with Puppet, which I’ll talk more about at the end of this post.)

The answer to this problem is to pre-seed debconf with the correct answers to it’s questions. To do this, first you need to install the package by hand:

    
$ apt-get install libnss-ldap
    

You’ll need to provide answers to the questions it asks, which we’re going to re-use later as the basis for our seed file. Next, make sure you have debconf-utils installed, and grab the answers to your questions:

    
$ apt-get install debconf-utils # Only if you need it
$ debconf-get-selections | grep libnss-ldap
libnss-ldap     libnss-ldap/rootbindpw  password
libnss-ldap     libnss-ldap/bindpw      password
libnss-ldap     libnss-ldap/dblogin     boolean false
# Automatically update libnss-ldap's configuration file?
libnss-ldap     libnss-ldap/override    boolean false
libnss-ldap     shared/ldapns/base-dn   string   dc=example,dc=com
libnss-ldap     libnss-ldap/rootbinddn  string   cn=admin,dc=example,dc=com
libnss-ldap     shared/ldapns/ldap_version      select   3
libnss-ldap     libnss-ldap/binddn      string   cn=proxyuser,dc=example,dc=net
libnss-ldap     shared/ldapns/ldap-server       string   ldapi:///
libnss-ldap     libnss-ldap/nsswitch    note
libnss-ldap     libnss-ldap/confperm    boolean false
libnss-ldap     libnss-ldap/dbrootlogin boolean true
    

Take the output of that and stick it in a file, say libnss-ldap.seed. You can now use that file to pre-seed a new system with those answers using debconf-set-selections:

    
$ debconf-get-selections | grep libnss-ldap > /tmp/libnss-ldap.seed
$ debconf-set-selections /tmp/libnss-ldap.seed
    

Or use ssh to pre-seed a new box:

    
$ cat /tmp/libnss-ldap.seed | ssh somehost debconf-set-selections
    

If you are using Puppet (and if you’re not, you should be) to manage your systems, you can use a recipe like this to automatically install packages that require interaction:

    
define seed_package($ensure = latest) {
  $seedpath = "/var/cache/local/preseeding" 
  file { "$seedpath/$name.seed":
    source => "puppet://$puppet_server/seeds/$lsbdistcodename/$name.seed",
    mode => 0600,
    owner => root,
    group => root
  }
  package { $name:
    ensure => $ensure,
    responsefile => "$seedpath/$name.seed",
    require => File["$seedpath/$name.seed"],
  }
}

class foobar {
  seed_package { "libnss-ldap":
    ensure => latest
  }
}
    

Some other resources on ways to use pre-seeding to great effect:

• Automating new Debian installations with preseeding at Debian Administration
• Preseed entry at the Debian Wiki

Happy unattended package installation! :)