HJK’s Adam Jacob was recently interviewed on Technometria with Phil Windley.

Here’s the abstract:

In his recent presentation at the Web 2.0 Expo in San Francisco, Adam Jacob talked about why a start-up needs an automated infrastructure. He covered the components necessary for any automated infrastructure to be successful and also presented use-cases. Along with Jesse Robbins, Adam joins Phil and Scott to talk about the automated infrastructure process.

They also discuss how iClassify and Puppet are used, and review the steps to automated infrastructure.

Check out the podcast here.

I recently gave a talk at the Web 2.0 Expo giving an overview of how (and why) we build automated infrastructure.

I’ll be giving a more technical version of his talk (titled “Building an Automated Infrastructure”) at the Velocity Conference, June 23-24 down in Burlingame, CA.

Velocity is going to be the best place to send your Operations people this year: they’ll be able to get all the best advice from some of the best operations folks on the planet, all in one place. (There is a discount code at the end of the slide deck!)

Hope to see you there!

In John Willis’ response to Puppet, iLike and Infrastructure 2.0, he poses the question:

Maybe I should ask this one more time… and why are people paying in excess of 3 to 5 million a year for this kind of software?

I think the answer revolves around two things.

People don’t realize that manual systems administration is a problem until it’s too late.

By “people” I mean everyone involved… many excellent systems administrators just don’t see the need for it. (“Why would I spend two months building out a configuration management infrastructure when I could just spend a week building my current systems by hand?”)

The result of this is that automation comes in too late in the game, after you’ve already got a couple of hundred (or couple of thousand) systems being managed by hand, huge teams of systems administrators, and entire support structures built around doing it by hand. If you’re a growing startup, this winds up impacting your agility and stability as a platform, and you maybe start looking for tools like Puppet to help out. You talk to companies like mine to help, or at least talking to Luke’s own Reductive Labs to help get you running quickly.

If you are the Fortune 1000, it’s another thing altogether. You start to look for a way out, and someone tells you that a software package and 3 to 5 million in licensing and consulting will get you to the end game. In this case, it’s hard to be nimble enough to realize that Open Source provides a better path here.

You have to see it to believe it.

If you ask anyone working in technology whether they want the ability to rebuild the entire infrastructure, from bare metal, in an afternoon the answer is yes.

What follows, though, is “but our infrastructure can’t work that way because of X”. Or “your systems must not be very complex.” As of today, if we took a poll, the majority of enterprise grade IT environments don’t have this kind of functionality. They’ve never even seen it. So they believe you when they say it, but they don’t have any confidence that it will really work out that way for them.

When you realize you must have it, and you already believe that it’s impressively difficult to do, having someone say it will only work with a 3-5 million investment and a fleet of consultants actually affirms what you already believe.

Your own experience tells you it is that hard, and it ought to require all of that investment. You want to believe it.

Lets hope that the growing movement of practical, real, automated infrastructures being built on Open Source software starts to change that perception.

John Willis, one of the founders of Gulf Breeze Software (an IBM Tivoli consulting house,) met up with Luke Kanies and interviewed him about Puppet.

In addition to lots of insightful commentary on how Puppet is constructed, and a nice compare/contrast with how Tivoli is built (and you would be hard pressed to talk to someone who knows more about how Tivoli is built than John Willis, I expect,) there is also a section about iLike and HJK:

Uncomfortable with his recent celebrity at conferences, Luke told me that he has difficulty measuring his successes because he has his head so deep in the development and services of Puppet. One of his better success stories is with iLike.com, a website that allows users to download and share music. When iLike created one of the first Facebook applications, it grew from about ½ million users to over 6 million in a week. Luke, being the entrepreneur that he is, asked how iLike planned to manage that growth. He discovered that a services company in Seattle was managing iLike.com’s infrastructure build out using Puppet. In fact, one of the owners of that company told Luke that he makes a healthy living installing Puppet. Luke admitted that he felt feel pretty good to know that other people can make a living from his product.

Well, Luke, we feel pretty good about the fact that we make a living with your product too. :)

Puppet enables us to get a huge jump-start on building automated, scaleable, easy to manage infrastructures for our clients. Using puppet, we:

1. Automate as much of the routine systems administration tasks as possible.
2. Get 10 minute unattended build times from bare metal, most of which is data transfer. Puppet takes it the rest of the way, getting the machines ready to have applications deployed on them. It’s down to two and a half minutes for Xen.
3. Bootstrap our clients production environments while building their development environment. I can’t stress how cool this really is. Because we are expressing the infrastructure at a higher level, when it comes time to deploy your production systems, it’s really a non-event. We just roll out the Puppet Master and an Operating System auto-install environment, and it’s finished.
4. Cross-pollinate between clients with similar architectures. We work with several different shops using Ruby on Rails, all of whom have very similar infrastructure needs. By using Puppet in all of them, when we solve a problem for one client, we’ve effectively solved it for the others. I love being able to tell a client that we solved a problem for them, and all it’s going to cost is the time it takes for us to add the recipe.

Puppet, today, is a tool that is good enough to handle the vast majority of issues encountered in building scalable infrastructures. Even the places where it falls short are almost always just a matter of it being less elegant than it could be, and the entire community is working on making those parts better.

Thanks, Luke.

Apt is a wonderful tool, but it does have its quirks. One of those is that it likes to ask you interactive questions during package installation. This makes total sense when a human is doing apt-get install foobar, but it causes all sorts of consternation when you want to automatically install packages. (You do this all the time with Puppet, which I’ll talk more about at the end of this post.)

The answer to this problem is to pre-seed debconf with the correct answers to it’s questions. To do this, first you need to install the package by hand:

    
$ apt-get install libnss-ldap
    

You’ll need to provide answers to the questions it asks, which we’re going to re-use later as the basis for our seed file. Next, make sure you have debconf-utils installed, and grab the answers to your questions:

    
$ apt-get install debconf-utils # Only if you need it
$ debconf-get-selections | grep libnss-ldap
libnss-ldap     libnss-ldap/rootbindpw  password
libnss-ldap     libnss-ldap/bindpw      password
libnss-ldap     libnss-ldap/dblogin     boolean false
# Automatically update libnss-ldap's configuration file?
libnss-ldap     libnss-ldap/override    boolean false
libnss-ldap     shared/ldapns/base-dn   string   dc=example,dc=com
libnss-ldap     libnss-ldap/rootbinddn  string   cn=admin,dc=example,dc=com
libnss-ldap     shared/ldapns/ldap_version      select   3
libnss-ldap     libnss-ldap/binddn      string   cn=proxyuser,dc=example,dc=net
libnss-ldap     shared/ldapns/ldap-server       string   ldapi:///
libnss-ldap     libnss-ldap/nsswitch    note
libnss-ldap     libnss-ldap/confperm    boolean false
libnss-ldap     libnss-ldap/dbrootlogin boolean true
    

Take the output of that and stick it in a file, say libnss-ldap.seed. You can now use that file to pre-seed a new system with those answers using debconf-set-selections:

    
$ debconf-get-selections | grep libnss-ldap > /tmp/libnss-ldap.seed
$ debconf-set-selections /tmp/libnss-ldap.seed
    

Or use ssh to pre-seed a new box:

    
$ cat /tmp/libnss-ldap.seed | ssh somehost debconf-set-selections
    

If you are using Puppet (and if you’re not, you should be) to manage your systems, you can use a recipe like this to automatically install packages that require interaction:

    
define seed_package($ensure = latest) {
  $seedpath = "/var/cache/local/preseeding" 
  file { "$seedpath/$name.seed":
    source => "puppet://$puppet_server/seeds/$lsbdistcodename/$name.seed",
    mode => 0600,
    owner => root,
    group => root
  }
  package { $name:
    ensure => $ensure,
    responsefile => "$seedpath/$name.seed",
    require => File["$seedpath/$name.seed"],
  }
}

class foobar {
  seed_package { "libnss-ldap":
    ensure => latest
  }
}
    

Some other resources on ways to use pre-seeding to great effect:

• Automating new Debian installations with preseeding at Debian Administration
• Preseed entry at the Debian Wiki

Happy unattended package installation! :)

When using runit with Upstart in Ubuntu 7.04, you’re going to run in to a couple of problems. The first is that the package doesn’t install cleanly without the presence of /etc/inittab, which Upstart doesn’t need. You can fix that with a simple:

    
$ sudo touch /etc/inittab
    

Now, you need to actually add runit to Upstart. You do this by putting the following in /etc/event.d/runsvdir:

    
start on runlevel 2
start on runlevel 3
start on runlevel 4
start on runlevel 5
stop on shutdown
respawn
exec /usr/sbin/runsvdir-start
    

This will cause runsvdir-start to run during runlevel 2, 3, 4, and 5, stop when the system is going to be shutdown, and respawn if it goes away.

If you are on Ubuntu 6.10, the syntax is a bit different:

start on runlevel 2
start on runlevel 3
start on runlevel 4
start on runlevel 5
stop on shutdown
respawn /usr/sbin/runsvdir-start

Catalyst has long had FastCGI support built in, but all of the recipes are for the much older mod_fastcgi. As a Debian user, and fan of software that’s still maintained, I prefer mod_fcgid.

What follows is a simple Apache 2 virtual host for a Catalyst application, using mod_fcgid:

<VirtualHost *:80>
    ServerName www.example.com
    ServerAlias example.com
    ErrorLog logs/www.example.com.error_log
    TransferLog logs/www.example.com.access_log

    # This should point at your myapp/root
    DocumentRoot /srv/myapp/root
    Alias /static /srv/myapp/root/static

    <Location /static>
        SetHandler default-handler
    </Location>

    Alias / /srv/myapp/script/myapp_fastcgi.pl/

    <Location />
        Options ExecCGI
        Order allow,deny
        Allow from all
        AddHandler fcgid-script .pl
    </Location>
</VirtualHost>

The problem we are trying to solve is allowing select groups of users to both:

A) Log In to a Windows Server via Terminal Services as a regular user.

B) Log In to a Windows Server via Terminal Services as a local administrator.

The process should be completely automated once it is in place, so that the only requirements are that Computers be put in the proper OU, and new Users added to the appropriate groups.

The entire mechanism can be implemented with only Windows Group Policy. Here’s how.

Step -1. Make sure your domain is at the right Domain Functional Level

In order to get the nested group functionality to work, the Domain must be in Windows 2000 Native mode or above. You can set this by doing:

1. Start -> Administrative Tools -> Active Directory Domains and Trusts
2. Right click on your Domain, and select “Raise Domain Functional Level”

Assuming you only have Windows 2003 domain controllers, go ahead and set it at Windows 2003. It needs to be at least Windows 2000 Native.

Step 0. Create a more useful MMC Snap In.

The default snap-in’s for Windows make things difficult for handling Group Policy, since you so often need to move back and forth between the group policy itself and Active Directory. So our first step is to create a consolidated MMC view, so we can do both in one window.

1. Start -> Run -> mmc
2. File -> Add/Remove Snap-in
3. Click “Add”
4. Select “Active Directory Users and Computers”, click “Add”
5. Select “Group Poloci Management”, click “Add”
6. Click “Close”, then click “OK”
7. Expand the “Console Root” mini-window.
8. Select “File” -> “Save”, Click your Desktop, and call it GPMCAD

Now, any time you want to play with Group Policy, you can just double click the GPMCAD.msc on your desktop.

Step 1. Create the Computer OUs

Each class of machine (“Webserver”, “Database”) needs to have it’s own OU, which we will use for setting the proper Group Policy.

1. Expand “Active Directory Users and Computers”
2. Expand your Forest, and Navigate to the place you want to create new computer OUs. (“sea/computers” in our example)
3. Create a new OU for a class of machines. Right click on the parent container, select “New”, then “Organizational Unit”. Give it a name (“Webservers”,) then click “OK”.
4. Now, right click on the computers you want to have populate this OU, and select “Move”. Navigate to the new OU, and click “OK”.

Step 2. Create the User Groups

For each class of machine, we’ll create two groups: one for Administrators, and one for regular Users.

1. Go to the OU where you want the groups located (“sea/groups” in our example.)
2. Right click on the OU, select “New”, then “Group”
3. In group name, put “Webserver Users”. (Where “Webservers” is our example group name; yours should match whatever the Computer OU you created before was.)
4. Under “Group Scope”, select “Universal”, and click “OK”
5. Right click on your new group, and select “Properties”
6. Click the “Members” tab, click “Add”, and type in the users you want to have non-administrator access to this host. When you’re done, click “OK”. (How to use this dialog is beyond the scope of this document, I think. ;)
7. Once you have all the users, click “OK”.

Now, repeat these steps, but call the group “Webserver Admins”, and ensure you only populate it with the users who should have Administrator privileges on this host.

Step 3. Create the new Group Policy

Now that we have our systems in a new OU, and our Authentication groups created, we need to create the group policy to apply our settings.

1. In the MMC, open “Group Policy Management” -> “Forest: yourdom.com” -> Domains -> “yourdom.com”. This will reflect the root of the Active Directory tree we were working with in Step 2.
2. Navigate to the Computer OU we created, right click, and select “Create and Link a GPO Here…”
3. Give it a name like “Webserver Permissions”.
4. Expand the OU we just right clicked on, exposing your new GPO. Right click on it, and select “Edit”.

Step 4. Edit the new Group Policy.

You should have just popped up the “Group Policy Object Editor.” First, lets set some limits on who can log in via the Console and Terminal Services.

1. Navigate to “Computer Configuration” -> “Windows Settings” -> “Security Settings” -> “Local Policies” -> “User Rights Assignment”.
2. Double click on “Allow log on locally”. Click “Define these policy settings”.
3. Click “Add User or Group”. Type “Administrators”, adding the local Administrators group.
4. Click “Add User or Group”. Hit “Browse”, and add “Domain Admins”, “Webserver Admins,” and “Webserver Users”. Click “OK”.
5. Now, click “OK” again to close this policy.

Repeat the above steps, but this time, double click on “Allow log on through Terminal Services.”

Now, you need to populate the local Administrators and Remote Desktop Users groups.

1. Navigate to “Computer Configuration” -> “Security Settings” -> “Restricted Groups”. Right click on “Restricted Groups”, and select “Add Group”.
2. Click “Browse”, and find your new “Webserver Admins” group, and click “OK.”
3. Click the “Add…” button next to “This group is a member of:”, and type “Administrators”. Click “OK”
4. Click “OK” to close the dialog.

Now, repeat the above, but for the “Webserver Users” group. When the time comes to set what group it should be a member of, make it “Remote Desktop Users”.

Close the Group Policy Object Editor dialog.

Step 5. Refresh the Group Policy on the clients, and test.

Depending on how often your Group Policy refreshes, you may have to wait a while. You can force the issue by running gpupdate /FORCE on the computers you want to update. Do that now, so we can make sure everything is working.

Now, try terminal serving as a User who is a member of the “Webserver Users” group. You should have access, but not Administrative privileges.

Next, try and log in as a member of the “Webserver Admins” group. You should have full administrative rights to the host.

Now it’s a Domain Administrators turn. Again, they should have full Administrative access.

Finally, try and log in as a user who isn’t in any of the above groups. You should get the “To log on to this remote computer..” speech.

Step 6. Rinse and Repeat.

For each class of systems you want to manage, repeat the above process. When you are done, you should have a super easy to manage domain structure, and getting the access privileges right for new hosts is as easy as moving them to the right place in Active Directory.

We've recently run in to a situation where we had a fairly large number of legitimate messages bounce due to a misconfigured MTA. Once we fixed the MTA, we needed to get all of those messages back out the door again. It was a job for CPAN.

    

use strict;
use warnings;        
use Mail::Box::Manager;
use Email::MIME;
use Email::Send;

foreach my $mbox (@ARGV) {
    my $mgr = Mail::Box::Manager->new;
    my $folder = $mgr->open(folder => $mbox);
    my $sender = Email::Send->new({mailer => 'SMTP'});
    $sender->mailer_args([Host => 'localhost']);

    foreach my $msg ($folder->messages) {
        PART: foreach my $part ($msg->parts('ALL')) {  
            my $body =  $part->decoded;
            if ($body =~ /This is the mail system at/) {
                next PART;
            } elsif ($part->decoded =~ /Action: failed/) {
                next PART;
            } else {
                my $email = Email::MIME->new(\$part->decoded);
                print "Sending to: " . $email->header("To") . "\n";
                $sender->send($email);
            }
        }
    }
}

The code uses a couple of different modules:

1. Mail::Box::Manager: handles opening a mailbox (in pretty much every common format) and extracting messages.
2. Email::MIME: lets you construct a new email from a MIME encoded message.
3. Email::Send: takes our Email::MIME object and sends it out (in this case, via SMTP, but you can choose your mechanism.)

So should you ever find yourself with a couple thousand erroneously bounced messages, don't fret. ;)

Avvo just unveiled their new platform for helping consumers find lawyers. We helped design and implement their production systems infrastructure, and we're pretty proud of how it turned out. Some of the press they received this morning include the The Seattle Times, John Cook's Venture Blog, The Seattle Post Intelligencer and CNET. (Here is a Google news search for Avvo; more are appearing all the time.) Avvo's got a great team, a cool product, and we couldn't be happier for them. Congratulations!