Dustin Collins


Read this first

DevOps: A House Divided

What is DevOps? DevOps is not about tools. It is about culture, enabling cross-functional teams to quickly deliver value in a sustained matter. It is hard to define DevOps, but blah blah alignment blah success blah blah joy. Three paragraphs later, no takeaways. How many posts have you read about DevOps that follow this format? Promoting this vague clickbait as guidance is only doing ourselves a disservice. We’re patting ourselves on the back so hard we fall over.

How did we end up here? I think it’s because we’re still not having real conversations between business units. We find ourselves now with two camps in the DevOps community: technical and non-technical. Either side shifts the conversation to where they feel most comfortable: the domain they work in and understand. Technical folks talk about tooling. Non-technical folks talk about culture. The divide here is artificial, but it...

Continue reading →

Integrating serverspec into your infrastructure testing workflow

I’m doing a presentation on how to use serverspec to create a tight feedback loop for infrastructure coding at the next Boston DevOps meetup Nov 19th.

Slides are here: http://slides.com/dustinmm80/serverspec

Continue reading →

Detecting Chef upload failures with Jenkins

Our Chef workflow at Essess looks like this:

  • A developer updates a cookbook and pushes changes to git
  • Jenkins is notified and runs the tests for the cookbook
  • On successful build, it triggers a job to push the updated cookbook to the Chef server.

We’re using hosted Chef, so we assumed this would be seamless. Nope. A developer would push, the jobs would trigger and Jenkins would report back green. Later on they would notice that the cookbook was still pinned to an old version in the environment. Returning to Jenkins, we see something like this in the job’s console output log:

Jenkins log

Well, that didn’t go as expected. You would think that if there was an upload failure we would get a non-0 exit code and Jenkins would report the build as failed. Not so. We use knife cookbook upload --all --force --environment ${env} to upload all cookbooks installed via berkshelf. env here is a Jenkins...

Continue reading →

Grabbing AWS credentials with bash

Let’s say you’re running on AWS and using IAM to assign your instances AWS keys dynamically. First off, if you’re not doing this you should. Why?

  • Rotating credentials in from IAM is much more secure than sharing your AWS account keys across your infrastructure. If someone gets your keys and you’re not using IAM, you’re screwed. If you’re rotating your keys via IAM you shouldn’t really ever have to know what they are in the first place.
  • By assigning IAM roles to your instances, you can control the granularity of that role’s permissions. You want to only allow your web servers access to one subfolder of a S3 bucket? Easy, through IAM.

Okay, so we’re all on board: If we’re using AWS we should use IAM roles to provide rotating access keys to our instances.

But what happens when you need those keys? I ran across this problem running unit tests on Jenkins for a Chef cookbook that uses...

Continue reading →

Infrastructure with Python

Like Python? Into DevOps?

Here are some tools I use make my life a little easier.


THE command line interface if you’re using AWS. Really nice documentation on how to talk to the different AWS services. I use this a lot as a glue library. For example, once Jenkins runs tests on a project, we tar.gz an artifact and use awscli to upload it to S3.


If you’re using AWS and need to get state or resources at runtime boto is what you want. The API is large, but well documented and composed.

botocore is a smaller low-level alternative, but I don’t care much about size when boto’s docs are good. botocore is the foundation for awscli.


Fabric is my go-to tool for remote execution. PyChef fetches the hosts I need and passes them to Fabric, which takes care of running any commands. This is a popular lib that has been around for a while; a lot of well-tested features.


A web...

Continue reading →

DevOps Resources

There are a lot of people on the web talking about every aspect of the DevOps movement, but it can be hard to find them. I’ve added some resources I use to a Github repo here. If you use any resources not on this list, feel free to open a pull request and we can build a great list of resources together.

Github: dustinmm80/devops_resources

My starter list:


Arrested DevOps

DevOps Cafe

Puppet Labs Podcast

The Cloudcast

The Food Fight Show

The Ship Show


Dzone DevOps zone



DevOps on Quora

LinkedIn DevOps group


Blogs (Orgs)

codeascraft (etsy)

Opscode Chef


Blogs (Personal)

Andrew Clay Shafer

Bryan Berry

Chad Fowler

Joe Hirn

John Allspaw

John E. Vincent

Len Lagestee

Mike Fiedler

Nathen Harvey

Patrick Debois

Tom Duffield


devopsdays on Vimeo


DevOps Weekly

Continue reading →

Chef integration testing with serverspec

Most resources discussing testing with Chef deal with unit testing. Unit testing your Chef recipes is a very good idea, but will only get you so far. You can have 100s of passing unit tests and your Chef converges will still fail. Integration testing can greatly increase the confidence you have in your Chef code. In this post, I’ll walk through how we’re currently doing it at Essess.

First, a clarification on Chef testing terms

I didn’t really understand how chefspec unit tests fit into the Chef testing process until I watched Seth Vargo’s talk at a June meetup on it. If you use Chef and haven’t watched it yet, check it out. He broke it down like this:

Unit testing in Chef is intention testing. When you run your chefspec tests, you are only testing whether you are telling Chef to do what you expected. When is this useful? When you have some complicated logic in your recipe or if you...

Continue reading →

Testing Python command line apps

Recently I wrote a Python command line app and had some trouble finding a good example of how to test it using best practices. Some ideas I ran across in my search:

  • Use a 3rd party CLI-testing solution like ScriptTest or a framework that includes testing like pyCLI.

I write my other tests with unittest and run them with nose. I don’t think there’s anything special about a CLI that requires the rest of my team to learn YATT (yet another testing tool).

  • Use sys.argv, subprocess or some combination of both.

This feels very hacky to me. Writing a Python CLI using sys.argv is a little silly considering the more robust options out there. argparse is in the standard library. 3rd party tools like clint and cliff are frameworks that seem promising. A testing solution that plays to the strength of these tools is ideal.

Personally, I use argparse. It does everything I need and its API is...

Continue reading →

Multi-VM Vagrant the DRY way

So Vagrant is great. If you’re not using it yet, you should check it out. It allows you to develop your applications in the same environment they’ll be running in.

I’ve been using Vagrant and Chef for a little over a year now, but I hadn’t used Ruby much before. I’ve written a lot of Chef cookbooks and Vagrantfiles since then. Writing a multi-VM Vagrantfile is much easier if you lean on Ruby to help you keep them shorter and more readable.

Here’s an example of a Vagrantfile with 3 Ubuntu LTS 12.04 machines:

  • clean - A base box, with nothing installed. For experimentation.
  • mongo - A MongoDB box, provisioned with chef-solo.
  • jenkins - A Jenkins CI box, provisioned with chef-solo.

There are 3 parts to the Vagrantfile.

1: Box definitions

Boxes can be configured with several options, most of them optional. The schema is defined at the top of the file.

=begin Example box JSON schema { :name...

Continue reading →