Our Chef workflow at Essess looks like this:

• A developer updates a cookbook and pushes changes to git
• Jenkins is notified and runs the tests for the cookbook
• On successful build, it triggers a job to push the updated cookbook to the Chef server.

We’re using hosted Chef, so we assumed this would be seamless. Nope. A developer would push, the jobs would trigger and Jenkins would report back green. Later on they would notice that the cookbook was still pinned to an old version in the environment. Returning to Jenkins, we see something like this in the job’s console output log:

Well, that didn’t go as expected. You would think that if there was an upload failure we would get a non-0 exit code and Jenkins would report the build as failed. Not so. We use knife cookbook upload --all --force --environment ${env} to upload all cookbooks installed via berkshelf. env here is a Jenkins...

Let’s say you’re running on AWS and using IAM to assign your instances AWS keys dynamically. First off, if you’re not doing this you should. Why?

• Rotating credentials in from IAM is much more secure than sharing your AWS account keys across your infrastructure. If someone gets your keys and you’re not using IAM, you’re screwed. If you’re rotating your keys via IAM you shouldn’t really ever have to know what they are in the first place.
• By assigning IAM roles to your instances, you can control the granularity of that role’s permissions. You want to only allow your web servers access to one subfolder of a S3 bucket? Easy, through IAM.

Okay, so we’re all on board: If we’re using AWS we should use IAM roles to provide rotating access keys to our instances.

But what happens when you need those keys? I ran across this problem running unit tests on Jenkins for a Chef cookbook that uses...

Like Python? Into DevOps?

Here are some tools I use make my life a little easier.

awscli

THE command line interface if you’re using AWS. Really nice documentation on how to talk to the different AWS services. I use this a lot as a glue library. For example, once Jenkins runs tests on a project, we tar.gz an artifact and use awscli to upload it to S3.

boto

If you’re using AWS and need to get state or resources at runtime boto is what you want. The API is large, but well documented and composed.

botocore is a smaller low-level alternative, but I don’t care much about size when boto’s docs are good. botocore is the foundation for awscli.

Fabric

Fabric is my go-to tool for remote execution. PyChef fetches the hosts I need and passes them to Fabric, which takes care of running any commands. This is a popular lib that has been around for a while; a lot of well-tested features.

Flask

A web...

There are a lot of people on the web talking about every aspect of the DevOps movement, but it can be hard to find them. I’ve added some resources I use to a Github repo here. If you use any resources not on this list, feel free to open a pull request and we can build a great list of resources together.

Github: dustinmm80/devops_resources

My starter list:

Podcasts

Arrested DevOps

DevOps Cafe

Puppet Labs Podcast

The Cloudcast

The Food Fight Show

The Ship Show

Aggregators/Q&A

Dzone DevOps zone

dev2ops

devops.com

DevOps on Quora

LinkedIn DevOps group

/r/devops

Blogs (Orgs)

codeascraft (etsy)

Opscode Chef

Puppet

Blogs (Personal)

Andrew Clay Shafer

Bryan Berry

Chad Fowler

Joe Hirn

John Allspaw

John E. Vincent

Len Lagestee

Mike Fiedler

Nathen Harvey

Patrick Debois

Tom Duffield

Videos

devopsdays on Vimeo

Newsletters

DevOps Weekly

Most resources discussing testing with Chef deal with unit testing. Unit testing your Chef recipes is a very good idea, but will only get you so far. You can have 100s of passing unit tests and your Chef converges will still fail. Integration testing can greatly increase the confidence you have in your Chef code. In this post, I’ll walk through how we’re currently doing it at Essess.

First, a clarification on Chef testing terms

I didn’t really understand how chefspec unit tests fit into the Chef testing process until I watched Seth Vargo’s talk at a June meetup on it. If you use Chef and haven’t watched it yet, check it out. He broke it down like this:

Unit testing in Chef is intention testing. When you run your chefspec tests, you are only testing whether you are telling Chef to do what you expected. When is this useful? When you have some complicated logic in your recipe or if you...

Recently I wrote a Python command line app and had some trouble finding a good example of how to test it using best practices. Some ideas I ran across in my search:

• Use a 3rd party CLI-testing solution like ScriptTest or a framework that includes testing like pyCLI.

I write my other tests with unittest and run them with nose. I don’t think there’s anything special about a CLI that requires the rest of my team to learn YATT (yet another testing tool).

• Use sys.argv, subprocess or some combination of both.

This feels very hacky to me. Writing a Python CLI using sys.argv is a little silly considering the more robust options out there. argparse is in the standard library. 3rd party tools like clint and cliff are frameworks that seem promising. A testing solution that plays to the strength of these tools is ideal.

Personally, I use argparse. It does everything I need and its API is...

So Vagrant is great. If you’re not using it yet, you should check it out. It allows you to develop your applications in the same environment they’ll be running in.

I’ve been using Vagrant and Chef for a little over a year now, but I hadn’t used Ruby much before. I’ve written a lot of Chef cookbooks and Vagrantfiles since then. Writing a multi-VM Vagrantfile is much easier if you lean on Ruby to help you keep them shorter and more readable.

Here’s an example of a Vagrantfile with 3 Ubuntu LTS 12.04 machines:

• clean - A base box, with nothing installed. For experimentation.
• mongo - A MongoDB box, provisioned with chef-solo.
• jenkins - A Jenkins CI box, provisioned with chef-solo.

There are 3 parts to the Vagrantfile.

1: Box definitions

Boxes can be configured with several options, most of them optional. The schema is defined at the top of the file.

=begin Example box JSON schema { :name...