I’ve been spoiled as a software engineer. When I write Python, Ruby or Javascript I have a REPL to try different ideas out. I can set breakpoints in PyCharm, RubyMine or the Chrome Devtools to inspect my stack at any point. The feedback cycle is short. I can TDD, even with infrastructure code. Developing code for hardware is not so straightforward. In this post I’ll share my current workflow for writing code for devices.

Hardware is messy; that’s part of the fun of it. In my free time I tinker with hardware: microcontrollers, sensors and robots. Writing code for Arduino and Particle devices can be frustrating. It can be difficult to tell why something isn’t working. Your code is perfect, but you have a broken connection somewhere or that new sensor you’re trying out was damaged in shipping. Or your code has a bug, but it’s hard to tell in the tangle of interrupts what your path of...

There is a recurring story about automation told by those looking at tech from the outside. It is a popular anecdote, and goes something like this:

Lisa is a developer at Initrode. She spent most of her day in databases and spreadsheets, cross-referencing them to create weekly reports the higher-ups needed. After some months doing this, she thought to herself “There has to be a better way!”. So she wrote a program to collate all data sources, perform the analysis needed, generate a PDF and email it on a schedule. Now she spends her time on her passion, rock climbing, and stills gets a paycheck every two weeks!

We’re doing DevOps now. It works. The Kool-Aid is drunk. We talk to robots with cheeky names in chatrooms to deploy new releases of our software. Servers confer with each other to promote leaders and cull low performers. Our task management software learns the cadence of our team...

I’ve been organizing the Boston DevOps meetup for almost two years now. When the organizer spot opened up I jumped at the chance. My experience DevOps-ing along in smaller organizations gave me a pretty narrow view of the movement and I wanted to educate myself. I also wanted to become better at public speaking (still working on that).

I’ve talked with a lot of people learning about, struggling with, and eventually getting somewhere with DevOps at different organizations. We’ve hosted presentations, panels and open spaces at several venues throughout Boston and Cambridge. Partnering with other local meetups, we’ve held discussions on everything from recruiting to wearables.

As always, YMMV, but here are some things I’ve learned organizing events for our community that weren’t obvious when I started.

1. Being welcoming takes consistent effort #

This really applies to any community, but...

What is DevOps? DevOps is not about tools. It is about culture, enabling cross-functional teams to quickly deliver value in a sustained matter. It is hard to define DevOps, but blah blah alignment blah success blah blah joy. Three paragraphs later, no takeaways. How many posts have you read about DevOps that follow this format? Promoting this vague clickbait as guidance is only doing ourselves a disservice. We’re patting ourselves on the back so hard we fall over.

How did we end up here? I think it’s because we’re still not having real conversations between business units. We find ourselves now with two camps in the DevOps community: technical and non-technical. Either side shifts the conversation to where they feel most comfortable: the domain they work in and understand. Technical folks talk about tooling. Non-technical folks talk about culture. The divide here is artificial, but it’s...

I’m doing a presentation on how to use serverspec to create a tight feedback loop for infrastructure coding at the next Boston DevOps meetup Nov 19th.

Slides are here: http://slides.com/dustinmm80/serverspec

Our Chef workflow at Essess looks like this:

• A developer updates a cookbook and pushes changes to git
• Jenkins is notified and runs the tests for the cookbook
• On successful build, it triggers a job to push the updated cookbook to the Chef server.

We’re using hosted Chef, so we assumed this would be seamless. Nope. A developer would push, the jobs would trigger and Jenkins would report back green. Later on they would notice that the cookbook was still pinned to an old version in the environment. Returning to Jenkins, we see something like this in the job’s console output log:

Well, that didn’t go as expected. You would think that if there was an upload failure we would get a non-0 exit code and Jenkins would report the build as failed. Not so. We use knife cookbook upload --all --force --environment ${env} to upload all cookbooks installed via berkshelf. env here is a Jenkins...

Let’s say you’re running on AWS and using IAM to assign your instances AWS keys dynamically. First off, if you’re not doing this you should. Why?

• Rotating credentials in from IAM is much more secure than sharing your AWS account keys across your infrastructure. If someone gets your keys and you’re not using IAM, you’re screwed. If you’re rotating your keys via IAM you shouldn’t really ever have to know what they are in the first place.
• By assigning IAM roles to your instances, you can control the granularity of that role’s permissions. You want to only allow your web servers access to one subfolder of a S3 bucket? Easy, through IAM.

Okay, so we’re all on board: If we’re using AWS we should use IAM roles to provide rotating access keys to our instances.

But what happens when you need those keys? I ran across this problem running unit tests on Jenkins for a Chef cookbook that uses...

Like Python? Into DevOps?

Here are some tools I use make my life a little easier.

awscli #

THE command line interface if you’re using AWS. Really nice documentation on how to talk to the different AWS services. I use this a lot as a glue library. For example, once Jenkins runs tests on a project, we tar.gz an artifact and use awscli to upload it to S3.

boto #

If you’re using AWS and need to get state or resources at runtime boto is what you want. The API is large, but well documented and composed.

botocore is a smaller low-level alternative, but I don’t care much about size when boto’s docs are good. botocore is the foundation for awscli.

Fabric #

Fabric is my go-to tool for remote execution. PyChef fetches the hosts I need and passes them to Fabric, which takes care of running any commands. This is a popular lib that has been around for a while; a lot of well-tested features.

There are a lot of people on the web talking about every aspect of the DevOps movement, but it can be hard to find them. I’ve added some resources I use to a Github repo here. If you use any resources not on this list, feel free to open a pull request and we can build a great list of resources together.

Github: dustinmm80/devops_resources

My starter list:

Podcasts #

Arrested DevOps

DevOps Cafe

Puppet Labs Podcast

The Cloudcast

The Food Fight Show

The Ship Show

Aggregators/Q&A #

Dzone DevOps zone

dev2ops

devops.com

DevOps on Quora

LinkedIn DevOps group

/r/devops

Blogs (Orgs) #

codeascraft (etsy)

Opscode Chef

Puppet

Blogs (Personal) #

Andrew Clay Shafer

Bryan Berry

Chad Fowler

Joe Hirn

John Allspaw

John E. Vincent

Len Lagestee

Mike Fiedler

Nathen Harvey

Patrick Debois

Tom Duffield

Videos #

devopsdays on Vimeo

Newsletters #

DevOps Weekly

Most resources discussing testing with Chef deal with unit testing. Unit testing your Chef recipes is a very good idea, but will only get you so far. You can have 100s of passing unit tests and your Chef converges will still fail. Integration testing can greatly increase the confidence you have in your Chef code. In this post, I’ll walk through how we’re currently doing it at Essess.

First, a clarification on Chef testing terms #

I didn’t really understand how chefspec unit tests fit into the Chef testing process until I watched Seth Vargo’s talk at a June meetup on it. If you use Chef and haven’t watched it yet, check it out. He broke it down like this:

Unit testing in Chef is intention testing. When you run your chefspec tests, you are only testing whether you are telling Chef to do what you expected. When is this useful? When you have some complicated logic in your recipe or if you...