Most resources discussing testing with Chef deal with unit testing. Unit testing your Chef recipes is a very good idea, but will only get you so far. You can have 100s of passing unit tests and your Chef converges will still fail. Integration testing can greatly increase the confidence you have in your Chef code. In this post, I’ll walk through how we’re currently doing it at Essess.

First, a clarification on Chef testing terms

I didn’t really understand how chefspec unit tests fit into the Chef testing process until I watched Seth Vargo’s talk at a June meetup on it. If you use Chef and haven’t watched it yet, check it out. He broke it down like this:

Unit testing in Chef is intention testing. When you run your chefspec tests, you are only testing whether you are telling Chef to do what you expected. When is this useful? When you have some complicated logic in your recipe or if you want to make sure you don’t introduce regressions. This kind of testing doesn’t tell you if your Chef converge will succeed or not. For that, we need to go beyond unit tests.

Enter integration testing

test-kitchen is what you need to do integration testing with Chef. It is still beta-state, but that doesn’t mean it’s not ready to use. It does mean that the documentation for the project is not great, and you might need to read through the project’s Github repo to figure out what’s going on.

Basically, this is what test-kitchen does:

• Launches a test machine (vagrant box, ec2 instance, LXC container, etc)
• Converges a given Chef run list on the machine
• Runs tests afterwards to make sure the box is in the state you expect

The tests are run by so-called bussers, test setup and execution frameworks.

test-kitchen ships with the bats busser, which allows you to write bash tests for your converged box. I got very frustrated with bats though; unless you’re a unix wizard it is very slow and unintuitive to write these tests in bash. There is a better way.

serverspec, it’s better

serverspec let’s you write your tests in Ruby in a more intuitive way. It’s syntax is similar to chefspec.

There are several resources types available to use.

The busser that test-kitchen uses depends on the folder structure you’re using. After test-kitchen converges, it will install the appropriate busser and use it to run your tests.

Enough theory, an example

Let’s say we want to launch an Ubuntu 12.04 server that is running MongoDB and RabbitMQ. This setup is arbitrary; you can use test-kitchen and serverspec to test any box you bring up.

This code is up at https://github.com/dustinmm80/serverspec-example.

You’ll need the vagrant-berkshelf plugin for test-kitchen to integrate with berkshelf. The setup.sh script in the repo will take care of this for you.

So, our .kitchen.yml file looks like this.

---
driver_plugin: vagrant
driver_config:
  require_chef_omnibus: true
  use_vagrant_berkshelf_plugin: true
  customize:
    memory: 1512
    cpus: 4

platforms:
- name: ubuntu12.04
  driver_config:
    box: ubuntu12.04
    box_url: https://opscode-vm-bento.s3.amazonaws.com/vagrant/opscode_ubuntu-12.04_provisionerless.box

suites:
- name: mongorabbit
  run_list: 
    - recipe[mongodb::10gen_repo]
    - recipe[mongodb]
    - recipe[rabbitmq]
  attributes:

We’re using the vagrant driver to launch a local virtualbox VM here.

Our serverspec tests will be located here: test/integration/mongorabbit/serverspec/. The mongorabbit dir maps to the suite we’re running tests against, and the serverspec dir lets test-kitchen know what busser it needs to run these tests.

We have 2 serverspec files here, one to test mongo and the other for rabbitmq. It’s a good idea to split functional parts of your server up into different test files. I test roles this way, so sometimes there are several different spec files.

mongo_spec.rb

require 'spec_helper'

# Mongo service
describe service('mongodb') do
    it { should be_enabled }
    it { should be_running }
end

ports = [27017, 28017]

ports.each do |port|
    describe port(port) do
        it { should be_listening }
    end
end

rabbitmq_spec.rb

require 'spec_helper'

describe service('rabbitmq-server') do
    it { should be_enabled }
    it { should be_running }
end

describe port(5672) do
    it { should be_listening }
end

For each service, we are testing that it is enabled for boot, running and listening on the ports it should be.

We also have spec_helper.rb, which configures how serverspec will run. Different options can be found in the serverspec docs. I just copied the one opscode uses in some of their cookbooks.

test-kitchen workflow

Since the docs around test-kitchen are not great yet, I’ll share my workflow when I’m writing serverspec tests.

I launch my test-kitchen box with this command:

test-kitchen -d never

The -d flag tells test-kitchen when to destroy the box. By default, it will destroy the box after a successful test run. This makes sense from a CI perspective, but not when developing tests. So I’m saying -d never to keep the box up. Otherwise, I’d have to wait for a full converge to run my tests again.

Once the box is up and converged, I write my tests and run this command to run them:

test-kitchen verify

If I want to log into the box to poke around I need to specify which box I mean. The naming schema is suite name + platform name, so in my example mongorabbit-ubuntu12.04. Use this command:

test-kitchen login mongorabbit-ubuntu12.04

Finally, when I am satisfied my tests are passing and I’m done working I destroy all the boxes.

test-kitchen destroy

Continuous integration with test-kitchen+serverspec

Launching Vagrant boxes on your CI server might not be the best or safest way to run your test-kitchen suites. I am using the kitchen-ec2 driver to launch EC2 instances and run the tests on them from Jenkins. test-kitchen runs after a cookbook has been pushed and its unit tests pass.

I’m sure there are many ways to do this and my process will surely look quite different a couple months from now.

Conclusion

test-kitchen and serverspec are awesome tools and let you really test your Chef code before releasing it out into the wild. Since adopting this approach, my team is much more confident that our servers are in a state to run our code when we push it.

Are you using a similar approach? What busser have you found to be the most useful? I would love to see more discussion in this area as these tools become better and more widely used.

Recently I wrote a Python command line app and had some trouble finding a good example of how to test it using best practices. Some ideas I ran across in my search:

• Use a 3rd party CLI-testing solution like ScriptTest or a framework that includes testing like pyCLI.

I write my other tests with unittest and run them with nose. I don’t think there’s anything special about a CLI that requires the rest of my team to learn YATT (yet another testing tool).

• Use sys.argv, subprocess or some combination of both.

This feels very hacky to me. Writing a Python CLI using sys.argv is a little silly considering the more robust options out there. argparse is in the standard library. 3rd party tools like clint and cliff are frameworks that seem promising. A testing solution that plays to the strength of these tools is ideal.

Personally, I use argparse. It does everything I need and its API is straightforward.

Case study

I want to write a Python CLI app that will ping an arbitrary set of EC2 instances on AWS that share a set of tags. It also takes some optional params, the AWS region and AMI.

Here’s the app code:

ping.py

import argparse

def ping(tags, region=None, ami=None):
    # some AWS/boto code here
    pass

def create_parser():
    parser = argparse.ArgumentParser(
        description='Ping a number of servers in AWS based on tags'
    )

    parser.add_argument(
        'tags', nargs='+',
        help='Tags to search for in AWS'
    )

    parser.add_argument(
        '-R', '--region', type=str, required=False,
        help='AWS region to limit search to'
    )

    parser.add_argument(
        '-A', '--ami', type=str, required=False,
        help='AWS AMI to limit search to'
    )

    return parser

def main():
    parser = create_parser()
    args = parser.parse_args()
    ping(args.tags, args.region, args.ami)

if __name__ == '__main__':
    main()

Pretty simple: one positional argument, tags, followed by 2 optional named parameters, region and ami. I’ve left the actual implementation blank; not needed for this discussion.

So, how should I test this? Above you can see I split the creation of the parser out to create_parser. I did this to make it easier to test. Here are a couple unit tests:

tests/test_ping.py

from ping import create_parser, ping
from unittest import TestCase

class CommandLineTestCase(TestCase):
"""
Base TestCase class, sets up a CLI parser
"""
@classmethod
def setUpClass(cls):
    parser = create_parser()
    cls.parser = parser

class PingTestCase(CommandLineTestCase):
    def test_with_empty_args():
        """
        User passes no args, should fail with SystemExit
        """                                    
        with self.assertRaises(SystemExit):
            self.parser.parse_args([])

    def test_db_servers_ubuntu_ami_in_australia():
        """
        Find database servers with the Ubuntu AMI in Australia region
        """
        args = self.parser.parse_args(['database', '-R', 'australia', '-A', 'idbs81839'])
        result = ping(args.tags, args.region, args.ami)
        self.assertIsNotNone(result)
        # Do some othe assertions on the result

In the above test file I set up a CommandLineTestCase class that will add the parser from ping.py before any tests are run. The tests then use the parser object on the class to test any number of arguments passed in. This is where the create_parser function in ping.py is key: I am able to create the parser used in the app and unit tests the same way. I can be sure if my tests pass my app will be have the same way.

I hope this is useful to some of you out there. I think unit tests are sometimes seen as a solved problem and there isn’t much discussion around them. They’re hard to teach because they’re so application specific. I’ll be writing about different approaches to testing in the future.

So Vagrant is great. If you’re not using it yet, you should check it out. It allows you to develop your applications in the same environment they’ll be running in.

I’ve been using Vagrant and Chef for a little over a year now, but I hadn’t used Ruby much before. I’ve written a lot of Chef cookbooks and Vagrantfiles since then. Writing a multi-VM Vagrantfile is much easier if you lean on Ruby to help you keep them shorter and more readable.

Here’s an example of a Vagrantfile with 3 Ubuntu LTS 12.04 machines:

• clean - A base box, with nothing installed. For experimentation.
• mongo - A MongoDB box, provisioned with chef-solo.
• jenkins - A Jenkins CI box, provisioned with chef-solo.

There are 3 parts to the Vagrantfile.

1: Box definitions

Boxes can be configured with several options, most of them optional. The schema is defined at the top of the file.

=begin
Example box JSON schema
{
    :name => :name_of_vagrant_box, #REQUIRED
    :ip => '10.0.0.11', #REQUIRED
    :synced_folders => [
        { '.' => '/home/vagrant/myapp' }
    ],
    :commands => [
        'touch /tmp/myfile'
    ],
    :vbox_config => [
        { '--memory' => '1536' }
    ],
    :chef_role => 'myrole',
    :chef_json => {
        'local' => true,
        'env_name' => 'development',
    }
}  
=end

boxes = [
    # Clean box, Ubuntu 12.04 LTS - for experimentation
    { 
        :name => :clean, 
        :ip => '33.33.33.80'
    },
    # MongoDB box, with Chef 'mongo' role
    { 
        :name => :mongo,
        :ip => '33.33.33.81',
        :chef_role => 'mongo'
    },
    # Jenkins master box, with Chef 'jenkins' role
    { 
        :name => :jenkins,
        :ip => '33.33.33.82',
        :vbox_config => [
            { '--memory' => '1536' }
        ],
        :chef_role => 'jenkins',
        :chef_json => {
            'jenkins' => {
                'server' => {
                    'user' => 'jenkins'
                }
            }
        }
    }
]

2: Global Chef JSON

All boxes will have this Chef JSON applied. The :chef_json parameter in the box definition will be merged with this, overwriting values if they conflict.

# Chef JSON config for all boxes
chef_json = {
    "local" => true,
    "env_name" => "development",
    "s3cmd" => {
        "user" => "vagrant"
    },
    "chef_client" => {
        "cron" => {
            "hour" => "4",
            "minute" => "0"
        }
    }
}

3: Vagrant configuration loop

Here is the main Vagrant configuration, that loops over the boxes defined in part 1 to create the Vagrant box definitions. If you’re provisioning your box via Chef it applies the chef_json hash and your box’s specific chef_json hash to apply attributes.

Vagrant.configure("2") do |config|
    boxes.each do |opts|
        config.vm.define opts[:name] do |config|
            # Box basics
            config.vm.box = "precise64"
            config.vm.box_url = "http://files.vagrantup.com/precise64.box"
            config.vm.network :private_network, ip: opts[:ip]
            config.ssh.forward_agent = true

            # Synced folders
            opts[:synced_folders].each do |hash|
                hash.each do |folder1, folder2|
                    config.vm.synced_folder folder1, folder2
                end
            end

            # VirtualBox customizations
            unless opts[:vbox_config].nil?
                config.vm.provider :virtualbox do |vb|
                    opts[:vbox_config].each do |hash|
                        hash.each do |key, value|
                            vb.customize ['modifyvm', :id, key, value]
                        end
                    end
                end
            end

            # Use a more recent version of Chef
            config.vm.provision :shell, :inline => "gem install chef --version 11.6.0 --no-rdoc --no-ri --conservative"

            # Run shell commands for box
            opts[:commands].each do |command|
                config.vm.provision :shell, :inline => command    
            end

            # Configure the box with Chef
            config.vm.provision :chef_solo do |chef|
                # Set paths to local Chef resources
                chef.provisioning_path = "/etc/chef"
                chef.cookbooks_path = ["../chef/cookbooks", "../chef/vendor/cookbooks"]
                chef.roles_path = "../chef/roles"

                # Add a Chef role if specified
                unless opts[:chef_role].nil?
                    chef.add_role(opts[:chef_role])   
                end

                # Merge the general Chef JSON with box-specific JSON
                chef.json = chef_json.merge(opts[:chef_json])
            end    
        end
    end
end

Benefits to writing your Multi-VM Vagrantfiles this way:

• Less boilerplate code. You only have to define one block inside Vagrant.configure("2")
• Boxes are defined at the top of the file; you don’t have to hunt around for them
• If 1 team member fully understands part 3 the others only need to think about parts 1 and 2

Of course you can do anything in your Vagrantfile that you can in any Ruby file, so I think you can extend this pattern to fit a lot of use cases. I am interested in suggestions on how to make this a better abstraction and hearing how you are tackling this problem.

The full Vagrantfile is up on Github.

Most modern applications make use of several 3rd party packages and libraries to get the job done. This is great, and saves you a ton of time. But in order to build a robust application/service you should use a package manager.

Why?

• Well-defined requirements

  You’ll be able to define which libraries you want and their specific versions. You don’t need to list out the libraries that those then depend on.

• Getting updates

  Using a new version of a library/tool is easy. You bump the version number in your requirements file, run the installer and you’re done.

• Easier development

  With package management set up, you can set up projects very easily. They also let you experiment a lot more quickly. You can download a package, try it out, and if it doesn’t work delete its line in the requirements file and start over. No searching in git repos or download links in other websites required. Bootstrapping someone new to the team can take less time.

• Smaller repos

  You don’t need to version code you didn’t write. Less code in the repo = faster VCS ops, less clutter.

Thankfully, there are tools that allow you to easily keep track of your 3rd party packages across the whole stack. I’ll go over the ones that I’m familiar with, starting with the top of the stack.

Frontend

Old and busted: Downloading your jQuery/Angular/YUI Javascript libs and copying them into your project.

There are a couple problems with this:

• Getting upstream releases is a manual process - go to site, find download link, download, extract and paste in project.
• Keeping 3rd party code in source control. Keeping track of the changes isn’t really helpful. Also it can very tempting to modify these libs.

Bower

Bower can manage all your front-end libraries. It is installed via node’s npm. We’ll get to this tool further down the stack. With bower, you define your dependencies with a bower.json file and set where they will be installed with .bowerrc. Bower works off of tags of repos on Github, so the list of available packages is immense. Bower will also automatically download any dependencies of your dependencies: you don’t have to hunt around for lower-level libraries when you run into errors.

Sindre Sorhus created a Bower package discovery site to help find things more easily.

Bower is by far the most adopted frontend package mangement tool and the are good, but there some competing projects.

Alternatives

• component - common.js components
• jam - require.js components, ‘brower-first’
• volo - package management + task helper
• npm + browserify - works with npm, for require.js components

Backend

If you’re running code on a server for your app/service, there is usually an accepted package management tool for each language. This is one of the first things I investigate when learning a new language. I’ll cover a few that I’ve used here.

Python

Use pip. It’s the package manager for Python. It is automatically installed if you use virtualenv. Requirements are defined line-by-line in a requirements.txt file.

Packages are listed on pypi. You can also download and install requirements directly from any source control repo with git.

Python has a few issues with package management; the format and tooling around packages has changed a lot in a little time and I feel like there is still a lot of confusion in the community. But it seems to be getting better. Another issue: many scientific packages can be hard to install, requiring a lot of extra steps.

node.js

npm is the way to go with node. So, if you’re using node as your backend, you can install bower with npm and generate a scaffold of your app with yeoman. Tooling bliss.

ruby

gem is the go-to tool for Ruby. You list your requirements in the Gemfile and you’re good to go.

Some nice tools in the Ruby dev workflow:

• bundler - Maintain a consistent environment for your apps
• rvm - Ruby Version Manager - isolate your Ruby environments

The Ruby community really has their game together here; it’s nice to work with the mature tools they have created.

go

Go has the great advantage of having its package management built-in. Running go get and any package’s repo URL will bring the package into your workspace. You can then use it in your app code. Go doesn’t have a central package repository, like Bower.

A serious disadvantage to this built-in tool is that you can’t specify a version. So you have to either fork the repo at a certain point or check specific versions of packages into source control. Yuck.

There are tools like gonuts out that let you specify versions, but they haven’t gained much traction yet and don’t list many packages. gopm looks promising, but its very new and not production-ready yet.

Something to consider when evaluating Go for your org.

Of course other languages have their tools (e.g., .NET has NuGet) but I’m not familiar with them. Investigating the status of package management for any language should be part of evaluating it for use in your projects.

Infrastructure

So, we’re using package management now for our frontend and backend. What about the servers that actually run our application? Good news everyone!

Chef

Chef is the configuration management tool I use. Chef has cookbooks, cookbooks have recipes.

Berkshelf is an awesome tool created by Riot Games that lets you tell Chef what cookbooks you need and their version. You can browse cookbooks at the Opscode community cookbooks site or just specify a repository and tag. Berkshelf will fetch all dependencies that cookbooks have and download them as well. This is a huge time-saver. You can then layer in your own cookbooks on top, in the library/application pattern described in this post.

I use Berkshelf because the tooling around it is great. There is a plugin for Vagrant, the GitHub repo is very active and the tool just makes sense to me.

If you’re using Puppet or another CMT, or just don’t like Berkshelf, there are tools out there for you. librarian can be combined with librarian-puppet to help you manage your Puppet modules.

Conclusion

Using package management across your whole stack just makes sense. It gives you more control over your system, speeds up development and will make the engineers on your team happier. The learning curve for these tools is not steep, in my experience. Once you’re familiar with the pattern that package management introduces you will really see the power of developing your applications and services using these tools.

Are you using package management in your projects? If so, how much of the stack is managed? Any great tools I missed?

Thanks for reading.

Python vs Go - Requests per Second

I’ve been working in Python for a few years now and have recently started to investigate Go, mostly as an experiment to alleviate some bottlenecks without having to scale out webservers.

I wrote a simple REST server in each language and used the ab tool to check the requests/second.

Python

server.py

from bottle import route, run

@route('/')
def home():
    article = {'name': 'A Royal Baby', 'body':'A slow news week'}
    return article

def main():
    run(host='localhost', port=8081)

if __name__ == '__main__':
    main()

Go

server.go

package main

import (
    "encoding/json"
    "fmt"
    "github.com/emicklei/go-restful"
    "io"
    "net/http"
)

func main() {
    ws := new(restful.WebService)
    ws.Route(ws.GET("/").To(hello))
    restful.Add(ws)
    fmt.Print("Server starting on port 8080\n")
     http.ListenAndServe(":8080", nil)
}

func hello(req *restful.Request, resp *restful.Response) {
    article := Article{"A Royal Baby", "A slow news week"}
    b, _ := json.Marshal(article)
    io.WriteString(resp, string(b))
}

type Article struct {
    Name string
    Body string
}

The Go version is definitely more verbose than the Python version, but I think it’s still pretty terse and to the point. The Python server is using the bottle framework, and the dict passed as a response is automatically serialized to a response. The Go server is using the go-restful package, which makes it easy to construct a RESTful API in Go.

The benchmark was run on a Macbook Pro with a 2.4Ghz i7 and 16GB of RAM.

This command was used:

ab -q -c 50 -n 1000 http://127.0.0.1:8080/ | grep "Requests per second"

The results

5 runs, average requests per second for 1000 requests

On average Go managed 13457 requests/second while Python managed 1311 requests/second.

In this test, Go was 10.26 times faster than Python while serving a JSON response.

Now, I know these scripts are very simple and missing logic that would be in real applications - a database connection for instance. And maybe there are more accurate ways to test this than ab. But I think these results are compelling enough to give you a reason to check out Go. In my experience so far, it carried over a lot of nice things about Python to a compiled language.

Anyone else run Python/Go benchmarks and want to share?

Here’s a solution I to creating a reusable dialog controller using the dialog directive from angular-bootstrap.

This can be a general purpose solution to creating dialogs, instead of having several controllers to open dialogs.

plnkr.co demo

Javascript

app.js

angular.module('MyApp').controller('OpenDialogCtrl',
    function ($scope, $dialog) {
        var dialog;

        $scope.openDialog = function (template, controller, args) {
            var opts = {
                backdrop: true,
                keyboard: true,
                backdropClick: true,
                templateUrl : '/app/html/partials/'+template+'.html',
                controller: controller,
                resolve: {
                    args: function() {
                        return args;
                    }
                }
            };
            dialog = $dialog.dialog(opts);
            dialog.open();
        };

        $scope.$on('closeDialog', function () {
            if (dialog) {
                dialog.close();
            }
        });
    }
);

angular.module('MyApp').controller('ListItemsCtrl',
    function ($scope, $rootScope, args) {        
        $scope.items = args.items;

        $scope.closeMe = function() {
            $rootScope.$broadcast('closeDialog');
        }
    }
);

The openDialog call takes 3 arguments

• template - the name of the HTML template for the dialog body, appended to the path in templateUrl - this can be changed to work with any project structure
• controller - the name of the controller for the dialog
• args - a Javascript object containing any items you’d like to pass from the controller/view to the dialog. This is injected into the dialog controller.

Instead of using the dialog.open().then(function(){}) pattern in OpenDialogCtrl we leave this up to the dialog’s controller (ListItemsCtrl in this case). The dialog controller broadcasts an event when it wants to close and the OpenDialogCtrl catches that event and closes. This assumes that you only have one dialog open at any time, something I think is reasonable.

HTML

index.html

<!doctype html>
<html>
<head></head>
<body ng-app="MyApp">

<a ng-controller="OpenDialogCtrl" ng-click="openDialog('listItemDialog', 'ListItemsCtrl', {items:['a', 'b', 'c']})">
    Open Dialog
</a>

</body
</html>

/app/html/partials/listItemDialog.html

<ul>
    <li ng-repeat="item in items">
       {{ item }}
    </li>
</ul>

<a ng-click="closeMe()">Close Me</a>

You can see the call here in index.html to open the dialog. We are passing items to the OpenDialogCtrl and then rendering the items in listItemDialog. Obviously we could do much more complex things here, like forms, but I think this simple example shows the power of this pattern.

Hope this helps! Leave your feedback below.

I think a lot of programmers feel that their job is to write code, not documentation. I think this is wrong, and leads to a lot of the misgivings devs have about documentation, namely:

1. Violates DRY (Don’t Repeat Yourself)

Docs should give context, not explain everything again. If you feel like you’re duplicating yourself, you either don’t need to doc this code (simple functions, etc) or you’re approaching it from the wrong angle.

2. Always out of date

This seems to be a product of the lack of priority assigned to documenting, the ‘not my job’ attitude. Some projects file bugs against doc errors the same they would code, and it works well for them. If you update some code, it takes very little time to update the docs as well.

They are not separate.

3. Time constraints

An easy excuse, and points to inadequate prioritization. 'Good enough' is 'good enough', but you need to know the things you should never skimp on:

• Code style
• Clarity
• Documentation

Compare these:

def swap_name(name):
    if name.count(' ') == 1:
        return ' '.join(name.split(' ')[::-1])
    return name

def swap_name(name):
    """
    Swap the first and last name of a person

    :param name: String of two words with a space, a full name
    :returns: A string with the 2 elements reversed
    """
    if name.count(' ') == 1:
        return ' '.join(name.split(' ')[::-1])
    return name

The 2nd is documented in ReST format, which means Sphinx can easily extract it from your code.

This is a simple example, but I think you can see that if you were to come across this function months or years after it was written, the 2nd would be much more helpful.

In a company setting, lacking docs lead to needing months to onboard new hires and development slowing down as systems become larger and more complex. Then, when the only dev on your team that understands X leaves, you’re left scrambling.

In an OSS project, lacking docs lead to people not using your project, discontinuation and obscurity.

Writing code that you can understand is almost never not enough; the code you write is never really yours but meant for sharing with other people. All the assumptions and decisions you made creating it are lost if you don’t write them down as you go.

This post is a reaction to the recent Reddit discussion Why do Programmers Hate Documenting?

A collection of tips from things I’ve seen in a lot of code bases.

Use print() as a function and not a statement

print('some string') # good
print 'some string'  # bad

print() has been moved to a function in Python 3 docs. It has been backported to 2.7.

Replace '' % (args) with ''.format(args)

'{0.name} {0.address} {0.city}'.format(k) # good
'%s %s %s' % (k.name, k.address, k.city)  # bad

String formatting with format() is replacing %. It is more powerful (argument unpacking, etc) and clear.

Make sure docstrings are using double quotes

def do(a, b, c): # good
    """
    does something
    """
    pass

def do(a, b, c): # bad
    '''
    does something
    '''
    pass

Docstrings with single quotes don’t follow ReST format, and can’t be picked up by Sphinx.

Make sure all other strings are single-quoted

quote = 'it isn\'t the gravity # good
quote = "it isn't the gravity" # bad

Single-quoted strings follows pep8, consistency is good across code base

Call functions with named parameters as often as possible

def rotate_head(x, y, z, reverse=False):
    pass

rotate_head(x=15, y=11, z=20, reverse=True) # good
rotate_head(15, 11, 20, True)                # bad

Sacrificing brevity for readability is okay in this case. Makes a big difference for code clarity.

Don’t use mutable types or functions as default keyword args

def filter_gradient(x, y, colors=None, timestamp=None): # good
    if colors is None:
        colors = []
    if timestamp is None:
        timestamp = datetime.now()

def filter_gradient(x, y, colors=[], timestamp=datetime.now()): # bad
    pass

Default parameters are bound at function declaration, and not execution. In the bad example above, multiple calls to filter_gradients would mutate colors every time. For timestamp, datetime.now() is not called every time the function is invoked. So timestamp would be identical every time you called it.

Be careful about commenting out code, crufts up codebase

Let git keep your history for you; if you comment out a block of code, consider just removing it instead. If you need it back, you can retrieve it from the git history. This keeps your code clean and minimizes confusion.

Functions or classes w/ 5 or more args can probably be split up

In the case where you are passing 5+ args to a function, it is probably better to split it up into either smaller functions or use **kwargs. If you use **kwargs be sure to document what keyword arguments will change the execution of your function.

For classes that are keeping the state of many things, first of all think about whether this needs to be a class. If it is a singleton, you probably can namespace it at the module level. If it needs to be a class, breaking it up into smaller classes and composing it using multiple inheritance can help manage the complexity. It will also make testing easier.

Keep lines at 79 lines or less

From PEP8, lines in python should be 79 lines or less. Some devs whine about this, but if you have a line longer than 79 chars, you’re probably trying to do too much in that line. Examples of this: complicated list comprehensions and appending long strings. List comprehensions should be simple (comprehendable) and ''.join(iterable) is the best way to join those strings. Each statement should read like a simple English sentence, not a paragraph or complex sentence.

Indent dicts and lists out, for readability

animals = ['tiger', 'bear', 'cat, 'dog', 'donkey',
'zebra', 'lion', 'wallaby', 'chipmunk']  # bad, make this all one line, or indent so each item has its own line

# same with dicts, but even more so

animals = { # good indentation
    'name': 'tiger'
    'species': 'feline',
    'age': 10
}

For clarity, indent out your dicts and lists if they start to get long.

Use correct data structures - tuples, lists, dicts, sets, deques

A lot of data structures in Python can be used interchangably, but each is suited to a specific purpose.

dict - use when you have key:value pairs and order doesn’t matter, use OrderedDict if order matters

set - use when you have homogenous items, they need to be unique, and order doesn’t matter

list - use when you have homogenous items, order matters, and its size won’t change much

tuple - use when position matters, heterogeneous items, and size never changes - can use namedtuple for clarity

deque - use when you would use a list, but you know you will be adding or removing elements from either end - much faster