Thursday, January 5, 2012

Setting up zc.buildout for tdd with nose.

What is buildout

Buildout is a (much underappreciated IMHO) system for building software, kind of similar to maven, leiningen, sbt, rake and othe build systems that exist for other languages. As usual with python (compared to maven, sbt and such), it is much more powerful and flexible than maven and is perhaps more similar to ant than maven, as it does not impose any conventions or a predefined workflow and can be made into whatever you need it to be.
The plugins for buidout are called recipies and there are a lot out there, but here we will be using just three:
- test runner. This will be the nose test runner, since nose is probably the most common way to run tests in python
- repl/ipython
- pydev project generation


The file is a project description file for distutils. The simplest is the following:
import os
import sys

from setuptools import setup, find_packages

here = os.path.abspath(os.path.dirname(__file__))

requires = [

setup(name='sample csv creator',
      description='sample csv creator',
        "Programming Language :: Python",
      install_requires = requires,

of course the name, version and description are to be filled out according to your needs. The requires is the dependency list, you put there all the eggs (artifacts in maven speak) that your project depends on, normally this means: - libraries for accessing databases - web frameworks such as pyramid, pylons or whatever have you - ORM like sqlalchemy

Bootstrap buildout

Download the bootstrap script and put it into your project directory:

then run it wih

this should download the setuptools and create the bin/buildout script, which you should launch next:


the important part is to launch it from the directory that contains the buildout.cfg file, that is your project direcory.

Create source folders

Create the src folder, inside it, create the csv folder and inside that create the tests folder (in case you would put your tests in a separate directory, which varies with the testing framework and coding paradigm you are going to be using for your project). 

Create eclipse pydev project

In order to use pydev for your development, you want to create the pydevproject file. In order to do that, you need to go into eclipse and create a pydev project. The filename location will be put into the buildout config in the next step. I usually put this in the main project directory.

Configure buildout

Create a directory .buidout in your home dir. In there create a file called default.cfg wit the following content
eggs-directory = /home/radaczyn/.buildout/eggs
download-cache = /home/radaczyn/.buildout/cache

This ensures that buildout behaves like maven and creates a local repo for eggs and downloads, that is it will not re-download the same artifact over and over again.

The next thing is to make the buildout configuration, like this:
parts = test
develop = .
pkgname = sample_csv_creator

recipe = zc.recipe.egg
eggs = ${buildout:pkgname}

recipe = pbp.recipe.noserunner
defaults =
eggs = ${app:eggs}

recipe =
eggs = ${app:eggs}

pydevproject_path = ${buildout:directory}/.pydevproject

the file should be named buildout.cfg and placed in the project directory.

Build it out

Launch bin/buildout from the project dir. You should see something like:
Creating directory '/media/disk/Documents/python/buildout-tutorial/parts'.
Creating directory '/media/disk/Documents/python/buildout-tutorial/develop-eggs'.
Develop: '/media/disk/Documents/python/buildout-tutorial/.'
Installing app.
Installing test.
Generated script '/media/disk/Documents/python/buildout-tutorial/bin/test'.
Installing pydev.

You will need to re-run this command in case you change:

  1. anything in the buildout.cfg
  2. dependencies of your project
To regenerate your scripts and pydevprojectfile.

Lather, rinse, repeat

Now, you know of course what TDD is, right? It's about writing tests for your SUT first. IN this case, I will just show you how to write a simple unit test 

Write a test

See some documentation for the testing frameworks in python, but let's use doctest for now (hence the --with-doctest switch in the noserunner configuration in buildout):
def foo(x):
        '''multiply input by 2.

        >>> foo(2)
        >>> foo('a')
        >>> foo([1, 2])
        [1, 2, 1, 2]

See it fail

bin/test - should fail, and you will see nice coverage report, also an html report is available in the coverage directory. 

Write some code

The should be changed to something like this:

def foo(x):
        '''multiply input by 2.

        >>> foo(2)
        >>> foo('a')
        >>> foo([1, 2])
        [1, 2, 1, 2]
        return 2 * x

See the test pass

Run bin/test. This should result in all tests clearly passing. Of course this is a very simplistic test and code, but you should get the idea. Now that you have the framework set in place, you can start running the test-fail-implement-pass-refatror cycle of TDD. You can also run your tests (with coverage and even see the coverage reports and highlighting from pydev).

REPL, ipython

You need to update the buildout config and add the following sections (pick the one you want of course):

recipe = zc.recipe.egg
eggs = ${app:eggs}
interpreter = repl

recipe                  = zc.recipe.egg
eggs                    = ipython
scripts                 = ipython
later you need to refer to them in the [buildout] section, choose either or both. After each change to the buildout.cfg you need to rerun bin/buildout to regenerate scripts, and then you can launch the given script.

Running the repl from those scripts ensures your code is on the PYTHONPATH and is impoartable.

Where to go from here?

As I said, there are many buildout recipies available, so you can really shape it into almost anything you want. A few examples:

Hope this give you some starting point to use buildout.