Thursday, December 20, 2012

Webapp development stack evolution (a.k.a Intellij + git + maven tomcat plugin = AWESOME!)

You know that +JetBrains' Intellij has a free and open source community edition, right? I have been using it on and off and it really rocks. First I'd like to start with my previous set of tools and workflow, so that you understand that it was not dead simple for me to switch IDE.

To start off, let me tell you that I am currently working on a webapp that is supposed to be deployed on tomcat.

Original stack

Back in the dark ages (about 2 years ago) I was using:
- SVN for version control
- eclipse
- maven from commandline and used mvn eclispe:eclispe to generate the eclipse project files
- mvn package to make war files

This had some major issues:
- any change in pom's required mvn eclipse:eclipse and reimport in eclipse
- any change in code required packaging and redeploying to tomcat. 
- I had frequently looked through the local history in search of not broken version of the codes

This was clearly a sub-awesome way to work.

Improvement #1 - m2e

Well, it turned out that there was a better way to integrate maven and eclipse, it's called m2e (formerly m2eclipse) and is bundled in eclipse nowadays. This made the eclipse:eclipse and reimport reduntant and was a major boost in productivity.

Improvement #3 - git!

This one is probably the most important thing. Git integrates with SVN nicely, and for me works like an unlimited undo tool. It also allows me to try out different ideas in the local branches and then easily merge the changes back to the upstream branch and push the changes to SVN. THis really rocks! Never again was I going through the local history in eclipse only to find out that I forgot some changes in one of the files. There was some drawback to that: switching between branches in git is painless, but switching them in eclipse is painful. Why? Because eclipse (despite the really awesome egit plugin) is not well suited for frequent switching of branches. Maybe it's because of the way m2e is built, but switching of a branch triggered a full rebuild of the entire workspace. Many times in the last year or 1,5 I have been waiting for 5-10 minutes for eclipse to "build workspace" only to find out that it re-triggered again. This sucked, but I somehow sticked with this way for over a year.

Improvement #4 - WTP and m2e integration

Now, I still build the webapp using mvn:package and deployed it manually to tomcat. This made the loop too long, so I was looking for a way to do that faster. The best solution I have found was WTP and m2e WTP integration, which basically installs a tomcat in your eclipse workspace, and starts tomcat with classpath set to the target directory of your maven projects. This means that your code changes can be visible immediately on your tomcat (if you have started it in debug mode, where the classes are reloaded on change) or you can just restart tomcat and see the results. Depending on the size of your project this can be a very fast process, definitely faster and less error prone from building war file and deploying it manually. 

Improvement #5 (and also regression #1) - Intellij

At times, even when editing things in eclipse, it would get sluggish. Also, sometimes the code completion capabilities would be... hmm. Sub-awesome is again the righ word - they were fine, don't get me wrong, but come'on to name a variable list? This is not what I want to name it. Knowing that the classes can get reloaded in debug mode made me think that maybe another IDE could also do the code hot-swap on tomcat. Why not try it? I have started using intellij about 3 months ago and it really rocks:
- the code completion is much better, it's instant, it's spot-on, it's everything that eclipse is and more. I don't know whether it is because intellij has all the classes and interfaces indexed or what but sometimes I feel like intellij is reading my mind.
the code refactoring is again everything that eclipse is (which is to say it is great) plus a little more. 
- switching branches and changing project structure was painless. Intellij immediately recognizes another branch, rebuild takes about 10s. Same goes for changing project structure (adding/removing modules etc). This is WAY faster than m2e.

There were some issues with intellij as well:
- there is no tomcat (or any container) integration in the community edition. This is really a pain, but I understand that JetBrains has to make money, too. It was actually the reason to give the paid trial version a try, but somehow it was not able to import my projects (!). So I was unable to find out how well (or how poor) the tomcat integration really is. With intellij 12 out of the door I will give it another try, as it might be a reason to actually pay for the full version instead of hacking the community edition

Improvement #6 (almost fixed regression #1) - maven tomcat plugin

Well, somehow through the prcoess of googling around and looking at stackoverflow, I have found a really decent way to improve the turnaorund time. The fastest way to deploy your webapp on tomcat is this:

mvn war:inplace tomcat:inplace

that builds a webapp in your src/main/webapp, then deploys that to tomcat. It does not build a war file! There are some caveats with this:
- you need to build your dependencies first. If you have some sort of grouping project, you are probabaly better off running this from your parent pom's directory
- you should not commit that to your repo. Probably the best way is to add src/main/webapp/WEB-INF/classes and src/main/webapp/WEB-INF/lib to .gitignore or svn:ignore properties.

Afterwards you can connect to the runnign tomcat with a debugger and do the hotswapping. When hotstwapping fails, you run the above mvn incantation (you can do that from within intellij as well), and you have your changes deployed within 1-2 seconds. 

Monday, December 17, 2012

Pidgin-sipe updated to 1.14.0

This time around I actually fixed the dependencies to build the package so that the voice and video support is built in. Hope it works fine for you, enjoy!

Just to recall:

Saturday, March 17, 2012

Friday, March 2, 2012

Test run order in surefire plugin

I just found this out today that the surefire plugin has a runOrder parameter. This is really cool, and perhaps the random should be the default value? It would help finding issues with test isolation (remember tests should be Fast, Independent, Repeatable, Self- validating, Timely). This would definitely help finding issues with the I part of this.

Thursday, February 2, 2012

Enabling video and audio calls in pidgin office communicator accout under Ubuntu oneiric (11.10)

Not sure if you are using office communicator, but I do @work. For a long time I was frustrated with the fact that the audio and video calls do not work. Fortunately, there is a way out.

Install required deps

sudo apt-get install libgstreamer0.10-dev libnice-dev libpurple-dev libnss3-dev libglib2.0-dev checkinstall
tar -xf pidgin-sipe-1.12.0.tar.bz2
cd pidgin-sipe-1.12.0/
./configure --with-vv
now you need to take a look at the output and make sure that it says:
Voice and video: enabled
because I might have forgotten some dependency. Anyway from now on (if you have the video enabled in the configure script) it is straightforward:
checkinstall -D make install
Voila! You need to restart pidgin and the audio and video should work. Maybe someone will make a ppa for this, but for now, there is none.
Update: I made the ppa myself. I don't have all the dependencies figured out yet, but the package is here:

and you also need to update pidgin to the one from pidgin ppa

Thursday, January 5, 2012

Setting up zc.buildout for tdd with nose.

What is buildout

Buildout is a (much underappreciated IMHO) system for building software, kind of similar to maven, leiningen, sbt, rake and othe build systems that exist for other languages. As usual with python (compared to maven, sbt and such), it is much more powerful and flexible than maven and is perhaps more similar to ant than maven, as it does not impose any conventions or a predefined workflow and can be made into whatever you need it to be.
The plugins for buidout are called recipies and there are a lot out there, but here we will be using just three:
- test runner. This will be the nose test runner, since nose is probably the most common way to run tests in python
- repl/ipython
- pydev project generation


The file is a project description file for distutils. The simplest is the following:
import os
import sys

from setuptools import setup, find_packages

here = os.path.abspath(os.path.dirname(__file__))

requires = [

setup(name='sample csv creator',
      description='sample csv creator',
        "Programming Language :: Python",
      install_requires = requires,

of course the name, version and description are to be filled out according to your needs. The requires is the dependency list, you put there all the eggs (artifacts in maven speak) that your project depends on, normally this means: - libraries for accessing databases - web frameworks such as pyramid, pylons or whatever have you - ORM like sqlalchemy

Bootstrap buildout

Download the bootstrap script and put it into your project directory:

then run it wih

this should download the setuptools and create the bin/buildout script, which you should launch next:


the important part is to launch it from the directory that contains the buildout.cfg file, that is your project direcory.

Create source folders

Create the src folder, inside it, create the csv folder and inside that create the tests folder (in case you would put your tests in a separate directory, which varies with the testing framework and coding paradigm you are going to be using for your project). 

Create eclipse pydev project

In order to use pydev for your development, you want to create the pydevproject file. In order to do that, you need to go into eclipse and create a pydev project. The filename location will be put into the buildout config in the next step. I usually put this in the main project directory.

Configure buildout

Create a directory .buidout in your home dir. In there create a file called default.cfg wit the following content
eggs-directory = /home/radaczyn/.buildout/eggs
download-cache = /home/radaczyn/.buildout/cache

This ensures that buildout behaves like maven and creates a local repo for eggs and downloads, that is it will not re-download the same artifact over and over again.

The next thing is to make the buildout configuration, like this:
parts = test
develop = .
pkgname = sample_csv_creator

recipe = zc.recipe.egg
eggs = ${buildout:pkgname}

recipe = pbp.recipe.noserunner
defaults =
eggs = ${app:eggs}

recipe =
eggs = ${app:eggs}

pydevproject_path = ${buildout:directory}/.pydevproject

the file should be named buildout.cfg and placed in the project directory.

Build it out

Launch bin/buildout from the project dir. You should see something like:
Creating directory '/media/disk/Documents/python/buildout-tutorial/parts'.
Creating directory '/media/disk/Documents/python/buildout-tutorial/develop-eggs'.
Develop: '/media/disk/Documents/python/buildout-tutorial/.'
Installing app.
Installing test.
Generated script '/media/disk/Documents/python/buildout-tutorial/bin/test'.
Installing pydev.

You will need to re-run this command in case you change:

  1. anything in the buildout.cfg
  2. dependencies of your project
To regenerate your scripts and pydevprojectfile.

Lather, rinse, repeat

Now, you know of course what TDD is, right? It's about writing tests for your SUT first. IN this case, I will just show you how to write a simple unit test 

Write a test

See some documentation for the testing frameworks in python, but let's use doctest for now (hence the --with-doctest switch in the noserunner configuration in buildout):
def foo(x):
        '''multiply input by 2.

        >>> foo(2)
        >>> foo('a')
        >>> foo([1, 2])
        [1, 2, 1, 2]

See it fail

bin/test - should fail, and you will see nice coverage report, also an html report is available in the coverage directory. 

Write some code

The should be changed to something like this:

def foo(x):
        '''multiply input by 2.

        >>> foo(2)
        >>> foo('a')
        >>> foo([1, 2])
        [1, 2, 1, 2]
        return 2 * x

See the test pass

Run bin/test. This should result in all tests clearly passing. Of course this is a very simplistic test and code, but you should get the idea. Now that you have the framework set in place, you can start running the test-fail-implement-pass-refatror cycle of TDD. You can also run your tests (with coverage and even see the coverage reports and highlighting from pydev).

REPL, ipython

You need to update the buildout config and add the following sections (pick the one you want of course):

recipe = zc.recipe.egg
eggs = ${app:eggs}
interpreter = repl

recipe                  = zc.recipe.egg
eggs                    = ipython
scripts                 = ipython
later you need to refer to them in the [buildout] section, choose either or both. After each change to the buildout.cfg you need to rerun bin/buildout to regenerate scripts, and then you can launch the given script.

Running the repl from those scripts ensures your code is on the PYTHONPATH and is impoartable.

Where to go from here?

As I said, there are many buildout recipies available, so you can really shape it into almost anything you want. A few examples:

Hope this give you some starting point to use buildout.