Wednesday, March 30, 2011

Releasing from a contuous integration

In the previous post I have described the setup of the continuously integrated system with a hint of problems that you typically encounter when trying to release several components that rely on each other's snapshot versions. Just to give you some idea about this, let's use a drawing:

now, this is a situation we have a releasable component D, which depends on A and B and C, which in turn both depend on A. This is of course a trivial example, imagine that there is another component that depends on A. How do you proceed with releasing (and probably more importantly, bugfixing) workflow for those? Well, let me outline the regular workflow you would have to follow:
  1. release the lowest level of dependencies (A) with standard maven release plugin
  2. change the poms of the intermediate projects (B and C) to point to the just-released version of A
  3. release those
  4. change the version of A in B and C's poms to include the snapshot version of A
  5. lather, rinse, repeat for just as many levels as you have in your project structure.
That's not really user-friendly, is it? I mean there is maven versions plugin that can automate some of those for you, but this still seems odd to use. Now, I found a couple questions related to the topic, and here's what we came up with:
  • identify releasable parts of the system. Make them as large as possible but not larger. The smaller you make them, the more work you will have to do in the next step
  • release them one by one and actually go through the 5 steps described above. Automate with versions plugin
I think the best way to prepare for new release would be to actually branch the whole repo when starting the release procedure in order for the rest of the team to continue working on the trunk (the release should be done from a last-successfull functional tests anyway, I guess). The idea of branches for release candidate seems like a good one too, so that you can then make a branch later if you want to create bugfix release or something to that sort. One of those questions also contained an idea that I found very interesting, namely having the parent pom (or actually grouping poms for that matter) as a separate directory that contains links to the other project's directories via svn:extenals. This seems like really cool solution, but perhaps there are some caveats there too?

Tuesday, March 15, 2011

Continuous integration and deployment using hudson and snapshot versions of artifacts

In my most recent project we are writing a rather large system that consists of at least a dozen components. We started off with true and agile...

big design upfront.

We sat down for a week with the clients and agreed on the interfaces... This seemed counterintuitive, especially in the perspective of agility and just doing the simplest thing that can possibly work. But it turned out to be quite useful. We started with a team of over 20 people that were working their assess off to achieve the goals of the sprint. They were working full speed from the start because of the head start that the interfaces definition meeting gave us. I cannot imagine agreeing on those interfaces as we go along. Especially if (as is in our case) the clients are not in the same office (and not in the same timezone for that matter). We would have never has any conclusive conversations and the thing would take forever to build. With the interfaces defined everybody could focus on the sprint goal. And the goal of the first sprint was:

an integrated system

To do that we relied on Hudson. The real cool feature of it is parsing the poms of the projects that it builds to find out the dependencies between them and to automatically construct build chains from there. So we ended up with a setup in which every commit to SVN triggers a shapshot deployment, which triggers downstream builds and deployments, which triggers building our webapp, deployment to tomcat and running functional tests on the deployed instance of the webapp. This is so insanely cool, that I want to shout out! Now, I know that most of the definitions of Continuous Integration are saying just that the build should be self-testing. They only talk about integration code, whereas I believe we should not only be integrating code (as in everything compiles and the unit/integration tests pass), but also a larger piece of the whole puzzle functionality (as in everything compiles + all the parts of a system we are building should compile and work together + all the funtional tests pass). This bit of integrating small pieces, to larger pieces, to yet larger pieces is what the continuous integration is all about. But every talk I've ever seen on this concentrated on integrating the code rather than functionality. Should I mention that pervasive testing is an absolute must to know whether any and all of those pieces work? Now, to be able to do functional tests you need to

continuously deploy

your stuff to the testing environment (be it a database instance, appserver instance, whatever) and run your functional tests there. In the maven case this means running everything off of a clean repository to eliminate issues with repo manager and things like that. This is of course simple with hudson (a.k.a Jenkins) + maven and the repo location setting.
The next part of the puzzle is

Releasing stuff.

This is a major struggle right now. In all the poms we have 0.1-SNAPSHOT dependency versions. This is cool for the continuous integration, but is a pain in the ass for releasing. There is a version plugin which can update the versions in poms to release versions, but in order to actually release the system, you need to create the release build chain yourself by hand. This is suboptimal, and I think that there must be a better solution to the problem. Does anybody have a solution?