Sunday, August 15, 2010

Can we measure *anything* in software teams?

Code complete, a great book about software construction, suggest starting with at least some measurements:
  1. number of lines of code written
  2. number of defects detected
  3. number of work months
  4. total dollars
Then the measurements should be standardized between projects and only after they are implemented across the company, they should be taken into account in understanding the process of development. These should influence your thinking about the process and what do the measurements improve. An example might be intoducing the continuous integration game into your hudson server. This will for sure make people optimize for this. In one of the projects we have set it up so that it takes into account checkstyle, PMD violations, broken builds, successful builds, tests failing and so on. Even though the game is still missing a couple of meaurements that could potentially be useful (such as: code coverage measurements, +10 points for each additional percentage of previously uncovered code or something to that extent), it immediately had an influence on what ended up in the repository. People tend to optimize for whatever measurement you are taking. This is a fact of life and I think that Joel is somewhat right in his judgement that the extrinsic motivation replaces an intrinsic one. Is there any way that we can measure things and still keep clean of the "people optimize for the measurement" thing?

Story points - measure of complexity or effort?

Ever since I started working in scrum I always has the impression that the story points estimation is just a rough estimation of effort that it would take to implement a given user story. Things like keeping the reference to prior sprints and stories helps to keep the numbers from growing towards infinity and using fibonacci numbers to estimate the stories were passed on to me from an experienced scrum master and I took them as if these were the rules of scrum. But recently I have began to wonder whether actually treating story points as a measure of effort makes sense.

Story points as effort
Using storypoints as a rough measure of effort has its proponents and it makes a lot of sense if you are doing a couple of things:

you have a longer term backlog definition
you are doing regular backlog grooming sessions, wherein you reestimate the story points for the stories that are on the product backlog
you are using the story points estimation for release planning
Especially the last item is important and leads me to the question: if you don't have a fixed release schedule/date? What if you deploy whatever you have developed as you go along? What if you don't have a longer term backlog defined and the priorities change rapidly? Does it also make sense to use the story point measure as means of estimating effort? Or...

Storypoints as a measure of complexity and uncertainty
Can we make another use of the story point measure? Can we make them into a measure of complexity and uncertainty about a given story? In this scenario we would be estimating complexity of a story and its uncertainty (like: we will be using a new technology that is supposed to be quite easy, but noone on the team has ever used it thus far). Could we then make the commitment based on either the task breakdown results or the story points estimation (whichever fills up first)? Or should the task breakdown and the storypoint estimation always go hand in hand?

Managerial view
I had not realized this until someone pointed that out to me recently, but whenever a manager sees any number that can be measured, his/her eyes start to shine. When there are numbers, the numbers can be measured and they can be compared! So we can compare different teams based on the numbers or perhaps individual programmers as well! Wow! This is how the story points that are supposed to be an aid for the team in planning the work and commitment become a reporting means for the higher management to judge the team's productivity.