In this post I would like to talk about three recurrent Java features I have changed my mind about in recent times: from loving them I have finished considering them as necessary evils you have to live with.

• Dependency injection
• Strong type system
• Object disorientation

Dependency injection

The dependency injection (DI) pattern is about separation of concerns. The concern of wiring object dependencies is extracted from objects and centralized in some kind of factory facility that, using some configuration information on how object are wired together, instantiates and configure these objects for you. This pattern is in the core of Spring and is implemented by Google Guice.

So why are DI solutions so important in Java? I think the main reason is because they are the only way to enable good unit testing. Sure there are other benefits, like minimizing coupling between objects or avoiding to write the same wiring boilerplate code again and again, but these are not as important as testing. While you will hardly find yourself in the need of changing a JDBC DAO for a JPA one, you will always need to mock dependencies when writing unit tests. And for doing so, you need to isolate these dependencies first.

And the Java new operator makes this task very difficult. It is just so rigid. You don’t have any mechanism for faking it once it is coded within your objects. The alternative usually implies implementing a full ecosystem of interfaces, their implementations enabling the dependency injection (in the form of constructors or setter methods), and factories for instantiating the configured objects. And since the boilerplate code this approach implies is considerable, you would rather use an external DI library that makes the work for you. So external DI solutions are good but, from a testing perspective, they solve a problem created by the Java new operator.

I realized of this problem when I tried to build something serious with Rails. The first thing I did was to look which dependency-injection solutions were available for Rails. I you look for DI in Rails there is a good chance you end up reading this article by Jamis Buck. That post summarized his clarifying experience with DI libraries in Ruby:

• He first created Copland: a port of the Java Hivemind.
• He then created Needle as a more Ruby-like approach to a DI solution.
• He finally concluded that DI frameworks are unnecessary in Ruby (he talks about DI tools, not about the pattern itself).

In my opinion, Ruby examples used by Jamis are not the best. I think a key aspect in Ruby is that new is just a class method. This means that classes in Ruby are by definition factories, and you can fake them directly when testing. Let me show you an example with a Person class that depends on Mouth, since to say something, a person has to open his mouth.

If you want to test this behavior with RSpec, you would do something like:

Since the Ruby new method is just a class method, it can be stubbed the way you want to inject the dependencies. This is a key aspect when testing with Ruby. You get used to it quickly because it is a natural thing from the programmer point of view. What is difficult is to go back to Java after knowing it, because of the big mess you have to create to achieve the same thing.

Another nice feature of Java DI solutions is that they manage the scope of objects for you. The usual default approach is to create an object each time is requested. But sometimes it is needed to have a singleton scope for some objects. Spring or Guice let you configure the scope in a declarative way and they manage the scope for you. The scope management is a production thing. For testing, you just create the objects you want to test directly.

If you hardcode a singleton object in Java you have to take care of cleaning its state between test cases. And this is just laborious and not elegant. DI frameworks offers a very nice solution for this problem. In Ruby, you still can test singleton classes because you can modify the Singleton module to expose a reset method that regenerates the single instance each time. This solution is possible by dynamic capabilities of Ruby. Again, I think it is nice just to be able to use the singleton object directly.

Another approach for the singleton problem is to have a global configuration object as a simple hash. In testing time you can override the contents of the hash. Again, Ruby syntax and symbols make to look nice something that seems scary if you come from the java World.

Regarding the scope problem, the solution offered by DI frameworks is probably more pure. They are also more complete. For example, Guice let you specify how the injected objects are created through the concept of providers. The thing is that with power comes complexity. Having many options is never for free. Even if you say: “I will only use a 10% of Guice” the rest of options are still there in the form of documentation and code. For example, they make you choose about what approach to take. And my point is that with a pure Ruby approach, without external tools, you can solve the most common cases in a much simpler way. I suppose it is a matter of taste, but I enjoy that approach much more.

Strong types system

I really don’t intend to talk about advantages and disadvantages of dynamic-typing and static-typing. But I would like to show an example of something that in my opinion is wrong. It is related to GWT concept of overlay types. It is a solution for managing JSON data as Java objects with GWT. This approach proposes to create a Java replication of the JSON data with getters/setters for the properties. The java code has to be mixed with the javascript code to be executed via JSNI. An example taken from the GWT page:

To me, it is like an implementation of the RY (Repeat Yourself) principle. I mean, do we need strong types that much? If this is the best abstraction you can do with Java of JSON data (and probably it is), then there must be something wrong with the language, because JSON data is only about structures of key/value pairs that can be nested.

Parsing JSON data with Ruby or Javascript is trivial. I will use the former in the next example. It shows how one line of javascript code can convert a JSON string into a plain Javascript object (using the JSON parsing utility of JQuery).

Object disorientation

The JSON example is a concrete example where dynamic typing just makes more sense. But to me, the essential problem of strong types is that they make more difficult to represent mental abstractions with code, because every single piece must fit in a very strict puzzle of types. And this means that you are forced to code a lot of things that have nothing to do with your domain, but with the type system of the language.

Take for example the next GWT design. In GWT, widgets that trigger click events implements an interface HasClickHandler. This interface defines the contract of registering ClickHandler objects, which are responsible of processing click events. This is a materialization of the Passive View pattern: a view without any state with controllers performing all the work. The idea is to have a very thin view that can be faked in controllers tests.

Discussions about the pattern apart, there is no any possible human explanation that justify the existence of HasClickHandler. And by “human” I am not being sarcastic. From a human point of view, it makes no sense. It makes sense under the constraints imposed by testing tools and, specially, by the Java language itself, where the only way to specify behavior in a modular way is through the addition of interfaces defining the contract.

Object disorientation is what happens when you go from “My view has a button I have to test” to “My view is stupid, I am going to create a HasClickHandler getter in its interface so I can fake it when testing”. As of today, no technical choice is free of this problem, but in the case of Java I think its effects are just overwhelming.

When considering using GWT a project I am working on, I was impressed by this presentation of Ray Ryan. I loved the principles and patterns discussed there for architecting the client part of a web application. The good thing about the presentation it that it focused on the concepts. The bad thing it that he showed very little example code.

It was when I saw how an example implementing those principles looked, that I decided I was going to study javascript hard. And it wasn’t because the authors didn’t make an excellent work with the example, but because I understood I wasn’t going to have fun programming that way. The bad thing is not that implementing the basic GWT “hello world” in the proposed way took 600 lines of code (without tests). The bad thing was how ugly everything looked, from top to bottom. And since having fun and beautiful code are two very important things to me, I discarded GWT, I studied javascript and JQuery hard, and worked a lot learning how to mount a good testing environment for this technologies and a Rails backend. And I don’t regret at all of how things are going. Of course, there are other problems but those will be for another post.

It is curious. 7 years ago, Eric Evan’s warned the world about how important was to focus on the domain on his seminar Domain-Driven Design book. Both design and code should reflect it as much as possible. In this time, despite of the big impact the book had in the Java community and all the discussions surrounding it, there is no still a mainstream implementation of many of the concepts he proposed. I wouldn’t say it is only due to Java being a poor language for representing the domain with code, but I think it has something to do with it.

Conclusion

I always find very difficult to explain what do I think is wrong with Java to other people. After writing this post, I still think the best way to understand it is to study other platforms and paradigms. In fact, the Java Platform offers nowadays a variety of languages, and some of them, like Groovy or Scala, are starting to attract a lot of attention. I still don’t know anything about Scala but I have used Groovy quite extensively and, although it has the same new operator problem that Java, it is a very nice alternative.

In my opinion, it just makes sense to have a server sending and receiving data in the form of JSON, and rich clients taking all the responsibility of rendering the UI and handling user interactions using HTML, CSS and Javascript. The key is that this approach enables you to architect the client part of your application. If some part of the client behavior is left to the server, it becomes more difficult to have a consistent architecture in the client part because concerns are mixed. And without a consistent architecture in the client, it is difficult to provide the user experience that is becoming more and more demanded in today web applications.

So, if you want to render everything in the client using Javascript, you need a good system for creating the HTML, and here it is where javascript templating systems appear. In this post I would like to explain how to test PURE Javascript templates using RSpec. I chose to use PURE because it is a production-ready system for rendering JSON data. The other solid approach seems to be JTemplates, but it doesn’t appear to be as actively maintained as PURE. There is also a proposal for including a templating system in JQuery that was initially submitted by Microsoft. There is already a demonstration implementation of the proposal but it is still in the incubation phase.

Tools

I am using RSpec and an amazing set of tools for running javascript from Ruby code:

• Harmony: which wraps Johnson and env.js letting you run javascript code in a DOM environment from ruby.
• Holy Grail: the Harmony plugin for Rails. It allows you to run javascript code in the context of your view tests.

Just follow the instructions in the sites or Harmony and Holy Grail for installing them. For the installation of Holy Grail with RSpec, read this article from Ken Mayer.

The example

For the sake of simplicity I will try to keep the PURE part as minimal as possible. I want to render the following person object.

And I will use the following PURE template:

Although in practice you will be using PURE directives for sure, for the example I will use the PURE auto-rendering mechanism.

What I want is to write a RSpec view spec to test that the template renders JSon as expected:

The helper method render_javascript_template receives a model object to render, and yields to a closure providing it with the result of the rendered template. The rendered text can be then checked using, in this case, Webrat matchers. The code of this helper method is shown bellow.

The render_javascript_template method has to do basically two things before rendering the templates using PURE:

• Rendering the template that contains the HTML of the template (in this example it is the ERB file itself).
• Including the required javascript files of PURE and JQuery.

The order of these steps is important: if you don’t invoke render before requiring the javascript files, JQuery won’t work properly.

The method that does the work is do_render_javascript_template. It basically does three things:

• It converts the json of the data to be rendered to a form suitable to be injected in the javascript runtime as a string literal (javascript doesn’t admit multi-lines literals).
• It then obtains the template CSS id that is going to be used to locate the template node we want to use. It obtains it from the file path of the template, which is specified in the outer describe of the spec.
• Finally it invokes the PURE auto-rendering process with the template. The only tip is that it appends an artificial container to the template because PURE fails if the template node hasn’t a parent. In the same way, before calling the JQuery function html(), an artificial root node is added because JQuery ignore root nodes with elements detached from the DOM.

Conclussions

While I test all my javascript code using JSpec, it didn’t fit well for testing my PURE templates. I had read about Harmony and I wondered if it would be possible to make the testing of PURE templates using RSpec and the powerful Webrat matchers. It happened to be not only possible, but a surprisingly fast and comfortable way of testing the templates. I am willing to see a templating system finally included with JQuery, but PURE is a powerful and fast choice that works today. I really think Javascript templates are here to stay.

The code of this post is availabe as a gist at github.

In my opinion, its current scoring in Amazon (4 stars) is difficult to understand. I would give it a 6 if I could. I think it is better than About Face 3: The Essentials of Interaction Design, also by Cooper. With this last one my expectations were very high but weren’t completely fulfilled (I think a 40% percent of the book was avoidable). But each page of The inmates… contains a lot of thoughts you can learn from. With many sentences I found myself thinking: ‘that’s totally obvious… but I hadn’t stopped to think about it until I read this’. And that’s exactly my favorite reaction when I am reading a technical book.

I would like to talk about the role of interaction design in two applications I use daily.

Lotus Notes: how not to design user interfaces

Lotus Notes is the mail client I have to use at work. There are so many wrong things about it that you can easily find web pages like Is Lotus Notes the worst business application ever?, Lotus Notes Sucks or Is Lotus Notes the world’s worst application?.

Why does this application provoke these kind of reactions?. I think the key is its user interface and the experience it produces in the poor user. Let me show you an example:

The screenshot shows how Lotus Notes let you manage your mail. It offers you 48 buttons with their 48 icons to do that (and the real scary part comes when you try to search for some functionality using the chaotic and non-standard menus). Managing your mail is about reading, writing, searching and organizing. None of these very fundamental goals is properly covered by Notes.

• Reading. Notes let you activate a “Preview Panel” that split the window in two parts, so you can read the contents of mails without opening them in new windows. It also draws in red color the mails you haven’t read yet. If you have your inbox empty (and I always keep it empty), when a new mail arrives, and you have the preview panel open, the mail is automatically marked as ‘read’. This behavior has made me miss several mails.

  Another good feature is that there is no obvious mechanism to force the checking of new mail. When some colleague has just send me an email I want to check, I close my inbox and open it again.

• Searching. This is where Notes really rocks. It makes searching your messages so difficult that I really think about it twice before doing it. It offers you two icons: a magnifier and some binoculars (you can look for them in the screenshot, if you like the ‘Where’s Wally?’ kind of games). Even today I can’t tell you the difference. I always use the keyboard shortcut that let me search in the subjects of the selected folder (so if I want to search in all my mail, I have to select the “All messages” folder).

  There is another “search in deep” functionality but it is so slow that I difficultly dare to use it. When you are used to Gmail, or you have seen the way Apple Mail does it, searching in Notes is like going back 20 years.

• Organizing. I like to archive most of the mails I receive. I have to do it in a local inbox, since my remote account has a limited size. I have tried hardly to find a shortcut for the action of archiving email, since I am always doing it. I haven’t being able to. I have googled for it, searched in the menus, looked for it in the help of Notes… Nothing. I am sure there is a way but I have been more than two years dragging the mail with my mouse, from the inbox, to the archive folder. There are like 10 buttons in the top of my inbox, each one with its shortcut, and no one of them could do this.

  Organizing in folders is also very difficult. The problems start even before the drag and drop process. If you try to reorganize the folders hierarchy you’ll know what I am talking about.

Allan Cooper talks about 14 factors that make software polite. Lotus Notes violates them all. For example, when you want to delete a file attachment of a message, an annoying modal dialog ask you whether you are sure, because it cannot be undone. Another example is the inability Lotus has to remember the state I left the workbench. Everyday I start Notes and it shows me the start panel. I always open my personal mail, and also the corporate mail of my team. Everyday I have to repeat the same operation and it annoys me, specially because Notes is not a lightweight and responsive product.

In a few months we will start using Microsoft Outlook at work. I haven’t used Outlook in the last 6 years but I am totally convinced I am going to be much more happy (because whatever Outlook does to manage your mail, it couldn’t be worse).

Mac OS X Time Machine: how complex systems can have wonderful user interfaces

If you have ever used some kind of Backup software, you can easily think about how many options you can configure: destination folder, backups periodicity, whether to compress them or not and the level of compression, scheduling, folders to include and exclude… Simply configuring the scheduling can make you visit several wizard pages choosing the days of the week, the exact time of each backup and so on. The next screenshot shows the configuration options of Time Machine, the Mac OS X backup system:

You don’t have to worry about configuring anything because the application makes all the decisions for you. And I loved that approach. Even in the case where the backup disk run out of space, the system warns you about it, telling you that it is going to start discarding old backups. It makes the decision for you and let you keep working.

And this way of doing things is totally logical because, after all, your most probable goals when you are using backup software don’t include spending your time creating enterprise-class backup configurations and schedules. What you probably want are two things: recovering some missing files occasionally, or restoring the full system in case of disaster. And both things can be done easily with the Time Capsule. And in the case of browsing the history of your files, the interaction design is totally outstanding.

• If you want to wait for jobs in Eclipse Junit Plugin tests, use this method instead of the proposed in the second edition of the book Eclipse: Building Commercial-Quality Plug-Ins.
• The global buddy class loading policy can affect performance seriously. It is logical and they warn you about it. I can confirm it is true.

Waiting for jobs in Eclipse JUnit Plugin Tests

The Eclipse PDE provides a launcher that let you test your plugins using JUnit. It basically instantiates a new Eclipse workbench and, when it is up, the tests are run inside it. When writing tests, a common need is waiting for jobs to finish. After launching a job for testing its results, the job is run in a separate thread, so the test code will continue running if you don’t stop it.

The book Eclipse: Building Commercial-Quality Plug-Ins is an excellent resource for learning Eclipse plugin development. It proposes some utility methods for using with Eclipse Plugin tests. One of this methods is indeed waitForJobs():

This implementation worked well for us until one day we decided to divide a plugin that had become too big. I started by creating a new plugin extracting some code of the plugin along with its tests. But the tests always failed launching a NoClassDefFoundError exception. I was totally convinced that this problem was related to the way we had configured class dependencies between plugins. We use Groovy and the exception was always launched from the groovy libraries referenced from groovy compiled code. I spend a lot of time trying to figure out what was going on. It was a colleague who pointed out that it seemed that the test code was not waiting for the job under test to finish. And he was right. After searching in google I found a more robust implementation of the waitForJobs() method:

Using this approach solved the problem. This would had nothing to do with performance if I hadn’t messed up with the class loading configuration trying to solve it. Basically I made many of our plugins to use a global buddy class loading policy and, once the problem was solved, I didn’t undo it. And, while we perceived a poor performance in our plugin, it wasn’t until recently that we discovered the cause was precisely that global policy (we thought it was a problem with the development installation of the system our product connects to). And this brings me to the next section.

Eclipse Buddy Class Loading

The Eclipse core platform runs on a sophisticated runtime for Java components (Equinox, an OSGI implementation). This runtime provides a powerful architecture for installing and running Java modules (bundles or plugins). For doing so, it redefines completely the class loading system of the Standard Java Platform. When running inside Eclipse, each bundle has its own class loader. Bundles declare in their manifests how do they relate to other bundles, so their class loaders know how to locate external classes.

When a bundle A depends on a bundle B it can access its exported classes with no problem. The problem comes when it is B who wants to load classes from A dynamically, but it can’t depend on it (because it would create a circular dependence, or just because it doesn’t make sense). For example, the Log4j bundle can’t depend on a bundle that provides a log4.properties configuration file but it still wants to locate it in execution time. We have a number of scenarios like this in our application, where some bundles need to load classes/resources from other bundles they don’t know in execution time.

In these situations Eclipse Buddy Class Loading come to the rescue (another approach, better but more complex, is to use extension points for allowing plugins to be extended in execution time). Buddy policies allow one bundle to declare that it needs to load classes from other bundles (buddies). When a bundle class loader fails to locate some class, it starts searching for it in the declared buddies. The system can configure a number of policies that determine the way classes are searched:

• dependent: search in bundles that directly or indirectly depend on the bundle.
• required: search in bundles that explicitly declare themselves as buddies of the caller bundle.
• global: search in all the bundles.

There are others policies available but I think the first two are the most common. Buddy policies are configured in the MANIFEST.MF file of the bundle that needs to load the classes. For example, in the manifest of the org.apache.log4j plugin you can find this:

Since it uses a registered policy, if you want your bundle to provide a properties file for log4j, you should declare your bundle as a buddy of Log4J.

With a dependent policy this last step is not necessary. As it is obvious, registered is more efficient than dependent, and this one is more efficient than global. In a product like our feature, which is big but not huge, when we changed from global to dependent we observed a huge performance boost, specially the first time the product functions were launched (which is when classes are loaded).

When you try to solve a mysterious bug for long time enough, you are so satisfied when yo do it that you don’t spend too much time blaming yourself for having produced it. And that was my state after this one…

It started when I tried to refactor some duplicated code in my controllers specs. It was related to the authentication system. Specs should test their behavior in both authenticated and insecure contexts, to be sure that anything happens if an anonymous user sends a POST action to your WorldDestroyerController. For example:

The next fragment was recurrent in all my examples.

For every example in the example group, it basically stubs the controller method that validates sessions in order to get a valid one when invoked (I am using authlogic). I wanted to create a macro context_authenticated so I could code the previous example like this:

My first attempt was to write a macro that included the before block and yielded to the block provided in the invocation:

The examples failed because the before(:each) fragment was not being invoked before executing them. To discover the problem I had to investigate a little bit about how the RSpec DSL works. Basically:

• Each context (alias for describe) creates a new ExampleGroup subclass.
• Each it (alias for example) creates a new instance of the current ExampleGroup subclass where it is invoked.

So the problem was related to the very nature of closures. In Ruby, a block of code (proc or lambda) maintains the bindings in effect for that closure. The bindings contains not only variable references, but also the self reference itself.

In my case, what I was doing was to submit a closure with the it examples to the macro. The execution context of this closure was the one where it was defined: the ExampleGroup subclass created by describe UserSessionsController.... However, inside the macro, a new ExampleGroup subclass was created, and the before block was associated with that example group. That was the reason why the the before code was not getting executed before the submitted examples. Once I understood this, the solution was simple:

What the previous code does is to change the evaluation context of the closure, so it get executed inside the ExampleGroup class created by the macro.

The title of the book refers to the the unit of effort man-month. It is used by (I want to think ‘old’) metrics to estimate and schedule software development. As Brooks explains in his book, while cost varies on the product of number of people and number of months, progress does not. It would do it if tasks in software development could be partitioned between individuals with no communication among them. When you take into consideration the need of communication between parts, the effort increases as n(n-1)/2. The famous Brook’s Law appeared in this book as an oversimplifying but eloquent way to capture the problem:

Adding manpower to a late software project makes it later

If you are interested in reading about the importance of communication in software development I recommend you to read Agile Software Development: The Cooperative Game. This book contains the most sensible discussion on the nature of software development that I have ever read. Cockburn defends that software development is a cooperative game of invention and communication.

Returning to the The Mythical Man-Month book, Another thing I loved is how it talks about the human factor in software development. In the chapter The surgical team, Fred Brooks says the following:

The conclusion is simple: if a 200-man project has 25 managers who are the most competent and experienced programmers, fire the 175 troops and put the managers back to programming

When I read this sentence I was totally amazed. Notice that the author was the project manager of the massive software system of IBM System/360 (OS/360). So more than 3 decades ago, a software developer with experience in developing huge programs, didn’t emphasized the need of ceremony and heavyweight recipes for software development. Instead he could clearly see what many people can’t today. He proposed that the most expert programmers should act as surgeons leading a surgical team. The rest of the team should assist them in order to maximize their effectiveness with the core tasks of writing the specs, designing, coding and testing.

Fred Brooks mentions a 1968 study that concludes that the best programmers are 10 times more productive that the worst ones. Robert L. Glass in Facts and Fallacies of Software Engineering talks about numbers ranging from 5 times better to 28 times better. Ignoring this fact is, in my opinion, the source of many deep problems in the spanish software development industry today. Companies don’t see technical talent as an asset to make more money, but simply like something academical.

Although these were the topics I enjoyed most, the book covers many other software development principles which are totally valid today: the importance of prototyping and the wicked nature of the discipline, the dangers of over-designing, the need of cohesive teams with good communication mechanisms… The visionary condition of the author was proved again with his famous article No silver bullet, which stated that nothing in the next 10 years (from 1986) would provide one order of magnitude improvement in software development. The article is included in the book by the way.

In conclusion, it is a book that I totally recommend. The principles it talks about are totally valid today and I really enjoyed reading about them considering the time the book was written. I am simply amazed by the fact that 34 years ago, someone was able to recapitulate so many true and concise ideas on software engineering, even before that software engineering, as a discipline, existed.

So I installed this extension, went to my physical library and added the technical books I own. Then I modified the PHP templates to fit my page theme and show the results in the way I wanted.

My last two Amazon’s orders have been quite big (the current USD/EUR exchange rate has something to do with it), so now I have 16 books waiting to be read in my library (and a few more in my Amazon’s wish list). In recent times I have focused my reading mainly in agile development, ruby on rails, good programming practices and, to a lesser extent, user experience design.

I intend to write reviews of the books I read in the form of blog posts. I’ll link back the reviews from the book backlog, using a Now Reading feature that let you associate a post number to book entries.

Imagine you have a file named say.script with the next contents:

In this DSL, you can say things that are visualized in the STDOUT. Executing this script with Ruby is very simple: just invoke eval() with the code to evaluate it in a context where the say() method exists.

If you want to evaluate the code in the context of some object (for example, an expression builder), you can write something like:

However, in Groovy, the GroovyShell.evaluate() method is not equivalent to Ruby’s eval() method. While both mechanisms can share state between the caller and the invoked code through the concept of bindings, in Groovy the evaluate() method is always resolved in the context of a Script object. This means that the implicit invocation context is not shared. If you try to execute this code:

You will obtain something like:

A workaround for this situation is to convert the script code into a closure. This way, the DSL code is evaluated but not executed. With the closure object you can make the invocation in the context you prefer. In the last example, code bellow will make it:

In the example, we wrap the code to execute with a closure syntax. In this way, when we call evaluate we obtain a closure object. With this closure object we can modify its delegate (invocation context) and then call it.

Finally, if we wanted to execute the code using a specific invocation context (something like Ruby’s instance_eval), we just have to modify the delegate object:

I don’t know of a better approach for solving this problem with Groovy. Initially I thought there will be a direct way, like in Ruby. But after hours of searching for it we didn’t find anything (which doesn’t mean it doesn’t exist, of course).

We started the development with the following SCRUM configuration:

• Iterations of 4 weeks.
• The Release Backlog is composed of user stories which are estimated in story points.
• The Iteration Backlogs are composed of the selected user stories for the iteration, which are decomposed into tasks, that are estimated in hours.

This approach is supported by several books, including the original Scrum book and the superb Mike Cohn’s Agile Estimating and Planning book. The original Scrum proposal recommends 4 weeks as the duration of each sprint (iteration). Regarding to the backlog contents, the original proposal talks about features as the main item for the product backlog, and tasks for the iteration backlog. Features would be estimated in days and tasks in hours.

Mike Cohn elaborated on this proposing to use user stories as the main product backlog artifact, and tasks for the iteration backlog. He proposes to estimate user stories with ideal days or story points, and tasks in hours. It is worth to notice that the user stories technique was popularized by Kent Beck in his seminar Extreme Programming Explained book. In the first edition of the book, Kent Beck proposed to estimate the size of stories with points (1,2 or 3). But in the second edition he advocates for using real time estimates. In the same way, he proposes splitting stories into tasks and says that developers should estimate them, although many XP practitioners prefer not to estimate tasks at all, or even to create stories as small as possible in order to eliminate the need for separate tasks.

After several weeks applying the mentioned approach, I realized that there were some aspects I didn’t like in our planning strategy. The process of splitting stories into tasks and, specially, the estimation of each task was a high time-consuming activity for the project and added little value. I think that this activity was becoming what Lean people call waste:

• Foreseeing estimations at the task level is, by definition, very inaccurate. It is not worth to think thoroughly about how many hours are you going to spend in implementing some subsystem and its tests. Specially if you do it one, two or three weeks before you do it. You’ll be wrong, because software development is a wicked problem.

• If you are going to manage inaccurate estimations at the task level, you have added no value to the story point estimations you already have. These can be converted into real time easily using your team speed so, why bother in the extra-effort?.

• In order to determine the tasks that need to be done for implementing a story, there are two contradictory forces:

  • If you don’t want to get deep in the design (after all, you are in an estimation meeting) you end up with a very similar set of tasks for many stories: design meeting, implementation, testing, documentation…

    Regarding to this, I found very artificial to estimate the hours you spend writing tests and implementing something. Can you separate both activities in a realistic way?

  • If you get deep in the design inside a planning meeting so you can create very specific tasks, then you are mixing two very different activities: design and planning. Each requires a very high amount of energy. I have found far more productive to postpone design discussions for late meetings where people can focus completely in the thing to be designed. When you are in a planning meeting you look at the whole number of stories from a very high point of view.

In my opinion, the main advantage of having tasks estimated in hours is that you can see how many hours are burnt daily in the sprint backlog, so it’s very clear how the team advances each day. However, only with the daily meeting and keeping the stories small you can perceive a good sprint progress information.

While the story-points estimations worked very well for estimating our release planning, I found that splitting and estimation of tasks with hours was very inaccurate and someway artificial.

So we decided to stop splitting stories into tasks (not only stop estimating them in hours, we stopped managing tasks as first-class planning artifacts). We want very small stories so we force ourselves to split them when necessary (while keeping them tracer bullets). And I don’t really miss tasks. Certainly, there are many times where it is useful to write down the high-level tasks that are needed for implementing some story, but we are fine capturing them as informal comments with the story.

In this process of improving our development method, we first used Excel templates for managing our backlogs. After that, we used Agilo: a very nice tool based on trac that can be easily deployed as a VMWare Appliance. It supported perfectly the estimation in story points and hours for the product backlog and the sprint backlog, respectively. When we changed our planning approach we started using Pivotal Tracker. Pivotal Tracker is an amazing tool. Funny enough, I proposed to their creators to have the possibility of dividing stories into tasks and estimating them in hours. Many people liked this feature, which is currently the number 8th in Pivotal Tracker Most popular topics. The first topic was add tasks to a story and they have already implemented it in a very nice and simple way (with no estimations of course).

In sum, in our current process:

• We have iterations of 2 weeks (we preferred a shorter delivery cycle)
• We manage user stories as the only planning artifact in our backlogs (release, product and sprint).
• We use tasks for clarification of some stories, not as a general planning item.