Distributed Agile

The question was asked on Linked In on how companies handle Agile development in distributed environment with outsourcing vendors. While team co-location is an ideal (I would say the easiest) environment for Agile development, it is not an option for many companies. Even if you do not outsource, chances are that your development team is spread out between multiple locations. Acquisitions are way too common in high-tech.

So if we have no say in a team’s location, the question really is: which is better - distributed Agile or distributed waterfall?

Personally, I got the answer to this question after implementing Agile a couple of years ago with development teams in Vancouver and Boston. It worked much better than the plan-driven approach we have used in the past.

You do need to modify some Agile practices, of course, by moving Agile process artifacts from their physical environment (white boards, sticky notes, …) to an online one. I used Microsoft SharePoint at that time. You can also look at commercial tools from Rally or Version 1. The way you communicate will change as well, as you supplement face to face meetings with IM, VOIP, videoconferencing, and wikis.

Agile techniques actually help you to address and mitigate the usual challenges of a distributed / off-shored project: lack of visibility on project status, delay in feedback cycle, loss of business and technical contexts, decrease in communication bandwidth, higher documentation overhead, and mistrust.

Short iterations, with a product demo and retrospective at the end, increase visibility of the project status and provide instant feedback as well as an opportunity for process adjustment. Customer involvement facilitates both a shared understanding of business context and communication between business people and the development teams.

Frequent team meetings help to build trust and improve communication on all levels and across different groups. Continuous test and integration cycles tell you where you are in the project.

Functional Test Driven Development helps in removing ambiguity from requirements and clearly communicating them to distributed teams. Fit (Framework for Integrated Testing) is a great open source tool that we are using (http://fit.c2.com/).

In my current company, Luxoft Canada, we successfully use Distributed Agile in a number of projects with teams located in the US, Canada, and Russia.

If you want to learn more about Distributed Agile read the book by Dean Leffingwell: “Scaling Software Agility, best practices for large enterprises”. Dean is a well known name in the s/w industry and was in charge of developing RUP at Rational Software.

He writes that “at scale, all agile development is distributed development. … even the largest or most distributed teams can achieve the faster time to market, higher productivity, and higher team morale that the agile methods provide.”

In the book Dean describes a case study of a Denver company called Ping Identity that is using SCRUM with Luxoft’s team in Moscow.

Agile is getting further acceptance in world of outsourcing and this trend is going to continue. Watch for new tools and techniques that are emerging to make it easier and more efficient.

On engineering practices in Scrum

The following couple of articles helped me to understand better the nature of the Scrum.

When is Scrum not Scrum? by Tobias Mayer and Differences Between Scrum and Extreme Programming by Mike Cohn

Both articles pay special attention to using engineering practices in Scrum. Both agree on importance of the practices and their value for the development process. More radical Tobias (he was expelled from the Scrum Alliance) thinks that they should be mandatory. Mike takes less categorical approach.

This is the bullet point list from "When Scrum is not scrum":

1. Product Owners are part of the team.
2. Two-week Sprints
3. Tasks are not measured in hours
4. Use of Taskboards rather than spreadsheets
5. Backlogs on the wall
6. Estimation Meetings
7. Insistence on Agile Engineering practices
8. The Scrum Master role is not always necessary

Here is an excerpt from Mike's article:

"I find true XP to be a small target off in the distance. If a team can aim at that and hit the bull’s eye, wonderful. If not, however, they are likely hacking (e.g., refactoring without any automated testing or TDD). Scrum is a big bull’s eye that on its own brings big improvements simply through the additional focus and the timeboxed iterations. That’s a good starting point for then adding the XP practices."


-Alex

Experiencing pair programming

At Luxoft Canada, working in a team of four developers we were able to pair and work on specific tasks that were taken from the Scrum task board.

Physical aspect of pairing involved having two developers (navigator and driver) sitting behind a computer connected to two sets of keyboard and mice. We managed to change our desks to rectangular desks to easily accommodate two programmers working behind the same desk.

When we began pair programming, some participants had no or little desire to participate. The only way to overcome this lack of desire was to experimentally try pair programming. I tried it and gradually grew into it and started seeing its tremendous benefits to the organization and us.

While working in a pair, there were cases in which we could not accept the other person’s point of view about solving a problem. For example, in one case I insisted that we needed to add more unit tests for a problem and my partner was insisting that what we have written was enough. We spent a few minutes explaining our positions and finally I had to compromise by not introducing the new unit test and leaving the decision to be made at a later time.

I paired with two people multiple times. On a few occasions the navigator programmer felt more skilled than the driver in performing some actions (as simple as using short keys) or the navigator was feeling s/he knows the area of the application better than the driver programmer. In these cases even though the navigator explained the techniques to the driver, it took a while for the driver to pick up the techniques and get used to using them. The navigator who had explained the tricks to the driver and still was seeing the driver using the old techniques felt frustrated. We learned that the navigator should explain the techniques and allow the driver to pick them gradually over time.

In other cases where we had some bad programming habits that pair programming surfaced them and gave us a chance to change them.

There were a couple of cases in which the driver would not give the navigator a chance to code. After a short period of time we all felt that this makes the navigator feel marginalized and lose his focus on the task. As soon as this problem was recognized, we made sure to switch the roles often.

In some cases in the driver seat, I was finding it difficult to explain what I meant when proposing a specific solution. I found it easier to take a minute to write the piece of code and show it my pair. Once, it involved modifying the code my pair had written a few minutes ago. It was good that he did not mind me touching his code and he was patient enough to let me take my time to explain what I meant using written code. I think the navigator should trust the driver and let him to take his/her time to explain his/her point either verbally or in written code.

I believe some of us occasionally create negative images of ourselves when we feel we are not functioning our best and start expressing the feeling verbally. I have done this and heard others doing the same. This usually does not have good outcomes. I have seen that expressing the frustration and creating negative images make things worse and affects the outcome of pair programming.

We made sure to have a few minutes of informal retrospective about pair programming experience among ourselves. This was done at the end of the day or when a task was completed. The talk about our achievements, what we really did well and what could be improved, educated us to do better each time.

For us:

1. Pair programming is an extremely efficient way to do general technical and application knowledge transfer.
2. Promotes team communication.
3. Results in better design.
4. Helps eliminating buggy code early or not introducing it in first place.
5. Increases team productivity since the team members will have undivided attention during coding.
6. Improves communication and collaboration skills of the team members.
7. Makes work more fun.

Fit from a Developer's Perspective

In an earlier post, Mandy wrote about the benefits of using Fit from a business owner’s perspective. I’d like to add to that and describe some of the benefits that I’ve seen as a developer working with Fit from deep in the trenches.

In our development process, we attempt to create Fit tests for new user stories up front in collaboration with business owners (let’s call this “Fit-driven development” - because we love the “X-driven” paradigms). This has several advantages for developers. First, we learn the language and rules of the business. This is crucial to any attempt at domain-driven design (see?), in which we carry the language and concepts of the business domain deep into our applications. Second, it gets us collaborating with the business owners on a day to day basis beyond the bounds of the SCRUM meeting. This builds knowledge of each other’s worlds and, more importantly, empathy, which is key to a building a positive working relationship. Third, we add to the test coverage in our system – tests that are usually at a level higher than unit tests, like integration or acceptance tests. All of this contributes to being able to get it right the first time.

Now, to be honest, we don’t always develop Fit-first but when we don’t I often find myself wishing that we had – for example, the time when I discovered that the language I used in my code was woefully inconsistent with the language the business uses, or the time when we implemented a core piece of logic and had to rewrite it when we learned that we had misinterpreted the partially-erased scribble on the whiteboard - shock! As a developer who must admit to rewriting code too frequently, tools, like Fit, that help to get it right the first time are whole-heartedly welcomed.

Extending Fit with a New Fixture

Fit is a very flexible tool for testing due in large part to the pre-defined fixture types (ActionFixture, ColumnFixture, RowFixture, fitlibrary’s DoFixture, etc.) This flexibility means it’s usually possible to write test tables in a way that is intuitive and easy to read. However, we’ve come across a situation where the test tables we want to write aren’t executable by any of Fit’s built-in fixture types.

The feature we’re working with is a Fit HTML table builder. Our test provides a fixture classname as input and expects a Fit HTML table as output. Here’s an example of what we wanted to test:



Unfortunately, when we ran this test using a ColumnFixture, Fit interpreted the nested table as more parsable cells rather than expected HTML output. And, while there are other ways to structure the test, such as using raw HTML, none of them seem very easy to work with.

What we really need is a new type of fixture, which interprets the table data in the last column as HTML content to be compared with the result of the fixture’s verification method (in the case above, table()).

public class HtmlTableFixture extends ColumnFixture {

@Override
public void check(Parse cell, TypeAdapter adapter) {
   String expected = cell.body;
   try {
       String result = (String) adapter.get();
       if (expected.equals(result)) {
           right(cell);
       } else {
           wrong(cell, result);
       }
   } catch (Exception e) {
       exception(cell, e);
   }
}

@Override
protected Binding createBinding(int column, Parse heads)
   throws Throwable {
   Binding binding = super.createBinding(column, heads);   
   if (column == columnBindings.length - 1) {
       binding = new HtmlTableBinding(binding);
   }
   return binding;
}

private static final class HtmlTableBinding
   extends Binding {

   private final Binding internalBinding;

   private HtmlTableBinding(Binding binding) {
       this.internalBinding = binding;
   }

   @Override
   public void doCell(Fixture fixture, Parse cell)
       throws Throwable {
       Unparse unparse = new Unparse(cell.parts);
       String html = unparse.text;
       cell.body = html;
       internalBinding.doCell(fixture, cell);
   }
}
}

Our fixture creates a custom binding, called HtmlTableBinding, which gets bound to the last cell in each row. This binding unparses the cell’s contents and stores them in the body of the cell’s Parse object. When the cell is later evaluated by check(), our fixture gets the expected value from the body of the cell's Parse object and compares it with the result from executing the method (via the TypeAdapter).

Running the test now gives us:



So, Fit is not only a flexible testing tool, it's also easily extensible for those corner cases where you might find that it doesn't give you exactly what you want.


* To understand why we need to unparse it helps to know how Fit reads HTML tables. When Fit is run on a file, the first thing it does is create a model of all of the table information, which it stores in a composite object called Parse. Each part of a table (i.e., <table>, <tr>, or <td>) is a nested Parse object within the composite. Unparsing is achieved by traversing the composite Parse and appending the HTML parts to a buffer.

public class Unparse {

public String text;

public Unparse(Parse parse) {
   text = unparse(parse);
}

private String unparse(Parse parse) {
   StringBuffer sb = new StringBuffer();
   sb.append(parse.tag);
   if (parse.body != null) {
       sb.append(parse.body);
   }
   if (parse.parts != null) {
       sb.append(unparse(parse.parts));
   }
   sb.append(parse.end);
   if (parse.more != null) {
       sb.append(unparse(parse.more));
   }
   return sb.toString();
}
}

Agile 2007 Conference

I recently found out that Stephen, Michael, Askhat and I are attending the Agile 2007 conference in Washington, D.C. August 13-16th. We hope to bring back some fresh ideas about Agile software development and lots of blog post material. Come say hi at the Luxoft booth!

Why We Scrum

I read an article today complaining that scrum was just another useless meeting and that the time would be put to better use speaking to the other developers about design issues one on one. I tend to disagree.
At a previous company, during our waterfall days, product managers tended throw requirements over the wall as they were busy. As a tester, I often found large discrepancies between my interpretation of the requirements and what the developer had coded into the product. On occasion I had to strong-arm developers into the product manager's office so we could come to a common understanding of a feature.
We had product development offices in Boston and Vancouver, so often the product manager, tester and developer were not co-located and had never met.
Scrum helped alot. When daily scrum meetings started up, we met our cross-department distributed team (face-to-face or over phone/video conference) and had a daily opportunity to discuss all the little questions and blockages that come up as you code and test. We came to a common understanding daily, and if we got astray from the feature vision the next day, it was still recoverable.
In conclusion, scrum breaks down the barrier of departments and lack of co-location, and gets a team communicating and collaborating. Cause many brains really are better than one.