Showing posts with label Scaling Agile. Show all posts
Showing posts with label Scaling Agile. Show all posts

Wednesday, September 23, 2009

Respect the elder bugs

Billy (below) is a known old bug. Old enough that it got a name and is well known to the development team.


Bugs, or defects in your software, are not to be treated as pets! Make sure you close bugs fast. Do not get used to bugs.

Agile software development (as typically is the case in the software industry) does create bugs. In fact I came across new Agilists that were surprised because Agile development practices also create bugs; in fact, I have seen more bugs (I am not talking about defects in production) in Agile development than in more traditional development.

But what I expect from a well-working Agile team is that:

  1. The number of bugs in production is low

  2. Bugs are closed very fast


Test Early and Often


Agile development practices such as continuous integration, test driven development, and automation facilitate the quick detection and elimination of bugs. Your customer tests (test validating the customer requirements) will detect any addition or change that breaks an existing test. Your developer tests (tests validating the design of your application) will detect anything that breaks the expected design and behavior of the application.

Disclaimer: for smaller teams, this will be as simple as identify and fix the recognized problem. But for very large teams, it might get a little more elaborated, and a bug is created to track the issue (at times, the issue itself is that there was no automated test).

Scaling Agile


Scaling Agile has a few nuances to it. And bug management and resolution is amongst the development practices that require special attention for a large Agile team. From my experience on a large Agile program, I second the importance to classify bugs by priority and severity.

Elisabeth Hendrickson wrote a very good article about it: Arguing Apples and Oranges.


But even with good bug classification and management, I noticed a few old (pet) bugs living in my large Agile program.

Consider the program had an average rate of 100 new bugs per week. Don’t be alarmed, it is a huge program. Imagine 100 developers continuously integrating code for 200 user stories per week. And an average rate of closing 100 bugs per week.

The depicted scenario can be of a very healthy Agile program. For instance, a small period of regression ensures the production readiness for the system. As Regression I mean the period allocated to stabilizing--fixing the bugs-- the next releasable artifact; not adding new features.

But how is this sample program creating old pet bugs?

Low priority, low severity bugs. These are the bugs that can live a long life.

If the reasoning for deciding the order to attack the bugs only considers severity and priority, you might be creating pet bugs. I did recognize a few elder pet bugs (for this reason) in my large agile program.

My recommendation is to adjust your bug triage formula. You should consider severity, priority, and the age of the bug.

I am sorry Billy (my old pet bug). Even though you were low priority and low severity, it is being embarrassing to have your small –almost inoffensive—existence in the system. I am sorry, but you have to go!

Wednesday, September 16, 2009

The Retrospective of Retrospectives

Feedback is one of the XP values and key mechanism for all the agile methodologies. For example, Scrum teams religiously have Retrospective at the end of each iteration. This enables the team to reflect and adapt, improving the process for the next iteration. Agile Teams learn from their successes and failures. Retrospectives provide an effective way to incorporate feedback into the project.

But what about large Agile teams? Once again I will use my current large Agile program as a case study:

Lately I have been working on a large Agile program. With the large Agile program, comes common large team challenges, and the need to adapt the Agile development and management practices to better suit the large team challenges.

Currently the program has around 180 people distributed over 12 teams. Typically, each team is formed by 15 people (project manager, business analyst, quality assurance and developers; for short: PM, BAs, QAs and Devs). All teams in the program are following Agile practices, such as 2 week Iterations, Daily Scrums and End of Iteration retrospectives.

The teams were getting good value out of the Retrospective Agile practice. But the improvements were done in a team level. And the overall program was losing focus on continuous improvement. At times, the action point from one team’s retrospective was dependent on other teams. Other times, two team retrospective action points were contradictory.

Another challenge was that the program level management was not able to attend every team retrospective. There were 15 retrospectives happening at the end of each Iteration, The program manager (and other people) would like to attend each team retrospective; but it was just not possible to be at several meeting at the same time.

Retrospective of Retrospectives


Similarly to how the daily scrum is extended to the Scrum of Scrums for large Agile teams, I came up with the idea of having a Retrospective of Retrospectives for large Agile teams.

The Retrospective of Retrospectives (RoR for short) meeting is held after each individual team retrospective. The participants (representatives of all team and the program) meet to discuss what went well and what to improve in the program level. The RoR focus is on improving the overall program output, and not on individual team performance.

RoR format


Data gathering: The top 5 items from each team retrospective is the start point for discussions and possible program action items.

Participants: Similarly to the Scrum of Scrums, all managers participate in this meeting. The attendance of another representative of each team is recommended; and it is encouraged to be in a rotation basis.

Context: The context for the RoR is slightly different than the context for individual team retrospectives. While the context for a end or iteration retrospective might be: what can we do to improve next iteration for our team, The RoR context is: What can we do as one team (the whole program team) to improve the overall output for the program.

Mechanics: Each team retrospective top items are read. Repetitions and alike items are grouped together. Then the items go for voting. (for example 3 votes per participant).

RoR top benefits


Below are the top benefits I noticed from the RoR.

Quick access to all teams – all managers and stake holders have visibility on the top items for each individual team. And all is available in only one meeting.

Focus on improvement – for both the individual teams and the program level. One common problem on large agile projects is that one loses focus on the overall program by constantly focusing on local improvements. It is like being around the trees and lose sight of the forest.

Common goal – All managers (and the selected team members) are periodically talking and acting on the overall goal of the program. The RoR participants carry the common goal message (and action points) back to their teams.

Everyone is heard– Any local team problem will be heard by all the teams. Even if something is specific for a team, the message (team retrospective item) will be heard by all teams.

Friday, July 17, 2009

CI, structural changes and merges

Be mindful of the amount of structural changes while in the period that will require more code merge effort.
This was the message I sent to the programmers on a large agile team I have been working with.

Scaling agile has a few caveats to it. This blog post is an awareness message for programmer, which I believe to be complementary to the Continuous Integration practice for large teams.

Below is the message sent to the large Agile team.

We are a large team (+120 programmers: devs and QAs)

Programmers are coding everyday; despite of release candidates and regression periods. How is it possible?

We branch (prior to a new release), then we merge back (after it has been released); all the time ( release branch strategy )…

Consider the following simplified example of a Release formed by 2 dev iterations and 1 regression iteration.

Sample Project Release and Iteration plan

Dev activity –-It1—It2—It3—It4—It5—It6—It7—It8—It9--ItN--
Regression –------------Reg 1----Reg2-----Reg3—-------------
Release sch –------------------Rel1-----Rel2—---Rel3----------


So let’s look at different moments in time:

It2:
Dev activity –-It1—It2
Regression –----------
Release sch –---------

All programmers are working in the main trunk. Any regression activity is happening in the main trunk. There is no release yet. The release candidate is the main trunk itself.


It3:
Dev activity –-It1—It2—It3
Regression –-----------Reg 1
Release sch –-----------------

The release is formed by two dev iteration and one regression iteration.
In this case, the work done for Rel1 is related to It1 , It2, and Reg1; but should not be impacted by It3 development. Therefore the need to branch out the codebase: main trunk, and R1 branch.

Humm… if there is a branch, there will be a merge later: structural changes will have to be merged back!

It4:
Dev activity –-It1—It2—It3—It4
Regression –------------Reg 1----
Release sch –-----------------Rel1

R1 has been released. The code is merged back to the main trunk. The R1 branch now should have very little activity (production support only), therefore it should not create any difficult merge.
Programmers are mainly working in the main trunk; no merge required (remember, we do Continuous Integration!). Go crazy, do the structural changes that you feel are amazing (btw, if it is not, CI will catch it).

It5:
Dev activity –-It1—It2—It3—It4—It5
Regression –------------Reg 1----Reg2
Release sch –-----------------Rel1-----

The development period for R2 is over (It 3 and It 4). The regression work should not be impacted by the development work from It5. A branch for R2 has been created. If there is a branch, the code will have to be merged later on… Please be mindful of structural changes as you will have to merge them.

Have you noticed the pattern?

In the sample presented above:
It 2, it4, It6 development is happening in the main trunk which won’t require large amount of merges later on.

It 3, it5, It7 development is happening in the main trunk, but there is lots of development happening for a release, and those will have to be merged back in the next Iteration.

So if you have a big restructure to be done; should you do it in It7 or It8? (Remember, there is no free lunch, you will have to merge it back!)