On Agile

Following a couple of other posts relating to agile (here & here), I thought I'd give my thoughts on the subject. 

There are numerous books and posts on the subject of Agile - searching for "Agile Development" on Google gives well over 1 million hits.  As someone who is hugely interested in how to improve software development, I've read a good deal of material on the subject (although, being fair, it would only be a fraction of a percent of those Google hits!).  What I've found very interesting is the number of different opinions as to what Agile means - it's pretty hard to find two books on the subject that agree on a definition of the term.

One thing that is for sure - Agile (whatever definition you may have) is no silver bullet.  Just being "Agile" does not guarantee success.  It does not guarantee the early delivery of rock-solid software to delirious customers. It does not guarantee that your staff will become highly motivated software ninjas overnight. Given that, what's all the fuss about?  The fuss is that previous software methodologies have got a woeful delivery record.  I'm not going to quote some specific number here, since there are a ton of research articles available which give differing statistics depending on exactly what is defined as failure.  However, the general range is anything from 30% - 85%.  Even if you take the best number there, it's clearly an unacceptable state for a multi-billion dollar industry.

Given the lack of consistency for what Agile means, and given the proliferation of processes that claim to be Agile (XP, SCRUM, Crystal etc.), what is the best way for you to move forward?  Alas, I don't have a silver bullet here either.  The best I can do is describe the process that I have gone through to get this stuff clear in my head.

The first step is to clearly understand what the key requirements are for your business.  Ignore software, ignore the process, just focus on what matters to the business.  For me, I came up with a single overriding principal:

To make a decent, long term, sustainable profit

I'm guessing that's not particularly contentious - any business for which that isn't a key principal is unlikely to last long.  However on it's own that's not really enough.  So I then pondered what aspects of the business would best support the primary principal.  This gave the following, in no particular order:

• To create solutions that give the customer the solution that he needs, when he needs it, and at a cost he is happy with
  • Happy customers mean repeat business and referrals, which are the best and most profitable form of sales. Plus I'd much rather be working with a customer who's happy rather than one who's not.

• To have control over the costs and timescales for each customer project
  • Not having control means that the customer doesn't know how much he is paying or when he's going to receive the goods.  Given that the software is probably only a part of a bigger project, the timescales in particular are critical - without confidence on delivery dates, how is the customer meant to schedule other aspects of the wider project such as training, marketing, manufacturing etc?  In addition, if we don't know when projects are due to complete it makes it very hard to commit to start dates new customers.

• To have a motivated, skilled team that share the same values
  • Motivated staff take care and pride in their work, which gives rich rewards in terms of the quality of the product.  They also tend to enjoy their work which reduces expensive staff turnover, and just makes the workplace a more enjoyable place to spend what amounts to a significant percentage of your life.

 

From this, I could then extract the aspect of Agile from the mass of books that I've read to provide, in effect, the essence.  These are the areas that I consider important, and why:

• Accept and embrace change.
  • Anyone who thinks that they can prevent change from occurring during a project of any size is, quite frankly, living on another planet.  Accept that change is essential to enable the software to meet the needs of the customer, and adopt a process that makes change as painless as possible.  The change may be requirements, it may be technical, it may be staffing, but whatever it is, it's going to happen.
• Develop in short iterations.
  • There's a load of important things about the iterative approach, so I'll expand on this below.
• Don't attempt up-front detailed design
  • Again, there's lots of empirical evidence that this doesn't provide any benefit.  By all means, do up front high-level architectural design - indeed, for the key structural aspects of the project (scalability, security, disaster recovery etc.), this is pretty much essential; getting those wrong or trying to reverse them into an existing code base can be very expensive and not something the "right click / refactor" is going to help with.  But the low level stuff is best done with the compiler.
• Make sure that testing is a first class citizen
  • Testing should being as soon as coding begins (indeed, if you want to do TDD, then it starts before the coding).  As far as possible, make the tests automated so that you can frequently run the full test suite.  It's inevitable (regardless of whether you attempt up-front detailed design or not) that, at times, you'll need to refactor parts of the code base to support new functionality.  At times like this, a large test suite gives a great safety net.  In addition, the tests (if well written) also act as a form of executable (and hence up to date) documentation.  Finally, and most importantly, testing as early as possible tends to promote a testable code base and gets quality in there from day one. 
• Empower your team to use their brains.
  • If you've any sense, you spend a lot of time recruiting the very best staff.  Recognize that, and let them shine.  If they are committed to the business, then trust them to make sane choices and don't try to micro-manage tem.  If they're not committed to the business, politely but firmly point them in the direction of the door.

Expanding on iterations, I think the following aspects are essential:

• At the start of an iteration, plan in detail what you are going to achieve.  For that plan to have any teeth, it is essential that changes are not allowed during an iteration.  For this reason, iterations should also not be too long - my experience suggests that between 2 and 4 weeks works well.
• During the iteration planning, ensure that the tasks being worked on are the most important to the project at this moment in time.  Don't do the trivial stuff whilst there are important things to be done (important can either be those items that give most business value, or those items that present the most technical risk)
• At the end of each iteration, deliver demonstrable, working software.  This keeps the team focused and gives a clear view of progress to date.  In addition if, god forbid, you fail to complete all the development tasks you will at least have a system that the customer could take.  And, since you worked in priority order, it should include most of the stuff that the customer considers important.  Telling the customer that you're not finished is never an easy conversation, but "We're not done, we estimate that we're about 80% of the way there, but here's a system in which all of the following function is complete and ready to go" is a much better chat than "We're not done, we estimate that we're about 80% of the way there.  Sorry, but there's nothing you can take yet because until we've done the other 20% nothing will work"
• At the end of each iteration, tasks are either done or not done.  It's notoriously hard to determine how much work is left on a task when it's not yet complete (how often have you heard the phrase "it's 80% done", only to then find it takes another 100% of the elapsed time so far to finish?).  In addition, done needs to be done.  Code written, all functionality complete, all tests written and passing.  Anything less is not done, and should hence be deferred to a future iteration.
• At the end of each iteration, evaluate what has gone well and what has gone badly.   Do more of the good things, and make changes to prevent the bad things from happening again.  I have seen a number of teams running iterations who recognize that they are not getting things done as quickly as they need, and whose response is "ok, we recognize that things aren't going well.  We'll try harder in the next iteration".  Trying harder at something that isn't working is unlikely to yield the results that you want.
• Don't queue up bugs.  The tasks that you've worked on to date are, by definition, the most important ones.  Bugs mean that they are not finished.  Fix the bugs.  If you don't, then at the end of the project you'll have a pile of important stuff that's not done.  The customer is not going to like that.

The key thing that I've observed with well-run iterations is that they tend to surface problems early in the project.  Pain is going to happen (what - you really think that nothing will go wrong?), and Early Pain is considerably more desirable than Late Pain.  Early Pain means that there's time to take corrective action. Late Pain is what kills projects.

Those are the things that I see as the essence of agile - it's not rocket science, it's just working smarter.  It's understanding that change happens, and making sure that you can handle it.  It's understanding that things go wrong, and making sure that they can be spotted and fixed as early as possible.  It's understanding that you have a hugely talented team, and using them.  It's no silver bullet though.

What else can I add?  Well, there are a few things that spring to mind:

• Requirements - up-front or iteratively?
• Contracts - fixed price or T&M?
• What methodology?  SCRUM?  XP?  Lean?
• We're not Agile, but want to be - how do we change?
• My customer doesn't want Agile - what are my options?

This has been a pretty long post, so although I've got things to say on those, I'll leave them for another day.

As a final remark, all of the above is just my opinion.  I've intentionally not put in references to books etc. - for each reference I find that says one thing, I've no doubt you can find a reference that says the opposite.  Such is the nature of our imprecise world.  I hope, however, that this does give some food for thought, and perhaps helps you through your own thought processes around how (or indeed if) to adopt Agile.

I would certainly appreciate any comments that folk might have...