Agile is more predictable

In a series of posts I’m going to examine some of the claimed benefits of agile methods – are they justified? My first post looked at the cost of development with agile, the second discussed speed. while the third addressed quality. Here I look at claims of un-predictability of agile.

What do we mean when we say agile methods sacrifice predictability for adaptability? Here I want to explore this commonly held belief – is it true?
In this context, predictability normally means the ability to deliver to a predetermined plan – predictable from the customer point of view.  The scope agreed is delivered with the necessary quality at the time and cost agreed.

But research shows that waterfall very rarely achieves this objective – in fact the record for plan-driven approaches is woeful! A now famous Standish report states that 31% of projects were cancelled before completion, while in 53% costs nearly doubled the  original budget. In fact only 16% came in on time and on budget. This report was from 1995 – when plan-driven was well established and agile methods had yet to make their appearance.

I would argue that in contrast agile methods bring predictability in several ways:

1) Timeboxed – Because agile treats scope rather than time as variable, ‘something’ will be delivered at the pre-determined milestone – the schedule does not slip.

2) Agile addresses high risk items early, like testing and end to end integration. These often prove to be the reasons plan-driven methods are so un-predictable.

3) Even if some scope is not delivered at the planned milestone, agile uses prioritisation by value to ensure the most important features are delivered (45% of features are never used according to the same Standish report)

4) Agile borrows from queueing theory to devise techniques to reduce variability in the development process – smoothing work flow through small, equally sized user stories greatly reduces queuing time and bottleneck formation in the process, delivering more reliable throughput.

5) By delivering potentially releasable code in each iteration, agile provides more visibility into real progress compared to the plan-driven approaches – this provides a more reliable basis for stakeholders to revise plans based on the reality of the project rather than any now incorrect plan.

6) Agile methods use inspect and adapt to adjust plans to emerging reality – regular reality checks mean more reliable predictions of milestone deliveries are possible.  In systems engineering terminology, such closed-loop systems are less reliant on every component of the system working in order to be predictable – instead they can compensate for unreliable components.

7) Agile methods encourage parallel work on various tasks and user stories – rather than analysis preceding development followed by test, these are performed concurrently.  Again, queueing theory shows us that this greatly reduces the variability of any process thereby increasing predictability.

Plan-driven methods can lend an illusion of control and predictability – and can serve political or covert roles by avoiding ‘whole-team’ accountability. But both experience and theory highlight that predictability is, contrary to common belief, not their strong-point.

Judging a book by its cover

In his now infamously named book “Balancing Discipline and Agility”, Barry Boehm, creator of the Spiral method, juxtaposed plan-driven and agile methods in terms of their discipline. His main point is that both planned and agile are valid methods depending on the context. His criteria for deciding which to us include the experience and skill of the developers, the criticality of the application, the culture of the organisation, the size of the team and the stability of the requirements. Many critics have suggested that what makes agile methods, or indeed any methods based on lean thinking (I believe agile is a lean method) work precisely because of their disciplined application. Although they must be modified to a certain context, that does not mean you can just cherry pick those practices that allow you continue with your current method pretty intact. That approach leads to a superficial adoption of agile when the underlying method does not support the core principles and values of lean and agile. However, I have another two critisisms of the ‘discipline or agile’ approach:

  • It assumes planned methods are more disciplined, or enforce discipline more effectively than agile ones – this is a fallacy. Research suggests that where planned methods are employed, a mere 6% of projects follow the defined method closely, whether that method is customised to a certain context or not  (see Fitzgerald, B. (1996). “Formalised systems development methodologies: a critical perspective.” Information Systems Journal 6(1): 3-23.)
  • It assumes there are only two basic types of method – a false dichotomy.  A recent Forrester Survey 2010 shows that only 13% of projects use waterfall methods, and 35% use various agile methods. But another 30% do not follow any method (ad-hoc) and 21% use iterative methods like RUP. Are methods other than agile and plan-driven ever valid? Boehm does not explicitly exclude them, and I see situations where they are valid in certain contexts.

Boehm has contributed hugely to the development of software engineering over the decades. But this book title gave rise to myths such as “agile can’t be relied on to produce quality and therefore critical systems” and “agile only works for experienced teams”. These myths are gradually being deconstructed as agile is scaled across enterprises composing large as well as critical projects, novice as well as expert developers. By focusing on the roots of agile in lean thinking, first generation agile methods can be extended to work effectively across a much greater array of contexts.

The Agile Innovation Dilemma

One the seminal events for the development of the agile software development movement was the 1986 publication in HBR of “The New New Product Development Game” by Nonaka and Takeuchi. Describing lean production principles applied to new product development, the paper introduced the metaphor of a rugby team where a clear goal, overlapping skill sets and joint accountability allow teams dynamically adapt and self-organise to achieve their objectives despite unforeseen setbacks and challenges. From this, the term scrum was used by Sutherland and Schwaber in 1995 to describe an incremental, team based approach to software development. In this way, agile development and innovating new products share a common lineage.
Agile methods have long been advocated in supporting innovation. They explicitly call for self-reflection and improvement of the method through retrospection. Close customer contact and an understanding of the business problem to be solved can help the development team develop more innovative solutions than if they were coding to a static functional specification. Proponents have written of ‘hyper-productive’ scrum exhibiting ‘punctuated equilibrium’ leading to discontinuous or radical innovations. However, agile practicioners offer an alternate view – the intense focus on delivering small increments of customer centric features in the immediate future undermines their ability to seek out alternate solutions, to ‘think outside the box’. This is what I refer to as the “agile innovation dilemma”. So what has gone wrong?

Agile methods promise, first and foremost, agility – that is, flexibility in responding to changing requirements, technology and markets. But in promoting agile, more traditional values such as productivity, quality and time to market are often to the fore.  In efforts to realise these benefits, agile implementations often become exercises in micro-managing development teams, with local optimisations coming to the fore. These are reflected in many of the ‘patterns of failure’ seen in agile projects, and I’ll be describing some of these in future posts. Unfortunatly, these tend to mitigate against innovation, resulting in ‘pressure cooker’ development environments where all focus is on delivering the next increment of functionality, where the product owner is breathing down your neck, and where doing the simplest thing can take precedence over doing the ‘right’ thing.

When did you last hear of agile being introduced to enhance the innovation of a team?

Agile delivers better Quality

In a series of posts I’m going to examine some of the claimed benefits of agile methods – are they justified? My first post looked at the cost of development with agile, while the second discussed speed. Here we address quality. Commentators often say agile methods are not suitable for life or mission critical systems – for some reason they believe the quality cannot be delivered reliably using agile. But there is a growing opinion that agile methods can deliver even better quality than planned approaches. I believe it’s not anything inherent in agile methods that can lead to lower quality, rather it is a lack of discipline in applying the method. Discipline is required to ensure high quality in any method, agile or not. Unfortunatly agile is perceived by some (unjustly) as not demanding discipline, and hence not ensuring consistent quality. Here are some reasons I think an informed, mindful and disciplined agile method can be considered more reliable in terms of quality than planned methods:

  • Quality is built in – not added afterwards: Iterative development encourages continuous test – at a minimum every iteration. In agile methods with automated test, this is increased to daily, or even hourly, or every time the code base changes. With test driven development, tests are developed and executed even before the development begins.  This encourages quality coding from the outset, and the repeated and up front emphasis on testing engenders a quality culture in the team
  • Waterfall methods seperate responsibility for development and test – they encourage an antagonistic relationship between developers under pressure to churn out features and QA who have the lonely job of policing the quality of the system.
  • QA is normally the last major task in a waterfall model, following requirements, analysis, design and development. Therefore, it is usually conducted in limited time and under severe pressure at the end of the project – conditions not conducive to rigorous and insightful test.

But quality is not confined to the coded implementation itself:

  • The quality of requirements are improved through close customer collaboration and face to face interaction. Detailed requirements are determined only when required, just before implementation, and therefore involve less ‘future gazing’. Requirements are described in a user centric fashion, and can be better understood by the customer than more technology centric descriptions usual in traditional methods.
  • The quality of the plan, and therefore the predictability of the project, is improved through continuous replanning for each iteration, by addressing high importance and high risk items early on (eg integration) and by the transparency offered by measuring progress through ‘done’ features and the value they represent to the business.
  • The quality of experience for stakeholders is improved: customers who get what they want earlier, sponsors who get happier customers, product management who get better visibility and more options for managing the development, and developers and testers who get a more motivating and sustainable work experience.

Taken together, I believe agile can deliver better quality software than planned methods. Agile is rooted in lean thinking, a set of philosophical tools that helped Japanese companies reach new levels of manufacturing and product development quality over several decades.  However, discipline in their application, as with any method, is not optional.

Extending the Scrum and Sprint Metaphors

Scrum uses the metaphor of a rugby team to convey the ‘directed autonomy’ of a ‘whole team’ working together to get the ball across the try line. Each member of the team has a role such as out-half, fullback or forward, but once the ball is in play, any member does whatever they can to progress towards the objective, regardless of their speciality. This is contrasted to a relay race team, where each member runs part of the race on their own, passing the baton at pre-determined points. This echos a waterfall process, with distinct roles and responsibilities and hand-offs between task specialities.
Another metaphor used in scrum is that of a sprint. During a sprint, distractions and interuptions to the team are minimised – the ScrumMaster has a specific responsibility to ensure they can focus purely on acheiving the preset sprint goals. This minimises context switching and multi-tasking which have been shown to be very wasteful, and increases the predictability of what will be delivered at the end of the sprint. But no-one can sprint all the time – if they try, then they are really just jogging. I like the sprint metaphor – there is one immediate, overwhelming objective and all distractions and other concerns are ignored until the sprint ends.  And during a sprint, ‘moving’ the goalposts is not allowed – no changes to the sprint goal or user stories and tasks is allowed (if this is necessary, then the sprint is cancelled). Just like a play in rugby, all focus is achieving one objective until the whistle blows.

But the ‘classic’ scrum method (Schwaber &  Beedle),  recommends each sprint is followed directly by another – sprints include ‘bookend’ events at the start and end, such as iteration planning, reviews, retrospectives, etc. Classic scrum also calls for just a little bit of distraction sprinkled throughout the sprint such as spending 5-10% of the time on ‘grooming’ the product backlog – allowing the next sprint backlog be prepared in advance of the subsequent sprint and therefore no need to dilly-dally before rushing into development again. Activities such as these require the focus of the team be diverted from the immediate sprint goal to focus on future stories and tasks.

I have a number concerns with this model.

  1. Distractions, multi-tasking and the consequent context switching are very disruptive to concentration and sap energy from a team. Try doing some real, focused work like writing a report or some complex code with constant interruptions such as calls and meetings about different topics – its very difficult to progress rapidly and steadily with the primary task. But people in the real world have to do ‘other’ stuff in the average workday: office communication sessions, performance appraisel with their boss, arrange or attend a presentation to another department, participate in the mentoring program, file travel expenses, attend the mandatory office security training session…..the list is looong. Sometimes a team will count on only 6 hours out of 8 being available for ‘real work’ because of these distractions. But how can you sprint efficiently if your not running 25% of the time? If you have to slow down, attend to something else, and then build up sprint speed again it is very wasteful.
  2. Agile purports to be an empirical control process – rather than planning the future based on experience of the past it instead uses an ‘inspect and adapt’ paradigm to continually refine the short-term plan based on present reality. In agile methods, this takes the form of getting feedback from stakeholders (customers, users) at the end of each iteration, and, based on that, building a plan for the next sprint. In many real world cases, the next sprint has already been substantially planned by the time the previous one is finished – team members have been grooming the backlog, breaking down stories and tasks, estimating, etc. Although this model impinges on the pure inspect-adapt model, many would see it as pragmatic – a sensible compromise to optimise utilisation of the team. However, I have seen this quickly lead to local optimisation (great for getting more work done, but not necessarily delivering more business value), premature investment in story elaboration, and erosion of team ownership of stories and estimates (since often one or two team members will work on stories for the next sprint while the remained deliver the current tasks). Reverting to the rugby metaphor, its like some players disengaging and setting up the next set piece before the play has ended.
  3. I discussed the idea of cadence in a previous post – cadence across teams and projects is a great basis for synchronising inter team communications. For example, in a multi-team project if one team is sprinting while the other is between sprints, there is little opportunity for joint reflection. If there is time set aside between sprints to synchronise between teams, and if the sprints start and end at the same time, it gives an opportunity for this communication. It also gives an opportunity to go ‘off-project’ and do all that ‘other stuff’ not directly related to the project. For example, what if performance reviews, training, presentation of company quarterly results, etc could only be scheduled between sprints – this would involve a cadence at the company level operating across all functions.

Again, like the rugby metaphor, I think a break from the flurry of play to take stock, tie your boot laces and plan the next move is an essential element of working effectively.

Agile Product Management

Agile is reaching a stage of maturity in some companies where attention is turning to how it can be scaled up from projects to products, portfolios and the whole enterprise.

In classic agile, where the focus is on one small, co-located team, product management is carried out by the ‘product owner’ role. In scrum, this role is responsible for the value being delivered to the customer, for the ROI. The tools they use include the product backlog, release planning, iteration planning and iteration review.  The product owner should have an intense interest in ensuring a common ‘Definition of Done’ is agreed across the team or multiple teams,  and work with the scrummaster to ensure it is always met. By ensuring each iteration only delivers ‘Done’ features gives transparency into progress as well as enabling various product release options at the end of each iteration (for example if a user story has been developed but not integrated and tested, it cannot be included in a release and so constrains the product owners options).

Each agile project must have one, and only one, product owner. Although this role can be supported by various domain experts (eg marketing, customer operations) or ‘Area Product Owners’ for large products, final decision making power must reside with one person.  With ownership of the product backlog, the product owner must work with teams to ‘groom’ the high level user stories, breaking them into small, even sized stories that can be implemented independantly and deliver ‘minimum marketable value’. They also work with the team in selecting the stories that go on each iteration backlog and in serving as or facilitating access to the ‘on-site customer’ for story elaboration. The product owner is also responsible for defining acceptance criteria and reviewing and accepting stories at the end of each iteration.

Many see agile as an implementation of lean principles in software development, and so talk of enterprise scale often refers more to lean than to agile. Here I discuss some of the emerging themes in organisational level agility:

  • Cadence: Agile introduces the idea of timeboxed iterations which are repeated over and over, each including planning, executing, delivering value and reflection and improvement. Methods like scrum recommend that these iterations are of constant duration, eg 2 weeks. This introduces a ‘cadence’ into the project – a smooth rythm which helps establish a continuous flow of value and facilitates ongoing coordination between project stakeholders.  This basic rythm of iteration development is also supplemented by sub-rythms, such as daily stand-ups, or hourly builds, and by meta-rythms, such as production releases every couple of  months. Just as these cadences provide a placeholder for synchronisation and communication between individuals in a team, they can also serve the same purpose for multiple team programs, multiple program portfolios and multi-portfolio organisations.
  • Options: Agile uses the idea of a prioritised backlog to ensure only the features that are most important at a particular time are worked on. Lean uses the ideas of delaying committment to the ‘last responsible moment’ and set based design. These mechanisms have similar aims – don’t constrain future options by what your doing today unless you really have to – that is, keep your options open.  This is especially powerful at the product, program or program management level – imagine if the program could be re-directed, or even terminated, at any point according to how market, technology or other contexts evolve.  By only investing resources in what can deliver value in the short-term, the program retains control of potential future expenditure. This is a compelling reason to move from projects to increments as a system development metaphor, as described in a previous blog.
  • Pull: Lean philosophy introduces the idea of pull, with the ‘kanban’ or ‘just-in-time’ material control mechanisms the most common examples.  Where traditional planning approaches schedule the pushing of materials, parts, customer requirements or software features to where the plan says they will be used next, pull systems use signalling built into the process to co-ordinate production and consumption along the value chain.  In manufacturing, a part isn’t produced until its needed by the next operation on the line. In software development, requirements aren’t elaborated and fixed until just before they are developed. In test driven development, code isn’t developed until a test fails. Effective pull relies on the even flow established by a cadence. And pull and associated in process signaling are core to self-organising teams and value chains – it eliminates the need for an ‘out-of-band’ signalling system represented by a ‘management layer’ assigning tasks, providing visibility through status reports, etc. Pull and the self-organisation it enables are key to scaling up agile development and are therefore essential to product, program and portfolio management that uses multiple teams.
  • Unified Build: In large enterprise development, many teams may be involved. Detailed coordination of multiple technical teams has always been a difficult and costly management task, with significant overhead in terms of meetings, documentation, negotiation, etc. With agile approaches, inter team coordination moves from the planning level to the code level – continuous integration and automated test are essential coordination mechanisms both within and between teams. Where products are being developed by multiple teams, continuous integration must build the entire end product, not just a subset of it.

Lean philosophy has been proven to scale effectively over decades in the worlds largest companies, the most famous example being Toyota. In the 80s’ and 90s’ western companies adopted many of these techniques in their manufacturing operations. Now there are moves to leverage the same underlying philosophies in development, with agile being a well progressed example in software development. Higher level organisational roles such as product management continue to evolve to the new philosophy and specific approaches will emerge over the coming years.

Agile and SMEs

I’ve been reading some material on attempts to implement agile methods in small companies and a recurring theme is that such firms don’t have the resources to invest in training & coaching and yet can’t afford to make mistakes when adopting an innovation like agile. Unlike larger companies, small organisations can’t launch process re-engineering and improvement programs, can’t run pilot projects which could fail, or experiment and learn by their own mistakes.
It got me thinking about how small and large firms are agile in seemingly different ways. Christen Claytonson introduced the idea of The Innovators Dilemma in his book of the same name which, simply put, says that the strategy of listening to your customer and innovating your product to move up the value chain is doomed to failure. This approach was espoused for decades with the tagline “the customer is always right”. The belief was that by delivering what the customer asked for made sure you were delivering what they wanted.
But Christenson showed through several examples that the customer doesn’t always know what they want, or even what its possible to get. He uses the example of mini steel mills in the US which undercut the larger mills by cheaply churning out simple steel products, while the established firms ignored them and focused on the higher margin work – following their strategy of moving up the value chain. But the mini mills gradually ate into the market, becoming more sophisticated themselves and aiming for higher value work. By the time the large mills realised what was happening their high cost, high margin business was collapsing. They had experienced “disruptive innovation”.
A more recent example, and one closer to home, is the rise and rise of low cost airlines like Ryanair – I don’t think any customer asked the airline to charge them for peanuts, or fly them to an airport an hour outside town, or charge them for luggage. But by listening to their customers, established airlines had missed the opportunity for low cost, mass air travel.

So what has this got to do with agile & SMEs? For small companies, business agility has been a powerful weapon – the ability to sense and exploit emerging opportunities, leverage their small size, co-location and lack of bureaucracy to move quickly has given them a competitive advantage versus the scale and resources of larger firms. But recently big firms have begun to invest in agility. Agile and lean philosophies are helping large firms transform from high overhead, hierarchical command and control organisations to more loosely federated small units with increased autonomy and their own sense and adapt mechanisms. If larger firms succeed in developing business agility (and I believe they are)  it will dilute a major competitive weapon of the small firm.

SMEs need to find a way to counter this movement before they get beaten at their own game.

“Agile Methods and Cloud Computing” or “When is a Project not a Project”

“Agile Methods  strive to deliver valuable, quality software faster and cheaper, while maintaining flexibility to harness change for competitive advantage.”

“Cloud Computing  strives to deliver valuable, quality software faster and cheaper, while maintaining flexibility to harness change for competitive advantage.”

Coming from entirly different perspectives, one of process, the other of technology, agile & cloud share a lot of similar aims.

Traditional waterfall methods are based on the idea of a “project” – with a defined deliverable, objective, duration and budget. At the beginning of the project, all these dimensions are agreed and the job of the Project Manager is to “work the plan” – to strive to make emerging reality match the predetermined plan. Much has been written about the shortcomings of this approach (“the future ain’t what it used to be”) and how trying to predict and plan for future events based on past experience is becoming increasingly difficult as the rate of change in the business and technology landscapes accelerate.

Agile methods emerged as an attempt to treat constant, unpredictable change in a more pragmatic way – accept its going to happen and work in a way that not only delivers in turbulent environments, but even ’embraces change for competitive advantage’.  As these methods have evolved, increasingly the emphasis is moving from “projects” to “increments” – from trying to deliver big lumps of functionality in one go, to delivering little bits in a constant ‘flow’. This has been made possible by reducing the ‘transaction costs’ of testing, building and deploying systems through continuous integration, automated testing, etc. Practices like test driven development and timeboxed iterations mean software is never far from being production ready. This means the “project” can be released, re-directed or even terminated at very short notice, while still delivering value to the customer according to the investment made.

This move from “project” centric to “increment” centric development, and the further reduction in the transaction costs of deploying to cloud hosted platforms, makes agile and cloud particularly good bedfellows.  Of course, this move away from large project based development has profound implications for other organisational processes, or even structures.  Pressures to adapt the organisation will be increased through the emergence of flexible technology platforms such as cloud computing.

So, when is a project not a project? When its one or more increments!

Agile is faster…

In a series of posts I’m going to examine some of the claimed benefits of agile methods – are they justified? My first post looked at the cost of development with agile. Here we address the speed of development.

Agile methods are often portrayed as delivering software faster than traditional, planned methods. Here are some of the ways agile gets value to market faster:

  • Note I use the word ‘value’ above rather than software – traditional methods don’t deliver any value to the customer until the entire product release is completed. But through incremental delivery, agile delivers the most valuable software features much earlier than this.
  • Through timeboxing development, agile facilitates concurrent design, test and development allowing the entire end to end process be radically compressed.
  • By breaking requirements into small ‘user stories’ it eliminates ‘batch dependancies’ where one feature can’t be completed or released until others in the requirement are also complete – the value ‘flows’.
  • Delivering the highest priority items first often means features included ‘just-in-case’ are never developed – they are superseded by new higher priority items as the business context evolves
  • Using smaller, evenly sized user stories and short iterations reduces variability and thereby  queuing times in complex systems such as software development teams
  • Methods such as Scrum & XP allow the team to work faster by explicitly eliminating distractions and interruptions – for example, in scrum the scrummaster is tasked with barring interruptions to the team and ensuring the sprint backlog is not changed in any way during the sprint. Other sources of distraction, such as being assigned to multiple projects at the same time, and changing team composition, are not encouraged.
  • Finally, although there is an up-front cost in providing automated test, continuous integration and other automated processes, these save time over the duration of the project and allow the team focus on delivering value with greater speed.

Knowledge Creation is at the Heart of Innovation

“The raison d’etre of a firm is to continuously create knowledge” – so say Nonaka and Takeuchi, creators of the most dominant theory on how organisations create knowledge. This sentiment is echoed by Peter Drucker, probably the most famous and respected thinker on management: “knowledge is the only meaningful resource”. And yet, in current discussions on the building of a smart or knowledge economy in Ireland, the focus is on innovation. So what is the relationship between knowledge and innovation?
Schroeder and Van de Ven provide the following definition:
The process of innovation centers on the temporal sequence of activities that occur over time in developing and implementing new ideas from concept to concrete reality’.
There are two major components to innovation – coming up with ideas and designs for something new, and implementing them in a ‘concrete’ way. The first of these is implicitly tied to knowledge – it takes knowledge of problems and possible fragments of a solution to come up with ideas and designs, and those ideas and designs are themselves new knowledge.
Therefore, innovation at an organisational level requires knowledge creation at an organisational level. I have just completed a white paper outlining Nonaka and Takeuchi’s framework of knowledge creation – the SECI framework. This should be of interest to anyone promoting innovation and knoweledge creation in their organisations and is available here