Extending the Scrum and Sprint Metaphors

Scrum uses the metaphor of a rugby team to convey the ‘directed autonomy’ of a ‘whole team’ working together to get the ball across the try line. Each member of the team has a role such as out-half, fullback or forward, but once the ball is in play, any member does whatever they can to progress towards the objective, regardless of their speciality. This is contrasted to a relay race team, where each member runs part of the race on their own, passing the baton at pre-determined points. This echos a waterfall process, with distinct roles and responsibilities and hand-offs between task specialities.
Another metaphor used in scrum is that of a sprint. During a sprint, distractions and interuptions to the team are minimised – the ScrumMaster has a specific responsibility to ensure they can focus purely on acheiving the preset sprint goals. This minimises context switching and multi-tasking which have been shown to be very wasteful, and increases the predictability of what will be delivered at the end of the sprint. But no-one can sprint all the time – if they try, then they are really just jogging. I like the sprint metaphor – there is one immediate, overwhelming objective and all distractions and other concerns are ignored until the sprint ends.  And during a sprint, ‘moving’ the goalposts is not allowed – no changes to the sprint goal or user stories and tasks is allowed (if this is necessary, then the sprint is cancelled). Just like a play in rugby, all focus is achieving one objective until the whistle blows.

But the ‘classic’ scrum method (Schwaber &  Beedle),  recommends each sprint is followed directly by another – sprints include ‘bookend’ events at the start and end, such as iteration planning, reviews, retrospectives, etc. Classic scrum also calls for just a little bit of distraction sprinkled throughout the sprint such as spending 5-10% of the time on ‘grooming’ the product backlog – allowing the next sprint backlog be prepared in advance of the subsequent sprint and therefore no need to dilly-dally before rushing into development again. Activities such as these require the focus of the team be diverted from the immediate sprint goal to focus on future stories and tasks.

I have a number concerns with this model.

  1. Distractions, multi-tasking and the consequent context switching are very disruptive to concentration and sap energy from a team. Try doing some real, focused work like writing a report or some complex code with constant interruptions such as calls and meetings about different topics – its very difficult to progress rapidly and steadily with the primary task. But people in the real world have to do ‘other’ stuff in the average workday: office communication sessions, performance appraisel with their boss, arrange or attend a presentation to another department, participate in the mentoring program, file travel expenses, attend the mandatory office security training session…..the list is looong. Sometimes a team will count on only 6 hours out of 8 being available for ‘real work’ because of these distractions. But how can you sprint efficiently if your not running 25% of the time? If you have to slow down, attend to something else, and then build up sprint speed again it is very wasteful.
  2. Agile purports to be an empirical control process – rather than planning the future based on experience of the past it instead uses an ‘inspect and adapt’ paradigm to continually refine the short-term plan based on present reality. In agile methods, this takes the form of getting feedback from stakeholders (customers, users) at the end of each iteration, and, based on that, building a plan for the next sprint. In many real world cases, the next sprint has already been substantially planned by the time the previous one is finished – team members have been grooming the backlog, breaking down stories and tasks, estimating, etc. Although this model impinges on the pure inspect-adapt model, many would see it as pragmatic – a sensible compromise to optimise utilisation of the team. However, I have seen this quickly lead to local optimisation (great for getting more work done, but not necessarily delivering more business value), premature investment in story elaboration, and erosion of team ownership of stories and estimates (since often one or two team members will work on stories for the next sprint while the remained deliver the current tasks). Reverting to the rugby metaphor, its like some players disengaging and setting up the next set piece before the play has ended.
  3. I discussed the idea of cadence in a previous post – cadence across teams and projects is a great basis for synchronising inter team communications. For example, in a multi-team project if one team is sprinting while the other is between sprints, there is little opportunity for joint reflection. If there is time set aside between sprints to synchronise between teams, and if the sprints start and end at the same time, it gives an opportunity for this communication. It also gives an opportunity to go ‘off-project’ and do all that ‘other stuff’ not directly related to the project. For example, what if performance reviews, training, presentation of company quarterly results, etc could only be scheduled between sprints – this would involve a cadence at the company level operating across all functions.

Again, like the rugby metaphor, I think a break from the flurry of play to take stock, tie your boot laces and plan the next move is an essential element of working effectively.

Agile Product Management

Agile is reaching a stage of maturity in some companies where attention is turning to how it can be scaled up from projects to products, portfolios and the whole enterprise.

In classic agile, where the focus is on one small, co-located team, product management is carried out by the ‘product owner’ role. In scrum, this role is responsible for the value being delivered to the customer, for the ROI. The tools they use include the product backlog, release planning, iteration planning and iteration review.  The product owner should have an intense interest in ensuring a common ‘Definition of Done’ is agreed across the team or multiple teams,  and work with the scrummaster to ensure it is always met. By ensuring each iteration only delivers ‘Done’ features gives transparency into progress as well as enabling various product release options at the end of each iteration (for example if a user story has been developed but not integrated and tested, it cannot be included in a release and so constrains the product owners options).

Each agile project must have one, and only one, product owner. Although this role can be supported by various domain experts (eg marketing, customer operations) or ‘Area Product Owners’ for large products, final decision making power must reside with one person.  With ownership of the product backlog, the product owner must work with teams to ‘groom’ the high level user stories, breaking them into small, even sized stories that can be implemented independantly and deliver ‘minimum marketable value’. They also work with the team in selecting the stories that go on each iteration backlog and in serving as or facilitating access to the ‘on-site customer’ for story elaboration. The product owner is also responsible for defining acceptance criteria and reviewing and accepting stories at the end of each iteration.

Many see agile as an implementation of lean principles in software development, and so talk of enterprise scale often refers more to lean than to agile. Here I discuss some of the emerging themes in organisational level agility:

  • Cadence: Agile introduces the idea of timeboxed iterations which are repeated over and over, each including planning, executing, delivering value and reflection and improvement. Methods like scrum recommend that these iterations are of constant duration, eg 2 weeks. This introduces a ‘cadence’ into the project – a smooth rythm which helps establish a continuous flow of value and facilitates ongoing coordination between project stakeholders.  This basic rythm of iteration development is also supplemented by sub-rythms, such as daily stand-ups, or hourly builds, and by meta-rythms, such as production releases every couple of  months. Just as these cadences provide a placeholder for synchronisation and communication between individuals in a team, they can also serve the same purpose for multiple team programs, multiple program portfolios and multi-portfolio organisations.
  • Options: Agile uses the idea of a prioritised backlog to ensure only the features that are most important at a particular time are worked on. Lean uses the ideas of delaying committment to the ‘last responsible moment’ and set based design. These mechanisms have similar aims – don’t constrain future options by what your doing today unless you really have to – that is, keep your options open.  This is especially powerful at the product, program or program management level – imagine if the program could be re-directed, or even terminated, at any point according to how market, technology or other contexts evolve.  By only investing resources in what can deliver value in the short-term, the program retains control of potential future expenditure. This is a compelling reason to move from projects to increments as a system development metaphor, as described in a previous blog.
  • Pull: Lean philosophy introduces the idea of pull, with the ‘kanban’ or ‘just-in-time’ material control mechanisms the most common examples.  Where traditional planning approaches schedule the pushing of materials, parts, customer requirements or software features to where the plan says they will be used next, pull systems use signalling built into the process to co-ordinate production and consumption along the value chain.  In manufacturing, a part isn’t produced until its needed by the next operation on the line. In software development, requirements aren’t elaborated and fixed until just before they are developed. In test driven development, code isn’t developed until a test fails. Effective pull relies on the even flow established by a cadence. And pull and associated in process signaling are core to self-organising teams and value chains – it eliminates the need for an ‘out-of-band’ signalling system represented by a ‘management layer’ assigning tasks, providing visibility through status reports, etc. Pull and the self-organisation it enables are key to scaling up agile development and are therefore essential to product, program and portfolio management that uses multiple teams.
  • Unified Build: In large enterprise development, many teams may be involved. Detailed coordination of multiple technical teams has always been a difficult and costly management task, with significant overhead in terms of meetings, documentation, negotiation, etc. With agile approaches, inter team coordination moves from the planning level to the code level – continuous integration and automated test are essential coordination mechanisms both within and between teams. Where products are being developed by multiple teams, continuous integration must build the entire end product, not just a subset of it.

Lean philosophy has been proven to scale effectively over decades in the worlds largest companies, the most famous example being Toyota. In the 80s’ and 90s’ western companies adopted many of these techniques in their manufacturing operations. Now there are moves to leverage the same underlying philosophies in development, with agile being a well progressed example in software development. Higher level organisational roles such as product management continue to evolve to the new philosophy and specific approaches will emerge over the coming years.

Agile and SMEs

I’ve been reading some material on attempts to implement agile methods in small companies and a recurring theme is that such firms don’t have the resources to invest in training & coaching and yet can’t afford to make mistakes when adopting an innovation like agile. Unlike larger companies, small organisations can’t launch process re-engineering and improvement programs, can’t run pilot projects which could fail, or experiment and learn by their own mistakes.
It got me thinking about how small and large firms are agile in seemingly different ways. Christen Claytonson introduced the idea of The Innovators Dilemma in his book of the same name which, simply put, says that the strategy of listening to your customer and innovating your product to move up the value chain is doomed to failure. This approach was espoused for decades with the tagline “the customer is always right”. The belief was that by delivering what the customer asked for made sure you were delivering what they wanted.
But Christenson showed through several examples that the customer doesn’t always know what they want, or even what its possible to get. He uses the example of mini steel mills in the US which undercut the larger mills by cheaply churning out simple steel products, while the established firms ignored them and focused on the higher margin work – following their strategy of moving up the value chain. But the mini mills gradually ate into the market, becoming more sophisticated themselves and aiming for higher value work. By the time the large mills realised what was happening their high cost, high margin business was collapsing. They had experienced “disruptive innovation”.
A more recent example, and one closer to home, is the rise and rise of low cost airlines like Ryanair – I don’t think any customer asked the airline to charge them for peanuts, or fly them to an airport an hour outside town, or charge them for luggage. But by listening to their customers, established airlines had missed the opportunity for low cost, mass air travel.

So what has this got to do with agile & SMEs? For small companies, business agility has been a powerful weapon – the ability to sense and exploit emerging opportunities, leverage their small size, co-location and lack of bureaucracy to move quickly has given them a competitive advantage versus the scale and resources of larger firms. But recently big firms have begun to invest in agility. Agile and lean philosophies are helping large firms transform from high overhead, hierarchical command and control organisations to more loosely federated small units with increased autonomy and their own sense and adapt mechanisms. If larger firms succeed in developing business agility (and I believe they are)  it will dilute a major competitive weapon of the small firm.

SMEs need to find a way to counter this movement before they get beaten at their own game.

“Agile Methods and Cloud Computing” or “When is a Project not a Project”

“Agile Methods  strive to deliver valuable, quality software faster and cheaper, while maintaining flexibility to harness change for competitive advantage.”

“Cloud Computing  strives to deliver valuable, quality software faster and cheaper, while maintaining flexibility to harness change for competitive advantage.”

Coming from entirly different perspectives, one of process, the other of technology, agile & cloud share a lot of similar aims.

Traditional waterfall methods are based on the idea of a “project” – with a defined deliverable, objective, duration and budget. At the beginning of the project, all these dimensions are agreed and the job of the Project Manager is to “work the plan” – to strive to make emerging reality match the predetermined plan. Much has been written about the shortcomings of this approach (“the future ain’t what it used to be”) and how trying to predict and plan for future events based on past experience is becoming increasingly difficult as the rate of change in the business and technology landscapes accelerate.

Agile methods emerged as an attempt to treat constant, unpredictable change in a more pragmatic way – accept its going to happen and work in a way that not only delivers in turbulent environments, but even ’embraces change for competitive advantage’.  As these methods have evolved, increasingly the emphasis is moving from “projects” to “increments” – from trying to deliver big lumps of functionality in one go, to delivering little bits in a constant ‘flow’. This has been made possible by reducing the ‘transaction costs’ of testing, building and deploying systems through continuous integration, automated testing, etc. Practices like test driven development and timeboxed iterations mean software is never far from being production ready. This means the “project” can be released, re-directed or even terminated at very short notice, while still delivering value to the customer according to the investment made.

This move from “project” centric to “increment” centric development, and the further reduction in the transaction costs of deploying to cloud hosted platforms, makes agile and cloud particularly good bedfellows.  Of course, this move away from large project based development has profound implications for other organisational processes, or even structures.  Pressures to adapt the organisation will be increased through the emergence of flexible technology platforms such as cloud computing.

So, when is a project not a project? When its one or more increments!

Agile is faster…

In a series of posts I’m going to examine some of the claimed benefits of agile methods – are they justified? My first post looked at the cost of development with agile. Here we address the speed of development.

Agile methods are often portrayed as delivering software faster than traditional, planned methods. Here are some of the ways agile gets value to market faster:

  • Note I use the word ‘value’ above rather than software – traditional methods don’t deliver any value to the customer until the entire product release is completed. But through incremental delivery, agile delivers the most valuable software features much earlier than this.
  • Through timeboxing development, agile facilitates concurrent design, test and development allowing the entire end to end process be radically compressed.
  • By breaking requirements into small ‘user stories’ it eliminates ‘batch dependancies’ where one feature can’t be completed or released until others in the requirement are also complete – the value ‘flows’.
  • Delivering the highest priority items first often means features included ‘just-in-case’ are never developed – they are superseded by new higher priority items as the business context evolves
  • Using smaller, evenly sized user stories and short iterations reduces variability and thereby  queuing times in complex systems such as software development teams
  • Methods such as Scrum & XP allow the team to work faster by explicitly eliminating distractions and interruptions – for example, in scrum the scrummaster is tasked with barring interruptions to the team and ensuring the sprint backlog is not changed in any way during the sprint. Other sources of distraction, such as being assigned to multiple projects at the same time, and changing team composition, are not encouraged.
  • Finally, although there is an up-front cost in providing automated test, continuous integration and other automated processes, these save time over the duration of the project and allow the team focus on delivering value with greater speed.

Knowledge Creation is at the Heart of Innovation

“The raison d’etre of a firm is to continuously create knowledge” – so say Nonaka and Takeuchi, creators of the most dominant theory on how organisations create knowledge. This sentiment is echoed by Peter Drucker, probably the most famous and respected thinker on management: “knowledge is the only meaningful resource”. And yet, in current discussions on the building of a smart or knowledge economy in Ireland, the focus is on innovation. So what is the relationship between knowledge and innovation?
Schroeder and Van de Ven provide the following definition:
The process of innovation centers on the temporal sequence of activities that occur over time in developing and implementing new ideas from concept to concrete reality’.
There are two major components to innovation – coming up with ideas and designs for something new, and implementing them in a ‘concrete’ way. The first of these is implicitly tied to knowledge – it takes knowledge of problems and possible fragments of a solution to come up with ideas and designs, and those ideas and designs are themselves new knowledge.
Therefore, innovation at an organisational level requires knowledge creation at an organisational level. I have just completed a white paper outlining Nonaka and Takeuchi’s framework of knowledge creation – the SECI framework. This should be of interest to anyone promoting innovation and knoweledge creation in their organisations and is available here

Agile vs. Lean – Timeboxes vs. Flow

Agile uses the idea of fixed timeboxes where the scope of work completed for an iteration is estimated based on a duration such as 2 weeks. Importantly, even if all the planned work for an iteration isn’t complete, what has been developed is built and delivered anyway – the iteration is timeboxed and the schedule does NOT slip.  This ensures there will be some value delivered to the customer, even if it falls short of what was planned. But in lean thinking, the focus is on flow – the continuous flow of value rather than the incremental flow. In a discrete production context, this manifests as a striving to reach a batch size of 1 – rather than a ‘production run’ on a machine making 1000 widgets, it makes just 1 as it is pulled from the subsequent operation. This move to smaller batches and therefore towards continuous flow is possible because of new manufacturing technology – for example a machine that can be instantly reprogrammed can easily switch from making one type of widget to another. Similarly, in software development, we strive for ever smaller increments, ever shorter iterations.  And, like production, this is enabled through technology such as automated testing and continuous integration. These technologies reduce the cost of testing, building and deploying and allow us deliver value in a way approaching the ideal – continuous flow – advocated by lean philosophy.

Agile is cheaper…

In a series of posts I’m going to critically review the common value propositions of agile – do the claims stand up to scrutiny?
Claim 1: Agile makes for cheaper software.
There are a number of justifications for this claim.

  • No more kitchen sink – with waterfall a common pattern is the tendency of the customer to squeeze in all possible requirements at the beginning of the process since the ‘change control process’ will make it very difficult to add requirements afterwards, and any changes would give a justification for the development team to slip the schedule. So lots of features, poorly understood and unlikely to be needed, are included up front. This leads to more complexity in the design which now has to take into account more features, delays starting time of development as it takes longer to get signed off requirements, and puts features with low or even no current business value on an equal priority for delivery as critical functions (since all must be delivered in a big release). The software delivered in the end therefore has more functionality than in agile – it costs more to produce.
  • Invest only in artifacts with business value: Agile values working software over documentation, but also values documentation with business value over that merely used to ‘run the project’. So time spent developing requirements specs, test plan documents, status reports, etc can be spent instead on great user guides, operations guides, regression test suites, etc.
  • Lower Opportunity Costs: Because agile delivers value earlier, it reduces lost opportunity costs and enables the business address opportunities in the market more quickly.
  • Lower Product LifeTime costs: The up front investment in continuous integration, automated test suites and refactored, well structured code has been shown to be rewarded several times over in reduced ongoing maintenance costs throughout the softwares life, and even extending that life beyond where ‘dirtier’ code becomes uneconomical to maintain.
  • Delayed Investment: By returning business value from early in the development lifetime, an earlier return on the investment is realised, and further investment can be delayed till its really necessary.
  • Global vs. Local Optimisation: A key lean principle is global over local optimisation. Large batches rather than continuous flow, specialisation rather than skills redundancy and other traditional tactics to increase resource utilisation aim to acheive optimisation of a particular task set such as development or testing. But it is the optimisation of the whole that is important and agile cuts costs by focusing on this global optimisation.

Are there other ways agile cuts costs? If you know more please share them…

Next blog I’ll look at how agile can speed time to value.

Agile vs. Lean

I often come across the question “Whats the difference between agile and lean, and how are they similar”. Big question!
There is an ongoing debate on the relationship of lean and agile methods and philosophies. Lean arose from Japanese production and is based on the two ‘pillars’ of continuous improvement and respect for people. It aims for elimination of waste, continuous, even flow and constant learning and striving for perfection (read innovation here). Agile arose from software development and focuses on emergent design, incremental delivery and continuous and disciplined planning. While lean has been highly developed for repeatable production tasks according to a preconceived design, it has also proved highly effective in development tasks where it aims to ‘outlearn the competition’. Agile has been used in development to create new designs and, as the name suggests, is specifically about change rather than repeatability.
These quite different lineages have been used to justify the stance that lean and agile are quite different approaches, and that even though Lean is now being applied to software development, they are mutually exclusive:
Kent Beck – “The values, principles, and practices of the two approaches are different, even though complementary.”
A contrary view is, however, emerging. Mary Poppendeick reflects the view that agile methods are an implementation of the lean philosophy in software development. Her seminal book “Lean Software Development: An Agile Toolkit” which first defined the lean approach to software development, reflects this in its title. This view is described well by Dean Leffingwell:
“I’ve been struck again and again by common principles – not just like principles or complementary principles – but virtually identical, fundamental, and immutable principles that provide the cornerstone values and philosophies of both lean and agile approaches. I’ve become more and more convinced that agile is a software instance of lean– purpose built for the unique challenges, intangibles and thought-based work of developing software as fast as efficiently as possible – but as lean as lean can be in that context.”
As this debate develops I hope to write a series of blogs exploring the issues and the debate, and do some ‘future-gazing’ as to how the emergence of lean will influence the evolution of agile development.

Agile Magic

With several companies I’ve worked with, agile has been seen as a silver bullet to solve problems of late delivery, bad quality and high cost. But agile is more than a set of practices – its a system of values and principles that need commitment and resources to bring success. Senior management will often say “this team is agile” just because they read the book, have daily stand ups and use words like sprint, iteration and user story. But this has been likened to a cargo cult, where performing agile ‘ceremonies’ are somehow construed as bringing benefits of a true agile implementation. Perhaps this is another aspect of the ‘illusion of control’ invoked by detailed planning and scheduling of projects in waterfall methods. And like the damage such illusion did to waterfall, blind belief in a sprinkling of agile magic dust can prove to be a major setback for true agile adoption.