Software Engineering
To understand how agile came to be popular in our industry it helps to understand what came before. When software came out of University Computer Laboratories and people with Computer Science degrees were working in companies using computers to build business applications, the industry looked to existing industries for inspiration of how to structure their work. The industry they chose was engineering. A number of principals were adopted from Engineering and applied to software projects.
Get it right first time
It is important to engineer a bridge so that it will not collapse when traffic volumes increase over the next 20 years and a Commercial Aeroplane must not come apart when under stress beyond what you could expect when flying in normal conditions.
Software was also expected to have no defects when used in production. If an error was found this was a major problem because software was delivered on physical media and we did not have the internet to deliver updates or patches. For business software, contracts would include penalty clauses for any down time resulting from software failures.
Rework is expensive
Software was also fully specified before any code was written. Business Analysts and System Architects would spend months detailing the requirements and Architecture of the application before any developer is given the details and allowed to build any functionality. Reworking code to accommodate changes in requirements was considered costly and complex and was avoided.
Rigorous process
Software development followed a well defined sequential process called the software development lifecycle with specific artifacts produced at the end of each step that fed into the next phase of development.
Software Development Lifecycle
Fixed Budget
A project budget allocated at the initiation stage was expected to be met regardless of size or complexity. When projects inevitably exceeded the budget due to unforeseen requirements or changes in cost of inputs it was considered a "budget blowout" and a failure.
Requirements Document Example
As a developer in a company that follows the software development lifecycle you would be provided with a business requirements document by a Business Analyst for the feature you were expected to implement. The document would follow a template similar to the one below.
The document contains all the rules and valid flows including alternative flows and error conditions. You were expected to translate this into the code. As a developer you would not communicate with the users of the software and may not understand the reasoning for the design, alternatives that were considered and discarded or tradeoffs made.
This process resulted in some high profile failures. One high profile failure in Australia was the Queensland Health payroll system.
The Queensland Health payroll scandal
The Payroll system was to provide accurate pay and rostering advice to 78,000 Queensland Health staff by July 1, 2008.
It was built by the Government owned CorpTech. The contract was to build a "whole-of-government human resources and finance solution" with a budget of $125 million. It went live in March 2010 after 10 "aborted attempts", despite very real doubts. It was not adopted by other government agencies.
When it went live almost 78,000 Queensland Health staff were underpaid, overpaid or not paid at all. More than five years later the cost to taxpayers was $1.2 billion. It was a compromised choice, and as the Commission of Inquiry said, "It was a catastrophic failure."
See SMH Article
Lean Manufacturing
People who were disillusioned with software engineering looked for other inspiration for ways to avoid the failures they were experiencing. They settled on a manufacturing technique pioneered by Taiichi Ohno at Toyota called Lean manufacturing. The goal was to reduce waste in a manufacturing system without sacrificing productivity. In Lean manufacturing the customer defines what is of value in terms of what they would pay for the product or service. If the customer is not interested in paying for a feature then it is waste and can be removed.
Principles of Lean
- Identify Value
Value is what the customer is willing to pay for. Design products to meet the needs of customers. Do not try to promote features that customers have not told you they value
- Map the Value Stream
Under all the steps and processes involved in taking a specific product from raw materials to delivering the final product to the customer.
Use customer value as a reference point. Identify all the activities that contribute to these values. Activities that do not add value to the end customer are considered waste.
Consider the complete life-cycle of a product. This includes the product’s design, the customers’ use of the product and the disposal of the product when it has reached the end of it's useful life.
- Create Flow
After the waste has been removed from the value stream the next step is to be sure the remaining steps flow smoothly. There must be no interruptions, delays, or bottlenecks in the process. Having clear visibility of the value stream and measuring how long the product remains at each step in the process is essential.
- Establish Pull
"Pull" is the central concept of “just in time” manufacturing or delivery. It is allowing the demand for a product to determine the rate at which they are provided and when inputs are needed. It means the customer can “pull” the product from you as needed. This can dramatically improve the time to market (or time to customer).
- Seek Perfection
It is important to remember lean is not a static system and requires constant effort and vigilance to perfect. There are always changes and improvements that can be made to any system.
Agile
The people who looked to lean manufacturing applied these principles in different ways and had different focuses. Some methodologies that became popular were extreme programming (XP), Scrum, Dynamic System Development Method (DSDM), Rapid Application Development (RAD), Crystal, Adaptive Software Development (ASD). When they came together they agreed on a set of values and principles that the different practices shared.
The Agile Manifesto
Individuals and interactions over processes and tools Working software over comprehensive documentation Customer collaboration over contract negotiation Responding to change over following a plan
12 Agile Manifesto Principles
- Customer satisfaction through early and continuous software delivery – Customers are happier when they receive working software at regular intervals, rather than waiting extended periods of time between releases.
- Accommodate changing requirements throughout the development process – The ability to avoid delays when a requirement or feature request changes.
- Frequent delivery of working software – Scrum accommodates this principle since the team operates in software sprints or iterations that ensure regular delivery of working software.
- Collaboration between the business stakeholders and developers throughout the project – Better decisions are made when the business and technical team are aligned.
- Support, trust, and motivate the people involved – Motivated teams are more likely to deliver their best work than unhappy teams.
- Enable face-to-face interactions – Communication is more successful when development teams are co-located.
- Working software is the primary measure of progress – Delivering functional software to the customer is the ultimate factor that measures progress.
- Agile processes to support a consistent development pace – Teams establish a repeatable and maintainable speed at which they can deliver working software, and they repeat it with each release.
- Attention to technical detail and design enhances agility – The right skills and good design ensures the team can maintain the pace, constantly improve the product, and sustain change.
- Simplicity – Develop just enough to get the job done for right now.
- Self-organizing teams encourage great architectures, requirements, and designs – Skilled and motivated team members who have decision-making power, take ownership, communicate regularly with other team members, and share ideas that deliver quality products.
- Regular reflections on how to become more effective – Self-improvement, process improvement, advancing skills, and techniques help team members work more efficiently.
Scrum
A key member of the scrum team is the scrum master. They are responsible for organising the various activities and tracking progress. They are a servant/leader in that they can make decisions but are also responsible for helping the team to deal with external factors that may limit the team's ability to deliver.
Scrum teams divide their work into stories. These represent a unit of deliverable functionality to the user of the application. The stories are prioritised from most valuable to least and worked on in that order. The stories are worked on in a series of sprints. These are timeboxed and typically last for 2-4 weeks.
During sprint planning the team will ensure they have the same shared understanding of what is required to complete each story and will estimate how much effort is required to complete the story.
The team will determine how many stories they will complete in the sprint. The team will be as honest as possible in their estimate and are expected to work together to meet that expectation. If the team fails to complete the work in the sprint it is the whole team who are responsible.
The scrum master ensures that progress is visible by tracking the progress of the team via a team board and charts. They will gather metrics over time and deliver their insights during team retrospectives.
Stories
A story contains all the information required to complete the functionality. It will contain the desired outcome in the form of:
As a
e.g. As a student I want to submit my exam answers So that they can be marked by my teacher
Acceptance criteria is provided to guide testing of the functionality. When the acceptance criteria is satisfied the story is done.
Estimation
Once the team has a clear understanding of the requirements for a story it is sized. A story is selected as a baseline. All other stories are sized relative to the baseline story. If the baseline story is estimated as a 5 then a story that is a 2 or 3 is roughly half the size of the baseline story.
The size is determined by the relative complexity, not how long someone may think it will take to complete. This is because it is not yet clear who will be working on the story and so different people with different experience could be working on it.
Each person is given a chance to size the story and only once each person has made their choice are the other members' estimates revealed to avoid influencing the other team members. If the estimates are similar it is recorded and the next story is assessed. If the estimates are varied then people are given a chance to explain their choice. This is valuable as it often surfaces misunderstandings or useful information that was missed when the story was originally discussed.
Burn Down Chart
The burn down chart is a useful tool for visualising the progress of the team towards their goal of completing the stories by the end of the sprint. A trend line is included to show the ideal number of story points remaining at the end of each day for the duration of the sprint.
It is the job of the scrum master to track the points left and plot the next day on the chart. This shows how the team is tracking. If they are falling behind it can mean that either the estimates were too optimistic or some other factor has caused the team to slow their velocity. It is up to the team to identify these issues and try to remove or mitigate them to keep their progress on track.
An example burndown chart where progress goes according to plan
A more typical chart where many stories are still in progress due to limited testing resources until the sprint is almost complete and the tester is able to finish testing the final stories
Kanban
The Kanban methodology is focused on flow. As in lean manufacturing the goal is to reduce waste and unnecessary delays. To that end the most important metric for a Kanban project is to have consistency in the process and reduce the time it takes for a story to move from the project backlog to being available for users providing the Just in Time delivery.
WIP
The main tool to achieve this is Work In Progress (WIP) limits. This is limiting the number of stories that can be in any particular part of the development process. By limiting the work in progress it is easier to spot where the bottlenecks are in the process and focus on removing them. If stories are being held up at any point the work ceases until they are moved.
Measuring Flow
To measure flow in a Kanban project a Cumulative Flow Diagram (CFD) is used. By measuring the number of stories in each column each day we can get useful metrics about the rate stories are moving through the system and where they are being held up and causing wasted time.
Lead Time
Time for a work item to be completed from when it enters the backlog. This can indicate that there are more items entering the backlog than the team can reasonably complete with the resources they have.
Cycle Time
Time for a work item to be completed from when it first moves into WIP. The goal is to reduce this time. Any waste in the system or bottleneck will cause this metric to increase over time. The team should assess this time regularly to identify areas where it can be improved.
Total WIP
All items that are being worked on at a point in time. This can show how close the system is to full capacity (all WIP limits are reached) this can indicate that the team is taking on too much work. Again if much of the WIP is work in the input queue then it may mean the team is under-resourced to tackle the work being allocated.
Conclusion
With a clear understanding of the objectives and tools of the agile methodology you use in your organisation your teams will be able to work together as a cohesive unit. They will solve the problems they face - both technical and organisational. With the right approach to deal with the challenges your team will have more time and energy to produce high quality work in less time.