Success stories and surveys support agile software development. In addition to anecdotal evidence there are little hard figures to guide our decision making. Here cumulative business value charts are used to describe the impact of agile and traditional choices to the bottom line. Focus is on visualizing the economic impact of individual agile practices and assumptions of their costs and profits.
A chart describing the cumulative earnings as a function of time is a basic tool for optimizing the return of investment, ROI. The line is higher all the time if we have multiple deliveries. It is still better if the more profitable increments are deployed the first. This advantage in time-to-market is clear also in the surveys. Double the results in half of the time assumes that earlier deliveries have business value and that the additional cost of more deliveries is small.
- It is not trivial to find minimum viable products that create value especially when organizations do not improve their businesses continuously but use large projects with non-negotiable late deadlines. Waste of the inventory of the partially done software is not as visible as the waste of tangible inventories.
- The cost of a delivery has been a show-stopper of agility. Business models were based on sales of new versions of software that were installed manually to each client computer. Each deployment package had to be tested and integrated manually. Without automation the costs can overcome the benefits of early deliveries.
The cost of change must be small if we proceed empirically and feedback directs our product and development process. Customers pay these changes when they create more value than the costs of delaying and implementing the the changes.
- Extreme programming proposes a set of practices like test driven development, re-factoring and pair programming that change the cost curve, cost of change, such a way.
- Waste of unnecessary features can be avoided by prioritizing product backlog items based on their business value and changing the order when appropriate.
Risks are an integral part of any design. In software development we do not commonly know what the users actually need and how we use new technologies that create the results. The cone of uncertainty is high. Reports of more than hundred-fold effort differences are common. Sales, savings and profit estimates are so inaccurate that it is quite common to ignore them altogether.
- Whole investment is lost if we do a wrong thing or fail in the implementation
- Customers try to protect themselves by fixed price projects. Costs increase because vendors have to buffer their offers especially if penalties are used. Competitive bidding weight visible price or over quality and total cost of ownership. The bidding game sets customers and vendors against each other.
- Incremental deliveries make risks visible and adaptation is easier.
- Agile approach is a natural risk management system, but it often fails when appropriate action is not taken. Organisation cultures must allow cancelling and redirecting projects.
- Queuing theory and theories of variation can lead to improved practices in handling unpredictable situations.
Traditional mass production uses specialized lowest costs work force, rigid processes and invests in tools. It locally optimizes the cost of the tasks that workers do and is often blind about the required administrative burden.
- Cost of learning can be significant. It unavoidable when we do something new. Open workspaces, pair programming and shared responsibilities are agile practices that accelerate learning. It would be good if that could break Brook’s law.
- Cost of teamwork is minimized in a cross-functional team in single location. Wastes of relearning, transfer of work, task switching and delays are minimized. Software development has dis-economics of scale because a lot of communication and coordination work is required.
- Utilization rate of the specialist become easily low. To avoid that developers take part of multiple simultaneous projects which often leads to coordination chaos.
- Cost of motivation is difficult to estimate but essential part of working life. Leadership style has an impact on the engagement of team members which may result in more value for the business.
- Traditional big front end design tries to minimize the cost of rework but real life shows that a large amount of work is needed to correct the bugs, integrate the components and deliver what users actually need.
Traditional and agile approaches have different assumptions of the relative costs of the parts of software development. Agile assumes high risks, high integration and collaboration needs, low predictability and low cost of change.
Example driven development is the best name to cover acceptance test driven development, behaviour driven development and specification by example approaches.
The concepts of acceptance test, testing and a tester have been defined long ago. I have found that changing their meaning is difficult. Even when we talk about test driven development we have to spend a lot of effort to explain that it is not done after programming and not by the separate testers.
The same happens if we talk about specifications. The old school uses user stories as a new way of writing specifications, which could be then send to a separate development team. Collaborative design is not happening as it should.
Though I like the idea of programming with natural languages, I think that they are too abstract and too prone to interpretations. Examples are a good way to explain things unambiguously and in a way that both the users and programmers see useful. Behaviour is something that programmers talk about, but for users that might not be as clear as examples.
With example driven development I want to emphasize collaborative design where users and developers work together using examples to clarify how users interact with computer programs to achieve business benefits that pay the return on investment.
Traditionally we have specifications, tests and code written to different purposes. They should not just be consistent but the same. In agile tests are specifications and in behaviour driven development code is moving closer to the tests.
The ultimate goal might be that we have only one description of the program that can be understood by both the users and the compilers. That is one goal of visual programming. The lack of programmers has been tried to be solved by moving the work to users. This approach has not been very successful because building large programs requires a lot of logical thinking and solutions to huge amounts of details. Software designers are not endangered species but programmers might be if the abstraction level of programming goes up.
Big front-end designs are bad, because they do not match the reality. The devils are really in the details. However, the cost of changes is considered too high and the illusion of predictability makes us close our eyes on reality.
In Scrum we say that we should focus on infrastructure and architecture in the first Sprints. Decisions about our software development environment, tools and architectures are not easily reversible. The situation is much worse in the construction business. If we are building a bridge or a tower of Eiffel we can’t start it again from the beginning if the base is not strong enough.
The case is not that bad in software development. We use software instead of electronic circuits just because it is easier to change software than hardware.
We need to have appropriate engineering practices to change Kent Beck’s famous cost curve. Extreme programming contains many practices that are needed to make the code easy to change. In addition to these we need solid architectural principles and rigorous attitude to quality.
PowerPoint architectures outlined at the first Sprints are not good enough. We need executable architectures and extensive testing with highest priory business functionality to make sure that the quality attributes, non-functional requirements, are OK, before we continue deeper.
We don’t stop to that. We require that the architectures are easy to change. Ease of refactoring and testing can be achieved with known design patterns.
My colleague has written a provocative blog entry about the current situation of agility.
Actually, we are promoting our new course: Agile Engineering Practices, which can be used as a part of Certified Scrum Developer curriculum.
My talk in Scum Gathering Amsterdam is now visible. See more about visual software design with themes and epics
The place: 10:00 – 11:00am on Tuesday, November 16 in Foyer
Synopsis: We have issues like user stories, themes, epics, UI mockups, business rules and acceptance tests that are used in creating our understanding of what to do and how. We groom product backlogs and have Sprint planning meetings and design tasks in Sprint backlogs. This IdeaCamp session pursues to tell us how to put these all together in real life projects.
The slides and result flip charts are now available at SlideShare. I like especially the idea of drawing users’ value stream with epics shown by one of the groups in the idea camp.
I bought some lean&agile books to read during my summer holiday. The list is not complete because I have already read quite many of them. I got Mike Cohn’s new Scrum book freely from the publisher. Thanks about that.
“Scaling Lean and Agile Development: Thinking and Organizational Tools for Large-Scale Scrum: Successful Large, Multisite and Offshore Products with Large-scale Scrum (Agile Software Development)”
“Agile Testing: A Practical Guide for Testers and Agile Teams (Addison-Wesley Signature)”
“Leading Lean Software Development: Results are Not the Point (Addison-Wesley Signature)”
“Agile Product Management with Scrum: Creating Products That Customers Love (Addison-Wesley Signature)”
“Coaching Agile Teams: A Companion for ScrumMasters, Agile Coaches, and Project Managers in Transition (Addison-Wesley Signature Series (Cohn))”
David J Anderson
ReWork: Change the Way You Work Forever”
“Lean Architecture: for Agile Software Development”
“Drive: The Surprising Truth About What Motivates Us”
Daniel H. Pink
“Switch: How to Change Things When Change is Hard”
A short history of waterfall
As experienced people remember the eve of software development was agile. Waterfall emerged because the prevailing management thinking embraced it. There was a quest for more rigor and predictability and we started to talk about software engineering which would improve our quality and productivity. More planning was an obvious solution when plans were failing.
When the systems grew the developers began to specialize, which was based on the prevailing mass production paradigm. Functional organizations were the norm everywhere. Because it was difficult to find skillful programmers, the tasks were divided so that cheaper and available labor could be used for defining, testing and documenting. Promotions to project managers enabled the traditional corporate ladder hierarchy.
Outsourcing was tried to solve the software crisis by decreasing the unit cost of the huge amount of work that was required for writing each single line of code only to find out that the distribution created another layer of complexity.
Massive process guides tried to catch every possible view of software development to make it predictable and repeatable. 80s was the era of methodologies. After that we got quality initiatives like CMM which created a quagmire of documentation and turf wars between them. It was still very difficult to tell how to create software because we have so many variations of it and its development. You had to write either something that is right and so generic that it does not help or something that must be applied or something which fits only to certain kinds of software development.
CHAOS reports told that the majority of software development project were not successful. Success factors were extracted and we found that the requirements and capability of making decisions are the keys to success. Systems thinking explains the failure of old straightforward initiatives. Specialization, outsourcing and rigorous processes answered well to one visible view of software development but they did not optimize the whole. Big front-end planning did not create better plans because the plans were frozen prematurely without appropriate testing and feedback.
In the 70s computers were less powerful than they are today. The programmer wrote the code to a paper based on which punch cards where created. Then the cards were read, compiled and finally executed. If anything went wrong the programmer had to make corrections and try again. The length of the development cycles might have been several hours if not days. Barry Boehm’s famous paper published in 1981 about software development economics was based on studies of the projects earlier than that. So it is natural that it concluded that the cost of a change is so huge that errors should be prevented at practically any cost. Reviews and careful table testing were the ways to prevent errors.
Waterfall was born due to the management paradigms but it is useful to consider the feasibility of modern iterative development using the technology of the past. Essential practices like continuous integration and automated testing were difficult but not outright impossible in all of the projects. Programming cycle times have not prevented agility after the punch card. The “mythical man-month” of Fred Brooks was written in 1975 but this was unfortunately not enough to change the history of software development.
My colleague at Tieturi, Arto Santala, has written a series of three blog entries about unit testing in EJB 3 environment. It is in Finnish but you might get the point through the code examples without our language.
As an idea unit testing has been known and used for decades but examples in blogosphere focus on simple situations instead of complex cases of real life. When we do unit and acceptance testing in industrial scale, we need to go deeper considerations about maintainability and performance of the test harness.
I found a revealing discussion at a popular suomi24.fi-site titled Ketterä IT-helvetti (agile IT hell ). It is amazing how often daily Scrum is practiced just opposite of Scrum’s basic idea of a self-organizing team. This 15 minutes is just for the developers to say hello to each other and coordinate days activities. They need to know the situation after yesterday’s work because software development is not predictable. Sometimes tasks estimated to take 2 days take 5 days and sometimes they are ready after one day. So everyone needs to know where the team is and agree on what to do next. The metaphor of chickens and pigs is meant to make sure that managers do not use that for micromanaging the people.
Command and control is bad because it destroys peoples’ natural work motivation. People including developers want to achieve something and autonomy to do it – not that they are given orders and controlled tightly. Command and control reduces productivity and creativity which is really harmful in an art like software development. Psychological studies have shown that time after another.
Pair programming can be done in a way that does not decrease the motivation. As the code is owned by team and every team member is responsible of changing any peace of it the team needs a way to teach the secrets of the code modules to everyone in the team. That is pair programming. It prevents harmful specialization and at the same time decreases the number of errors in the code. Your pair is not watching and controlling what you are programming but you two work together to create great software. Two creative minds coding and discussing together is a way to learn from each other and do a great work. Pairs are changed normally on daily basis so that different angles on the problem can be thought.
Pair programming should not be a stressful situation like the one described in suomi24. The team should discuss about the situation in its retrospective and decide how to continue to find the the joy of work that agility gives when it is practiced at its best.
Like most of Finns I have my vacation this month. As typical Europeans we have quite long vacations and sometimes people think that it is a kind of waste that reduces productivity.
It is well admitted that people need sleep to be productive. Overtime and trashing at work is quite typical in our profession. But still we ignore the agile principle of sustainable pace. After months of overtime people are so burned down that it is a miracle if they get any significant output.
The time that has not been allocated to a specific purpose might become the most productive part of all. Vacations are wonderful places for that provided of course that you do not have a tight schedule for your holiday. You may guess that I do not behave like that. Actually I live one day at a time. I have about 50 topics to this blog, several books to read and few home computers to upgrade. Most importantly I have now time for exercise. Weather has been good for walking and for work in the garden. I do not run marathons like some of my colleagues.
I have always combined walking and thinking. New ideas pop up when I walk. That is wonderful time considering them thoroughly.
I have a mini-laptop, Asus EEE 901 with 20GB SSD using Ubuntu 9.04 to be exact. It is an excellent tool for surfing the web and maintaining this new web site. I appreciate its small size, long battery life and connectivity. This is my first really personal computer – it is with me all days long. Now it is possible to take notes and have them with me. It is my extended memory.
I remember the word creativity in the title, so let’s talk about that now. One part of creativity is brainstorming – a wild burst of new ideas. It is only possible if you have free non-allocated time. In a tight schedule you pick the most obvious solutions and hurry on to get things done. During the vacations you have that luxury. I do not mean that you should spend your holidays to thinking work related things but people like me do it anyway not because someone pays for it but because it is fun.
I read Malcolm Gladwell’s book Outliers It was well worth of its cost – 7 euros, because it is well written an easy to read. The ideas were not actually new, but getting an idea of how successful people are made is made clear. Bill Gates was one of his examples.
First of all you need luck to succeed. For example Bill Gates was one of the lucky few young people who had unlimited access to computer time in 1968. Second very important thing is 10 000 hours of hard work. Creating a professional skill in any area of life requires 10 years of enthusiastic learning. You have to be lucky to have a good “university” for that. It was Hamburg for the Beatles.
Success is not as much of IQ as employers are used to think. An old study of Lewis Terman, a professor of psychology, has shown that the success of people with high IQ is not as good as we typically believe. The geniuses that he found did not succeed much better than ordinary people. Gladwell conjectures that IQ has a threshold value. Your IQ must be good enough to get in into good universities but above that other things are more important. Creative thinking is more open ended than an ability to solve puzzles that have a single right solution.
Now thinking about growing good software developers. Most important thing is to have that 10 000 hours of work. It is also important to have multifaceted experience. A 10 year career of COBOL-programming in same domain area or even a piece of software is not more than 10 times one years experience. I even wonder what has kept the person in a position where he can’t learn anything new.
I emphasize attitude, the passion to do ones work. It is a known hiring guideline to hire the attitude and train the skill, but this is much easier to say than actually do. The same frustrated looser that you fired may become a passionate star of your competitor. So, it is not so much about selecting people but about creating a corporate culture. An that is really hard, especially when your business environment is difficult.
Agile and lean software development is all about people. It is not about processes and tools as should remember from Agile Manifesto. Nevertheless, the people focus on the Scrum process and kanban in Lean believing that the change of process is the silver bullet. I admit that I have seen remarkable increases in teams’ motivation when they have adopted Scrum, but I assume that the correlation does not mean a causal relationship, especially if we ignore the human part of agility.
In this post I have intentionally ignored what Gladwell said about cultural background. Read that from his book 🙂