New lean and agile books
Jul/100
I bought some lean&agile books to read during my summer holiday. The list is not complete because I have already read quite many of them. I got Mike Cohn’s new Scrum book freely from the publisher. Thanks about that.
“Scaling Lean and Agile Development: Thinking and Organizational Tools for Large-Scale Scrum: Successful Large, Multisite and Offshore Products with Large-scale Scrum (Agile Software Development)”
Craig Larman;
“Agile Testing: A Practical Guide for Testers and Agile Teams (Addison-Wesley Signature)”
Lisa Crispin;
“Leading Lean Software Development: Results are Not the Point (Addison-Wesley Signature)”
Mary Poppendieck;
“Agile Product Management with Scrum: Creating Products That Customers Love (Addison-Wesley Signature)”
Roman Pichler;
“Coaching Agile Teams: A Companion for ScrumMasters, Agile Coaches, and Project Managers in Transition (Addison-Wesley Signature Series (Cohn))”
Lyssa Adkins;
“Kanban”
David J Anderson
ReWork: Change the Way You Work Forever”
Jason Fried;
“Lean Architecture: for Agile Software Development”
James Coplien;
“Drive: The Surprising Truth About What Motivates Us”
Daniel H. Pink
“Switch: How to Change Things When Change is Hard”
Chip Heath;
Origin of waterfall
May/100
A short history of waterfall
As experienced people remember the eve of software development was agile. Waterfall emerged because the prevailing management thinking embraced it. There was a quest for more rigor and predictability and we started to talk about software engineering which would improve our quality and productivity. More planning was an obvious solution when plans were failing.
When the systems grew the developers began to specialize, which was based on the prevailing mass production paradigm. Functional organizations were the norm everywhere. Because it was difficult to find skillful programmers, the tasks were divided so that cheaper and available labor could be used for defining, testing and documenting. Promotions to project managers enabled the traditional corporate ladder hierarchy.
Outsourcing was tried to solve the software crisis by decreasing the unit cost of the huge amount of work that was required for writing each single line of code only to find out that the distribution created another layer of complexity.
Massive process guides tried to catch every possible view of software development to make it predictable and repeatable. 80s was the era of methodologies. After that we got quality initiatives like CMM which created a quagmire of documentation and turf wars between them. It was still very difficult to tell how to create software because we have so many variations of it and its development. You had to write either something that is right and so generic that it does not help or something that must be applied or something which fits only to certain kinds of software development.
CHAOS reports told that the majority of software development project were not successful. Success factors were extracted and we found that the requirements and capability of making decisions are the keys to success. Systems thinking explains the failure of old straightforward initiatives. Specialization, outsourcing and rigorous processes answered well to one visible view of software development but they did not optimize the whole. Big front-end planning did not create better plans because the plans were frozen prematurely without appropriate testing and feedback.
In the 70s computers were less powerful than they are today. The programmer wrote the code to a paper based on which punch cards where created. Then the cards were read, compiled and finally executed. If anything went wrong the programmer had to make corrections and try again. The length of the development cycles might have been several hours if not days. Barry Boehm’s famous paper published in 1981 about software development economics was based on studies of the projects earlier than that. So it is natural that it concluded that the cost of a change is so huge that errors should be prevented at practically any cost. Reviews and careful table testing were the ways to prevent errors.
Waterfall was born due to the management paradigms but it is useful to consider the feasibility of modern iterative development using the technology of the past. Essential practices like continuous integration and automated testing were difficult but not outright impossible in all of the projects. Programming cycle times have not prevented agility after the punch card. The “mythical man-month” of Fred Brooks was written in 1975 but this was unfortunately not enough to change the history of software development.
Unit testing EJB 3
Feb/100
My colleague at Tieturi, Arto Santala, has written a series of three blog entries about unit testing in EJB 3 environment. It is in Finnish but you might get the point through the code examples without our language.
As an idea unit testing has been known and used for decades but examples in blogosphere focus on simple situations instead of complex cases of real life. When we do unit and acceptance testing in industrial scale, we need to go deeper considerations about maintainability and performance of the test harness.
Command and control with pair programming
Nov/091
I found a revealing discussion at a popular suomi24.fi-site titled Ketterä IT-helvetti (agile IT hell ). It is amazing how often daily Scrum is practiced just opposite of Scrum’s basic idea of a self-organizing team. This 15 minutes is just for the developers to say hello to each other and coordinate days activities. They need to know the situation after yesterday’s work because software development is not predictable. Sometimes tasks estimated to take 2 days take 5 days and sometimes they are ready after one day. So everyone needs to know where the team is and agree on what to do next. The metaphor of chickens and pigs is meant to make sure that managers do not use that for micromanaging the people.
Command and control is bad because it destroys peoples’ natural work motivation. People including developers want to achieve something and autonomy to do it - not that they are given orders and controlled tightly. Command and control reduces productivity and creativity which is really harmful in an art like software development. Psychological studies have shown that time after another.
Pair programming can be done in a way that does not decrease the motivation. As the code is owned by team and every team member is responsible of changing any peace of it the team needs a way to teach the secrets of the code modules to everyone in the team. That is pair programming. It prevents harmful specialization and at the same time decreases the number of errors in the code. Your pair is not watching and controlling what you are programming but you two work together to create great software. Two creative minds coding and discussing together is a way to learn from each other and do a great work. Pairs are changed normally on daily basis so that different angles on the problem can be thought.
Pair programming should not be a stressful situation like the one described in suomi24. The team should discuss about the situation in its retrospective and decide how to continue to find the the joy of work that agility gives when it is practiced at its best.
Vacations and creativity
Jul/090
Like most of Finns I have my vacation this month. As typical Europeans we have quite long vacations and sometimes people think that it is a kind of waste that reduces productivity.
It is well admitted that people need sleep to be productive. Overtime and trashing at work is quite typical in our profession. But still we ignore the agile principle of sustainable pace. After months of overtime people are so burned down that it is a miracle if they get any significant output.
The time that has not been allocated to a specific purpose might become the most productive part of all. Vacations are wonderful places for that provided of course that you do not have a tight schedule for your holiday. You may guess that I do not behave like that. Actually I live one day at a time. I have about 50 topics to this blog, several books to read and few home computers to upgrade. Most importantly I have now time for exercise. Weather has been good for walking and for work in the garden. I do not run marathons like some of my colleagues.
I have always combined walking and thinking. New ideas pop up when I walk. That is wonderful time considering them thoroughly.
I have a mini-laptop, Asus EEE 901 with 20GB SSD using Ubuntu 9.04 to be exact. It is an excellent tool for surfing the web and maintaining this new web site. I appreciate its small size, long battery life and connectivity. This is my first really personal computer - it is with me all days long. Now it is possible to take notes and have them with me. It is my extended memory.
I remember the word creativity in the title, so let’s talk about that now. One part of creativity is brainstorming - a wild burst of new ideas. It is only possible if you have free non-allocated time. In a tight schedule you pick the most obvious solutions and hurry on to get things done. During the vacations you have that luxury. I do not mean that you should spend your holidays to thinking work related things but people like me do it anyway not because someone pays for it but because it is fun.
How to grow a good developer
Jun/090
I read Malcolm Gladwell’s book Outliers It was well worth of its cost - 7 euros, because it is well written an easy to read. The ideas were not actually new, but getting an idea of how successful people are made is made clear. Bill Gates was one of his examples.
First of all you need luck to succeed. For example Bill Gates was one of the lucky few young people who had unlimited access to computer time in 1968. Second very important thing is 10 000 hours of hard work. Creating a professional skill in any area of life requires 10 years of enthusiastic learning. You have to be lucky to have a good “university” for that. It was Hamburg for the Beatles.
Success is not as much of IQ as employers are used to think. An old study of Lewis Terman, a professor of psychology, has shown that the success of people with high IQ is not as good as we typically believe. The geniuses that he found did not succeed much better than ordinary people. Gladwell conjectures that IQ has a threshold value. Your IQ must be good enough to get in into good universities but above that other things are more important. Creative thinking is more open ended than an ability to solve puzzles that have a single right solution.
Now thinking about growing good software developers. Most important thing is to have that 10 000 hours of work. It is also important to have multifaceted experience. A 10 year career of COBOL-programming in same domain area or even a piece of software is not more than 10 times one years experience. I even wonder what has kept the person in a position where he can’t learn anything new.
I emphasize attitude, the passion to do ones work. It is a known hiring guideline to hire the attitude and train the skill, but this is much easier to say than actually do. The same frustrated looser that you fired may become a passionate star of your competitor. So, it is not so much about selecting people but about creating a corporate culture. An that is really hard, especially when your business environment is difficult.
Agile and lean software development is all about people. It is not about processes and tools as should remember from Agile Manifesto. Nevertheless, the people focus on the Scrum process and kanban in Lean believing that the change of process is the silver bullet. I admit that I have seen remarkable increases in teams’ motivation when they have adopted Scrum, but I assume that the correlation does not mean a causal relationship, especially if we ignore the human part of agility.
In this post I have intentionally ignored what Gladwell said about cultural background. Read that from his book 
Agile acquisition and fixed price
Jun/090
I have been quite busy lately and not been very active with this blog. However, discussion about agile acquisition has become more acute because there are few failed purchases in public discussion. I mean AKE and TEO, who’s purchasing procedures have been questioned by Valtiontalouden tarkastusvirasto. See more (in Finnish)
General public and politicians typically react to problems by requiring more front-end design and rigor to the procedures. This makes the situation worse, because it increases unnecessary costs and bureaucracy.
At first we need to understand that hiring a designer is different than buying something that can be accurately defined before the purchase. When you buy design work minimizing the cost is not the only and not even the most important consideration. There are huge quality differences in the various solutions offered by the vendors. Comparing them is not straightforward and easy. One of them could propose a COTS solution, another something based on SOA and 3rd one a fully tailored solution. Obviously, it is not possible to define exact requirements without any idea of the implementation. Actually defining the exact requirements is exactly what the development project is supposed to do. If I knew exactly what I want I would not need the designer. The designer is hired for contributing while we work together to create the forthcoming software.
Because software development is a common endeavor with the buyer and the vendor, asking a fixed price is somehow skewed. I, as a buyer would probably get the maximum price with all of the risks included in the prise. If the competitive pressure keeps the cost too low, my vendor can very easily compromise the quality. It is also useful for me that my organization has an interest to lower the costs by removing unnecessary features. Prioritization is rarely happening in fixed price projects.
But I still have a budget, idea of return of investment. To keep the costs in bay I need to collaborate with the vendor to spend the money wisely. Monthly deliveries of done increments of functionality are a safe bet. If the vendor can not deliver, I have to stop the project (and have that allowed in the contract) and try something else.
Agile acquisition
Jun/090
I have added a comment to Agile Finland’s forum
Neuvottelumenettely needs promotion
The Finnish law of public acquisition does not require that we do competitive bidding using comprehensive requirement specification. Public buyers does not, however, know the allowed negotiation procedure (Neuvontamenettely) that is more suitable for purchasing software development services. So, agile community should do promotion to improve the situation.
Read more details in the discussion chain. I have corrected the word neuvontamenettely to neuvottelumenettely.
Scrum Team availability
May/090
If you read XP or Scrum books or online sources you find that team is a stable group of developers who work full-time to accomplish the goal. The recommended way of estimating its capacity is to use historical velocity and team members report effort remaining in hours to have daily updates of the burn down charts.
In agile estimation the estimate of the complexity of the features, often expressed in story points, is the easy part though not trivial. In my thesis I have a small experience report about the variability of the feature sets, but I will leave that now. Instead I want to talk about the variability of the team’s capability to implement what they have promised.
In real life the organizations are not so lean that they have dedicated teams for the development. Instead, each developer has responsibilities in multiple projects and also critical maintenance work that must be done as soon as possible. An of course, they are also vacations, trainings, corporate administration and common meetings.
Some the capacity variation mentioned in the previous chapter is known or can be estimated in the Sprint planning. Using just velocity averages assumes that this variability is irrelevant and averages out in a large number of Sprints. But if we know that most of developers have their vacation in July, then it is just reasonable to take that into account and adjust the promises accordingly. In my own Sprints I have asked every developer individually how many person days is he/she available for Sprint work. That is best guess of the capacity at the moment. An then the team limits the work into that. Naturally, there is a mathematical relation between velocity in story points and capacity in person days. So you can still use story points as your unit if you want.
We could argue about the best guess of the capacity. In historical average velocity we take into account the average changes of the availability of the developers, because we count only the story points of realized features. It could be better estimate of the capacity if the developers have optimistic bias in the Sprint planning. But in averages we don’t have known capacity deviations as we have if we ask the capacity from the developers.
The real variability of the capacity occurs during the Sprint. It is not enough that team members report the effort remaining of each task. Suppose that I have promised to be available 15 person days in the Sprint and I get an urgent need to put 10 days to something else. Then it is quite natural that I tell that as an impediment in the daily Scrum as soon as I know about it. The consequence might be that a feature is taken away from the Sprint. Obviously the burn down chart would have told the same thing but not so early.
I think that it is relevant to take availability variations into account in Agile projects, because they have impact on the stress level of the developers with all of its consequences. The variability has also consequences in releases especially when we work in a project which has many teams that depend on each other. Realistic daily view of situation is needed.
That’s it today. A long story of a small but important detail.
Agile adoption strategy
May/090
In information security area, it is common to talk about strategy. I have used the following paper that condenses what is important: Information security strategy.
The case of agile adoption is similar though the details are different. In both cases we need some funding to make deep changes in the way we work every day. To even start we need decisions about who does and what and why just that. Large bureaucratic organizations love their process guides and still more they want to spend in developing tools for the forthcoming needs. It is not a coincidence that we have so many agile tool vendors. In the following I adapt the previous paper to agile adoption.
A strategic approach that works
1. Implement baseline practices
Typical organizations have a dysfunctional practices or a few of them. Starting with something that gives immediate and visual results is something that is needed to justify spending. Starting with Scrum-type retrospectives is a good choice because it gives us subjective measures of how successful we are when we continue. I like to call retrospectives Japanese quality circles to emphasize that agile adoption is a quality improvement initiative
2. Analyze your value stream
Now it is time to start what the consultants and coaches are so eager to sell you. Analysis will give you ideas of improvement. At least you must check the impediments of your agile adoption. If you don’t have support of your customer or your upper management, it might be very difficult to succeed. I like Admed Sidky’s agile adoption framework which gives you the order in which you should implement the agile practives.
3. Prepare business case for the improvement(s)
Now you must justify the adoption of the individual practices. Note, of course that they depend on each other. Creating a business case for each adoption step also gives you a view in which order your organization should go ahead. Of course you must consider risks as well.
4. Initiate your improvement programme
Now you are on your way with real money and as always with people and organizations you can’t predict what happens. So you must use empirical process control to steer your way. You need to establish the part of organization that facilitates transparency, inspections and adaptations. Becoming public is a necessary part of this programme. People, all of them, are an essential part of this. They make the adoption successful, not the things ivory towers can deliver.
5. Manage and deliver the programme
If your agile adoption goes as normal, more and more of your problems will surface. Some of them can be handled by the teams, but some others need corporate level development projects. Of course, your programme has multiple deliveries.
6 Review progress
In continuous improvement initiatives like this you must have your Deming cycle installed so that you can really facilitate your way to success.
This is it this time - a draft that can be extended. Comments are allowed in this blog. Because I do not visit very often, it may take time before I publish them.