New lean and agile books

I bought some lean&agile  books to read during my summer holiday. The list is not complete because I have already read quite many of them. I got Mike Cohn’s new Scrum book freely from the publisher. Thanks about that.

Craig Larman: “Scaling Lean and Agile Development: Thinking and Organizational Tools for Large-Scale Scrum: Successful Large, Multisite and Offshore Products with Large-scale Scrum (Agile Software Development)”.

Lisa Crispin: “Agile Testing: A Practical Guide for Testers and Agile Teams (Addison-Wesley Signature)”.

Mary Poppendieck: “Leading Lean Software Development: Results are Not the Point (Addison-Wesley Signature)”

Roman Pichler: “Agile Product Management with Scrum: Creating Products That Customers Love (Addison-Wesley Signature)”.

Lyssa Adkins: “Coaching Agile Teams: A Companion for ScrumMasters, Agile Coaches, and Project Managers in Transition (Addison-Wesley Signature Series (Cohn))”.

David J Anderson: “Kanban”.

Jason Fried: “ReWork: Change the Way You Work Forever”.

James Coplien: “Lean Architecture: for Agile Software Development”.

Daniel H. Pink: “Drive: The Surprising Truth About What Motivates Us”.

Chip Heath: “Switch: How to Change Things When Change is Hard”.

Origin of waterfall

A short history of waterfall

As experienced people remember the eve of software development was agile. Waterfall emerged because the prevailing management thinking embraced it. There was a quest for more rigor and predictability and we started to talk about software engineering which would improve our quality and productivity. More planning was an obvious solution when plans were failing.

When the systems grew the developers began to specialize, which was based on the prevailing mass production paradigm. Functional organizations were the norm everywhere. Because it was difficult to find skillful programmers, the tasks were divided so that cheaper and available labor could be used for defining, testing and documenting. Promotions to project managers enabled the traditional corporate ladder hierarchy.

Outsourcing was tried to solve the software crisis by decreasing the unit cost of the huge amount of work that was required for writing each single line of code only to find out that the distribution created another layer of complexity.

Massive process guides tried to catch every possible view of software development to make it predictable and repeatable. 80s was the era of methodologies. After that we got quality initiatives like CMM which created a quagmire of documentation and turf wars between them. It was still very difficult to tell how to create software because we have so many variations of it and its development. You had to write either something that is right and so generic that it does not help or something that must be applied or something which fits only to certain kinds of software development.

CHAOS reports told that the majority of software development project were not successful. Success factors were extracted and we found that the requirements and capability of making decisions are the keys to success. Systems thinking explains the failure of old straightforward initiatives. Specialization, outsourcing and rigorous processes answered well to one visible view of software development but they did not optimize the whole. Big front-end planning did not create better plans because the plans were frozen prematurely without appropriate testing and feedback.

In the 70s computers were less powerful than they are today. The programmer wrote the code to a paper based on which punch cards where created. Then the cards were read, compiled and finally executed. If anything went wrong the programmer had to make corrections and try again. The length of the development cycles might have been several hours if not days. Barry Boehm’s famous paper published in 1981 about software development economics was based on studies of the projects earlier than that. So it is natural that it concluded that the cost of a change is so huge that errors should be prevented at practically any cost. Reviews and careful table testing were the ways to prevent errors.

Waterfall was born due to the management paradigms but it is useful to consider the feasibility of modern iterative development using the technology of the past. Essential practices like continuous integration and automated testing were difficult but not outright impossible in all of the projects. Programming cycle times have not prevented agility after the punch card. The “mythical man-month” of Fred Brooks was written in 1975 but this was unfortunately not enough to change the history of software development.

Unit testing EJB 3

My colleague at Tieturi, Arto Santala, has written a series of three blog entries about unit testing in EJB 3 environment. It is in Finnish but you might get the point through the code examples without our language.As an idea unit testing has been known and used for decades but examples in blogosphere focus on simple situations instead of complex cases of real life. When we do unit and acceptance testing in industrial scale, we need to go deeper considerations about maintainability and performance of the test harness.

Innovation requires skill, motivation and courage

Innovation is not just brainstorming with fancy techniques. Truly new business ideas need well grown skill, deep motivation to struggle through all the forthcoming difficulties and courage to put enough effort to make it successful.

A stereotype of an inventor is a mad man who works to create a perpetual motion machine. He has limited education to base his inventions but decades of hard work is a kind of substitute of that. Chances of success are limited because it is easier to be creative at the spearhead of science. Motivation is a required because innovation is not just heureka moments in brainstorming sessions but persistent work to try different things, ideas and variations.

Many of the ideas we get are indeed doable, but not commercially feasible or the manufacturing process is not feasible.I believe in intrinsic motivators, but money is not totally out of the table, because we need money to fund the materials and inventor’s time for a long journey of trials and errors – inventors use empirical process control. Many useful inventions are out of reach of individuals, because of expensive equipment that is needed. On the other point view, governments typically overspend because too much money is chasing too few ideas. Just think of ITER.

Courage is important because new innovations break our barriers of thoughts. Something that we believed, was found to be not true. Saying that is not easy. It is easier to let it be. So, employees have no reason to innovate. They get their salaries without and hunger overcomes the desire to create something great. People have ideas but they do not go ahead with them, because that will do harm. Remember that the majority of the ideas are not that great when studied more. And if it happens to be a success story, it is not just fair that your employer gets it for free. You might get a 100 € bonus 🙂

Universities are thought to be sources of innovation. Ph.D’s have the theoretical knowledge to synthesize new ideas but current world is very eager to make sure that the big companies get the results. Tight research funding narrows the focus of the researchers. You must complete your Ph.D in four years. To do so, you take part of a large team that is sure of about the results when they start.

BTW. I found an academic study of this topic. Creativity and innovation in the Systems Engineering process [PDF].

Merry Christmas and Happy New Year

I wish you merry Christmas and happy new year 🙂 It has been surprisingly strong upturn after the economic downturn in 2009. I have  been running CSM trainings one after another – the public ones are just a peak of the iceberg.

Based on the statistics this blog is not dead though I have not written anything for a while. The backlog of topics that I would like to write grows larger and while I read a lot – web, books and discussion forums I get still more issues to comment. I don’t make a promise for the new year to be more active even though I feel that way now in my Christmas leave.

Less extreme Scrum

Agile software development has often seen as an foreign element leading to chaos in Virginia Satir’s model. Current situation of Scrum is indeed somehow chaotic. Key persons are arguing against each other and it has been very demanding to define what Scrum actually is? Ken Schwaber’s ScrumGuide is our current definition but it is not ready for a Scrum Alliance’s multi-choice exam.

There are many different versions or variations of Scrum which may or may not be under a common framework. Common thing in these is that they are more realistic and less extreme. It has been visible for a while that radical extreme Scrum has given space to real life. Some of these modifications clearly belong under the title “Scrum but” but some others are coming from Schwaber and Sutherland, who are the authorities who define Scrum.

In the London Gathering Schwaber introduced product backlog refactoring meetings where team collaborates with the product owner to create actionable product backlog items. In Munich Gathering Sutherland emphasized that the user story must be ready for the Sprint, which means that a good enough specification exists to continue.

The main difference here between Rational Unified Process and Scrum is in the collaboration between the developers and the product owner. In RUP there are strong roles and artifacts that are just handed to next group of people. Product owners’ role has got more content by the discussion of release planning – still obscurely defined time-box of Scrum.We are now also openly talking about undone features and telling that team might not be able to complete a user story to the point that it can released. So we add a stabilization Sprint where undone features are completed before the release.

Situational leadership model was my favorite in the 80’s. In addition to Tucman’s model it was also well visible in the Munich Gathering. Actually that means that we have to admit that leaders can’t just assume that teams self-organize when they are just empowered to do so. Sometimes directive forms of leadership are needed though we understand the drawbacks of command and control.

Schwaber’s integration teams that he introduced in his book Enterprise and Scrum have not been heard much in talks of the Scrum people. The case studies of real life projects tell clearly that defined organizations are used, not just self-organizing flocks of business and technical people. The question of scaling agility is interesting and clearly not solved in a way that theoreticians and practitioners can both accept. Actually, I would like to asses (or rather measure) different agile solutions instead of arguing about right or wrong Scrum.

Scrum gathering Munich

I am taking part to Scrum gathering this week. This is fully booked though the economy is down. The talks that I have heard so far are encouraging and enjoyable. I am here to see friends and to hear their stories and experiences. There are many issues that I should rise (in this blog and otherwise) but I the thing near my heart is collaborative design. Developers are still struggling with functional organizations. Product owners don’t talk with the developers and we have subgroups of developers who communicate only with specifications. This might be because of the early phase of the agile adoption but understanding the cross-functional teams might be difficult too.

Losing weight with Scrum

Scrum is simple and hard. So is losing weigth for the majority of people.

The secret of losing weight is that you eat less. That is that simple, but doing it is hard.We can apply empirical process control to control one’s weight. Let’s have a daily weighting every morning to create a burn down chart of your body mass. Then you tell your group of peers what did you actually eat yesterday and what are you planning to eat today. This is basically how weightwatchers do it.

Eating less is hard and so is software development. Deciding what do (that is called prioritizing), limiting work to capacity under pressure and keeping the commitment are tough things to do in real life.

Disclosure: My own weight is normal and has always been.

September 2009

Work is continuing behind the scenes. August 2009 was surprisingly busy to me and to my customers. Nordea bank announced today that the economic downturn is over. This is something that the stock prices have predicted about a half a year. To those of you who are not familiar with its behavior, I like to remind that highest peek of unemployment normally occurs about half a year after the lowest economic output and that stock prices change direction before it.We have still bumpy road ahead if the bottom is in the Winter. Businesses lack money, cash to invest. Reorganizing takes time when a lot of people is involved. Inventing new ideas might be happening when we have time for that but realizing them is difficult. We need to ask permissions.

Antti Tarvanen opened a new blog about agile acquisition that is julkiset ohjelmistohankinnat in Finnish. I try to be active there, too. ScrumAlliance starts exams 1.10.2009. Find more at ScrumAlliance.org.I have few courses before that. So getting a CSM status without the exam is still possible, but everyone assures that passing the exam will be easy. I have now new versions of my favorite tools Enterprise Architect and Netbeans. EA is now capable of generating code from statechart diagrams. Netbeans 6.7.1 has interesting things in JavaFX, SOA and Webservices. I have been working with Debian flavors of Linux at home and I think that the progress in the open source area is also changing the world.

That is it now, folks. At 1:20 AM.

Lean opportunity

Organizational issues come into discussion every time we talk about agile adoption. Creating a really lean and agile organization has been found very difficult. It has been asked whether that is even possible, not to talk about being sustainable.

In current economic situation savings are the appealing part of lean thinking. The more important part, empowering the ordinary people, is hard to justify. On the contrary, power is restricted to the (upper) management especially when we talk about costs. Layouts are also used to save.

Fortunately, the availability of software development jobs is still quite good, which means that volunteered change of ones job is the way to do that. To many organizations becoming lean means layouts. Some of the them even get rid of unnecessary management layers. But, the big but. Really working together as teams and collaborating parties is far away if we are not talking about Japan where jobs are for life (maybe still). Layouts do bad to morale and motivation and organizations look like frozen. No innovations, no new ideas, just unspoken fear and risk avoidance.

Obviously, becoming lean means losing fat, but it also means that you will become more energetic, capable. I am speaking now both the people in individual level and the organizations.We have learned that economic downturns will end some day. So, the tactic of waiting has become popular. We assume that we can just sleep the bad time away and continue as usual. Unfortunately that is not how the market economy works. Downturns are the time for restructuring the businesses. Some old ways of doing business will be gone forever and new businesses have to be started. You lose your opportunity if you dig to your pothole. So, instead of just keeping your old cows and killing the calves, you should consider doing just the opposite. Getting money from your cows is of course important especially when you are running out of cash, but old cows do not milk forever. You must have courage to take your changes.

Here I am talking about the opportunities of lean and agile software development. Being effective and capable of utilizing the opportunities is always fashionable. Now we have the theories of lean thinking available.  The question is,  do you take the opportunity or does your competitor.