Origin of waterfall

A short history of waterfall

As experienced people remember the eve of software development was agile. Waterfall emerged because the prevailing management thinking embraced it. There was a quest for more rigor and predictability and we started to talk about software engineering which would improve our quality and productivity. More planning was an obvious solution when plans were failing.

When the systems grew the developers began to specialize, which was based on the prevailing mass production paradigm. Functional organizations were the norm everywhere. Because it was difficult to find skillful programmers, the tasks were divided so that cheaper and available labor could be used for defining, testing and documenting. Promotions to project managers enabled the traditional corporate ladder hierarchy.

Outsourcing was tried to solve the software crisis by decreasing the unit cost of the huge amount of work that was required for writing each single line of code only to find out that the distribution created another layer of complexity.

Massive process guides tried to catch every possible view of software development to make it predictable and repeatable. 80s was the era of methodologies. After that we got quality initiatives like CMM which created a quagmire of documentation and turf wars between them. It was still very difficult to tell how to create software because we have so many variations of it and its development. You had to write either something that is right and so generic that it does not help or something that must be applied or something which fits only to certain kinds of software development.

CHAOS reports told that the majority of software development project were not successful. Success factors were extracted and we found that the requirements and capability of making decisions are the keys to success. Systems thinking explains the failure of old straightforward initiatives. Specialization, outsourcing and rigorous processes answered well to one visible view of software development but they did not optimize the whole. Big front-end planning did not create better plans because the plans were frozen prematurely without appropriate testing and feedback.

In the 70s computers were less powerful than they are today. The programmer wrote the code to a paper based on which punch cards where created. Then the cards were read, compiled and finally executed. If anything went wrong the programmer had to make corrections and try again. The length of the development cycles might have been several hours if not days. Barry Boehm’s famous paper published in 1981 about software development economics was based on studies of the projects earlier than that. So it is natural that it concluded that the cost of a change is so huge that errors should be prevented at practically any cost. Reviews and careful table testing were the ways to prevent errors.

Waterfall was born due to the management paradigms but it is useful to consider the feasibility of modern iterative development using the technology of the past. Essential practices like continuous integration and automated testing were difficult but not outright impossible in all of the projects. Programming cycle times have not prevented agility after the punch card. The “mythical man-month” of Fred Brooks was written in 1975 but this was unfortunately not enough to change the history of software development.

Unit testing EJB 3

My colleague at Tieturi, Arto Santala, has written a series of three blog entries about unit testing in EJB 3 environment. It is in Finnish but you might get the point through the code examples without our language.As an idea unit testing has been known and used for decades but examples in blogosphere focus on simple situations instead of complex cases of real life. When we do unit and acceptance testing in industrial scale, we need to go deeper considerations about maintainability and performance of the test harness.

Innovation requires skill, motivation and courage

Innovation is not just brainstorming with fancy techniques. Truly new business ideas need well grown skill, deep motivation to struggle through all the forthcoming difficulties and courage to put enough effort to make it successful.

A stereotype of an inventor is a mad man who works to create a perpetual motion machine. He has limited education to base his inventions but decades of hard work is a kind of substitute of that. Chances of success are limited because it is easier to be creative at the spearhead of science. Motivation is a required because innovation is not just heureka moments in brainstorming sessions but persistent work to try different things, ideas and variations.

Many of the ideas we get are indeed doable, but not commercially feasible or the manufacturing process is not feasible.I believe in intrinsic motivators, but money is not totally out of the table, because we need money to fund the materials and inventor’s time for a long journey of trials and errors – inventors use empirical process control. Many useful inventions are out of reach of individuals, because of expensive equipment that is needed. On the other point view, governments typically overspend because too much money is chasing too few ideas. Just think of ITER.

Courage is important because new innovations break our barriers of thoughts. Something that we believed, was found to be not true. Saying that is not easy. It is easier to let it be. So, employees have no reason to innovate. They get their salaries without and hunger overcomes the desire to create something great. People have ideas but they do not go ahead with them, because that will do harm. Remember that the majority of the ideas are not that great when studied more. And if it happens to be a success story, it is not just fair that your employer gets it for free. You might get a 100 € bonus 🙂

Universities are thought to be sources of innovation. Ph.D’s have the theoretical knowledge to synthesize new ideas but current world is very eager to make sure that the big companies get the results. Tight research funding narrows the focus of the researchers. You must complete your Ph.D in four years. To do so, you take part of a large team that is sure of about the results when they start.

BTW. I found an academic study of this topic. Creativity and innovation in the Systems Engineering process [PDF].

Merry Christmas and Happy New Year

I wish you merry Christmas and happy new year 🙂 It has been surprisingly strong upturn after the economic downturn in 2009. I have  been running CSM trainings one after another – the public ones are just a peak of the iceberg.

Based on the statistics this blog is not dead though I have not written anything for a while. The backlog of topics that I would like to write grows larger and while I read a lot – web, books and discussion forums I get still more issues to comment. I don’t make a promise for the new year to be more active even though I feel that way now in my Christmas leave.

Losing weight with Scrum

Scrum is simple and hard. So is losing weigth for the majority of people.

The secret of losing weight is that you eat less. That is that simple, but doing it is hard.We can apply empirical process control to control one’s weight. Let’s have a daily weighting every morning to create a burn down chart of your body mass. Then you tell your group of peers what did you actually eat yesterday and what are you planning to eat today. This is basically how weightwatchers do it.

Eating less is hard and so is software development. Deciding what do (that is called prioritizing), limiting work to capacity under pressure and keeping the commitment are tough things to do in real life.

Disclosure: My own weight is normal and has always been.

September 2009

Work is continuing behind the scenes. August 2009 was surprisingly busy to me and to my customers. Nordea bank announced today that the economic downturn is over. This is something that the stock prices have predicted about a half a year. To those of you who are not familiar with its behavior, I like to remind that highest peek of unemployment normally occurs about half a year after the lowest economic output and that stock prices change direction before it.We have still bumpy road ahead if the bottom is in the Winter. Businesses lack money, cash to invest. Reorganizing takes time when a lot of people is involved. Inventing new ideas might be happening when we have time for that but realizing them is difficult. We need to ask permissions.

Antti Tarvanen opened a new blog about agile acquisition that is julkiset ohjelmistohankinnat in Finnish. I try to be active there, too. ScrumAlliance starts exams 1.10.2009. Find more at ScrumAlliance.org.I have few courses before that. So getting a CSM status without the exam is still possible, but everyone assures that passing the exam will be easy. I have now new versions of my favorite tools Enterprise Architect and Netbeans. EA is now capable of generating code from statechart diagrams. Netbeans 6.7.1 has interesting things in JavaFX, SOA and Webservices. I have been working with Debian flavors of Linux at home and I think that the progress in the open source area is also changing the world.

That is it now, folks. At 1:20 AM.

Lean opportunity

Organizational issues come into discussion every time we talk about agile adoption. Creating a really lean and agile organization has been found very difficult. It has been asked whether that is even possible, not to talk about being sustainable.

In current economic situation savings are the appealing part of lean thinking. The more important part, empowering the ordinary people, is hard to justify. On the contrary, power is restricted to the (upper) management especially when we talk about costs. Layouts are also used to save.

Fortunately, the availability of software development jobs is still quite good, which means that volunteered change of ones job is the way to do that. To many organizations becoming lean means layouts. Some of the them even get rid of unnecessary management layers. But, the big but. Really working together as teams and collaborating parties is far away if we are not talking about Japan where jobs are for life (maybe still). Layouts do bad to morale and motivation and organizations look like frozen. No innovations, no new ideas, just unspoken fear and risk avoidance.

Obviously, becoming lean means losing fat, but it also means that you will become more energetic, capable. I am speaking now both the people in individual level and the organizations.We have learned that economic downturns will end some day. So, the tactic of waiting has become popular. We assume that we can just sleep the bad time away and continue as usual. Unfortunately that is not how the market economy works. Downturns are the time for restructuring the businesses. Some old ways of doing business will be gone forever and new businesses have to be started. You lose your opportunity if you dig to your pothole. So, instead of just keeping your old cows and killing the calves, you should consider doing just the opposite. Getting money from your cows is of course important especially when you are running out of cash, but old cows do not milk forever. You must have courage to take your changes.

Here I am talking about the opportunities of lean and agile software development. Being effective and capable of utilizing the opportunities is always fashionable. Now we have the theories of lean thinking available.  The question is,  do you take the opportunity or does your competitor.

8 years of lean software development in Tieturi

Kalle Huhtala’s talk about practical experiences of lean software development is now down-loadable from the site of the Projektipäivät. It is in Finnish, sorry. Unfortunately I could not take part of this day but knowing Kalle, I am sure that it was a good talk.

Tieturi is a training and coaching house but we do some in-house software development to create solutions for eLearning. These projects are quite small and Kalle’s team serves multiple customers at a time. RePa is an excel sheet that is used to manage the tasks and resources. It can be classified as a kanban, because there is only one list of things to do.

A dedicated one hour weekly meeting is used to have the coordination discussions in addition what happens ad hoc in the team room.

Experiences of RePa are quite good as you can assume when you hear that we have used that for eight years. Our process has helped us to balance the work in progress and resources that we have. Limiting work to capacity is important because failures or false promises are not an option in small projects that we do.

ELearning is the answer that queue theory gives to trainer’s value stream. You can start learn as soon as you have bought the solution and you can proceed on your own when you have time. There is no need to gather large numbers of students to classes. The solutions are smaller than several days that you typically spend to a class. You can spend just minutes to learn something that is very specific to your actual need. For example, if you want to know how to use links in WordPress, you could just start a video that teaches to you to do it.

Vacations and creativity

Like most of Finns I have my vacation this month. As typical Europeans we have quite long vacations and sometimes people think that it is a kind of waste that reduces productivity.

It is well admitted that people need sleep to be productive. Overtime and trashing at work is quite typical in our profession. But still we ignore the agile principle of sustainable pace. After months of overtime people are so burned down that it is a miracle if they get any significant output. The time that has not been allocated to a specific purpose might become the most productive part of all.

Vacations are wonderful places for that provided of course that you do not have a tight schedule for your holiday. You may guess that I do not behave like that. Actually I live one day at a time. I have about 50 topics to this blog, several books to read and few home computers to upgrade.

Most importantly I have now time for exercise. Weather has been good for walking and for work in the garden. I do not run marathons like some of my colleagues. I have always combined walking and thinking. New ideas pop up when I walk. That is wonderful time considering them thoroughly.

I have a mini-laptop, Asus EEE 901 with 20GB SSD using Ubuntu 9.04 to be exact. It is an excellent tool for surfing the web and maintaining this new web site. I appreciate its small size, long battery life and connectivity. This is my first really personal computer – it is with me all days long. Now it is possible to take notes and have them with me. It is my extended memory.

I remember the word creativity in the title, so let’s talk about that now. One part of creativity is brainstorming – a wild burst of new ideas. It is only possible if you have free non-allocated time. In a tight schedule you pick the most obvious solutions and hurry on to get things done. During the vacations you have that luxury. I do not mean that you should spend your holidays to thinking work related things but people like me do it anyway not because someone pays for it but because it is fun.

Blogger’s Information Security

Let’s talk a little about my experiences with the first months of this blog. One of the most important reasons to start to use my own domain was to get rid of legacy email-addresses that gather a lot of spam. Changing my email-address everywhere was not as difficult as it sounds, because most of the incoming mail was from mailing lists that I have subscribed and they had means to change that address. Oddly enough, one of large economic magazines, Talouselämä, has a mailing list which has no way to cancel the subscription or I did not find it.

The amount of spam was actually less than what I had thought –  less than spam 10 mails a day. Filters are quite capable of classifying these as spam and the ones that get through are easy to spot visually. Hardest thing is to find people who send me email only few times a year. That requires a good address book and/or good scanning of your email database.

I have registered a lot of software during the last decades and some of them might want to contact me. At least one book store that I used more than 10 years ago, persistently sends their offers to me. These ones are quite close to spam, but there has been a real business relationship.

Starting a blog introduced me to another problem – comment spam.  Akismet spam filter is quite effective and in this blog I am afraid that some real comments gets filtered. I am sorry about that.  But still, a good comment has something to do with the post that you are commenting. Just, “I liked that”, does not.

Because I have had Russian at school, you can’t lure me with Kyrillic letters.I am an advocate of freedom of speech especially in Internet. See more about this topic in EFFI . So, I prefer anonymous comments without any emails and home pages. Links are considered harmful in information security point of view. So, I still welcome comments.