A short history of waterfall
As experienced people remember the eve of software development was agile. Waterfall emerged because the prevailing management thinking embraced it. There was a quest for more rigor and predictability and we started to talk about software engineering which would improve our quality and productivity. More planning was an obvious solution when plans were failing.
When the systems grew the developers began to specialize, which was based on the prevailing mass production paradigm. Functional organizations were the norm everywhere. Because it was difficult to find skillful programmers, the tasks were divided so that cheaper and available labor could be used for defining, testing and documenting. Promotions to project managers enabled the traditional corporate ladder hierarchy.
Outsourcing was tried to solve the software crisis by decreasing the unit cost of the huge amount of work that was required for writing each single line of code only to find out that the distribution created another layer of complexity.
Massive process guides tried to catch every possible view of software development to make it predictable and repeatable. 80s was the era of methodologies. After that we got quality initiatives like CMM which created a quagmire of documentation and turf wars between them. It was still very difficult to tell how to create software because we have so many variations of it and its development. You had to write either something that is right and so generic that it does not help or something that must be applied or something which fits only to certain kinds of software development.
CHAOS reports told that the majority of software development project were not successful. Success factors were extracted and we found that the requirements and capability of making decisions are the keys to success. Systems thinking explains the failure of old straightforward initiatives. Specialization, outsourcing and rigorous processes answered well to one visible view of software development but they did not optimize the whole. Big front-end planning did not create better plans because the plans were frozen prematurely without appropriate testing and feedback.
In the 70s computers were less powerful than they are today. The programmer wrote the code to a paper based on which punch cards where created. Then the cards were read, compiled and finally executed. If anything went wrong the programmer had to make corrections and try again. The length of the development cycles might have been several hours if not days. Barry Boehm’s famous paper published in 1981 about software development economics was based on studies of the projects earlier than that. So it is natural that it concluded that the cost of a change is so huge that errors should be prevented at practically any cost. Reviews and careful table testing were the ways to prevent errors.
Waterfall was born due to the management paradigms but it is useful to consider the feasibility of modern iterative development using the technology of the past. Essential practices like continuous integration and automated testing were difficult but not outright impossible in all of the projects. Programming cycle times have not prevented agility after the punch card. The “mythical man-month” of Fred Brooks was written in 1975 but this was unfortunately not enough to change the history of software development.