The Fall of Waterfall

Does your development methodology send you down Niagara Falls in a barrel?

Ever wonder why there are so few failed bridge-construction projects when most software is developed using a similar process? Conventional wisdom would dictate that building bridges would be harder than building Web sites. In this day and age, when you can probably count your company's truly successful IT projects on one hand, engineering's tried-and-true waterfall process just might not be ideal for software development.

How We Got Here

When I think of software development project management pains, these symptoms top my list of significant failure indicators:

  • While you're at it: A symptom that causes project scope to spiral out of control. A false belief in ever-diminishing marginal feature costs often make business users act like kleptomaniacs on the "Supermarket Sweep" game show when formulating requirements.
  • Once-in-a-lifetime opportunity: Business users often feel that future enhancement requests will sit on an infinite backlog. This fear tends to make them inclined to ask for everything they ever anticipate needing and hesitant to prioritize anything below "critical."
  • Lifetime project opportunity: A strict waterfall methodology with a long development life-cycle limits adaptability to business changes or ideas. This is why the system users needed yesterday never gets used when they have to wait six months to get it.
  • Long-and-dirty solution: The waterfall methodology forces the solution to be implemented with a design that seemed appropriate at the beginning of the project, but usually ends up being either over- or underengineered, resulting in poor performance and excessive maintenance costs.
  • Stuff that we rightfully get blamed for: Common mistakes in the engineering process, such as misestimating, shortcutting, or de-emphasizing quality, are accentuated by the waterfall methodology.
  • Hate/hate relationship: The "bond" formed between IT and business stakeholders based on an accumulation of tensions that result from project difficulties and subsequent failures. This adversarial relationship usually further degrades from project to project, exacerbating the issues caused by the waterfall methodology.

The Pros and Cons

Still true today, and as described in Frederick P. Brooks' The Mythical Man-Month: Essays on Software Engineering, Anniversary Edition (Addison-Wesley, 1995), large-scale software development projects executed with a waterfall approach are rarely successfully implemented on time or on budget in such a way that the spirit of the solution is realized. This has increased the popularity of alternative approaches such as eXtreme Programming, Unified Process, Feature-Driven Development, SCRUM, and others. Contrary to the waterfall approach, these methodologies strive for increased agility by breaking a large project down into smaller repeated iterations of the development life cycle. Iterative approaches combat the shortcomings listed previously with:

  • A development cycle with shorter repeated iterations. Implementing aspects of the system followed by requests for additional user feedback or requirements repeatedly will result in fewer deviations between the user's vision of the system and its actual implementation.
  • Ability to focus on high-priority functionality. Agile methodologies provide more opportunities to shift a project's direction and emphasis to stay in concert with ever-changing business needs.
  • Forced revisiting of past work for quality improvement. Effective developers not only develop a system that meets today's feature requirements but also design for ease of maintenance and extensibility. Often within a waterfall project with a looming deadline, a developer sacrifices future maintainability in order to implement the requested features. The iterative model encourages devoting time during the iterations to improve the quality of what was developed during the previous iteration, resulting in more stable and easier maintained systems.
  • Project outcomes have less variability. Using an iterative methodology means fewer "I hope a miracle occurs to bring this all together in the end" pits in the stomach because the process is much more controlled. Misestimating by 20 percent during a four-week iteration is much less problematic than missing by the same percent in a nine-month waterfall project.

For all the advantages of using an alternative approach, however, many organizations are resisting changing their deficient existing processes. The primary concerns that keep organizations from embracing these new methodologies are:

  • Logistics of stakeholder interaction. EXtreme Programming, by far the most well known of these methodologies, requires that the business stakeholders be in close proximity to the system developers so there can be interaction on a continual basis to prioritize new features and provide feedback. In general, assuming that the business stakeholder has a primary job outside of the project, this interaction is not always feasible.
  • Logistics of IT resource allocation. Many organizations have very specialized IT roles (such as a data modeler who works with the team during the design phase), and it would be difficult to allocate the diverse set of necessary resources with an iterative schedule where there involvement would be needed much more frequently.
  • Financial objections. Business users and financial analysts like to know what they are getting, and for how much. The iterative approach's quintessential response to this question is "you're going to get as many of the high-priority features that we are able to do in the timeframe and budget that are available to us." Needless to say, this usually doesn't go over very well with the CFO. The fact that these investments usually require ROI justification makes the financial objections even more germane. Additionally, a large project can be capitalized, meaning that the business can pay for it as a one-time expenditure and in most cases depreciate the solution's cost over time with less of a negative effect on short-term profits. In some cases, the expenses associated with a full-time staff that continuously develop a solution in an iterative fashion would be difficult to capitalize in the same manner.
  • Perception issues. Many of the issues that have tainted the use of iterative methodologies relate to perception. Many still believe that "eXtreme programmers" are college dropouts who say "dude" a lot and can't be counted on to be at work during high tide. However, these prejudices often lead to "It isn't for our company" dismissals without truly understanding the methodology's issues and benefits.
  • Effort required. The iterative approach requires a significant amount of effort and management because the timeframes are shorter. If a developer is out for a week during a four-week iteration, it is a big loss to the project. Many feel that the iterative approach is too demanding because it focuses on consistent workload rather than workloads with very high variability.
  • Cost of change. Finally, those who are convinced of the merits of agile methodologies typically quiver at the idea of trying to justify the cost and effort involved to make the transition. These hard and soft costs often include new training, new tools, different resource allocations, resource changes, and the loss of productivity during the transition.

How We Can Improve

I believe that for most organizations looking to use alternative methodologies, a controlled evolution is the best mechanism for a transition to an agile methodology. To alleviate some of the short-term issues with the waterfall methodology and to assist in a migration toward an agile approach, I recommend:

  • Reducing projects' scope and team size to about one half of your current larger projects. Break up larger efforts into multiple projects.
  • Allocating 20 to 30 percent of the project budget to perform post release refactorings, stabilization, improvements, developer training, and documentation.
  • Budgeting like waterfall but developing in an iterative fashion. A good friend suggested this idea, which works well to allow projects to be prebudgeted, but still benefit from an iterative approach.
  • Doing more interim releases and demos in an iterative fashion. Enhance your ability to get early feedback and raise your team's confidence in meeting deadlines by releasing on a frequent basis. Much like completing a project, the most satisfying culmination to an iteration is in deployment — even if it's to only a subset of users.
  • Staffing the projects with dedicated resources. Resources allocated as part time contribute little and often detract from progress.

If you build systems that, like bridges, have only one primary purpose or feature, the waterfall methodology should be fine. But if you're looking to make your systems do things that bridges can't, like increase feature sets every six to eight weeks and cost less to maintain over time, it's time to upgrade your methodology.

Robert Northrop [[email protected]] is a director of design and development with Tallan, a professional services company specializing in developing custom technology solutions for its clients.

We welcome your comments on this topic on our social media channels, or [contact us directly] with questions about the site.
Comment  | 
Email This  | 
Print  | 
More Insights
Copyright © 2021 UBM Electronics, A UBM company, All rights reserved. Privacy Policy | Terms of Service