Ever wonder why there are so few failed bridge-construction projects when most software is developed using a similar process? Conventional wisdom would dictate that building bridges would be harder than building Web sites. In this day and age, when you can probably count your company's truly successful IT projects on one hand, engineering's tried-and-true waterfall process just might not be ideal for software development.

How We Got Here

When I think of software development project management pains, these symptoms top my list of significant failure indicators:

• While you're at it: A symptom that causes project scope to spiral out of control. A false belief in ever-diminishing marginal feature costs often make business users act like kleptomaniacs on the "Supermarket Sweep" game show when formulating requirements.
• Once-in-a-lifetime opportunity: Business users often feel that future enhancement requests will sit on an infinite backlog. This fear tends to make them inclined to ask for everything they ever anticipate needing and hesitant to prioritize anything below "critical."
• Lifetime project opportunity: A strict waterfall methodology with a long development life-cycle limits adaptability to business changes or ideas. This is why the system users needed yesterday never gets used when they have to wait six months to get it.
• Long-and-dirty solution: The waterfall methodology forces the solution to be implemented with a design that seemed appropriate at the beginning of the project, but usually ends up being either over- or underengineered, resulting in poor performance and excessive maintenance costs.
• Stuff that we rightfully get blamed for: Common mistakes in the engineering process, such as misestimating, shortcutting, or de-emphasizing quality, are accentuated by the waterfall methodology.
• Hate/hate relationship: The "bond" formed between IT and business stakeholders based on an accumulation of tensions that result from project difficulties and subsequent failures. This adversarial relationship usually further degrades from project to project, exacerbating the issues caused by the waterfall methodology.

The Pros and Cons

Still true today, and as described in Frederick P. Brooks' The Mythical Man-Month: Essays on Software Engineering, Anniversary Edition (Addison-Wesley, 1995), large-scale software development projects executed with a waterfall approach are rarely successfully implemented on time or on budget in such a way that the spirit of the solution is realized. This has increased the popularity of alternative approaches such as eXtreme Programming, Unified Process, Feature-Driven Development, SCRUM, and others. Contrary to the waterfall approach, these methodologies strive for increased agility by breaking a large project down into smaller repeated iterations of the development life cycle. Iterative approaches combat the shortcomings listed previously with:

• A development cycle with shorter repeated iterations. Implementing aspects of the system followed by requests for additional user feedback or requirements repeatedly will result in fewer deviations between the user's vision of the system and its actual implementation.
• Ability to focus on high-priority functionality. Agile methodologies provide more opportunities to shift a project's direction and emphasis to stay in concert with ever-changing business needs.
• Forced revisiting of past work for quality improvement. Effective developers not only develop a system that meets today's feature requirements but also design for ease of maintenance and extensibility. Often within a waterfall project with a looming deadline, a developer sacrifices future maintainability in order to implement the requested features. The iterative model encourages devoting time during the iterations to improve the quality of what was developed during the previous iteration, resulting in more stable and easier maintained systems.
• Project outcomes have less variability. Using an iterative methodology means fewer "I hope a miracle occurs to bring this all together in the end" pits in the stomach because the process is much more controlled. Misestimating by 20 percent during a four-week iteration is much less problematic than missing by the same percent in a nine-month waterfall project.