QSM Study Examines the Best and Worst of Software Development Projects

A study of "the Best and Worst in Class" for software development projects was released today by Quantitative Software Management, Inc.. The results offer many insights, including the assessment that the best in class projects are 3.37 times faster to market and 7.48 times cheaper than the worst.

QSM, which has been compiling detailed data across all major industry segments for nearly 20 years, said the study represents the latest three-year time slice for the data.

The costs and duration associated with being "best" or "worse" are dramatic: the construction and test portion (the build phase) of the project for best in class was around 3.6 months/$200,000, while the worst in class took a year and rang up $1.4 million. The total lifecycle costs (the requirements analysis plus the build phase) are also meaningful: just above 7.5 months/$300,000 total for the best in class, the worst in class taking over two years at a budget-busting $2.2 million.

Four major factors contributed to the results, and only one was technology-related. The rest concerned management and technology approaches. The factors included: controlling requirements change; highly-skilled people with good functional knowledge of the application domain; effective tooling; and effective project leadership.

"Controlling change is the most important element to successful development projects," said Doug Putnam, managing partner of QSM, Inc. "An inability to adroitly manage change can be the enemy of productivity and quality. Effective leadership creates a culture where change is well-managed by highly skilled teams with good domain knowledge."

Michael Mah, managing partner of QSM Associates added, "It may seem ironic that, while tools are always a significant factor as productivity drivers, they ranked only third as an influencing factor for the quality of the projects. The study drives home the point that, despite its technology focus, software development in the end is a people activity. There can be no technology substitute for training and management."

QSM catalogued the demographics behind the statistics: 536 IT projects sampled from 31 companies in 16 countries, including 16 industries and 9 sub-sectors, ranging from Aerospace to Financial IT. There were 179 programming languages represented, the most predominant being Cobol, Visual Basic, Java, C/C++, Powerbuilder and J2EE. The types of projects included ERP, e-commerce/internet, billing, customer care, funds management, and inventory control, on mainframe and client/server platforms.

The average-sized project consisted of 30,000 new plus changed source lines of code (or about 600 function points). The elapsed time of the averaged sized project was 13.5 calendar months. Over this period of time, it was staffed at its peak by a team of 7.4 people, expending 55 person months of effort throughout the 13.5 month schedule. Best-in-class developers are 1.86 times faster to market and 2.72 times cheaper than "average" software developers, QSM said.

From a quality and reliability perspective, the mean-time-to-defect (MTTD) for the best and the worst projects was only modestly acceptable for both. At initial deployment, neither class of project would run bug-free for longer than a day at a time. It would typically take two to four months of additional debugging for both classes of projects to exhibit better than two weeks of bug-free reliability. This includes all classes of bugs, from trivial/cosmetic defects to serious and critical defects.