Why Your Software Development Process Is Broken - InformationWeek
Government // Enterprise Architecture
07:55 PM
Connect Directly

Why Your Software Development Process Is Broken

It doesn't matter if you're building the next hot iPhone app or tweaking an in-house ERP system. If you don't want to be roadkill, fundamental changes need to be made.

If software is in fact eating the world, you have two choices: Be a predator, or be prey. Sadly, unlike nature's sleek hunters, our organizations are rarely coordinated, athletic and nimble. Instead, we're like too many people shoved into a horse costume trying to coordinate our movements so we don't fall over. Maybe 10 developers just blew five weeks building a feature that no customer ever asked for, or maybe your rock star dev team just finished a migration to a cloud service that doesn't meet security requirements. Whatever. Point is, the idea that your developers and product managers can work in tandem to bring down a gazelle (read: win new business) is almost humorous. Programmers have either too little or too much control over the software development process, with both approaches leading to inefficient and ineffective practices.

The solution is simple and has worked in open source for years: Install a "benevolent dictator" as pack leader, and put nontechnical product managers out to pasture.

Look, in the beginning, your company probably had one developer, and life was good. The benefits of customized software over whatever you were doing before were dramatic, the low-hanging fruit so low and so sweet that any code jockey with a copy of Cobol/Visual Basic/PHP (depending on the decade) could make great strides. Then someone decided to sell your software, and jaguar became wildebeest.

Your Problem: Too Much Control

Once you started selling software, it became clear that programmers didn't want to talk to customers, even if executives wanted them to, which they didn't. Workloads skyrocketed, and programmers started writing what was easy rather than what would maximize profit ... er, customer satisfaction. So the CEO brought in product managers who would (in the immortal words of Office Space), deal with customers so the engineers don't have to.

As a result, we have inbound product managers (who figure out what needs to be developed and tell the programmers how to build it) and outbound product managers (who figure out how the product needs to be "communicated to the market" and also have significant input into what needs to be developed). These are now the "customers" your software developers serve -- a recipe for failure unless all product managers have an intimate knowledge of your software and how programmers build it. Otherwise, the process becomes, fundamentally, a one-way pipeline:

1. Customer [who knows what's required, describes potential solution] talks to ...

2. Product manager [who may or may not write specifications that actually reflect requirements] talks to ...

3. Developer [who knows existing product and how to fulfill the specifications product manager generates] talks to ...

No one. It's a dead end.

The best software results from a conversation between the person who has the requirements (the customer) and the person writing the specifications (the programmer). That's because there needs to be give and take -- the person writing the specifications ideally knows the art of the possible (based on available tools) and what will fit with the existing software, and needs to be able to talk through a number of implementation options with the person who has the requirements. If the product manager doesn't know how to program (and how existing software is structured and designed), then the product manager is just an impediment to that critical conversation.

Most organizations try to solve this problem using agile development. This is admittedly a better option than what was in place before, but really all it amounts to is an admission that adding a product manager layer resulted in bad software. So instead, we should break development into small chunks and have the customer continually resteer and redefine requirements (based on actual software or prototypes that we spend lots of time developing) until we get something someone wants to buy, or at least close enough. Never mind all the wasted dev time and technical debt incurred on our way to a mediocre solution; at least we're no longer throwing away products that took six months to define and 18 months to develop under "waterfall" methods, right?

The developers don't have enough control. You need to fix that.

Your New Problem: Too Much Control?

You might say, "At least developers don't have too much control, which would be worse." But you'd be wrong. While it's true that developers have too little control over specifying what needs to be done and how, they have too much control over back-end decisions that product managers don't care about. Have you ever wondered how an architecture gets to have six different database software packages, five different storage systems, four different caching layers, three different languages and two different Web servers? It's not because that was the best design. It's because new technology is cool, and developers like playing with new technology. (See: "Your Ninja 10X Rockstar Developer Is Force-Feeding Bacon to Your Lean Startup.")

The developers have too much control. You need to fix that.

Why You Need A Benevolent Dictator

By moving programmers farther from the core business reality, we both lose the benefit of having them speak directly with end users and allow them to sink into bad habits, like choosing technologies because they're cool instead of because they best serve the business. It's a double whammy. The good news is that there is a solution. The bad news? It's going to require some organizational upheaval.

The "benevolent dictator" concept has anchored the open source movement for years; there, the title is usually Benevolent Dictator for Life, but the concept is essentially that this person is in charge of both the product and how development will proceed. A benevolent dictator is capable of writing technical specifications and is large and in charge of both the product managers and the programmers.

And by "capable of writing technical specifications," I really mean "knows how to program and has been a software developer for some number of years."

And ultimately, this is the core problem. For whatever reason, we often don't put product managers in charge of developers, and we don't make sure that they can program, and we certainly don't let them call BS when developers go down the "let's do this because it's cool" path. Consider this LinkedIn discussion on "What is the most important qualification for a product manager?" I'm sorry -- "visionary" isn't the right answer. Nor are any of the other options, or any of the comments in the thread.

Don't take my word for it. Look at organizations that build really good software. Steve Jobs is an interesting example, because he made himself the customer, and then argued with his developers for what he wanted (see, for example, how Apple's iDVD was designed). Ryan Singer, a product manager with 37signals, which makes Web-based collaboration apps, including Basecamp, has the same outlook. In a recent interview, Singer said the company "doesn't do research." Its approach is to "make things we want, and to make them the way that we want them."

It's certainly possible to have a benevolent dictator product manager who doesn't dream in binary -- as long as he or she can define specifications clearly enough to deliver the vision and has the anti-BS detectors needed to keep the architecture efficient. For example, Marty Cagan, author of what many consider the product manager bible (Inspired: How to Create Products Customers Love), made a list of famous product managers and what they have in common. All are former software developers, save three: Steve Jobs; Fred Smith of FedEx; and Michael Dell, who was hacking hardware instead of software. Cagan himself was a software developer before he managed products at Netscape, AOL and eBay, and although he is not as adamant as I am about the need for programming experience, he does urge programmers to talk directly to customers. (How often does that really happen?)

Prove Me Wrong

I would love for someone to school me on how organizations with nonprogrammer product managers who interact with development teams that are under a different management structure can succeed, because there are many of them. It would be great if they could produce excellent software efficiently under that model. Please, tell me how wrong I am and what I don't understand, because I would love to give more cheerful advice than, "You need to significantly restructure and restaff" when people complain about product mediocrity and development inefficiency. But my gut says companies that fail to kill the current product manager model will end up as dinner.

Comment  | 
Print  | 
More Insights
Newest First  |  Oldest First  |  Threaded View
<<   <   Page 2 / 2
User Rank: Ninja
9/14/2013 | 2:31:26 PM
re: Why Your Software Development Process Is Broken
You are missing a few key stakeholders and roles in the software development process:
- business analysts: they compensate the lack of tech knowledge of the product manager and write detailed requirements documents to be consumed by anyone involved with development
- QA: they cover both testing and quality advocacy (the "A" in "QA") and should be talked to and talking to product manager and BA before any developers come into the fold
- customer support: who better than support to tell you what is wrong with your software and what is needed? And who better to make a final call if a new version / product can go out as is without generating a lot of follow up cost?

Lastly, make developers at least handle upper level support cases. In most cases developers will never use the software they make in a production setting. Shielding them from the pain points that the customers encounters cannot lead to anything good. Nothing straightens out a developer more than a complaining customer...after all, developers didn't want to hear what QA, BA, and support told them all along. The biggest fault that can be made is leave all decisions to developers, they know how to write marvelous code, the rest not so much.
User Rank: Ninja
9/13/2013 | 11:50:32 PM
re: Why Your Software Development Process Is Broken
I wholeheartedly agree on that point. IT certainly needs strong leadership to prevent it. However, so too the business needs someone to curtail the relentless amplification of process complexity because it tries to chase every customer "shiny thing" regardless of how distant they wander from the core business.
User Rank: Apprentice
9/13/2013 | 11:00:48 PM
re: Why Your Software Development Process Is Broken
anyone remembers Steve Jobs?
D. Henschen
D. Henschen,
User Rank: Author
9/13/2013 | 7:25:54 PM
re: Why Your Software Development Process Is Broken
This seems like an experienced, in-depth diagnosis from somebody who has been there and back. I'm not a developer, but it seems pretty straightforward that more direct interaction between developers and users would be a good thing: fewer intermediaries = more reliable communication.

But would it not be equally effective to have developers AND product managers sit in on the same requirements-gathering interviews with the customers (even if the PMs aren't technical)? Does the benevolent dictator really have time to oversee decisions by all developers and all PMs?
User Rank: Ninja
9/13/2013 | 5:04:24 PM
re: Why Your Software Development Process Is Broken
Seems to me you will still have a problem as old as software itself: Which customer do you listen to? Customer A says "got to have this, essential to use". Customer B says "I don't need that, just makes things more complicated.".
So you take a poll and 56% of customers agree with Customer A and the rest with Customer B. Is it as simple as going with majority? That may not get those customers to contribute any more revenue. But it may irritate the other 44% to look elsewhere, costing you revenue.
Is that world developers, or even benevolent dictators, need to live in? Seems like where Sales & Marketing earns their keep to me. Somehow they have to work together.
User Rank: Ninja
9/13/2013 | 4:49:39 PM
re: Why Your Software Development Process Is Broken
I'm not arguing that developers only develop things without ROI; I'm arguing that developers--left unchecked--will often choose to develop things in a way that often uses unideal technology solutions (either using too many new/different/cool technologies or using same-old, same-old technology without thinking about what is available out there).
User Rank: Ninja
9/13/2013 | 4:29:02 PM
re: Why Your Software Development Process Is Broken
I disagree that only developers invent features with no ROI. There's plenty on the business side who fire up the waaaambulance because they don't want to prove the viability of a new business model with a small-scale manual procedures before requesting millions of IT software enhancements. Then when the business model doesn't work, IT is left with a new dimension of complexity and tons of dead code. Rinse and repeat and is it any wonder why custom software is riddled with dead code and no-longer-used features that are continually maintained each time they fail an automated test case?

Example: A former employer hired someone from a similar LOB who arrived with thunderous ideas. After getting chummy with the CEO on a business trip that demonstrated the success of these ideas elsewhere, the CEO immediately ordered IT to support these new ideas in several pilot locations. Despite IT's plea for an small-scale manual tests to prove and refine the process with limited IT support, we spent six months feverishly adding features to support what was billed as something that was going to not only attract new customers looking for these new uber-convenient services but it would also more than pay for itself and allow for additional investments in price to beat competitors on loss leaders. (Queue applause sign)

Not only was the pilot a failure but there was the belief that if we give it enough time, it has to eventually work. (Sound familiar Microsoft?) More locations were added which only exacerbated the complete failure it already was. It was finally killed after the new services became the target of con artists who made an already costly mistake even that much more of a dead end.

Without a doubt the complexities added to the organization's software still exist and continue to plague their software maintenance staff.

Granted, sometimes new business models are only a success when combined with technological automation. However, I've seen quite a few crazy ideas that wreck IT portolio plans because some new hot shot upper management type joins the company with big but unproven ideas.
<<   <   Page 2 / 2
How Enterprises Are Attacking the IT Security Enterprise
How Enterprises Are Attacking the IT Security Enterprise
To learn more about what organizations are doing to tackle attacks and threats we surveyed a group of 300 IT and infosec professionals to find out what their biggest IT security challenges are and what they're doing to defend against today's threats. Download the report to see what they're saying.
Register for InformationWeek Newsletters
White Papers
Current Issue
Annual IT Salary Report 
Base pay for IT professionals has remained flat this year with a median annual salary of $88,000 for staff and $112,000 for management. However, 58% of staff and 62% of managers who responded to our survey say they're satisfied with their compensation. Download this report to find out which positions earn the highest compensation.
Twitter Feed
InformationWeek Radio
Archived InformationWeek Radio
Join us for a roundup of the top stories on InformationWeek.com for the week of November 6, 2016. We'll be talking with the InformationWeek.com editors and correspondents who brought you the top stories of the week to get the "story behind the story."
Sponsored Live Streaming Video
Everything You've Been Told About Mobility Is Wrong
Attend this video symposium with Sean Wisdom, Global Director of Mobility Solutions, and learn about how you can harness powerful new products to mobilize your business potential.
Flash Poll