It doesn't matter if you're building the next hot iPhone app or tweaking an in-house ERP system. If you don't want to be roadkill, fundamental changes need to be made.
If software is in fact eating the world, you have two choices: Be a predator, or be prey. Sadly, unlike nature's sleek hunters, our organizations are rarely coordinated, athletic and nimble. Instead, we're like too many people shoved into a horse costume trying to coordinate our movements so we don't fall over. Maybe 10 developers just blew five weeks building a feature that no customer ever asked for, or maybe your rock star dev team just finished a migration to a cloud service that doesn't meet security requirements. Whatever. Point is, the idea that your developers and product managers can work in tandem to bring down a gazelle (read: win new business) is almost humorous. Programmers have either too little or too much control over the software development process, with both approaches leading to inefficient and ineffective practices.
The solution is simple and has worked in open source for years: Install a "benevolent dictator" as pack leader, and put nontechnical product managers out to pasture.
Look, in the beginning, your company probably had one developer, and life was good. The benefits of customized software over whatever you were doing before were dramatic, the low-hanging fruit so low and so sweet that any code jockey with a copy of Cobol/Visual Basic/PHP (depending on the decade) could make great strides. Then someone decided to sell your software, and jaguar became wildebeest.
Your Problem: Too Much Control
Once you started selling software, it became clear that programmers didn't want to talk to customers, even if executives wanted them to, which they didn't. Workloads skyrocketed, and programmers started writing what was easy rather than what would maximize profit ... er, customer satisfaction. So the CEO brought in product managers who would (in the immortal words of Office Space), deal with customers so the engineers don't have to.
As a result, we have inbound product managers (who figure out what needs to be developed and tell the programmers how to build it) and outbound product managers (who figure out how the product needs to be "communicated to the market" and also have significant input into what needs to be developed). These are now the "customers" your software developers serve -- a recipe for failure unless all product managers have an intimate knowledge of your software and how programmers build it. Otherwise, the process becomes, fundamentally, a one-way pipeline:
2. Product manager [who may or may not write specifications that actually reflect requirements] talks to ...
3. Developer [who knows existing product and how to fulfill the specifications product manager generates] talks to ...
No one. It's a dead end.
The best software results from a conversation between the person who has the requirements (the customer) and the person writing the specifications (the programmer). That's because there needs to be give and take -- the person writing the specifications ideally knows the art of the possible (based on available tools) and what will fit with the existing software, and needs to be able to talk through a number of implementation options with the person who has the requirements. If the product manager doesn't know how to program (and how existing software is structured and designed), then the product manager is just an impediment to that critical conversation.
Most organizations try to solve this problem using agile development. This is admittedly a better option than what was in place before, but really all it amounts to is an admission that adding a product manager layer resulted in bad software. So instead, we should break development into small chunks and have the customer continually resteer and redefine requirements (based on actual software or prototypes that we spend lots of time developing) until we get something someone wants to buy, or at least close enough. Never mind all the wasted dev time and technical debt incurred on our way to a mediocre solution; at least we're no longer throwing away products that took six months to define and 18 months to develop under "waterfall" methods, right?
The developers don't have enough control. You need to fix that.
Your New Problem: Too Much Control?
You might say, "At least developers don't have too much control, which would be worse." But you'd be wrong. While it's true that developers have too little control over specifying what needs to be done and how, they have too much control over back-end decisions that product managers don't care about. Have you ever wondered how an architecture gets to have six different database software packages, five different storage systems, four different caching layers, three different languages and two different Web servers? It's not because that was the best design. It's because new technology is cool, and developers like playing with new technology. (See: "Your Ninja 10X Rockstar Developer Is Force-Feeding Bacon to Your Lean Startup.")
The developers have too much control. You need to fix that.
Why You Need A Benevolent Dictator
By moving programmers farther from the core business reality, we both lose the benefit of having them speak directly with end users and allow them to sink into bad habits, like choosing technologies because they're cool instead of because they best serve the business. It's a double whammy. The good news is that there is a solution. The bad news? It's going to require some organizational upheaval.
The "benevolent dictator" concept has anchored the open source movement for years; there, the title is usually Benevolent Dictator for Life, but the concept is essentially that this person is in charge of both the product and how development will proceed. A benevolent dictator is capable of writing technical specifications and is large and in charge of both the product managers and the programmers.
And by "capable of writing technical specifications," I really mean "knows how to program and has been a software developer for some number of years."
And ultimately, this is the core problem. For whatever reason, we often don't put product managers in charge of developers, and we don't make sure that they can program, and we certainly don't let them call BS when developers go down the "let's do this because it's cool" path. Consider this LinkedIn discussion on "What is the most important qualification for a product manager?" I'm sorry -- "visionary" isn't the right answer. Nor are any of the other options, or any of the comments in the thread.
Don't take my word for it. Look at organizations that build really good software. Steve Jobs is an interesting example, because he made himself the customer, and then argued with his developers for what he wanted (see, for example, how Apple's iDVD was designed). Ryan Singer, a product manager with 37signals, which makes Web-based collaboration apps, including Basecamp, has the same outlook. In a recent interview, Singer said the company "doesn't do research." Its approach is to "make things we want, and to make them the way that we want them."
It's certainly possible to have a benevolent dictator product manager who doesn't dream in binary -- as long as he or she can define specifications clearly enough to deliver the vision and has the anti-BS detectors needed to keep the architecture efficient. For example, Marty Cagan, author of what many consider the product manager bible (Inspired: How to Create Products Customers Love), made a list of famous product managers and what they have in common. All are former software developers, save three: Steve Jobs; Fred Smith of FedEx; and Michael Dell, who was hacking hardware instead of software. Cagan himself was a software developer before he managed products at Netscape, AOL and eBay, and although he is not as adamant as I am about the need for programming experience, he does urge programmers to talk directly to customers. (How often does that really happen?)
Prove Me Wrong
I would love for someone to school me on how organizations with nonprogrammer product managers who interact with development teams that are under a different management structure can succeed, because there are many of them. It would be great if they could produce excellent software efficiently under that model. Please, tell me how wrong I am and what I don't understand, because I would love to give more cheerful advice than, "You need to significantly restructure and restaff" when people complain about product mediocrity and development inefficiency. But my gut says companies that fail to kill the current product manager model will end up as dinner.