Government // Enterprise Architecture
Commentary
9/11/2013
07:55 PM
Connect Directly
LinkedIn
Google+
RSS
E-Mail
50%
50%
Repost This

Why Your Software Development Process Is Broken

It doesn't matter if you're building the next hot iPhone app or tweaking an in-house ERP system. If you don't want to be roadkill, fundamental changes need to be made.

If software is in fact eating the world, you have two choices: Be a predator, or be prey. Sadly, unlike nature's sleek hunters, our organizations are rarely coordinated, athletic and nimble. Instead, we're like too many people shoved into a horse costume trying to coordinate our movements so we don't fall over. Maybe 10 developers just blew five weeks building a feature that no customer ever asked for, or maybe your rock star dev team just finished a migration to a cloud service that doesn't meet security requirements. Whatever. Point is, the idea that your developers and product managers can work in tandem to bring down a gazelle (read: win new business) is almost humorous. Programmers have either too little or too much control over the software development process, with both approaches leading to inefficient and ineffective practices.

The solution is simple and has worked in open source for years: Install a "benevolent dictator" as pack leader, and put nontechnical product managers out to pasture.

Look, in the beginning, your company probably had one developer, and life was good. The benefits of customized software over whatever you were doing before were dramatic, the low-hanging fruit so low and so sweet that any code jockey with a copy of Cobol/Visual Basic/PHP (depending on the decade) could make great strides. Then someone decided to sell your software, and jaguar became wildebeest.

Your Problem: Too Much Control

Once you started selling software, it became clear that programmers didn't want to talk to customers, even if executives wanted them to, which they didn't. Workloads skyrocketed, and programmers started writing what was easy rather than what would maximize profit ... er, customer satisfaction. So the CEO brought in product managers who would (in the immortal words of Office Space), deal with customers so the engineers don't have to.

As a result, we have inbound product managers (who figure out what needs to be developed and tell the programmers how to build it) and outbound product managers (who figure out how the product needs to be "communicated to the market" and also have significant input into what needs to be developed). These are now the "customers" your software developers serve -- a recipe for failure unless all product managers have an intimate knowledge of your software and how programmers build it. Otherwise, the process becomes, fundamentally, a one-way pipeline:

1. Customer [who knows what's required, describes potential solution] talks to ...

2. Product manager [who may or may not write specifications that actually reflect requirements] talks to ...

3. Developer [who knows existing product and how to fulfill the specifications product manager generates] talks to ...

No one. It's a dead end.

The best software results from a conversation between the person who has the requirements (the customer) and the person writing the specifications (the programmer). That's because there needs to be give and take -- the person writing the specifications ideally knows the art of the possible (based on available tools) and what will fit with the existing software, and needs to be able to talk through a number of implementation options with the person who has the requirements. If the product manager doesn't know how to program (and how existing software is structured and designed), then the product manager is just an impediment to that critical conversation.

Most organizations try to solve this problem using agile development. This is admittedly a better option than what was in place before, but really all it amounts to is an admission that adding a product manager layer resulted in bad software. So instead, we should break development into small chunks and have the customer continually resteer and redefine requirements (based on actual software or prototypes that we spend lots of time developing) until we get something someone wants to buy, or at least close enough. Never mind all the wasted dev time and technical debt incurred on our way to a mediocre solution; at least we're no longer throwing away products that took six months to define and 18 months to develop under "waterfall" methods, right?

The developers don't have enough control. You need to fix that.

Your New Problem: Too Much Control?

You might say, "At least developers don't have too much control, which would be worse." But you'd be wrong. While it's true that developers have too little control over specifying what needs to be done and how, they have too much control over back-end decisions that product managers don't care about. Have you ever wondered how an architecture gets to have six different database software packages, five different storage systems, four different caching layers, three different languages and two different Web servers? It's not because that was the best design. It's because new technology is cool, and developers like playing with new technology. (See: "Your Ninja 10X Rockstar Developer Is Force-Feeding Bacon to Your Lean Startup.")

The developers have too much control. You need to fix that.

Why You Need A Benevolent Dictator

By moving programmers farther from the core business reality, we both lose the benefit of having them speak directly with end users and allow them to sink into bad habits, like choosing technologies because they're cool instead of because they best serve the business. It's a double whammy. The good news is that there is a solution. The bad news? It's going to require some organizational upheaval.

The "benevolent dictator" concept has anchored the open source movement for years; there, the title is usually Benevolent Dictator for Life, but the concept is essentially that this person is in charge of both the product and how development will proceed. A benevolent dictator is capable of writing technical specifications and is large and in charge of both the product managers and the programmers.

And by "capable of writing technical specifications," I really mean "knows how to program and has been a software developer for some number of years."

And ultimately, this is the core problem. For whatever reason, we often don't put product managers in charge of developers, and we don't make sure that they can program, and we certainly don't let them call BS when developers go down the "let's do this because it's cool" path. Consider this LinkedIn discussion on "What is the most important qualification for a product manager?" I'm sorry -- "visionary" isn't the right answer. Nor are any of the other options, or any of the comments in the thread.

Don't take my word for it. Look at organizations that build really good software. Steve Jobs is an interesting example, because he made himself the customer, and then argued with his developers for what he wanted (see, for example, how Apple's iDVD was designed). Ryan Singer, a product manager with 37signals, which makes Web-based collaboration apps, including Basecamp, has the same outlook. In a recent interview, Singer said the company "doesn't do research." Its approach is to "make things we want, and to make them the way that we want them."

It's certainly possible to have a benevolent dictator product manager who doesn't dream in binary -- as long as he or she can define specifications clearly enough to deliver the vision and has the anti-BS detectors needed to keep the architecture efficient. For example, Marty Cagan, author of what many consider the product manager bible (Inspired: How to Create Products Customers Love), made a list of famous product managers and what they have in common. All are former software developers, save three: Steve Jobs; Fred Smith of FedEx; and Michael Dell, who was hacking hardware instead of software. Cagan himself was a software developer before he managed products at Netscape, AOL and eBay, and although he is not as adamant as I am about the need for programming experience, he does urge programmers to talk directly to customers. (How often does that really happen?)

Prove Me Wrong

I would love for someone to school me on how organizations with nonprogrammer product managers who interact with development teams that are under a different management structure can succeed, because there are many of them. It would be great if they could produce excellent software efficiently under that model. Please, tell me how wrong I am and what I don't understand, because I would love to give more cheerful advice than, "You need to significantly restructure and restaff" when people complain about product mediocrity and development inefficiency. But my gut says companies that fail to kill the current product manager model will end up as dinner.

Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
Page 1 / 2   >   >>
AndrewT197
50%
50%
AndrewT197,
User Rank: Apprentice
9/19/2013 | 4:10:27 AM
re: Why Your Software Development Process Is Broken
Not a bad article, but I think it is brave to try bring a generic model to all development environments. The article starts with "It doesn't matter if you're building the next hot iPhone app or tweaking an in-house ERP system." then jumps to the mutually exclusive advantages of either side.

ERP -> many locked in restrictions
iPhone App -> solving a specific need

Major design differences.

A product manager needs to be technical if a specific need is being addressed (most open source projects) in other instances where many customers exists, the product manager needs to mediate AS a nonbiast customer without knowing the technical side, but more focused on the business side.

The problem that brews today is learning programming is a long hard ongoing task and most developer / programmers you meet don't actually learn how technologies work they believe a language is the answer to getting things done. Now today many programmers short cut the learning curve (university / training and resources are expensive) because they need to earn money quickly.

We all must start somewhere, but the funds allocated in startups are minimal so few people have to do the job of massive corporations. Also most startups are based on an idea or market need, so many times the starting point has a customer profile in mind and not a real paying customer.

In a massive world it would go like this. Notice how late the technical guys come in, but in small worlds, the technical guys must solve problems from other directions.

Note advertising is not sales.

Strategic managers talk to Marketing
Marketing talks to Advertisers

Advertisers talk to customers (potential and existing)
Customers talk to Sales
Sales talk to marketing
Marketing talks to Strategic managers
Strategic managers talk to systems analysts
Analysts talk to Product managers
Analysts talk to Test department
Analysts talk to Design department
Design department talks to Senior developers
Senior Developers talk to junior developers.

Development is a long term thing. Short cuts generate startups but ultimately long term is professionalism.
jemison288
50%
50%
jemison288,
User Rank: Moderator
9/17/2013 | 1:45:44 PM
re: Why Your Software Development Process Is Broken
Yes, and also, you better make sure your benevolent dictator is (a) smart, (b) motivated by the right things, and (c) truly benevolent.
ChrisMurphy
50%
50%
ChrisMurphy,
User Rank: Author
9/17/2013 | 12:49:17 PM
re: Why Your Software Development Process Is Broken
A response to my tweet of this story makes a good point about needing to motivate your coders: "Even if you have a benevolent dictator, ALL need to be sold out to creating value!"
WKash
50%
50%
WKash,
User Rank: Author
9/17/2013 | 12:46:57 PM
re: Why Your Software Development Process Is Broken
One other reason it makes sense for developers to get closer to their customers, and not just under the direction of a product manager, is the pain large enterprises face in having to too-frequently patch their software. For the US Marine Corps, as noted in our story, "Software Patches Eat Government's Lunch, " http://www.informationweek.com... , the cost of testing and applying patches is a huge tax on IT operations at large enterprises.
ChrisMurphy
50%
50%
ChrisMurphy,
User Rank: Author
9/16/2013 | 3:17:33 PM
re: Why Your Software Development Process Is Broken
More companies are learning these hard lessons as they do more customer-facing software development. Customer-facing mobile apps, in-vehicle features and touchscreens, B-to-B e-commerce sites -- smartphones and tablets are forcing companies to hone higher-level software development skills than they needed when the "customer" was an employee as compared with now when it's for the end customers who pay the bills.
BrainiacV
50%
50%
BrainiacV,
User Rank: Apprentice
9/16/2013 | 2:37:00 PM
re: Why Your Software Development Process Is Broken
My take on development is that programmers have to interact with end users to the point they become experts, if just for a day, to understand the requirements and then use their knowledge of software systems to transcend the requirements. As Steve Jobs once said, if Henry Ford had asked the customers what they wanted, they would have asked for a faster horse. Many times the requirements that come my way are mired in manual processes. In effect they want to pave the cow path, because it is the only thing they know. Understanding the end result will allow you to take things to the next level that the end user has no concept of. It gets past the incremental requests because the end user didn't know you could do something so they pitch their requests low and you find yourself writing (and rewriting) code that would have been much more efficient if they had asked you for what they wanted in the first place. I tell them to tell me their wildest fantasy, I say we will scale it back to what can be done and in a reasonable amount of time, with an eye towards the future. To use an architectural analogy (I studied to be an architect before being seduced by the dark side of programming), don't tell me you will want a wet bar on the other side of the room AFTER I've laid the concrete floor. Software is plastic, but as systems get built, they start to develop rigidity. I've inherited systems that had to be rewritten because the previous system had developed the rigidity of concrete.
arigney
50%
50%
arigney,
User Rank: Apprentice
9/16/2013 | 12:31:11 PM
re: Why Your Software Development Process Is Broken
In a perfect world you are absolutely correct however really important large software systems are being built by large companies like that have a vested interest in only providing the customer what they say or think what they want rather than evaluating and analyzing and working with them to produce the software that they actually need. Also governments and big clients don't really care who is building their software unless they are big enough to sue. So until agile development standards are accepted or enforced by law we will still see large software projects failing/costing their governments etc a lot more than they should with sub-standard quality and performance.
jemison288
50%
50%
jemison288,
User Rank: Moderator
9/14/2013 | 4:39:05 PM
re: Why Your Software Development Process Is Broken
It is true that I'm saying doesn't address the disagreeing-customers problem. Although I would argue that having a good benevolent dictator would help resolve those issues by having a clear vision of the product and a clear understanding of whether the feature fits with that vision or doesn't (or how to make it fit with that vision).
jemison288
50%
50%
jemison288,
User Rank: Moderator
9/14/2013 | 4:37:48 PM
re: Why Your Software Development Process Is Broken
Doug -- I think that Marty Cagan mainly agrees with this vision that you're discussing; let's have more people involved all along. My concern that is PMs are not technical enough to really manage a software product properly, and that responsibility is too diffuse--there isn't one throat to choke in the bad PM situation that I describe above.
jemison288
50%
50%
jemison288,
User Rank: Moderator
9/14/2013 | 4:36:17 PM
re: Why Your Software Development Process Is Broken
Very good points. I am railing here against a certain vision of product management, and I do not outline a complete solution that takes into account all stakeholders.
Page 1 / 2   >   >>
Register for InformationWeek Newsletters
White Papers
Current Issue
InformationWeek Elite 100 - 2014
Our InformationWeek Elite 100 issue -- our 26th ranking of technology innovators -- shines a spotlight on businesses that are succeeding because of their digital strategies. We take a close at look at the top five companies in this year's ranking and the eight winners of our Business Innovation awards, and offer 20 great ideas that you can use in your company. We also provide a ranked list of our Elite 100 innovators.
Video
Slideshows
Twitter Feed
Audio Interviews
Archived Audio Interviews
GE is a leader in combining connected devices and advanced analytics in pursuit of practical goals like less downtime, lower operating costs, and higher throughput. At GIO Power & Water, CIO Jim Fowler is part of the team exploring how to apply these techniques to some of the world's essential infrastructure, from power plants to water treatment systems. Join us, and bring your questions, as we talk about what's ahead.