Big Government Software Projects: 11 Lessons Learned - InformationWeek
IT Leadership // CIO Insights & Innovation
08:06 AM
Connect Directly
Moving UEBA Beyond the Ground Floor
Sep 20, 2017
This webinar will provide the details you need about UEBA so you can make the decisions on how bes ...Read More>>

Big Government Software Projects: 11 Lessons Learned

Think you have political battles and scope-creep issues? Here's how one IT leader developed a large-scale information system in the bureaucracy of the Department of Veterans Affairs.

6. Find a champion and use the clout as needed
Enterprise-scale software systems by definition cross the fiefdoms of many chieftains, each of whom can kill your small but determined caravan as it travels through. The larger the organization, the smaller and less significant your caravan can appear to each chief. Although it is important to understand the incentives of any chief you depend on, there are some who will only be swayed by superior firepower. For such situations there is no substitute for a champion at the highest levels of the organization. It was not by chance that the first people to give their blood samples and health data to the Million Veteran Program were the highest levels of VA leadership. Neither was it coincidence that several resistant chiefs were reminded of the origins of samples on an as-needed basis. 

7. Write your plan in sand
We were never more confident about what the system would have to do than the day before development began. It went downhill from there as the "business" learned from trial and error how to recruit and enroll enormous numbers of veterans from across the country. This story is familiar to those involved with software development, but it can be blasphemous to those in government project planning.

Most large-scale efforts to build things in government (and elsewhere) require a book-thick collection of specifications and requirements to be signed off in advance of "breaking ground." The whiplash of the dotcom era taught many of us in IT that, although the waterfall method might lead to perfectly capable buildings or fighter jets, the malleable nature of software requires a different kind of planning. As our understanding of the complexity of the project grew and as the business learned just what was possible, our detailed 12-month and even six-month project plans became works of fiction. We learned that three months was about the longest period we could predict deliverables with some confidence. We kept six- and 12-month plans and submitted them as instructed to the appropriate overseers, but shifted them to 20,000- and 40,000-foot views.

Those timelines will vary by project, but the lesson is that in complex projects with novel or unfamiliar business processes, it's essential to use project plans at several levels of granularity and to revisit them often. Note: This is very different from constantly revisiting project goals. If goals change frequently, you've got bigger problems. 

8. Establish one ultimate business owner
An enterprise-scale system means a long list of stakeholders, each with valuable opinions on how your software should work. We were blessed and cursed by an overabundance of talented and devoted stakeholders with opinions that varied, appropriately, with their worldviews and priorities. We dutifully identified and catalogued such priorities. However, the only way we could sustain progress was by establishing one ultimate business owner who could settle disputes and trump others in a timely manner. With some effort we convinced all stakeholders that the MVP project director -- the primary day-to-day business owner -- would be the arbiter of conflicting priorities. We made clear that delays in reaching consensus would delay deliverable software.

Getting everyone to honor the arrangement wasn't a simple task, even when agreed to in theory. Several influential stakeholders in the habit of being listened to continued "suggesting" alternative approaches directly to the development team, resulting in hours-long distractions. Only when it became clear that the development team would only change direction with the MVP project director's consent did the number of direct requests for changes stop. Enforcing this required empowering every member of the development team to politely convey this process to individuals holding higher rank, and standing firmly behind them when they did.

9. Prepare for an evolving software development method
Before this project, I'd already treated software development methods such as agile, waterfall, and rational unified process like traffic signs in Boston -- more suggestion than law. What happened in this project, though, was a full metamorphosis from one process to another.

In the earliest stages of the project, we worked in full-fledge agile mode. We benefited tremendously from being able to yell across the hall to the people ultimately responsible for using the system. They were quick to jump in, debate a finer point of development, challenge assumptions, and hash out problems in a hurry. We did rapid prototyping in a short amount of time.

As the project team grew, the project touched more systems and the demands for deliverables increased from above. The emphasis shifted from "how might this work?" to "make sure it works." The interrelated nature of the systems meant that a change to the mail routine now affected the site coordinator application, which affected our reporting system, and so on. We needed to add structure and a greater emphasis on quality assurance and testing.

Where frequent conversations with MVP operations partners were an enormous asset in the earliest stages, now such conversations became a liability. I actually had to curtail unannounced visits by MVP operations and politely ask this same group of people that had been working so successfully with us in agile mode for months to visit less frequently. We went from two-week coding sprints between releases to six- to eight-week runs, and we formalized requirements-gathering sessions. The close relationship the combined team had fostered in our agile days probably lessened the hurt and suspicion as I chased people away from our developers (sometimes literally). Looking back, the ability to prepare for such a shift in advance would have been helpful.

10. Let the business dictate priorities and developers explain the consequences
The software team naturally develops strong opinions on project priorities after months of automating businesses processes. I learned, however, that expressing those opinions creates a slippery slope. Because business owners have limited experience with software development, they find it frustrating that seemingly simple changes can't be made the way they once were to paper-based processes. Why does a simple request like changing the order of mailings cause a two-week delay in deployment? Amid such frustration, any perception that the software team is dictating business priorities pours fuel on that fire.

If we came to the opinion while reviewing priorities that option A was a higher priority than option B, we learned not to advocate for one or the other. Instead we found it far more effective to outline the effects of one choice versus another from as unbiased a position as possible. This seemingly subtle discrimination made a world of difference.

11. Protect your project from outsiders until it's ready to leave the nest
Our project's survival, in no uncertain terms, depended on reaching 100,000 enrolled within one year of deployment. With that ultimatum in mind we treated everything else as a threat to our existence.

Survival meant we had to resist efforts by others, even within our own organization, to subsume, host, develop, or partner with our program until we had a running, working system. It was a double-edged sword, because many of the people suggesting expanding our scope were great supporters and in a position to advance the team's agenda. Their suggestions were exactly the type of work we really did want to be involved with -- such as becoming the platform for recruitment and enrollment across the VA, or integrating with existing, patient-facing, web-based platforms. How could we protect the current effort while not alienating the group that would offer future opportunities?

At first, I tried to explain the downstream effects of taking on a new element, the time it would take to expand the data model, how slightly divergent business processes would increase development and testing time, and so on. It sounded like IT jargon. I needed a better way to protect the project's timelines while not burning bridges.

Remember when I said that not dictating priorities made a world of difference? Rather than reject any proposal to join forces, switch hosts, or incorporate new goals, I learned to express enthusiasm -- which was easy because they really were projects I wanted to take on, at some point. Then I would explain how the new plans could delay our 100,000-enrollment goal. Unfortunately, I was sure to relay, decisions to change project goals and deadlines felt above my pay grade, but with permission we could make it happen. Not surprisingly, our priorities remained unchanged as our 100,000-enrollment goal came from the top.

Today, as MVP recruits the next 200,000 enrollees and the first samples get queued up for processing, MAVERIC is transitioning the hosting of the MVP recruitment and enrollment systems to the VA's Office of Information Technology. There, it will be dressed with the proper fittings for a production system of its age. Like a nervous parent I'll watch from a distance, hoping we raised it right and trusting that that's enough. I'll be forever grateful for the opportunity and thankful that we were able to avoid making the wrong kind of headlines. But frankly, I'm pretty excited about becoming an empty nester.

A most sincere thanks to visionary, mentor, and friend Dr. Louis Fiore and the MAVERIC team for making so many interesting projects possible.The views expressed in this article are the author's and not those of the Department of Veterans Affairs.

2 of 2
Comment  | 
Print  | 
More Insights
Newest First  |  Oldest First  |  Threaded View
User Rank: Apprentice
11/24/2013 | 5:08:51 PM
Re: Hard Earned Insight
yump. Gather requirements and go underground until done. While it smacks of waterfall, it's necessary to freeze, at some point, and, if you got it right, let the developers deliver. I used this time manage development and to go around and convince various stake holders that what we were doing was their idea ("Gee, I'm so sorry I argued with you, you were right." They had forgotten everything but the argument, because it was all ego.). Using agile builds trust and is a good way to gather requirements, but on a really complex project, you have to freeze and deliver. Managing expectations is what it is all about during that phase.
User Rank: Author
11/19/2013 | 7:02:55 PM
Valuable points
There are some truly valuable lessons here, but especially your point about establishing one ultimate business owner, so stakeholders can't undermine the project; and your point about letting the business dictate the priorities and developers explain the consquences.  This is the ultimate frustration for many end users is a system designed by developers instead of designed for the business users. Thanks for sharing these.
User Rank: Apprentice
11/19/2013 | 3:21:20 PM
Re: Public v. Private
Thanks for the thoughtful question.  I'd love to learn from others whether such outreach would fly in publically traded companies.  That said, most academics live inside of a "publish or perish" paradigm which, despite the altrusitic nature of medical research, can lead to pretty competitive environment.  No surprise here - whether in business, govt, or academia it helps to be referred.  We asked "who else should we be talking to?" and "would you mind if we said you referred us?" It went a long way.
User Rank: Author
11/19/2013 | 10:32:37 AM
Re: Hard Earned Insight
The point Chris raises is something we practiced with the relaunch of on this new platform:
We drew in many business-unit ideas and contributions early on in the project, but at some point the project manager (kudos to Joe Wyka) narrowed the process to a few select stakeholders. We still have lots more to do and tweak in version 2.0 of this project, but we didn't let a succession of proposed incremental improvements slow down the launch. 
Alison Diana
Alison Diana,
User Rank: Moderator
11/19/2013 | 10:08:04 AM
Public v. Private
I wonder what, if any, differences there would be on a list compiled by someone at a similar position in a public company. For example, calling people to learn about "what they'd have done differently" might be a lot more challenging if you work in the private sector, where competitors certainly won't want to share their valuable (and competitive) information! Although I'd guess you can find similar size companies in different industries to discuss implemenations...
User Rank: Author
11/19/2013 | 9:33:01 AM
Hard Earned Insight
This is must-read advice for anyone who leads projects. What I found really valuable: The advice that at 1 stage of a project you need to draw in lots of business-unit ideas and contributions, but then later shut those same people out so the team can deliver. Subtle, powerful insight.
How Enterprises Are Attacking the IT Security Enterprise
How Enterprises Are Attacking the IT Security Enterprise
To learn more about what organizations are doing to tackle attacks and threats we surveyed a group of 300 IT and infosec professionals to find out what their biggest IT security challenges are and what they're doing to defend against today's threats. Download the report to see what they're saying.
Register for InformationWeek Newsletters
White Papers
Current Issue
IT Strategies to Conquer the Cloud
Chances are your organization is adopting cloud computing in one way or another -- or in multiple ways. Understanding the skills you need and how cloud affects IT operations and networking will help you adapt.
Twitter Feed
Sponsored Live Streaming Video
Everything You've Been Told About Mobility Is Wrong
Attend this video symposium with Sean Wisdom, Global Director of Mobility Solutions, and learn about how you can harness powerful new products to mobilize your business potential.
Flash Poll