Big Government Software Projects: 11 Lessons Learned

6. Find a champion and use the clout as needed
Enterprise-scale software systems by definition cross the fiefdoms of many chieftains, each of whom can kill your small but determined caravan as it travels through. The larger the organization, the smaller and less significant your caravan can appear to each chief. Although it is important to understand the incentives of any chief you depend on, there are some who will only be swayed by superior firepower. For such situations there is no substitute for a champion at the highest levels of the organization. It was not by chance that the first people to give their blood samples and health data to the Million Veteran Program were the highest levels of VA leadership. Neither was it coincidence that several resistant chiefs were reminded of the origins of samples on an as-needed basis. 

7. Write your plan in sand
We were never more confident about what the system would have to do than the day before development began. It went downhill from there as the "business" learned from trial and error how to recruit and enroll enormous numbers of veterans from across the country. This story is familiar to those involved with software development, but it can be blasphemous to those in government project planning.

Most large-scale efforts to build things in government (and elsewhere) require a book-thick collection of specifications and requirements to be signed off in advance of "breaking ground." The whiplash of the dotcom era taught many of us in IT that, although the waterfall method might lead to perfectly capable buildings or fighter jets, the malleable nature of software requires a different kind of planning. As our understanding of the complexity of the project grew and as the business learned just what was possible, our detailed 12-month and even six-month project plans became works of fiction. We learned that three months was about the longest period we could predict deliverables with some confidence. We kept six- and 12-month plans and submitted them as instructed to the appropriate overseers, but shifted them to 20,000- and 40,000-foot views.

Those timelines will vary by project, but the lesson is that in complex projects with novel or unfamiliar business processes, it's essential to use project plans at several levels of granularity and to revisit them often. Note: This is very different from constantly revisiting project goals. If goals change frequently, you've got bigger problems. 

8. Establish one ultimate business owner
An enterprise-scale system means a long list of stakeholders, each with valuable opinions on how your software should work. We were blessed and cursed by an overabundance of talented and devoted stakeholders with opinions that varied, appropriately, with their worldviews and priorities. We dutifully identified and catalogued such priorities. However, the only way we could sustain progress was by establishing one ultimate business owner who could settle disputes and trump others in a timely manner. With some effort we convinced all stakeholders that the MVP project director -- the primary day-to-day business owner -- would be the arbiter of conflicting priorities. We made clear that delays in reaching consensus would delay deliverable software.

Getting everyone to honor the arrangement wasn't a simple task, even when agreed to in theory. Several influential stakeholders in the habit of being listened to continued "suggesting" alternative approaches directly to the development team, resulting in hours-long distractions. Only when it became clear that the development team would only change direction with the MVP project director's consent did the number of direct requests for changes stop. Enforcing this required empowering every member of the development team to politely convey this process to individuals holding higher rank, and standing firmly behind them when they did.

9. Prepare for an evolving software development method
Before this project, I'd already treated software development methods such as agile, waterfall, and rational unified process like traffic signs in Boston -- more suggestion than law. What happened in this project, though, was a full metamorphosis from one process to another.

In the earliest stages of the project, we worked in full-fledge agile mode. We benefited tremendously from being able to yell across the hall to the people ultimately responsible for using the system. They were quick to jump in, debate a finer point of development, challenge assumptions, and hash out problems in a hurry. We did rapid prototyping in a short amount of time.

As the project team grew, the project touched more systems and the demands for deliverables increased from above. The emphasis shifted from "how might this work?" to "make sure it works." The interrelated nature of the systems meant that a change to the mail routine now affected the site coordinator application, which affected our reporting system, and so on. We needed to add structure and a greater emphasis on quality assurance and testing.

Where frequent conversations with MVP operations partners were an enormous asset in the earliest stages, now such conversations became a liability. I actually had to curtail unannounced visits by MVP operations and politely ask this same group of people that had been working so successfully with us in agile mode for months to visit less frequently. We went from two-week coding sprints between releases to six- to eight-week runs, and we formalized requirements-gathering sessions. The close relationship the combined team had fostered in our agile days probably lessened the hurt and suspicion as I chased people away from our developers (sometimes literally). Looking back, the ability to prepare for such a shift in advance would have been helpful.

10. Let the business dictate priorities and developers explain the consequences
The software team naturally develops strong opinions on project priorities after months of automating businesses processes. I learned, however, that expressing those opinions creates a slippery slope. Because business owners have limited experience with software development, they find it frustrating that seemingly simple changes can't be made the way they once were to paper-based processes. Why does a simple request like changing the order of mailings cause a two-week delay in deployment? Amid such frustration, any perception that the software team is dictating business priorities pours fuel on that fire.

If we came to the opinion while reviewing priorities that option A was a higher priority than option B, we learned not to advocate for one or the other. Instead we found it far more effective to outline the effects of one choice versus another from as unbiased a position as possible. This seemingly subtle discrimination made a world of difference.

11. Protect your project from outsiders until it's ready to leave the nest
Our project's survival, in no uncertain terms, depended on reaching 100,000 enrolled within one year of deployment. With that ultimatum in mind we treated everything else as a threat to our existence.

Survival meant we had to resist efforts by others, even within our own organization, to subsume, host, develop, or partner with our program until we had a running, working system. It was a double-edged sword, because many of the people suggesting expanding our scope were great supporters and in a position to advance the team's agenda. Their suggestions were exactly the type of work we really did want to be involved with -- such as becoming the platform for recruitment and enrollment across the VA, or integrating with existing, patient-facing, web-based platforms. How could we protect the current effort while not alienating the group that would offer future opportunities?

At first, I tried to explain the downstream effects of taking on a new element, the time it would take to expand the data model, how slightly divergent business processes would increase development and testing time, and so on. It sounded like IT jargon. I needed a better way to protect the project's timelines while not burning bridges.

Remember when I said that not dictating priorities made a world of difference? Rather than reject any proposal to join forces, switch hosts, or incorporate new goals, I learned to express enthusiasm -- which was easy because they really were projects I wanted to take on, at some point. Then I would explain how the new plans could delay our 100,000-enrollment goal. Unfortunately, I was sure to relay, decisions to change project goals and deadlines felt above my pay grade, but with permission we could make it happen. Not surprisingly, our priorities remained unchanged as our 100,000-enrollment goal came from the top.

Today, as MVP recruits the next 200,000 enrollees and the first samples get queued up for processing, MAVERIC is transitioning the hosting of the MVP recruitment and enrollment systems to the VA's Office of Information Technology. There, it will be dressed with the proper fittings for a production system of its age. Like a nervous parent I'll watch from a distance, hoping we raised it right and trusting that that's enough. I'll be forever grateful for the opportunity and thankful that we were able to avoid making the wrong kind of headlines. But frankly, I'm pretty excited about becoming an empty nester.

A most sincere thanks to visionary, mentor, and friend Dr. Louis Fiore and the MAVERIC team for making so many interesting projects possible.The views expressed in this article are the author's and not those of the Department of Veterans Affairs.