Big Government Software Projects: 11 Lessons Learned - InformationWeek

InformationWeek is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

IT Leadership // CIO Insights & Innovation
08:06 AM
Connect Directly

Big Government Software Projects: 11 Lessons Learned

Think you have political battles and scope-creep issues? Here's how one IT leader developed a large-scale information system in the bureaucracy of the Department of Veterans Affairs.

The technical difficulties surrounding the launch of might eventually prove to be one of the most consequential failed government IT rollouts of our time. It joins a long line of large-scale IT failure stories. Yet despite the finger pointing and political circus surrounding the launch, government will continue to require new information systems in order to fulfill its obligation to provide services to its citizens. We need to learn from these megaprojects.

For the last five years I had the great fortune of leading the information technology portion of what has been called one of the "5 Most Innovative Government Big Data Projects." The Department of Veterans Affairs' Million Veteran Program (MVP) set out to enroll and gather the health and biological information of 1 million veterans to facilitate faster and more cost-effective discovery of the relationships between human genes and our health. To make this possible we developed several integrated applications that pull from multiple systems across the VA to manage the mail, calls, surveys, schedules, samples, and logistics across several departments within this giant government bureaucracy.  

Today, MVP systems have helped enroll more than 200,000 veterans, and I'm personally off to a new set of challenges, making this a good time to reflect back on lessons learned. Some of what follows are descriptions of what we did right. Most, however, are lessons learned the hard way. Of course, no two IT projects are the same, and what we did isn't directly comparably to However it's my hope that some of my experiences in attempting to garner resources, build teams, identify barriers, and negotiate solutions across institutional silos will be useful to people involved in large-scale system development and implementation.

The need for more data
First, here's a little about the project to give you some context. Our DNA affects everything from how we process nutrients to how we react to illness and process drugs. But which of our 30,000 genes make us more likely to suffer from, say, alcoholism? Figuring out the patterns that matter requires the participation of enormous numbers of patients -- a factor that has limited scientific discovery. When the Honorable Eric Shinseki was appointed Secretary of Veterans Affairs by President Obama, one of the "Transformative Initiatives" he invested in was the Million Veteran Program, meant to address the shortage of DNA for research and to ensure our veterans receive the cutting-edge care that can come from genomic discovery.

With a core of nine to 12 people we developed a series of applications that on a weekly basis send out 20,000 pieces of mail, process approximately 2,000 blood samples and surveys, field 2,000 calls in a call center, and manage consent forms from 40 VA hospitals. This project will only truly be considered a success when it leads to better healthcare for veterans. That said, MVP's recruitment and enrollment systems are working, and the first genomic samples are being shipped off for processing. 

Here are 11 lessons we learned along the way.

1. Chance favors the prepared
The organization that did this work, the Massachusetts Veterans Epidemiology Research & Information Center, or MAVERIC, is a multi-disciplinary research-and-development organization with experience conducting large-scale research projects for the VA. When Secretary Shinseki declared his intent to transform the VA, we had already presented a whitepaper and held conversations with research leaders on the importance of capturing and making genomic data reusable. We invested the time of a small team to develop prototypes to hash out critical details before any real dollars were approved to develop the infrastructure for MVP. By investing a small amount of resources in the potential "next big thing" for your organization, the worst thing that can happen is that your team is engaged and sharpening its skills. If you guess right, you've got a head start when leaders look for projects to fund. With luck, you get to help set the agenda. 

2. Talk to people who've done it
Rather than just rely on vendors, whitepapers, and a key contact or two in the early research stages, we did an all-out cold-call campaign to everyone we could find with experience on similar projects. We identified some 15 organizations and asked about the scope, the resources required, how they defined success, and what they planned to accomplish in the next six months. A great question, we learned, was "What would you do differently if you had it to do over?" This might seem intrusive or even inappropriate, but we found professionals quite happy to share and grateful to have new contacts doing similar work. In several cases we were hosted for half-day or all-day visits.

The research helped in three ways. It previewed the technical and political challenges we would face. It gave us confidence in our early estimates of timelines and required resources. And it provided contacts for future advice. And we also realized that, when it came to building a system to enroll enormous cohorts, no one had all the answers. This made us more comfortable hacking through the inevitable problems as they arose. 

3. Hire great people…
Hiring highly motivated and talented individuals is the single biggest reason we made good progress. The best hires shared one trait: All were very strongly attracted to the mission. The chance to build information systems that can improve lives is rare and important enough that people even found it worth accepting a government salary and the inevitable red tape that comes with government work. 

4. …and remove people who aren't great, quickly
There will be people who seem terrific on paper and even during interviews but for whatever reason don't work out. It's very tempting to hope that things will change or to ignore the issue altogether. Instead, after you've taken the right steps to rectify the situation, these individuals should be removed as soon as it becomes clear to you and key team members that things aren't working out. This was almost unheard of in the government organization I worked for, and I wasn't excited to break from tradition. However, sticking with a poor fit longer than absolutely necessary has a reverberating effect. A results-oriented team quickly realizes when one individual is hurting its combined efforts and grows resentful.

5. Insource first, outsource selectively
The nature of our project required developing and releasing several different applications in rapid succession. My initial plan was to charge my in-house team, with their close proximity to the business, to the most complex and time-sensitive modules. I assigned development of smaller but critical modules to consulting firms to develop in parallel. This turned out to be a mistake. After several disappointing results it became clear that the blame belonged not with our external partners (many of whom were excellent) but to the structure I had chosen to accomplish the work.

I am now firmly in the camp of Fred Brooks and his "mythical man month" idea: In a nutshell, nine women cannot make a baby in one month by each carrying in parallel. My initial setup meant that our in-house team was completing its modules and then correcting the shortcomings of contractor deliverables. The project in its early stages was simply too complex and amorphous to be successfully parallelized. Instead, we shifted to a model where development of every module was led by in-house developers, with contractors used to supplement our in-house team. We brought project management in-house and instead contracted for embedded, onsite developers, analysts, and QA people -- whose work was overseen by the in-house team. This change improved communication and teamwork, and it let us quickly spot sub-par contractors and replace them with high performers. This oversight is especially important in government where the decision of which vendor to partner with is often determined by contracting offices, not the person responsible for delivering the project. After the change, the quality and quantity of deliverables increased substantially.

We welcome your comments on this topic on our social media channels, or [contact us directly] with questions about the site.
1 of 2
Comment  | 
Print  | 
More Insights
Newest First  |  Oldest First  |  Threaded View
User Rank: Author
11/19/2013 | 7:02:55 PM
Valuable points
There are some truly valuable lessons here, but especially your point about establishing one ultimate business owner, so stakeholders can't undermine the project; and your point about letting the business dictate the priorities and developers explain the consquences.  This is the ultimate frustration for many end users is a system designed by developers instead of designed for the business users. Thanks for sharing these.
User Rank: Author
11/19/2013 | 10:32:37 AM
Re: Hard Earned Insight
The point Chris raises is something we practiced with the relaunch of on this new platform:
We drew in many business-unit ideas and contributions early on in the project, but at some point the project manager (kudos to Joe Wyka) narrowed the process to a few select stakeholders. We still have lots more to do and tweak in version 2.0 of this project, but we didn't let a succession of proposed incremental improvements slow down the launch. 
User Rank: Author
11/19/2013 | 9:33:01 AM
Hard Earned Insight
This is must-read advice for anyone who leads projects. What I found really valuable: The advice that at 1 stage of a project you need to draw in lots of business-unit ideas and contributions, but then later shut those same people out so the team can deliver. Subtle, powerful insight.
InformationWeek Is Getting an Upgrade!

Find out more about our plans to improve the look, functionality, and performance of the InformationWeek site in the coming months.

Becoming a Self-Taught Cybersecurity Pro
Jessica Davis, Senior Editor, Enterprise Apps,  6/9/2021
Ancestry's DevOps Strategy to Control Its CI/CD Pipeline
Joao-Pierre S. Ruth, Senior Writer,  6/4/2021
IT Leadership: 10 Ways to Unleash Enterprise Innovation
Lisa Morgan, Freelance Writer,  6/8/2021
White Papers
Register for InformationWeek Newsletters
Current Issue
Planning Your Digital Transformation Roadmap
Download this report to learn about the latest technologies and best practices or ensuring a successful transition from outdated business transformation tactics.
Flash Poll