Big Government Software Projects: 11 Lessons Learned
Think you have political battles and scope-creep issues? Here's how one IT leader developed a large-scale information system in the bureaucracy of the Department of Veterans Affairs.
The technical difficulties surrounding the launch of Healthcare.gov might eventually prove to be one of the most consequential failed government IT rollouts of our time. It joins a long line of large-scale IT failure stories. Yet despite the finger pointing and political circus surrounding the launch, government will continue to require new information systems in order to fulfill its obligation to provide services to its citizens. We need to learn from these megaprojects.
For the last five years I had the great fortune of leading the information technology portion of what has been called one of the "5 Most Innovative Government Big Data Projects." The Department of Veterans Affairs' Million Veteran Program (MVP) set out to enroll and gather the health and biological information of 1 million veterans to facilitate faster and more cost-effective discovery of the relationships between human genes and our health. To make this possible we developed several integrated applications that pull from multiple systems across the VA to manage the mail, calls, surveys, schedules, samples, and logistics across several departments within this giant government bureaucracy.
Today, MVP systems have helped enroll more than 200,000 veterans, and I'm personally off to a new set of challenges, making this a good time to reflect back on lessons learned. Some of what follows are descriptions of what we did right. Most, however, are lessons learned the hard way. Of course, no two IT projects are the same, and what we did isn't directly comparably to Healthcare.gov. However it's my hope that some of my experiences in attempting to garner resources, build teams, identify barriers, and negotiate solutions across institutional silos will be useful to people involved in large-scale system development and implementation.
The need for more data First, here's a little about the project to give you some context. Our DNA affects everything from how we process nutrients to how we react to illness and process drugs. But which of our 30,000 genes make us more likely to suffer from, say, alcoholism? Figuring out the patterns that matter requires the participation of enormous numbers of patients -- a factor that has limited scientific discovery. When the Honorable Eric Shinseki was appointed Secretary of Veterans Affairs by President Obama, one of the "Transformative Initiatives" he invested in was the Million Veteran Program, meant to address the shortage of DNA for research and to ensure our veterans receive the cutting-edge care that can come from genomic discovery.
With a core of nine to 12 people we developed a series of applications that on a weekly basis send out 20,000 pieces of mail, process approximately 2,000 blood samples and surveys, field 2,000 calls in a call center, and manage consent forms from 40 VA hospitals. This project will only truly be considered a success when it leads to better healthcare for veterans. That said, MVP's recruitment and enrollment systems are working, and the first genomic samples are being shipped off for processing.
Here are 11 lessons we learned along the way.
1. Chance favors the prepared The organization that did this work, the Massachusetts Veterans Epidemiology Research & Information Center, or MAVERIC, is a multi-disciplinary research-and-development organization with experience conducting large-scale research projects for the VA. When Secretary Shinseki declared his intent to transform the VA, we had already presented a whitepaper and held conversations with research leaders on the importance of capturing and making genomic data reusable. We invested the time of a small team to develop prototypes to hash out critical details before any real dollars were approved to develop the infrastructure for MVP. By investing a small amount of resources in the potential "next big thing" for your organization, the worst thing that can happen is that your team is engaged and sharpening its skills. If you guess right, you've got a head start when leaders look for projects to fund. With luck, you get to help set the agenda.
2. Talk to people who've done it Rather than just rely on vendors, whitepapers, and a key contact or two in the early research stages, we did an all-out cold-call campaign to everyone we could find with experience on similar projects. We identified some 15 organizations and asked about the scope, the resources required, how they defined success, and what they planned to accomplish in the next six months. A great question, we learned, was "What would you do differently if you had it to do over?" This might seem intrusive or even inappropriate, but we found professionals quite happy to share and grateful to have new contacts doing similar work. In several cases we were hosted for half-day or all-day visits.
The research helped in three ways. It previewed the technical and political challenges we would face. It gave us confidence in our early estimates of timelines and required resources. And it provided contacts for future advice. And we also realized that, when it came to building a system to enroll enormous cohorts, no one had all the answers. This made us more comfortable hacking through the inevitable problems as they arose.
3. Hire great people… Hiring highly motivated and talented individuals is the single biggest reason we made good progress. The best hires shared one trait: All were very strongly attracted to the mission. The chance to build information systems that can improve lives is rare and important enough that people even found it worth accepting a government salary and the inevitable red tape that comes with government work.
4. …and remove people who aren't great, quickly There will be people who seem terrific on paper and even during interviews but for whatever reason don't work out. It's very tempting to hope that things will change or to ignore the issue altogether. Instead, after you've taken the right steps to rectify the situation, these individuals should be removed as soon as it becomes clear to you and key team members that things aren't working out. This was almost unheard of in the government organization I worked for, and I wasn't excited to break from tradition. However, sticking with a poor fit longer than absolutely necessary has a reverberating effect. A results-oriented team quickly realizes when one individual is hurting its combined efforts and grows resentful.
5. Insource first, outsource selectively The nature of our project required developing and releasing several different applications in rapid succession. My initial plan was to charge my in-house team, with their close proximity to the business, to the most complex and time-sensitive modules. I assigned development of smaller but critical modules to consulting firms to develop in parallel. This turned out to be a mistake. After several disappointing results it became clear that the blame belonged not with our external partners (many of whom were excellent) but to the structure I had chosen to accomplish the work.
I am now firmly in the camp of Fred Brooks and his "mythical man month" idea: In a nutshell, nine women cannot make a baby in one month by each carrying in parallel. My initial setup meant that our in-house team was completing its modules and then correcting the shortcomings of contractor deliverables. The project in its early stages was simply too complex and amorphous to be successfully parallelized. Instead, we shifted to a model where development of every module was led by in-house developers, with contractors used to supplement our in-house team. We brought project management in-house and instead contracted for embedded, onsite developers, analysts, and QA people -- whose work was overseen by the in-house team. This change improved communication and teamwork, and it let us quickly spot sub-par contractors and replace them with high performers. This oversight is especially important in government where the decision of which vendor to partner with is often determined by contracting offices, not the person responsible for delivering the project. After the change, the quality and quantity of deliverables increased substantially.
How Enterprises Are Attacking the IT Security EnterpriseTo learn more about what organizations are doing to tackle attacks and threats we surveyed a group of 300 IT and infosec professionals to find out what their biggest IT security challenges are and what they're doing to defend against today's threats. Download the report to see what they're saying.
IT Strategies to Conquer the CloudChances are your organization is adopting cloud computing in one way or another -- or in multiple ways. Understanding the skills you need and how cloud affects IT operations and networking will help you adapt.