Big Government Software Projects: 11 Lessons Learned - InformationWeek

InformationWeek is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

IT Leadership // CIO Insights & Innovation
08:06 AM
Connect Directly

Big Government Software Projects: 11 Lessons Learned

Think you have political battles and scope-creep issues? Here's how one IT leader developed a large-scale information system in the bureaucracy of the Department of Veterans Affairs.

The technical difficulties surrounding the launch of might eventually prove to be one of the most consequential failed government IT rollouts of our time. It joins a long line of large-scale IT failure stories. Yet despite the finger pointing and political circus surrounding the launch, government will continue to require new information systems in order to fulfill its obligation to provide services to its citizens. We need to learn from these megaprojects.

For the last five years I had the great fortune of leading the information technology portion of what has been called one of the "5 Most Innovative Government Big Data Projects." The Department of Veterans Affairs' Million Veteran Program (MVP) set out to enroll and gather the health and biological information of 1 million veterans to facilitate faster and more cost-effective discovery of the relationships between human genes and our health. To make this possible we developed several integrated applications that pull from multiple systems across the VA to manage the mail, calls, surveys, schedules, samples, and logistics across several departments within this giant government bureaucracy.  

Today, MVP systems have helped enroll more than 200,000 veterans, and I'm personally off to a new set of challenges, making this a good time to reflect back on lessons learned. Some of what follows are descriptions of what we did right. Most, however, are lessons learned the hard way. Of course, no two IT projects are the same, and what we did isn't directly comparably to However it's my hope that some of my experiences in attempting to garner resources, build teams, identify barriers, and negotiate solutions across institutional silos will be useful to people involved in large-scale system development and implementation.

The need for more data
First, here's a little about the project to give you some context. Our DNA affects everything from how we process nutrients to how we react to illness and process drugs. But which of our 30,000 genes make us more likely to suffer from, say, alcoholism? Figuring out the patterns that matter requires the participation of enormous numbers of patients -- a factor that has limited scientific discovery. When the Honorable Eric Shinseki was appointed Secretary of Veterans Affairs by President Obama, one of the "Transformative Initiatives" he invested in was the Million Veteran Program, meant to address the shortage of DNA for research and to ensure our veterans receive the cutting-edge care that can come from genomic discovery.

With a core of nine to 12 people we developed a series of applications that on a weekly basis send out 20,000 pieces of mail, process approximately 2,000 blood samples and surveys, field 2,000 calls in a call center, and manage consent forms from 40 VA hospitals. This project will only truly be considered a success when it leads to better healthcare for veterans. That said, MVP's recruitment and enrollment systems are working, and the first genomic samples are being shipped off for processing. 

Here are 11 lessons we learned along the way.

1. Chance favors the prepared
The organization that did this work, the Massachusetts Veterans Epidemiology Research & Information Center, or MAVERIC, is a multi-disciplinary research-and-development organization with experience conducting large-scale research projects for the VA. When Secretary Shinseki declared his intent to transform the VA, we had already presented a whitepaper and held conversations with research leaders on the importance of capturing and making genomic data reusable. We invested the time of a small team to develop prototypes to hash out critical details before any real dollars were approved to develop the infrastructure for MVP. By investing a small amount of resources in the potential "next big thing" for your organization, the worst thing that can happen is that your team is engaged and sharpening its skills. If you guess right, you've got a head start when leaders look for projects to fund. With luck, you get to help set the agenda. 

2. Talk to people who've done it
Rather than just rely on vendors, whitepapers, and a key contact or two in the early research stages, we did an all-out cold-call campaign to everyone we could find with experience on similar projects. We identified some 15 organizations and asked about the scope, the resources required, how they defined success, and what they planned to accomplish in the next six months. A great question, we learned, was "What would you do differently if you had it to do over?" This might seem intrusive or even inappropriate, but we found professionals quite happy to share and grateful to have new contacts doing similar work. In several cases we were hosted for half-day or all-day visits.

The research helped in three ways. It previewed the technical and political challenges we would face. It gave us confidence in our early estimates of timelines and required resources. And it provided contacts for future advice. And we also realized that, when it came to building a system to enroll enormous cohorts, no one had all the answers. This made us more comfortable hacking through the inevitable problems as they arose. 

3. Hire great people…
Hiring highly motivated and talented individuals is the single biggest reason we made good progress. The best hires shared one trait: All were very strongly attracted to the mission. The chance to build information systems that can improve lives is rare and important enough that people even found it worth accepting a government salary and the inevitable red tape that comes with government work. 

4. …and remove people who aren't great, quickly
There will be people who seem terrific on paper and even during interviews but for whatever reason don't work out. It's very tempting to hope that things will change or to ignore the issue altogether. Instead, after you've taken the right steps to rectify the situation, these individuals should be removed as soon as it becomes clear to you and key team members that things aren't working out. This was almost unheard of in the government organization I worked for, and I wasn't excited to break from tradition. However, sticking with a poor fit longer than absolutely necessary has a reverberating effect. A results-oriented team quickly realizes when one individual is hurting its combined efforts and grows resentful.

5. Insource first, outsource selectively
The nature of our project required developing and releasing several different applications in rapid succession. My initial plan was to charge my in-house team, with their close proximity to the business, to the most complex and time-sensitive modules. I assigned development of smaller but critical modules to consulting firms to develop in parallel. This turned out to be a mistake. After several disappointing results it became clear that the blame belonged not with our external partners (many of whom were excellent) but to the structure I had chosen to accomplish the work.

I am now firmly in the camp of Fred Brooks and his "mythical man month" idea: In a nutshell, nine women cannot make a baby in one month by each carrying in parallel. My initial setup meant that our in-house team was completing its modules and then correcting the shortcomings of contractor deliverables. The project in its early stages was simply too complex and amorphous to be successfully parallelized. Instead, we shifted to a model where development of every module was led by in-house developers, with contractors used to supplement our in-house team. We brought project management in-house and instead contracted for embedded, onsite developers, analysts, and QA people -- whose work was overseen by the in-house team. This change improved communication and teamwork, and it let us quickly spot sub-par contractors and replace them with high performers. This oversight is especially important in government where the decision of which vendor to partner with is often determined by contracting offices, not the person responsible for delivering the project. After the change, the quality and quantity of deliverables increased substantially.

We welcome your comments on this topic on our social media channels, or [contact us directly] with questions about the site.
1 of 2
Comment  | 
Print  | 
More Insights
Newest First  |  Oldest First  |  Threaded View
User Rank: Apprentice
11/24/2013 | 5:08:51 PM
Re: Hard Earned Insight
yump. Gather requirements and go underground until done. While it smacks of waterfall, it's necessary to freeze, at some point, and, if you got it right, let the developers deliver. I used this time manage development and to go around and convince various stake holders that what we were doing was their idea ("Gee, I'm so sorry I argued with you, you were right." They had forgotten everything but the argument, because it was all ego.). Using agile builds trust and is a good way to gather requirements, but on a really complex project, you have to freeze and deliver. Managing expectations is what it is all about during that phase.
User Rank: Author
11/19/2013 | 7:02:55 PM
Valuable points
There are some truly valuable lessons here, but especially your point about establishing one ultimate business owner, so stakeholders can't undermine the project; and your point about letting the business dictate the priorities and developers explain the consquences.  This is the ultimate frustration for many end users is a system designed by developers instead of designed for the business users. Thanks for sharing these.
User Rank: Apprentice
11/19/2013 | 3:21:20 PM
Re: Public v. Private
Thanks for the thoughtful question.  I'd love to learn from others whether such outreach would fly in publically traded companies.  That said, most academics live inside of a "publish or perish" paradigm which, despite the altrusitic nature of medical research, can lead to pretty competitive environment.  No surprise here - whether in business, govt, or academia it helps to be referred.  We asked "who else should we be talking to?" and "would you mind if we said you referred us?" It went a long way.
User Rank: Author
11/19/2013 | 10:32:37 AM
Re: Hard Earned Insight
The point Chris raises is something we practiced with the relaunch of on this new platform:
We drew in many business-unit ideas and contributions early on in the project, but at some point the project manager (kudos to Joe Wyka) narrowed the process to a few select stakeholders. We still have lots more to do and tweak in version 2.0 of this project, but we didn't let a succession of proposed incremental improvements slow down the launch. 
Alison Diana
Alison Diana,
User Rank: Moderator
11/19/2013 | 10:08:04 AM
Public v. Private
I wonder what, if any, differences there would be on a list compiled by someone at a similar position in a public company. For example, calling people to learn about "what they'd have done differently" might be a lot more challenging if you work in the private sector, where competitors certainly won't want to share their valuable (and competitive) information! Although I'd guess you can find similar size companies in different industries to discuss implemenations...
User Rank: Author
11/19/2013 | 9:33:01 AM
Hard Earned Insight
This is must-read advice for anyone who leads projects. What I found really valuable: The advice that at 1 stage of a project you need to draw in lots of business-unit ideas and contributions, but then later shut those same people out so the team can deliver. Subtle, powerful insight.
AI Regulation: Has the Time Arrived?
John Edwards, Technology Journalist & Author,  2/24/2020
Fighting the Coronavirus with Analytics and GIS
Jessica Davis, Senior Editor, Enterprise Apps,  2/3/2020
IT Careers: 10 Job Skills in High Demand This Year
Cynthia Harvey, Freelance Journalist, InformationWeek,  2/3/2020
White Papers
Register for InformationWeek Newsletters
Current Issue
IT Careers: Tech Drives Constant Change
Advances in information technology and management concepts mean that IT professionals must update their skill sets, even their career goals on an almost yearly basis. In this IT Trend Report, experts share advice on how IT pros can keep up with this every-changing job market. Read it today!
Flash Poll