Report Blasts Problem-Plagued Cancer Research Grid

The National Cancer Institute is reevaluating an expansive computing grid used for cancer research, following an assessment by its board of scientific advisers that the project is riddled with problems. NCI has invested more than $350 million in the project, called caBIG, including nearly $200 million during the past two years.

CaBIG, which stands for the Cancer Biomedical Informatics Grid, grew out of a pilot project launched in 2004 whose goal was to provide shared computing for biomedical research and to develop software tools and standard formats for information exchange. NCI-funded cancer centers across the U.S., including Mayo Clinic Cancer Center and Sloan Kettering Institute for Cancer Research, have access to caBIG via 145 network nodes. CaBIG comprises nine "workspaces," which support work in life sciences research, clinical trials, tissue banks and pathology, and other areas.

Following his appointment in July, NCI director Harold Varmus initiated a four-month review of caBIG, and that review was completed in February by a working group of the Institute's board of scientific advisers. The working group, in a report issued last month, found that fewer than a dozen cancer centers are using caBIG's clinical data management tools or its cloud computing infrastructure. And while acknowledging that caBIG has had a "positive influence" on creating standards for data exchange, the working group criticized the program for straying from its original goals and for sprawling into an overly complex "software enterprise" of more than 70 applications.

CaBIG is administered by NCI's Center for Biomedical Informatics and IT. The 10-member working group, comprising experts with advanced degrees in science and medicine, concluded that caBIG's program management structure has become bureaucratic and suffers from "very high" overhead costs. The program grew too rapidly without careful prioritization or a cost-effective business model, according to the working group. Lead contractors are Booz Allen Hamilton, SAIC-Frederick, and Sapient.

The NCI board issued 10 recommendations for corrective action. They include an immediate moratorium on all caBIG software development, including projects underway through commercial contracts; a one-year moratorium on all new projects and contracts; an audit of caBIG expenditures to date; and the creation of an independent oversight committee.

The working group's report details a long list of problems with the program. Some caBIG applications cost millions of dollars to build. The price tag for one, caArray, an array data management system, was $9.2 million. Other software is overdesigned, difficult to use, or lacking support and documentation. For these and other reasons, a majority of 32 "bench to bench" life sciences research tools developed for caBIG have had limited use or impact, according to the report.

Only seven of the 51 NIC-funded cancer centers participating in caBIG use the program's cloud computing infrastructure, caGrid. Twenty others are in the process of accessing caGrid, but that brings the total to just over half of those centers. Among the issues cited as obstacles are that caGrid is too complex and hasn't been adequately tested for security. Even the University of Chicago, which helped develop caGrid, isn't using it.

"If there was ever an example of how not to run a program, this may be it," says Michael Biddick, president and CTO of Fusion PPT, a systems integrator that works with government agencies. Biddick is an InformationWeek contributing editor who has written about caBIG. "The failure to link the mission objectives to the technology shows how important user acceptance and buy-in can be," he says.

Efforts to build a user community around caBIG, which include strategic planning meetings and monthly conference calls, were also found to be lacking. An annual meeting intended to advance the program was described in the report by one unidentified attendee as a "contractor feeding frenzy." In addition to the three prime contractors, about 20 other vendors are involved with caBIG.

The NCI report describes caBIG's program administration as having a "very complex organizational structure" and says that information on the program's budget has been inconsistent and difficult to decipher. The working group determined that money paid to Booz-Allen Hamilton, SAIC-Frederick, and Sapient over the past seven years exceeds $60 million and is "probably significantly more." It estimates total spending on caBIG to be at least $350 million for fiscal years 2004 to 2010, including about $100 million in federal stimulus funds.

CaBIG officials, in an effort to be "nimble" in delivering new capabilities, awarded subcontracts without peer review, contrary to standard practice within the National Institutes of Health. (The National Cancer Institute is part of the National Institutes of Health, which is an agency of the U.S. Department of Health and Human Services.) "There has been a complete lack of independent program review and oversight," the working group reports.

Biddick, of Fusion PPT, says that approach allowed new tasks to get started quickly but did so without important checks and balances. "Conflicts of interest that existed between the project oversight and implementation violate Project Management 101," he says.

The report also maintains that caBIG's contractors don't have enough expertise in cancer research, resulting in a perception among researchers that the contractors steered funding toward pet projects, such as cloud computing, rather than their needs. The working group expressed concern that this trend continues with ongoing projects, such as the development of a roadmap for caGrid 2.0. "The roadmap, goals, and timeline are opaque to the community," according to the report.

Andrea Califano, chairman of the working group and a professor of systems biology at Columbia University, puts part of the blame for caBIG's problems on a "layer of management" that exists between the caBIG user community and leadership team, resulting in miscommunications and a disconnect between the program's objectives and actions. He says adoption was also hampered by an installed base of software tools from Oracle, Velos, and other vendors that are used by 70% or more of cancer centers. Califano estimates it can cost cancer centers up to $30 million in personnel and other expenses to integrate caBIG software with their existing computing environments.

The White House's Office of Management and Budget is attempting to rein in problem-plagued federal IT projects through its TechStat review program. The goal is to "re-scope" complex projects before they become funding sinkholes. The feds started by looking at financial system upgrades and other high-risk projects, and they're now applying the oversight more broadly. CaBIG hasn't been subject to TechStat review but would be a strong candidate.

NCI says it's evaluating what to do next with caBIG. "The report has significant recommendations, all of which we are taking seriously," an NCI spokesman told InformationWeek in an email, in response to our inquiry. "At this juncture, however, decisions have not yet been finalized."

CaBIG's three main contactors scrambled to respond to the working group's report and our requests for comment. A Booz Allen spokeswoman referred questions to NCI but said Booz Allen "is very proud of the work we do for the National Cancer Institute and our contributions to the important caBIG project." Sapient declined to comment, while SAIC-Frederick was unable to answer questions in time for this article.

The report concludes that caBIG's original goals remain relevant and recommends refocusing on them. In addition to its progress on data exchange and applications interoperability, the working group found caBIG to be providing valuable support for software developed by the academic research community. But the working group also determined that caBIG suffers from poor execution and an unsustainable business model, and it recommends that further development of caBIG-branded software be discontinued.