Virginia Still Disrupted By IT Outage

Some government agencies in Virginia were still experiencing service disruptions Tuesday, despite the repair of a storage network that last week caused a major IT system failure statewide.

Virginia Gov. Bob McDonnell has ordered a formal investigation into a network outage that started last Wednesday and affected 27 of 89 state agencies.

"I am not pleased that our employees and citizens have experienced this disruption in service," he said via an e-mailed statement Tuesday. "It is crucial we learn what happened and why in order to ensure that such occurrences are prevented in the future."

The Virginia department of motor vehicles (DMV) is one of three agencies for which problems persist, and which was "heavily impacted" by the failure, according to a statement issued Monday by Virginia secretary of technology Jim Duffey.

According to the DMV's website, the agency on Tuesday is unable to process driver's licenses or ID cards in its 74 customer service centers. The agency said those are the only transactions still affected by the network failure.

The department of taxation and the state board of elections also are still having problems due to the outage, according to Marcella Williamson, a spokeswoman from the Virginia Information Technologies Agency (VITA), which oversees the state's IT operations.

VITA said on Friday that it had successfully repaired the EMC DMX-3 storage area network (SAN) responsible for the outage. Thirteen percent of the state's file servers failed as a result of the SAN going down.

At the time, the agency said it was busy bringing servers back online and restoring services, work that continued over the weekend into Tuesday. In a progress report sent via e-mail Monday, VITA said that the agencies still experiencing problems are those with "the largest and most complex databases... These databases make the restoration process extremely time consuming. The unfortunate result is the agencies will not be able to process some customer transactions."

In his statement, Duffey said that the "events that led to the outage appear to be unprecedented," a comment he attributed to EMC. "The manufacturer reports that the system and its underlying technology have an exemplary history of reliability, industry-leading data availability of more than 99.999% and no similar failure in one billion hours of run time," he said. EMC did not immediately return a request for comment on Tuesday.

The statewide system failure is a major black mark against VITA and Northrop Grumman, which won a $2.3 billion contract in 2005 to work with VITA to maintain Virginia's computer and communication systems. The outage is the second serious IT disruption to plague the state since the Northrop deal, a move that was deemed as risky and has been criticized by state officials.