Virginia May Lose Data From OutageVirginia May Lose Data From Outage

IT operations in Virginia are more or less back to normal after an unprecedented network failure, but the state is still working "around the clock" to recover data, state CIO Sam Nixon said. He said that data loss has been "minimal" but that hopefully all of it will be recovered once recovery operations are completed.

Nixon also is agency head at Virginia Information Technologies Agency (VITA), which was created in 2003 to oversee and modernize the state's IT operations. A more than week-long disruption in IT operations and services for 27 state government agencies ended Thursday. The ordeal started last Wednesday when an EMC DMX-3 storage area network (SAN) went down, taking 13 percent of the state's file servers with it.

By Friday, VITA and Northrop Grumman, which partners with VITA to maintain IT networks for the state, had repaired the SAN. However, it took until Tuesday for nearly all affected agencies to be back online and to business as usual. Even then, the Virginia Department of Motor Vehicles, department of taxation and state board of elections still experienced disruptions. Officials said the complexity and size of their databases caused recovery to take longer for these agencies.

On Thursday Northrop, which signed a controversial $2.3 billion contract with VITA in 2005, formally apologized for the failure and said it would do everything it could to avoid such a scenario in the future. "Northrop Grumman deeply regrets the disruption and inconvenience this has caused state agencies and Virginia citizens," said Linda A. Mills, corporate vice president and president, Northrop Grumman Information Systems, in a statement. She said that while the "disruption to the business of state government has ended," the firm's investigation into the problem has not. "Our efforts to examine the factors that contributed to the initial problem and the lengthy restoration of service is just beginning," Mills said.

Northrop also plans to reimburse Virginia for "the reasonable costs of an assessment" of the problem, "as it is an essential and responsible measure that will strengthen our ability to protect against future issues," she said. Revisions to Northrop's own plan to improve process and response time for restoring agency operations also will occur as a result of the outage, Mills said.

Mills also acknowledged criticism to the state's decision to partially outsource IT operations to a private company, which has been widespread since its inception. The outage was the second major one since the two joined forces to handle IT operations for Virginia. "The partnership has experienced its share of obstacles, not unusual with large transformation programs," Mills said. "But with this modernized system Virginia and her citizens should find themselves years ahead of other states with the service provided by its IT infrastructure."

FURTHER READING: Virginia Still Disrupted By IT Outage Virginia Repairs SAN Failure That Caused Statewide Outage Lost USB Device Forces Virginia Into Damage Control, Again Northrop Grumman Relies On Shared-Services IT Model Virginia Outsourcing Deal Bears Careful Watching