Northrop Grumman To Pay $4.7 Million For Virginia OutageNorthrop Grumman To Pay $4.7 Million For Virginia Outage

Top 20 Government Cloud Service Providers

Northrop Grumman will pay nearly $5 million to the Commonwealth of Virginia for a network outage last summer that lasted a week and affected multiple state agencies.

A third-party review by Agilysys evaluating the incident put much of the blame on the contractor for the outage that began on Aug. 25, for failing to follow best practices and not responding appropriately when replacing hardware that had failed. Northrop has already paid $250,000 for the report's creation.

"This agreement brings closure to this incident, and provides the commonwealth with an improved information technology infrastructure that will reliably support Virginia's citizens and agencies in the years ahead," said Gov. Bob McDonnell in a statement.

Northrop Grumman has a $2.3 billion contract to work with the Virginia Information Technologies Agency (VITA) to look after communications and computer services for the commonwealth. VITA was created in 2003 to maintain and modernize Virginia's IT operations.

The money will be used to make improvements to Virginia's network infrastructure and repair any loss incurred by the outage, according to the commonwealth.

The payment breaks down in the following way: Northrop will make monthly invoice credits over the next 24 months that total $1.91 million to make up for the direct costs incurred by Virginia as a result of the disruption.

An additional $2.09 million will be spent by the contractor to make operational improvements to database backup, storage systems, system monitoring, database monitoring, network monitoring, and capacity management, according to the commonwealth.

Northrop Grumman also will pay $750,000 more to implement and fund new technology to protect commonwealth data, as well as address additional recommendations made in the independent review.

"We are pleased to have reached an agreement with the commonwealth regarding the disruption caused by the August 2010 outage," said Northrop spokeswoman Christy Whitman, in an e-mailed statement.

She added that Northrop is working with VITA to implement new procedures, policies, and other preventative measures to ensure a similar incident does not occur in the future.

Virginia's trouble began last Aug. 25 when two memory boards that were meant to back each other up failed. Analysis by EMC, the manufacturer of the boards, said a so-called "electrical over-stress condition at the component level" caused the dual failure, which resulted in a loss of data, according to the independent report by Agilysys.

Following that, "human error during the memory board replacement process resulted in the incurred extended outage," the report found.

The disruption affected 13% of Virginia's branch file servers and 26 of 89 executive branch agencies, 16 of which reported financial impact due to the outage. The Virginia department of motor vehicles, which was unable to process driver's licenses or ID cards in its 74 customer service centers during the outage, was the most visibly affected of the agencies.

Virtualization can hide performance issues from traditional tools. This report provides guidelines to help you understand the hurdles that virtualization puts in front of APM, and how to get over them. Download it here. (Free with registration.)