Playing For KeepsPlaying For Keeps

Until Sept. 11, an executive's worst nightmare might have involved the loss of a key building to fire or a natural disaster such as an earthquake or tornado. But the thought of hundreds of employees dying in a terrorist attack was unfathomable, as was the notion that a single event could indefinitely interrupt such basic business necessities as telecommunications and transportation.

That's why a companywide approach to disaster recovery-also referred to as business-continuity planning-has become a higher priority at many companies. "Business continuity will have higher-level visibility and higher-level expectations," says Bill Godfrey, chief technology officer at Dow Jones & Co., publisher of The Wall Street Journal. It's a lesson all too clear to the New York business-information provider, which on Sept. 11 lost its main offices near the World Trade Center and had to evacuate hundreds of editors, reporters, and support personnel. To get a newspaper out that night, Dow Jones initiated its business-continuity plan by hastily setting up alternate offices, installing 100 workstations, ordering additional network capacity, and counseling employees as they adjusted to the new working conditions. It will take months of rebuilding before Dow Jones can return to its original offices.

"Our management knows we have to invest" in business-continuity planning, Godfrey says. "We won't get away on the cheap." Now, all production systems will be backed up and tested, he says, and the company is looking to hire a director to focus solely on disaster recovery.

Companies directly affected by the events of September aren't the only ones giving business-continuity planning more serious thought. According to an information Research survey this month of 250 IT and business managers responsible for their companies' business-continuity and/or disaster-recovery plans, fielded by PricewaterhouseCoopers, more than half the managers surveyed say their companies will increase spending on business-continuity planning in the next 12 months-including 10% who expect a significant jump in expenditures.

There's certainly room for improvement. According to the survey, only 64% of companies with business-continuity plans extend their efforts across the entire enterprise. High costs and limited time are serious challenges to comprehensive planning, and most businesses don't regularly update their plans, nor do they disseminate them throughout the company.

Perhaps most disturbing, evidence points to a disconnect between business managers and IT managers, and between companies and their partners, concerning business-continuity planning. Three in five survey respondents say business continuity and disaster recovery don't involve a collaborative partnership between business and IT leaders. And few companies bother to share business-continuity plans with business partners, key vendors, or service providers.

Business-continuity plans must anticipate the chain reaction a single, dramatic failure can create throughout a company and outside it. Doing so requires the cooperation of a company's management at all levels, as well as the involvement of most, if not all, of its value-chain partners. While many companies have spent some time and effort developing such plans, few have a comprehensive grasp of all elements.

Companies that provide business-continuity planning and disaster-recovery services, perhaps not surprisingly, report a considerable surge in inquiries since Sept. 11. "The sight of all that paper fluttering down after the towers collapsed hit home with people," says Pat McAnally, marketing director at SunGard Planning Solutions, the consulting arm of SunGard Data Systems Inc., a provider of disaster-recovery programs.

Though viewed by some as primarily an IT concern, business continuity spans a range of operations to ensure that offices and plants continue to function-or can be quickly replaced-in the event of a disaster. Among the elements survey respondents have covered in their plans: data and technology recovery; facilities management; procedures for public relations and damage control; and communications with customers, suppliers, business partners, and investors. Half of the respondents even include alternate sites for receiving and sorting mail.

Cost is a key concern in business-continuity planning, with roughly half the respondents citing capital expense as a possible hurdle in the effective management of their programs. Companies spend about 1% to 2% of their IT budgets on business-continuity planning, several CIOs say. Even heightened awareness is unlikely to change those percentages by much.

A lack of time is the most common barrier. Nearly six in 10 managers surveyed contend that time constraints prevent them from instituting an effective business-continuity plan. Blue Cross and Blue Shield of North Carolina's business-continuity planning has to compete for attention with other large projects, says Fred Goldwater, CIO at the Chapel Hill, N.C., health insurer. "We have only so much time and human calories for doing work," he says.

Snap-on Inc., a $2 billion-a-year Kenosha, Wis., toolmaker took a year and a half to develop and execute a business-continuity plan, CIO Al Biland says. To get the project rolling, Biland retained disaster-recovery services provider Comdisco Inc. to interview scores of Snap-on managers. "We started by giving Comdisco a chart of the entire organization," Biland recalls. "It's a difficult process-not something you can do in a one-hour meeting."

But the process can pay dividends when the unexpected occurs. Last year, a tornado struck the town of Algona, Iowa, where Snap-on manufactures toolboxes. The winds blew off part of the plant's roof, and heavy rains damaged the Hewlett-Packard servers inside that ran Baan enterprise resource planning applications, Microsoft Outlook and Exchange, and Autotime, a payroll application. The IT department shipped similar servers it used for quality-assurance testing at Snap-on's headquarters to Algona, an eight-hour, 440-mile trip. The servers were up and operating the next day, even before shaken workers returned to the factory floor. Deploying servers typically used for application development, testing, and quality assurance isn't uncommon as a backup for failed systems; it's "the ace in the hole in business continuity," Biland says.

Similar to Snap-on's experience, many companies with business-continuity plans find these efforts aren't an idle exercise. According to information Research's survey, more than a quarter of respondents have had to use their plans in the last 12 months. Of those, more than two-thirds describe the crisis that precipitated the use of those plans as severe or extremely severe.

It's surprising, then, that nearly 40% of companies say it will take days or longer to bring records back online if disaster wipes out their companies' main data stores. While 14% say they could cut over to a hot-backup or standby system instantly, 7% concede it would take longer than a week, if they were able to get the data online at all.

That may be because sophisticated data recovery isn't cheap. Commerzbank AG's decision last summer to upgrade its backup system proved a wise-but-expensive investment for the U.S. arm of the $21.4 billion-a-year German bank, whose New York operations were on the 33rd floor of Two World Financial Center and are temporarily quartered in the New York suburb of Rye, N.Y.

Until a few months before the tragedy, the bank had been relying on tape libraries for its technology backup. But tape needs to be stored and then moved from one place to another. "For large amounts of data, tape alone wasn't practical in a crisis," says Richard Arenaro, corporate VP and regional manager of Windows and Unix systems. To assure quicker backups of data, Commerzbank began using EMC Corp.'s Symmetrix storage hardware and Remote Data Facility software. The Symmetrix system lets a company copy data while a system is up and running; it isn't cheap, but it's regarded as the gold standard of data replication.

Even at the cost of a few million dollars, the investment paid for itself on Sept. 11. "On tape, we would've been looking at at least 20 hours" to retrieve data says Gene Batan, VP and North American manager of systems and IT. That would have been unacceptable, because the bank's U.S. operation moves $30 billion daily in money transfers. Without the Symmetrix system in place, "we wouldn't have known our position that day." Even with it, the bank lost four hours, but that time was spent making sure employees were safe and wrestling with which data would be moved to Commerzbank's backup system.

Deciding which data needs to be recovered first during a crisis can be tricky. But Snap-on's Biland points out that data and systems related to transactions and cash flow usually rise to the top of the list. "You have to take a cold, hard look at where the cash is being generated," he says. "For non-mission-critical systems, the recovery time is longer." At Snap-on, contingency plans provide for quick restoration of systems that accept orders; pick, pack, and ship products; and bill dealers. Noncritical systems such as decision-support and accounting applications that don't have a direct impact on customers are lower priority.

IT and business managers often see things differently when it comes to business-continuity planning. While nearly half of corporate managers say it's a collaborative venture, only a third of IT executives concur. "In a lot of cases, business management forgets they have to do anything," says Jamie Bachant, manager of corporate disaster-recovery services at Lockheed Martin Enterprise Information Systems, the Denver IT arm of Lockheed Martin Corp. "They assume that if the computer's up and running, everything is hunky-dory."

Bachant wants business managers to be more proactive in developing the contingency plans that directly affect their operations and in challenging solutions proposed by IT managers. For example, Bachant's team might suggest a plan that gets certain systems up and running within 72 hours of a disaster. "They say it's OK, when they really needed 24 hours," Bachant says. "It's a holdover from the glass-house days when people felt IT would take care of them." With that in mind, Bachant has asked top executives from each of Lockheed's five business units to work with his disaster-recovery team to keep management and IT in step with each other about the company's preparedness.

One way to boost collaboration between business managers and IT is to establish action groups that include both. The idea isn't new; a number of companies have grouped business managers with IT staffers to ensure IT systems deliver what the business needs. Yet, only 2% of the 250 companies surveyed say that a joint IT-business committee has ownership of their companies' continuity program.

Not everyone thinks the lack of a formal joint committee for such planning means there's no cooperation. One such company is Group Health Inc., the largest not-for-profit health insurer in New York, with 2.4 million subscribers and annual premiums of $1.6 billion.

Ownership, creation, and maintenance of Group Health's disaster-recovery program rests squarely with the IT department, but others are involved. Chief operating officer Donna Lynne determines the order in which systems will be restored in the event of an emergency. Assistant VP Art Louise, who oversees disaster recovery and facilities management for the insurer, meets at least every other week with the company's risk manager, Laura Wolf, to discuss business-continuity planning. All departments designate a middle manager to actively plan and participate in semiannual testing of the Manhattan company's disaster-recovery plan.

"Senior management made it very clear that disaster recovery is a corporate issue," Louise says. "It's not an individual business-unit issue. Everybody succeeds or everybody fails."

Phil Bloodworth, global partner in charge of business-continuity planning at PricewaterhouseCoopers, understands why business managers look to IT as the leader in contingency planning. "Historically, IT was way out in front of the curve. It understood the need to back up and recover systems," he says. "That's not new; it's been done for years." But, as computing has been decentralized and distributed, the ownership of business-continuity plans is shifting to the business manager. "IT is now the facilitator, not the driver, of business-continuity planning," Bloodworth says.

Only 12% of survey respondents say IT "owns" their business-continuity plan, but 32% say IT maintains it. More than 40% place the ownership of business-continuity planning with corporate executives. Merrill Lynch & Co., the New York financial-services company, created a business post-director of global contingency planning-to oversee business continuity. The job is held by Paul Honey, who politely refers a visitor to someone else if the question is about technology.

"We worry about all aspects: implementing a strong technical program, a strong business program, and a strong crisis-management program," Honey says. "We have a mix of technical, security, facility-management, and business people, and everyone talks to one another."

Merrill Lynch's main office was located in the World Financial Center, now part of the ruins that are Ground Zero. Some 9,000 employees evacuated the area; 8,000 were working in a new location the following Monday, in time for the reopening of the stock markets. "It wouldn't have been possible without cooperation from people representing all parts of the business," Honey says.

Sometimes, a business-continuity plan must be amended as it's being invoked. That's exactly what the New York Board of Trade had to do after the Sept. 11 attacks destroyed its 13 trading pits in the World Trade Center.

Fortunately, the exchange already had a well-thought-out business-continuity plan, the result of the 1993 terrorist bombing of the Trade Center. After that attack, the New York Board of Trade commissioned a study that determined its customers-primarily traders of cocoa, coffee, sugar, cotton, and orange juice-would lose $3.5 million in business each day trading was halted. So the exchange developed plans to operate elsewhere in the event of a disaster and hired Comdisco to build a backup facility. All told, the exchange spent $1.75 million for equipment and pays Comdisco $300,000 annually to manage the site. That's why, one day after the September attacks, the Board of Trade was up and running at its backup site in Long Island City, across the East River from midtown Manhattan. The exchange had two rings, 13 pits, and 1,000 commodities traders working in two-hour shifts.

But the Board of Trade never expected such utter destruction and hadn't foreseen the need for a lengthy stay at its emergency site. That forced VP of IT Bob Gaba to revise the contingency plans. "Because we built for a short stay, we hadn't fully populated it with phones," Gaba says. "We ordered a telecommunications switch from Siemens on the 15th; they built it in 48 hours, and we had it the next day." The following week, the commodities exchange had 600 communications lines. Now, it runs up to 1,800 lines, 30 T1 lines, and ISDN, from multiple carriers.

Without the business-continuity process, the exchange could have been out of business when it lost its World Trade Center trading pits. "This is an extremely competitive business," Gaba says. "If we're down a week, Chicago commodities exchanges could offer our contracts, and one exchange in Europe was already going to move in on our business." By January, Gaba says, the Board of Trade plans to have six more pits operating in Long Island City.

Gaba also credits his company's relatively smooth recovery to practice, including monthly tests to ensure the emergency site was viable and trial runs every 60 days that included logistics and the relocation of employees. Such regular oversight of business-continuity plans is vital. Again surprisingly, more than half the companies surveyed wait prolonged periods before reviewing their business-continuity plans. Three in 10 survey respondents say they review their plans annually, with nearly one-quarter doing so less frequently, including 16% that cite no regular review schedule. While 20% say continuity reviews are continuous, another 22% say they reassess their plans more than once a year. Lockheed hasn't had a real disaster that Bachant can recall, but it tests its backup and recovery process about 15 times a year.

Businesses not only need to consider how often they review plans, but also who gets copies of the plans. Generally, it's the people within the company responsible for executing contingency plans. About half the respondents say all managers, including executives, as well as selected staff members receive copies.

Some companies, including Snap-on, require key managers and employees to keep copies of contingency plans at their homes. But only 12% of respondents say such documents are distributed to their entire workforce, an approach that doesn't sit well with Blue Cross and Blue Shield of North Carolina. The insurer makes its business-continuity plan available only to those responsible for business continuity, CIO Goldwater says. But the company is in the process of revising its plans and has decided it will place copies of them on the company intranet as well as on notebook computers. It will also print hard copies in case an event prevents access to the intranet.

Brady Corp., a $546 million-a-year global manufacturer of signs, labels, and specialty tapes, has left it to each business unit to develop its own business-continuity plan. In some cases, the plans fill a 300-page binder; in other instances, only a page or two. "Our goal down the road is to come up with a standard package for all sites," says Gary Laskiewicz, the Milwaukee company's IT security manager.

Like many other IT managers, Laskiewicz says he hopes the events of Sept. 11 persuade senior executives to allot more money to business-continuity projects. Brady already mirrors its key SAP ERP applications and data between two data centers in Milwaukee. But the company hasn't budgeted for real-time backup of so-called bolt-on systems, such as shipping, Laskiewicz says. And though Brady maintains a contract with IBM to provide iSeries servers in the event of a disaster, the company hasn't contracted for enough servers to meet all of its needs. "We think it's the right time to make the pitch," he says.

Still, he's philosophical about the outcome. "It's my job to let them know of our vulnerabilities," Laskiewicz says of senior management. "If they accept that, great; if not, they'll have to allocate more money. It's their decision." It's a decision companies are giving more weight to, because the ramifications have been made so shockingly apparent. -with Helen D'Antoni