Casino CIOs Put Their Chips On IBM's EServer ISeriesCasino CIOs Put Their Chips On IBM's EServer ISeries

You might think that the companies whose operations are far from the Sept. 11 terrorist targets have less to worry about when it comes to business-continuity planning. Take away natural disasters and you might assume that business continuity would be even less important. Think again.

Las Vegas casinos may be safe from hurricanes and blizzards, and the mountain of money that moves around town may not have the international impact that financial institutions in New York, Chicago, or San Francisco have, but continuous availability is a must in this round-the-clock industry.

To prevent a shutdown of computer systems, Aladdin Resort & Casino and other casinos in the gambling mecca rely on high-availability computers that guarantee constant uptime. Even brief downtime can disrupt operations and provoke guests to take their chips elsewhere. "If it happened often, we'd have serious customer-loyalty problems," says Chad Seegmiller, Aladdin's director of IT.

The high-availability server of choice for many gaming establishments is IBM's eServer iSeries, formerly known as the AS/400. The server comes with a deeply integrated OS/400 operating system and DB/400 database. "The AS/400s run 96% of this town," says Steve Vollmer, IT director at the $590 million Venetian Resort Hotel and Casino, which looks as if it were in the middle of Venice, Italy: An ersatz canal with floating gondolas winds through the casino.

Vollmer says the September terrorist attacks reminded him and Venetian's top management that their business-continuity processes were well worth the money. "Our plan is already solid," Vollmer says. "Our tapes go out 24-by-7 in armored trucks to a facility unknown to us, run by a company that doesn't want its name out."

Still, Vollmer admits that solid doesn't mean invulnerable. "If something like [the September attacks] happened to us, I'd get another AS/400 and get running in a bomb shelter," he says. "I hope it never happens."

Recently, a small number of companies using iSeries systems have had hard-disk failures, raising questions about whether the server's reputation as high-available hardware is deserved. Advanced Marketing Services Inc., a San Diego book publisher and distributor that counts the Harry Potter series among its properties, experienced multiple disk failures earlier this year.

"I've had eight disk outages since the beginning of the year," says AS/400 administrator Alan Batricevich. Advanced Marketing mirrored data on its iSeries 820-9406 servers and installed a temporary software fix that IBM provided at the end of July that seemed to solve the problem.

But in late August, it had another disk failure. "I have every IBM temporary fix available that we're told would resolve these failures," says Batricevich, yet the problem persists. The eighth crash would have cost the book distributor millions of dollars if not for the mirroring. "I still don't have that warm, fuzzy feeling" about the system being up all the time, he says.

IBM doesn't know the exact reason for Advanced Marketing's problem but says its patches should fix the problem. Mirroring or replicating data on a twin server should prevent downtime, the vendor says.

The breakdowns seem to happen as data moves between the operating system and the input/output processor. If a disk goes down on a standalone system without mirroring, the whole system could crash. Some customers were down for more than a day before their systems were fixed. When data isn't available, companies can lose a lot of business.

BCC Technologies Inc., a storage-systems vendor that sells to iSeries owners, buys disk drives from IBM's Information Technology Group. "We don't see [hardware or mechanical] problems with the drives, so we believe it's a problem with OS/400 and the drives," says Dave Breisacher, CEO of BCC. Monitoring the controller of the storage system connected to the iSeries is a way to enforce parity among multiple drives so no drive is left out, cut loose, or killed by the I/O processor, Breisacher says.

For now, IT executives at some casinos believe their wager on the iSeries server will continue to pay off. That's because the casinos, which also use high-availability software, mirror their servers. If one server crashes because of disk-drive failure or other reasons, another one will be able to take over.

High availability is part of the always-on Las Vegas culture. Take a 2 a.m. stroll into a casino along the Strip-the five-mile-long avenue ablaze in neon lights that runs through the city's main casino-hotel district-and don't be surprised to see gaming tables packed with gamblers. As regulars hit a lucky streak, casino systems automatically update their records for all sorts of customer-loyalty rewards. The players never have to take their hands off the cards or dice.

"The 400 is the best database. It never re-indexes, and it never crashes. I'm 24-by-7-by-365 or dead," Vollmer says. The Venetian's two iSeries 830-2320 servers also run the hotel, casino, slots, inventory-purchasing, reservations, financial, call-accounting, payroll, time, and attendance applications.

Manual processes can't duplicate the automated ones that track guests' room accommodations, gaming-table winnings, loyalty-points accumulations, and personal preferences such as the type of entertainment and meals guests enjoy.

"The AS/400's history of 99.5% availability is OK for some, but a half-day outage would be disastrous for us," says Seegmiller at Aladdin, which has decor inspired by the desert wonders of 1001 Arabian Nights. "Losses from manual processes and procedures could add up quickly." To protect against such losses, the casino runs two iSeries 720-9406s. "Our player-loyalty cards update the AS/400 instantly; if customers don't get points to their account, it impacts how they'll interact with us," Seegmiller says.

To help ensure constant uptime, gambling spots such as Aladdin, Venetian, and Station Casinos use software from Vision Solutions Inc. to replicate data between servers.

Station mirrors two iSeries 820-2397 servers that run identical sets of casino, financial, and hotel applications. Some 250,000 transactions go through the 2397s daily, typical for a midsize casino such as Station, which had $850 million in revenue last year. "As soon as a transaction is complete on the primary system, it replicates the transaction to the secondary system, and we're never more than a few transactions behind," Station IT director Greg Dauenhauer says.

Like Aladdin, Station-an old-school casino that's shy on glitz but a favorite of local gamblers-tracks revenue from its iSeries servers, which let customers accumulate points and receive gifts from the casino, such as free stays, golf clubs, or credits with which to gamble more. "The system is essential to our operations," Dauenhauer says. "We run our business on it."

The iSeries' popularity lies in its reputation for reliability and low operating cost, says Dan Kusnetzky, an analyst at International Data Corp. "It's self-contained, very reliable, and reduces the most critical cost," he says. "It's easy to manage because it's so tightly integrated, so it reduces the people cost."

The tight, sometimes hard-wired, integration between hardware, operating system, database, and applications means most iSeries customers avoid management costs that can be seven times the cost of the system, as is the case on Unix and Windows platforms, Kusnetzky says.

Because each iSeries system typically carries an up-front price of hundreds of thousands of dollars, management, development, and maintenance costs could end up costing casinos millions of dollars each year. Luckily for cost-conscious customers, the self-contained management and tight integration mean minimal maintenance and management costs.

The iSeries has become the de facto high-availability server for the Las Vegas casino industry because the AS/400 is designed to run packaged applications with minimal programming and maintenance. Casinos, which are relatively small when compared with major companies, have limited IT staffs-90 for Station, 28 for Venetian, and 18 for Aladdin. Software makers such as Inter-American Data for hotel applications, Stratton Warren Software for inventory management, and Infinium Software for financials have for years been writing low-maintenance products that run on the AS/400.

The iSeries fits perfectly into the casinos' game plans. Casino operators would rather spend money on promoting their gaming properties than on IT. You can bet the house on it.