Opening Holiday Weeks Show Uptime Isn't Easy For Online RetailersOpening Holiday Weeks Show Uptime Isn't Easy For Online Retailers

What if, in the days leading up to Christmas, a crush of shoppers forced a retailer to lock its doors during peak business hours? Unimaginable--but that's exactly what happened to varying degrees at the Web stores for Wal-Mart, Macy's, and other retailers as the holiday e-commerce season got off to a blazing start. Here's what's more shocking: There's no guarantee it won't happen again during the all-important shopping days ahead, when even bigger throngs are expected at e-retail sites.

You'd think retailers would have licked the problems of site downtime and slow response times by now, even during the holiday rush. Yet while everything else associated with online shopping keeps rising--the number of visitors, sales volume, site sophistication, and consumer expectations--site performance is still a wild card. As companies load up on Ajax- and Flash-powered interactive features that can stress sites, simple uptime and performance could be the differentiators this season.

Wal-Mart's failure on Black Friday, the day after Thanksgiving, was the most stunning. Walmart.com was down for a total of about 10 hours that day, according to Internet monitoring firm Keynote Systems, forcing it to greet shoppers at times with a "come back later" notice. "I'm afraid it was too much of a good thing on Friday," a Wal-Mart spokesman says. Wal-Mart expected order activity to be double the level on last year's Black Friday, but it came in at seven times the previous year's volume. Wal-Mart set big online goals for this holiday season, having spent 13 months adding faster checkouts and an interactive toy section, the kind of features it hopes will lure about 300 million visitors this holiday season. Not if the door's closed it won't.

Macy's site performed poorly for about nine hours on Black Friday, according to Keynote, and was down for about an hour that day and then again part of the following Tuesday. Zappos' and Foot Locker's sites also had some performance problems. Keynote says most sites, including Wal-Mart's, recovered Monday and didn't have major problems that day.

Stores that kept their sites up did blockbuster business. Online buying on so-called Cyber Monday--thought of as the start of the Web shopping season, the theory being people start buying from their computers at work stuff they saw in stores over the weekend--totaled $608 million, up 26% from the same day last year. That made it the biggest single day in retail e-commerce history, says comScore Networks. On the Friday after Thanksgiving, online retail spending was up 42% over the previous year, comScore says.

WHEN THINGS GO WRONG

With the Web bringing in increasingly significant sales, why are retailers still dropping the ball on site performance? In most cases, they aren't. Many conduct load testing, forecasting, and monitoring using tools from companies such as Keynote and Gomez, and they use content distribution networks from companies such as Akamai Technologies and Netli to speed performance. They build their e-commerce infrastructures with an eye on performance, with redundancy at common failure points. But sites can still fail.

One reason is that the various teams--marketing, site designers, QA testers--too often aren't watching or sharing the same metrics, says Matthew Poepsel, VP of professional services at Gomez. If groups worry only about their metrics--marketing pushes a promotion, for instance, without being sure site design and capacity can handle it--you have "individual success but collective failure," he says. When a problem hits, Poepsel says, 80% of the recovery time is spent identifying the problem. "Once you know what the problem is, you can get it fixed pretty quickly, unless it's a sheer bandwidth issue and you need to cut a new contract," he says.

While Barnes & Noble's site has been performing fine in recent days, the problems of some other retailers don't surprise site CTO David Willen. "It just shows the incredible complexity it takes to power these Web sites," he says.

Barnes & Noble, like a lot of retailers this year, ramped up its site's complexity by adding more Macromedia Flash, Ajax, and other Web 2.0 technologies to jazz up the customer experience. Willen's team used Ajax--combining asynchronous JavaScript language and XML--so customers can open product details within the screen, without clicking over to a new page. But the enhancements require more processing power, more system analysis, and more load testing.

Circuit City built some Web 2.0 technologies into its site this year but was selective to avoid problems--a "delicate balance of customer experience and performance," says Kevin Hewett, manager of e-commerce infrastructure. "In some cases, the trade-offs aren't always acceptable." Redcats USA, whose brands include Chadwick's of Boston and the Lane Bryant catalog, has made similar trade-offs with its use of Ajax, says Bob Viscovich, senior VP of online technology.

CUSTOMERS EXPECT PERFECTION

Customers are ruthlessly unforgiving of poor performance. In a Gomez survey of 1,173 online shoppers, 53% say they'll switch to a competitor if a site takes too long to load, and 21% will call customer service.

Wal-Mart's testing doesn't appear to have been out of line with industry standards. When testing their systems for load capacity in preparation for Black Friday, retailers typically used twice the capacity their sites had on the same day last year, says Keynote, which equates that level to a 600% increase over an average shopping day outside of the holidays. Wal-Mart declined to discuss what caused its problems.

Rivals lie in wait for such outages. Circuit City's Hewett describes it as "a traffic windfall" if a competitor goes down during peak loads. Circuit City cut it close last year, he says: When a major competitor's site failed, it picked up a rush of new customers that stressed the site, though it stopped just short of failure. There's less time than ever to react to shifts in traffic, too. "We tend to get a traffic surge within seconds of refreshing content on the site that offers updated promos and pricing," Hewett says.

MORE TO DO

Circuit City is working more closely with its server provider, Sun Microsystems, to develop capacity planning, including using Sun's consulting services. Hewett says moving the company's Web application platform to BEA's WebLogic proved a boon in performance, lowering the server memory and processing power it needs. It also uses Akamai's and Keynote's services and products.

In addition to working with Keynote and Gomez, Best Buy recently began using TeaLeaf Technology to capture clickstream information so it can re-create the customer experience, says Judy Diesen, VP of business services at BestBuy.com. Best Buy overhauled its site for the holidays, including adding a gift center that lets people shop by price, age, and gender, and e-mail wish lists for presents.

There's still a lot more to be spent online in the holiday shopping season. While Cyber Monday gets the press, peak spending usually comes in mid-December. Online shopping isn't big enough to make or break most retailers; Wal-Mart stores did $18.8 billion for the four weeks through Black Friday, while comScore predicts the whole U.S. e-retail industry will do $24 billion this holiday season. Still, retailers know they must get this right. That $24 billion would mark 24% growth from last year. An untold number of in-store buyers did their comparison shopping online first. And retailers are adding on to their sites--Wal-Mart, the country's largest DVD seller, last week said it would start selling movie downloads from its site sometime next year. For its most recent quarter, Barnes & Noble's online sales were about 6% of total sales. "At this time of year, it can be a meaningful amount of revenue," says Willen. "... That's why the stakes are so high." And that's why poor site performance and downtime could become the grinch that stole Christmas--or at least a painfully large chunk of it.

-- with W. David Gardner