Overstocked With DataOverstocked With Data

Overstock.com Inc., the online closeout retailer of brand-name merchandise, is migrating from an Oracle 9i database to Oracle 10g and shifting its reporting from operational systems to a data warehouse.

The latter is necessary because Overstock's business has been growing 100% a year, and its operational systems were struggling under their combined customer-transaction and reporting loads, says Shawn Schwegman, senior VP of technology. With 14 million to 18 million hits a month, Overstock's Web site handles an information and transaction load equivalent to Sears' or J.C. Penney's Web sites, he says.

The demands on its transactional systems were becoming so heavy that Overstock found itself "turning off one reporting function after another" to keep customer-response times satisfactory, but that created other problems. "Basically, you're dead in the water if you can't report on operations," Schwegman says.

With no window of time to shut down operations and move data, the 24-by-7 online merchandiser had to decide how to migrate to Oracle 10g and implement a new reporting system without affecting customers.

That meant Overstock needed a high-volume data-management tool to migrate data into Oracle 10g and then load selected data into the new NCR Corp. Teradata data warehouse, Schwegman says. Overstock evaluated data-migration toolsets from GoldenGate Software, Quest Software, and DataMirror. All three offer real-time data migration, Schwegman says, but the company chose GoldenGate's transactional-data-management tools for their ability to extract data from Oracle database log files rather than the database itself.

"Most ETL [extract, transform, and load] tools add 10% to 15% performance overhead to the production environment. Because GoldenGate can read the Oracle log files, its overhead is less than 4%," Schwegman says.

Overstock uses up to 12 Oracle database clusters, with the largest consisting of four eight-way servers. That cluster alone represents a $1 million technology investment, including the cost of application software, and reducing performance overhead by 10% saves Overstock $100,000, Schwegman says. His total relational database investment is more than $5 million, so saving 10% of that promises to be a significant amount.

Overstock purchased the GoldenGate tools two months ago. In the first phase of the project, it has been migrating from Oracle 9i to 10g, with GoldenGate handling the data transfers without any database downtime.

The online retailer also has been building a Teradata data warehouse with Business Objects SA reporting software replacing the 30,000 lines of custom reporting code previously used by Overstock managers and analysts. The company started to assemble the data warehouse in April, a project that normally takes six months or more, but Schwegman expects to finish the work in less time. The system now has 25 early users, "executives and a set of data hounds, power users," Schwegman says. In another month, the total will be 250 users, he adds.

When the data warehouse is finished, GoldenGate will continue moving data from the operational Oracle systems into the data warehouse "in near real time to provide reporting and analysis for our environment," Schwegman says.

Needing real-time analysis for customer-relationship management was one reason Overstock created its own in-house reporting system after it was founded in 1999. Now, as a maturing online retailer--Overstock is the No. 6 or 7 retail Web site in terms of the number of unique visitors, Schwegman says--the company must move away from homegrown systems to systems that march in step with its growth rather than impede it.