Why MapReduce Matters to SQL Data WarehousingWhy MapReduce Matters to SQL Data Warehousing

Greenplum and Aster Data have both just announced the integration of MapReduce into their SQL MPP data warehouse products. So why do I think this could be a big deal? The short answer is "Because MapReduce offers dramatic performance gains in analytic application areas that still need great performance speed-up." The long answer goes something like this.

The core ideas of MapReduce are:• For large problems, parallel computing is much more cost effective and/or feasible than the alternatives. • If you shoehorn programs into a certain very simple framework - namely that you're limited to only having map and reduce steps - then building a general execution engine that gives parallelism "for free" is straightforward. • A lot more problems can be solved within that framework than one might at first expect. In essence, you can do almost anything to a single record* - that's a map step. But you are sharply limited in how you combine information about multiple (often intermediate) records - that's a reduce step. Still, reduce steps let you do counts, sums, or other aggregations. That, plus the general power of map steps, makes MapReduce useful for at least three major classes of applications:

1. Text tokenization, indexing, and search