Ian Ayres: Success Starts With Random TestsIan Ayres: Success Starts With Random Tests

"If you're not relying on the twin pillars of regression analysis and random tests, you're making a big mistake." This was a key point delivered this morning at the Gartner BI Summit by keynoter Ian Ayres in his talk on "Why Thinking-By-Numbers is the New Way to Be Smart." Drawing from his book "Super Crunchers," Ayres said random testing, in particular, is the best way to get into statistical algorithms, and he guaranteed the measure will give you "at least a 5 percent to 10 percent improvement in any measure you care about."Ayres' larger point was that it's time for companies to move away from gut feel, tribal knowledge and intuition in decision making and move instead toward crunching the numbers. He cited dozens of interesting mainstream "Super Cruncher" examples in which statistical analyses are being applied to decisions:

Epagogix is helping the movie industry predict film revenue before they go into production by coding up scripts, stars, directors and other known variables and then doing regression analysis against historical results. Pre production, Epagogix predicted that the Nicole Kidman film "The Interpreter" would gross $69 million, and, in fact, it ended up grossing $73 million - a more accurate forecast than most movie studios are used to.

Professor Orley Ashenfelter of Princeton caused a stir in the wine industry by predicting the quality of wines based on statistical analysis of growing conditions (average temperatures, rainfall, etc.) compared with historical growing conditions for top wines.

eHarmony does regression analysis of personality characteristics to suggest possible matches that other dating sites with simpler methods might miss.

Farecast predicts whether airline ticket prices will go up or down based on regression analysis of historical pricing patterns, and it even displays a confidence level and sells fare insurance on its predictions.

Delaycast predicts the likelihood of flight delays based on historical patterns tied to up-to-the-minute variables such as weather conditions.

A big believer in random tests, Ayres used the technique in picking the name of his book, finding that Web ads for "Super Crunchers" outperformed "The End of Intuition" (another name he had in mind) by 63 percent.

"One reason I like random tests is that the results are so easy to communicate, and easier it's easier for people to understand and believe than regression analysis," said Ayres.

Ayres recommended Web sites as the natural place to start random testing (and he touted the free Google Web Optimizer as a helpful tool). Want to find out if online marketing drives or cannibalizes offline sales? Start by randomly turning on and off the online marketing campaigns at random times in randomly chosen cities. Keep track of the impact, and once you have the results, segment the results to see if the impact varied among different groups. Ayres cautioned practitioners not to throw off the results by picking test cities or times in a non-random fashion. rather than determining everything will a random selection tool such as a coin toss.

Even if the humans involved in a decision are "subject matter experts," Ayres says countless studies show that they're more accurate than statistical analyses only about 6 percent of the time, usually when there are few variables. As the number of variables and complexity increases, empirical analysis wins every time."If you're not relying on the twin pillars of regression analysis and random tests, you're making a big mistake." This was the key message delivered this morning at the Gartner BI Summit by keynoter Ian Ayres in his talk on "Why Thinking-By-Numbers is the New Way to Be Smart." Drawing from his book "Super Crunchers," Ayres said random testing will give you "at least a 5 percent to 10 percent improvement in any measure you care about."