Breakthrough Analysis: In BI Deployments, Methodology Does MatterBreakthrough Analysis: In BI Deployments, Methodology Does Matter

As a business intelligence practitioner, I feel like an IT Odysseus. I steer clear of data-warehouse design partisanship, which will suck you in like Charybdis' whirlpool. I also try to minimize losses to the many-headed Scylla that threatens application developers: an endless obsession over J2EE and .Net, unified process, aspect-oriented programming, model-driven development, agile methods and a host of other doctrinal matters. I find that value resides on a higher plane where what counts most are clear, correct, timely and usable results. Business insight, not methodology, pays the bills.

Nonetheless I've come to realize that BI is much more than a set of software tools and analysis objects. Each BI project embodies a transformational process designed to tease useful knowledge from data. Nail the application of BI practices and you can spend more time creating business value.

So I now tend to consciously repeat similar technical and management steps in each new project I take on. I have become what I have scoffed at, a methodologist.

I've discovered that most BI vendors have formal methodologies of their own. They've been hiding in plain sight, some for years. Take Informatica's Velocity methodology, which dates to 1999. Velocity combines best practices, application templates and a standard work breakdown structure. It's fairly generic--only portions are vendor-specific--and Informatica invites systems integrators to pull elements into their own methodologies. Although it is designed to ensure that customer systems go live, Velocity actually helps drive sales, not only of professional services but also of software, says Jon Herstein, professional services senior director.

At text-mining vendor ClearForest, the ClearPath methodology is more than a sales differentiator. "Our problem as a vendor is not that we're not winning enough deals; it's that there are not enough deals out there," says Jay Henderson, director of product and corporate marketing. Providing a framework for application definition and deployment lets the company suggest compelling end-user benefits that create opportunities.

IT methodologies languish if they don't adapt to computing innovations. Informatica is updating Velocity to accommodate new product features as well as advances in high-availability and grid computing, data-quality management and data-integration techniques. The company plans to release version 6 later this year. And ClearPath addresses a hot issue: bringing text analytics into the BI fold.

Innovations and emerging concerns, including the need to handle text, are motivating a collaborative upgrade to CRISP-DM, the Cross Industry Standard Process for Data Mining. Data-mining technology has greatly matured since the methodology was launched in 1999, according to Duncan Ross, advanced analytics specialist at CRISP-DM co-sponsor Teradata. Data volumes are way up and many new types of data source have come online. Users now want to integrate formerly self-contained analytics in line-of-business applications to integrate data mining into operational systems.

CRISP-DM was initially funded by a mid-1990s European Union grant, and unlike approaches crafted by vendor professional-service groups, it was designed to be tool- and vendor-neutral. Teradata and co-sponsor SPSS take neutrality seriously and have assembled a special interest group that includes users, consultancies and rival vendors to plan the modernization. Participants at the recent first SIG meeting included ClearForest and data miners Fair Isaac, KXEN, Salford Systems and SAS. The collective hopes to roll out CRISP-DM 2.0 in six months. Immediate goals are to improve model deployment and add monitoring. The group wants to "help people understand how to make data mining a mainstream business activity," Ross says.

SPSS built CRISP-DM support into its data-mining tools and uses it as the basis for training. Unlike the other analytics methodologies I surveyed, CRISP-DM has been widely adopted by customers, according to Colin Shearer, senior vice president for market strategy at SPSS. No doubt that's thanks both to the methodology's openness and its tool integration.

CRISP-DM and other initiatives have convinced me that my own experience isn't unique. If you want to do BI well, methodology does matter.

Seth Grimes is a principal of Alta Plana Corp., a Washington-based consultancy specializing in large-scale analytic computing systems. Write to him at [email protected].