Bias And The Credibility Of IT Industry Analysts ReloadedBias And The Credibility Of IT Industry Analysts Reloaded

information's February 6 cover story on the credibility of IT industry analysts is the gift that keeps on giving, primarily in the way of phone calls and E-mails telling me I got it right or wrong. information has decided to give back. Later this month, we'll publish our next installment on this subject. But rather than rehash the same old arguments about whether analyst firms have the best interests of the IT executive in mind, we'll be talking about where IT analysis is heading. Here's a hint: blogs and bias will play a major role.No one is immune to bias--from the most seasoned analyst VP, "fellow," or research director at the big firms to the most ruggedly independent conspiracy theorist flooding the blogosphere with what he or she considers to be straight talk. The most important thing is to understand any analyst's bias, or angle, and factor that into your own interpretation of the analysis. I'll be addressing this in this blog and another to be posted shortly.

I'm not going to pretend that this is an original idea, but I'd like to present some thoughts from some of the preliminary reporting I've done for my upcoming story. Many of these sources have been newer analyst firms that launched around the time of the dot-com bubble. Others are well-known pundits who've recently ventured off on their own. Still others make extensive use of blogs to create a two-way dialogue that attempts to draw perspective from all corners of cyberspace.

Analyst firms launched in recent years seem to have gone out of their way to avoid the appearance of bias. Like many of his peers launching analyst firms, Rich Stiennon doesn't write vendor-sponsored white papers for his new firm, IT-Harvest, and he says he has sold financial interests in his previous employer, anti-spyware maker Webroot Software, as well as Reflex Security and Whale Communications, an application access and security provider on whose advisory board Stiennon used to sit. A June 2004 press release from Reflex Security boasted that Stiennon "highlighted" the company for its intrusion prevention system technology at a Gartner IT security summit, which would be troubling if Stiennon, a former Gartner security analyst, was an investor at the time.

"The hardest part of starting up a research firm is getting noticed," Stiennon told me. IT-Harvest focuses on IT security, with an emphasis on writing research reports. While Stiennon makes his most basic research available for free on the Web, he charges clients, mostly IT vendors and VCs, a $5,000 annual subscription fee for access to harder-to-get stats, such as how much product sales revenue a specific vendor is making in a particular region of the world. "Free content flies in the face of the conventional analyst industry, but it's used today to gain an audience," he says.

Rob Enderle, one of Stiennon's peers from their days with the elite analyst firms, has also hung his own shingle after years as a research fellow with Giga Information Group, which Forrester Research bought in early 2003. Enderle told me there's a number of valid criticisms about analysts: They at times restate the obvious, make unsubstantiated claims, and are prone to being biased toward certain vendors. "Boy, there's almost no way to get around bias," he told me. "I take revenue from vendors. It's fair to say that impacts me in some way." Like analysts in many other smaller firms, Enderle says he invests his personal finances in large, diverse funds rather than individual companies.

Of course, the very mention of Enderle in my blog leaves me open to rabid criticism that I'm just another lazy journalist. That's your call to make, but I'd like to point out that I was critical of Enderle's appearance at an August 2004 SCO Group conference and said so at the time.

My point is that bias affects everyone. During the course of researching my story, I had an E-mail exchange with James McGovern, chief security architect with investment and insurance firm The Hartford, on this subject. As soon as analyst firms sign commercial IT vendors as clients, the appearance of bias becomes a concern, he wrote me. This isn't exclusive to more established analyst firms, but "we would certainly hope that all consultants carefully analyze a wide range of solutions and strictly avoid using pay-for-play filters or other questionable approaches," wrote McGovern, who writes two different blogs, one about enterprise architecture and another about home theaters.

Everyone's got their own personal BS meter, and I encourage you to share yours. In part II of this blog, you'll see how smaller firms, academia, and bloggers cope with the bias issue. Stay tuned.