Letter Drop: Shady Stats; Playing Up the PositiveLetter Drop: Shady Stats; Playing Up the Positive

Richard Starnes challenges, "Anyone with an elementary understanding of statistics and surveys knows the CSI-FBI survey is statistically questionable."

information Staff, Contributor

November 21, 2006

2 Min Read
information logo in a gray background | information

Shady Stats

I am an economist with a strong background in statistical analysis, now working full time in computer security. I am appalled that the CSI and FBI continue to circulate these kinds of results without properly qualifying them (Dashboard: "Bucking the Hype, IT Security Losses Decline"), but it's even more frightening to see the media continue to consume it. The headline should have emphasized that companies' reporting behavior is changing and we don't have very good data on what's really going on.

The big story is the lack of useful data to determine if things are getting better or worse. If we got rid of the police and the means for reporting crime, there would seem to be a drop in crime, but it would be the reporting process, not the underlying reality. If we want to know about cybercrime and cyberexploits, government, industry, law enforcement and educational institutions would do well to address our common ignorance on what's really happening out there.

The reason there is no basis for making judgments about trends over time from this data is that it is not a time-series sample, in which the same people in the same companies using the same methods and definitions report each year. It is a completely voluntary sample with different participants, different definitions and different perspectives each year. So we don't know if the change is the result of a different sample or a real-world shift. This is pretend science and is bad for the industry. It should not be reported this way without clarification, and it certainly gives no support for the arguments on the level or nature of malicious activities over time.

Ken Kousky
Saginaw, Mich.
[email protected]

You talk about the hype surrounding information security and cite the latest CSI-FBI study that claims financial losses are dropping along with information security budgets. Anyone with an elementary understanding of statistics and surveys knows the CSI-FBI survey is statistically questionable. They do not publish the methodology of the survey, but the response size alone will tell you the error rate is above acceptable limits. I see little change to indicate this is no longer the case.

Richard Starnes
London
[email protected]

Play Up the Positive

There is a paradox of speed (Dashboard: "Should You Speed Up BI? Not So Fast!" September 2006, ): You can speed up the steps yet still go no faster. The Concorde, for example, cut flying time from New York to Paris in half, yet door-to-door travel time decreased only 15 percent due to airport time and traffic congestion. However, downplaying the positive impact real-time BI could have by fretting over OLTP response times and Ferraris towing boats misses the point. Analytics embedded into operational processes are the real payoff for BI. The problem there is data warehouses and bloated BI tools--business as usual.

Neil Raden
Santa Barbara, Calif.
[email protected]

Read more about:

20062006
Never Miss a Beat: Get a snapshot of the issues affecting the IT industry straight to your inbox.

You May Also Like


More Insights