Surviving The Spam StormSurviving The Spam Storm

Despite the assurance offered by Bill Gates two years ago that "two years from now, spam will be solved," the spam deluge is getting heavier.

Spam rose 59% between September and November, according to Postini, a managed messaging service. The company processed 70 billion e-mail connections for customers during that period and 91% of it was spam. In the past year, the company says, the daily volume of spam has risen by 120%.

For anti-spam vendors, surviving the spam deluge has been a matter of natural selection. "The market for anti-spam technologies has shaken out and there are a handful of companies that are still delivering good-quality products," says Scott Petry, founder and CTO of Postini. "But a lot of the less well capitalized companies that weren't able to keep up with the tactics have fallen by the wayside."

Refuge from the spam storm can be found online, at a managed e-mail service. "If you think about a doubling in the spam rate and a significant increase in the amount of image spam, that means that the mail server that the customer is running has to work harder to process more and more load, that load being garbage," explains Petry. "So any solution that allows you to move the burden of processing out to the Internet, as opposed to the customer's network, is going to yield quality of service results to the person trying to run the mail server."

The network, it turns out, is the best defense against the bot armies that spew endless pitches for porn and pills.

Postini can track and block messages in real-time because of the sheer volume of data passing through its network. "You can identify in real-time who's being good and bad when you have lots of volume," Petry says.

Google has access to similar information and also makes effective use of it. "There's a lot more spam being sent today," says Keith Coleman, Gmail product manager. "But with Gmail, we've been able to keep more of it out of the inbox over time."

Google uses not only its search technology in its spam eradication efforts, but also the votes of its users, a strategy the company is known for employing to improve the relevancy of its search results. The "Report Spam" button in the Gmail inbox and the "Not Spam" button in the spam folder let users tell Google when e-mail has been misclassified.

"That basically lets us use user feedback as the primary input for our classification system," says Coleman. "We do some static analysis of messages, but letting users tell us what's good and what's bad turns out to be very, very useful."

Petry is skeptical of this approach as a means of identifying unwanted e-mail since spam messages, like snowflakes, are unique these days. "If you have users submitting spam, by the time you update the signature, the spam has changed," he says. "What we see today is [that] truly no spam is alike. Every spam is randomized or composed in a manner that's designed to break any of those reference models."

But Google also uses reported information to assess sender reputation. "When users get spam in their inboxes and report it, that feedback goes into the IP reputation of the sender," says Coleman. "We track that over time and it turns out to be a very useful indicator of spamminess."

In conjunction with its homegrown machine learning and search systems, and sender authentication schemes such as DomainKeys and SPF, user input appears to work for Google's Gmail.

Google Gmail engineer Bradley Taylor more or less throws down the gauntlet to spammers in his assessment of his company's approach. "We have nearly perfect information about how spammy various IP addresses are based upon how often users mark and unmark spam," he explains in a research paper. "So, that's what we use. We don't need manual whitelists. If you want your mail to get through, just authenticate and behave yourself, and we'll take good care of you. And if you misbehave, we'll know that, too, of course, and take 'care' of you also."

Such bravado, however, is unlikely to deter spammers. Petry expects to see more sophisticated threats online. "I challenge anybody to find a threat that has diminished," he says.

Perhaps in two years.