A Challenge For The AgesA Challenge For The Ages

Three years ago I talked to an IRS official about an ambitious project that would make it possible for people in the agency to instantly retrieve individual and business tax records. It was a conversation that taught me a lot about what a hugely difficult task data integration can be. Now, scores of government agencies on the national, state and local levels are trying to find ways to share and analyze data that relates to homeland security. If data integration at the IRS was a hard task, comprehensive integration and analysis of security data across all levels of government is going to be absolutely Herculean.

The database standardization project at the IRS, the official told me, involved pulling together information that resided in virtually every storage format known to mankind -- including mainframes, reel-to-reel tape, digital audio tapes and optical storage devices, among many others. The repositories used different means of data security, they were physically far-flung, and their ability to communicate with applications and each other was uneven or nonexistent. At the time, the agency projected it would take 10 years and $10 billion to complete its planned IT modernization initiatives.

Three years later, every time I see something on the news about efforts underway to make various security agencies work together and share information, I reflect back on that IRS interview. Homeland security involves many government departments and agencies, not just one. It involves many levels of government, not just the federal level. If the IRS's legacy systems were varied, the systems in place at the many entities that contribute to homeland security give new meaning to the word "diverse." I haven't even mentioned the varied formats of the data itself. And that's just the integration part: If the IRS's project involved actual analytics, it was trifling compared to what U.S. security imperatives will demand.

Some agencies, at least, are working to meet those demands. We hear about it piecemeal, but real efforts are underway. Law enforcement agencies around Washington are using wireless technology to integrate data from various jurisdictions. The FBI is deploying analysis software in its anti-terrorism efforts. And more money has been earmarked to secure the nation's ports. Not all the news is good: The Government Accountability Office said last month that the FBI lacks a coordinated plan to modernize its IT systems, and a report this month from the Department of Homeland Security's Inspector General office criticized the department for "not playing a lead role" in consolidating terrorist watchlist information.

The things we learn from the news media will inevitably make up only a tiny fraction of the whole story. That's as much a comment on the sheer scale of the job at hand as it is on the limitations of the media. As anyone familiar with data integration knows, there's a lot more to it than meets the eye. As anyone familiar with security knows, it's usually not a good idea to show your hand to your adversaries.

For all its infuriating inefficiency, arbitrary protocols and catty interagency rivalries, the U.S. government still managed to successfully partner with the private sector to build the Hoover Dam, the Liberty Ship fleet, the national highway system and the Apollo 11 lunar module. It absolutely is no exaggeration whatsoever to say that constructing a comprehensive means of accumulating, sharing and analyzing its myriad security data will be a task that rivals those. It will be carried out more quietly than the accomplishments that have preceded it. But it's got to be done.