Specialized Database Captures, Analyzes Time-Sequenced DataSpecialized Database Captures, Analyzes Time-Sequenced Data

Historis, a specialized database system from Logical Information Machines, is a system designed to do one thing well: deal with time series data, such as stock prices or a patient's vital signs over the course of a day.

If you need to know the price the price of GE common shares at 11:59 a.m., 12:59 p.m., and 1:59 p.m. on a certain day, you can get the information from the enterprise Oracle or IBM DB2 relational database, but it will have to send the data to a data warehouse or invest in building a data cube with different views of time-stamped data.

You can get the answer faster using 64-bit Historis, according to Tony Kolton, CEO of Logical Information Machines and a former Chicago Board of Options Exchange trader. It can time-stamp data as it arrives down to a thousandth of a second, and it has built-in analysis tools to look at sequences of data within a given time period. Relational databases can do these things as well, but Historis is built differently than the rows and columns of a relational table. It is more of a hierarchical database, with data flowing in and stored as large, sequential files, with metadata captured about the data as it arrives.

Kolton claims the system can insert 250,000 records a second and extract 220,000 records a second. Its performance "stays the same, no matter how many time series you have in the database," he said in an interview.

Data can be time-stamped down to one thousandth of a second, although each piece of data isn't literally labeled with a time. Rather, the time of data acquisition can be calculated for masses of data using metadata that reflects the pace of data capture. Once a formula is captured, the time of arrival for masses of data can be calculated using the metadata, he said.

Historis data schema store disk locations of where timed series of data are stored, speeding the retrieval process. Different time series can be retrieved without going through the process of executing lengthy relational database joins, where SQL pulls together data from different tables in the database. Historis can manipulate 2 to 3 terabytes of data in near-real-time without performance degradation, said Stephen Johnston, LIM's director of server software.

It has its own query language, MQL, which uses English-like statements to ask questions of the database, without requiring programming skill or knowledge of relational's SQL query language. Users still need to master a basic command set, such as Show, When, and the Boolean operator, And.

Kolton wanted a timed-series database as an options trader after going through the stock market crash of 1987. He emerged more or less unscathed, but the trader next to him lost $12 million over the course of an hour. He and other traders needed a way to analyze incoming data closer to real time.

"There's intelligence hidden in a sea of data," but by the time traders analyze it, it tends to be too late for active traders to do anything with it, Kolton said. Historis makes the information more useful to parties forced to do their jobs in real time, with information streaming at them from their environment.

He founded Logical Information Machines in his garage in Chicago in 1988. His trading experience may be one reason the new system has found traction at some financial services and trading firms, including Chevron, ConAgra, Goldman Sachs, Fidelity Investments, Pacific Gas & Electric, and UBS Securities, according to Johnston.

Historis was launched by LIM in 1990; the 64-bit version of the system was introduced at the end of July and runs under Solaris and Linux. Historis is priced at $67,500 per processor for a lifetime server license. A Solaris or Linux server running Historis requires a minimum of 1 Gbyte of memory and 50 Gbytes of disk.

A data warehouse service based on Historis, Market Information Machine, can be linked to a stock market, energy trading, or financial instrument database and leased at a rate of $5,000 a month.

Oracle offers a real-time database system -- TimesTen -- that works in conjunction with its flagship Oracle database. It gets its high-speed characteristics by operating in a server's random access memory, avoiding the latencies of going to disk.