As more agencies explore a new form of database technology called in-memory databases, contractors and other technology companies are stepping up with solutions.
In-memory databases promise faster transaction speeds than standard relational database management systems.
Raytheon Co. of Lexington, Mass., is incorporating two such databases for some of the shipboard electronic and combat systems of the Navy DDG 1000 Zumwalt Class Destroyers, which the integrator is helping Northrop Grumman Corp. build.
And last month, In-Q-Tel, the private venture capital firm created by the CIA, made a strategic investment in StreamBase Systems Inc., a Lexington, Mass.-based provider of in-memory database software and associated analysis tools.
In-memory databases are optimized for use in the working memory of machines. Usually, databases are stored in main memory, typically on hard drives. When new material is generated, it is written to disk first, and when a query is made of the database from a program, that material has to be read off the disk.
In contrast, in-memory databases reside entirely in the working memory, or RAM, of a server or cluster of servers — though they can be archived on disk. Material is only written to disk later, if at all.
“The working dataset that the application will be using is resident, or persistent in memory,” said Patrick Moor, head of government contracting and manufacturing for Ants Software Inc. of Burlingame, Calif., one of the companies chosen for the Raytheon work.
The other company was TimesTen, another inline database company now owned by Oracle Corp., of Redwood Shores, Calif.
RAM works faster than hard drives, although it also is far more expensive on a per-byte basis. It also is volatile, meaning the data is lost once the power is shut off. But because these in-memory databases reside in RAM, they are generally able to ingest hundreds of thousands of transactions per second. They also can be queried more rapidly.
“For many applications where you need to capture, react to and analyze that data instantaneously, a database is just too slow,” said Bill Hobbib, vice president of marketing at StreamBase. With a traditional relational database management system, “you are storing the data before you query it. We can query the data at the moment it arrives.”
In tests, StreamBase has shown that its software can ingest as many as 500,000 messages per second, whereas an RDBMS can, at most, take in about 3,000 messages per second, Hobbib noted.
For its destroyer work, Raytheon’s Integrated Defense Systems was looking for a database that could ingest a lot of information from radar and sonar systems. Raytheon even generated an acronym to describe the environment, CRUD — create, replicate, update and delete — said Paul Rivot, a director of competitive technologies at IBM, which is supporting Raytheon’s work.
Traditionally, to tackle the problem, government contractors would write a custom program that would run an entire database in working memory in such a way that new material wouldn’t be written to disk first. “It was so expensive, and you would have to custom develop it for each application,” Rivot said. “Using an off-the-shelf memory and database product, it is obviously cheaper.”
The Ants software combines an in-memory database with regular RDBMS, so material can be stored through regular SQL commands. For its own software, StreamBase also kept close with SQL as well. It keeps the basic syntax, programming primitives and declarative nature of SQL. But the StreamBase software extends the language with additional capabilities, such as handling data that arrives out of sequential order, matching complex sequential patterns and detecting patterns over periods of time.
A database programmer could learn StreamBase extensions in about a day, Hobbib said. No standards body oversees the company’s extensions, but StreamBase is looking into that possibility.
In-memory databases are not the only way to accelerate transaction and analysis speeds. You could also make the hard drive much faster.
Texas Memory Systems Inc., of Houston, offers hard storage systems that appear to an operating system as hard drives yet consist entirely of much-faster RAM units. When outfitted with a standard RDBMS, such solidstate drives could process hundreds of thousands of transactions per second, said Woody Hutsell, executive vice president of Texas Memory Systems. Each unit can hold as much as 128G, and they can be tethered together for more capacity.
Washington Technology associate editor David Hubler contributed to this story. Joab Jackson is assistant managing editor for technology with Government Computer News.
