NIST Updates Software Testing ToolNIST Updates Software Testing Tool

Best Government Websites

The National Institute of Standards and Technology (NIST) has updated a two-year-old software bug-detection tool with a new user interface and a new feature to improve testing efficiency.

The latest release of the institute's Advanced Combinatorial Testing System (ACTS) -- which uses a NIST-developed approach to testing -- includes a new feature that specifies relationships between parameters being tested. The feature can help testers eliminate unnecessary evaluation of possible code interactions that aren't valid, such as using Microsoft's Internet Explorer on a Linux system, according to NIST.

The release also includes a new tutorial for using the tool called Practical Combination Testing, a method developed by NIST researchers that ACTS employs. The testing method checks for bugs caused by the interaction of between two and six variables.

The tutorial includes concepts for using software tools to generate combinational tests and was written so even undergraduates in computer science or engineering can understand it, according to the institute.

Developed in 2008 by NIST researchers and available for free, ACTS takes a unique approach to bug testing by leveraging NIST research that found nearly all software failures are caused by only six or less variables.

Much of the time, software testing is done by checking combinations of input actions in pairs, and by testing as many of those pairs as possible. However, researchers at NIST said this isn't practical because it's time consuming and also could itself crash an application.

NIST researcher Rick Kuhn, working with others from the institute, found that between 70% and 95% of bugs are the result of only two variables interacting, and nearly 100% are triggered by no more than six. Kuhn worked with researcher Jeff Yu Lei and students from the University of Texas at Arlington to develop ACTS based on this concept.

NIST provided an example to illustrate the difference between traditional bug testing and ACTS. Testing all possible interactions for a product with 34 on and off switches would require 17 billion tests, according to the institute. However, ACTS can test all three-way interactions using only 33 tests, and all six-way combinations with just 522 tests, numbers that are far lower than 17 billion.

Since 2008, NIST has delivered ACTS to 465 organizations and individuals – including academics and governments. In addition to IT organizations – which comprise about half of ACTS users – the tool also is popular in the financial, defense, and telecommunications sectors, according to the institute.

NIST currently is working with Lockheed Martin to study how the company uses ACTS to test its software applications and will publish the results when research is complete.