Developers' Tool Improves Open Source Security, Trims DefectsDevelopers' Tool Improves Open Source Security, Trims Defects

Source code scans, such as those being performed on open source projects for the Department of Homeland Security, have become an important new tool in eliminating bugs at many of the projects, open source developers say.

At the same time, the scans are illustrating that average open source projects are, well, average, when it comes to creating bugs. Commercial code and some of the open source projects under review are showing one code defect or security exposure per 1,000 lines of code. But the best projects are showing a defect rate far lower than that. They're also illustrating how open source, unlike some commercial teams, is willing to air its defects and clean them up quickly.

"We log into their Web site each week and see what they've found," said Jeremy Allison, lead developer and co-founder of the Samba open source project. The code scanner, Coverity's Prevent SQS, "has got an inhuman eye for detail. It's like having the most persnickety programmer in the world looking over your shoulder," Allison said in an interview. Samba is included with most distributions of Linux and converts a Linux server into a machine that can talk with Windows.

The Samba team has fixed 228 bugs found by Prevent SQS and still has 11 findings to review and determine whether they're real bugs. The team inspects each finding because Prevent SQS returns false positives, and Samba developers notify Coverity when an alleged bug is actually good code so it can improve the tool's scanning ability.

"Their false-positive rate is low enough for it to be an extremely valuable tool," Allison added. The findings were complicated by the fact that Samba switched its change management system from Subversion to GIT, and broke the scripts that Coverity used to download the most recent builds or compilations of Samba. For a while, no bugs showed up because Prevent was rescanning previously fixed code. Then it reached the right repository and many days of new development showed up with a bunch of new defects.

But on the whole, "I was quite pleased with what Coverity said about us," Allison said. The scanning results show Samba with a defect rate of 0.024 per 1,000 lines of code instead of the average 1 per 1,000.

Not everybody has felt that way, as results of the scans have been aired by information. "This story is just free and open source software bashing," said a reader in a submitted comment. Many readers wanted to see a comparison of open source to commercial code, but proprietary software companies are secretive about their defect rates. "Seems seriously slanted," said a commenter. "Steve Ballmer, is that you?" said another. But the scans show a lot about how open source projects are run. When the scans were first conducted, starting in March 2006, some projects were pained to discover a new workload consisting of dozens of bugs on their hands.

Spring, the popular Java development framework, has not been asked to participate in the scans, but lead developer Rod Johnson said an unexpected report from Coverity, listing hundreds of alleged bugs, "could overwhelm many projects. This kind of load could be a real issue for projects reliant on volunteer effort, where developers would need to give up evenings or weekends or change their personal schedule to deal with it."

Allison agreed, saying, "The first time it happens, it's awful." But he also said that once the team has fixed "all the gross errors, it becomes part of your daily routine" and leads to cleaner code.

Johnson said Spring hasn't been asked to participate in the scans but said "we would be open to it."

The PostgreSQL database project was another open source effort that showed up as exceptionally clean, with an error rate of 0.041 per 1,000 lines of code in a project with 909,148 lines. But lead integrator Bruce Momjian said that's because it has a lot of those persnickety developers who are willing to pay attention to everybody else's details. The Coverity scans are improving the overall quality of the code, he said.

The FreeBSD project found Coverity's scans useful enough that it brought the Prevent product in-house and conducts the scans itself, adjusting Prevent to reduce the false positives it finds. False positives are the bane of code scanners and reduce their effectiveness. A target program is scanned as static source code -- it's not running -- which allows the scanner to conclude that certain code conditions that could never occur in a running program are nevertheless a defect, just like one that could. Prevent SQS tries to eliminate such false positives.

Coverity, however, failed to say that FreeBSD also was scanning and cleaning up bugs on its own server rather than the Coverity site that lists the results of its scans. So FreeBSD shows up on the Scan Coventry Web site with 605 bugs, with only six verified and none of them fixed.

FreeBSD Security Officer Colin Percival wrote to information: "The analysis performed by the FreeBSD project -- fixing, by my count, about 200 bugs in the past four years, and marking many more as false positives -- isn't reflected on the Coverity Scan website, leading to confusion such as yours." Likewise, members of the Firebird versioning open source database project, which is listed with 195 unverified and unfixed defects, were quick to point out that the errors were actually resident in a piece of auxiliary code used for character set conversion, the International Component for Unicode. That project is sponsored and supported by IBM and is not under Firebird's control, pointed out Paul Beach, a Firebird project developer and administrator.

"The Firebird project reviewed the Coverity results almost immediately when they were published [in March 2006] and found that the report isn't actually related to the Firebird engine," Beach wrote in response to information's article.

The scanner being used to check open source projects for code defects is the product of four years of research at Stanford University. In November 2003, much of the research team left to found Coverity in San Francisco. The team included Andy Chou, now chief scientist at Coverity; Benjamin Chelf, now CTO of Coverity; and Dawson Engler, Coverity co-founder and associate computer science professor at Stanford. What Prevent SQS has done in the first phase of its Department of Homeland Security project is perform 12 checks on the open source projects selected for review. They include forward nulls, where a pointer variable has a value of zero, referring to a nonexistent memory space and causing the program to crash; negative returns, where a function returns a negative value that is not expected and causes unpredictable results, such as using a negative value to index an array, which places the array in an unpredictable memory space; and dynamic overruns, where a piece of dynamically allocated memory is written to, but its size is not respected, said David Maxwell, Coverity's chief open source strategist.

Prevent also checks for memory leaks, where memory is allocated to create a software object but never reclaimed for the system when the object is disposed of. It "sizechecks" or looks for a pointer that has been cast as a data type too large for the memory assigned to hold it, causing a memory overwrite. Eleven open source projects have been scanned for the 12 defects and cleaned up the resulting discovered defects sufficiently to advance to a second "rung" or new phase of Coverity checking. They were: the Perl, PHP, Python, and Tcl scripting languages; Samba; Amanda backup and recovery project; NTP, the Network Time Protocol, which coordinates correct timing between two dissimilarly timed systems; OpenPAM, the open source method of aggregating multiple user authentication schemes; OpenVPN, the open source VPN; and Overdose, a Yahoo chat client.

Coverity called attention to the 11 as it announced an advanced round of checking for those projects that had completed two rounds of checks. Rung 0 and Rung 1 completed 12 defect checks. The Coverity checking engine is now capable of 60 checks, but not all of them will be immediately applied to the 11 projects. There will be Rungs 3, 4, and 5 as well. Maxwell said Coverity is trying to pace the amount of defect information it throws at open source projects so that they don't get distracted or overwhelmed by the lists of what the Prevent checkers are finding.

The results being cited come from scans automatically conducted on each build of an open source project, which occurs at least weekly and often several times a week. What the scan results don't show is the number of false positives in the supposed bug list. Until developers on an open source team check out each Prevent finding, it may or may not be a bug.