Mozilla is trying to refute the notion that the buggier the software, the less secure it is.
The open-source maker of the Firefox browser is creating a publicly available metrics model that software providers can use to measure the relative security of their products.
Mozilla itself will leverage the model to measure the efficiency of both its development process and the response by Mozilla and its users to Firefox security issues, said independent security consultant Rich Mogull of Securosis, who was tapped to lead the project.
“This is less focused on providing a public number,” Mogull told SCMagazineUS.com on Monday. “It is more about how Mozilla themselves can better track their security effort. The goal of these metrics is to improve Mozilla's ability to understand how they perform security development and respond to security issues and keep their users as safe as possible.”
Window Snyder, chief of security at Mozilla, said she has wanted to achieve this since she arrived at the company about two years ago.
“One of the things I've always wanted to see is a more refined approach to evaluating the security of a project over time,” she told SCMagazineUS.com.
That means limiting the emphasis on traditional risk criteria, such as vulnerability counts, she said. At Mozilla, for example, the public is encouraged to find bugs in Firefox, which naturally drives the numbers up.
Microsoft, on the other hand, does not publicly disclose all of its flaws; in fact, sometimes the software giant includes fixes as part of a service pack update and never reveals vulnerability details.
“You never know if you're comparing apples to apples,” Snyder said.
Mogull said basing security on bug counts is flawed because few vulnerabilities are publicly exploited, and most are unknown until the moment a patch is released.
“Just how many bugs make it into something doesn't necessarily measure the security and may or may not give you any indication how well your security development process is,” he said.
Mogull said the model – a preliminary version (xls) has been released and users are encouraged to supply feedback – will be based on statistics that allow Mozilla to study things such as when a bug was found in the development lifecycle, which tools were used to find it and how quickly users updated to the latest patch.
Snyder – who admits there is no way to achieve an “absolute level of security” – said the first step is to create a baseline model that will let Mozilla track improvements or declines over time and determine the cause.
She said the model can be used by other organizations trying to bolster security.
"They don't have to reinvent the wheel,” Snyder said. “They can use it in their own environments and modify it for their own security projects.”