False positives may be a nuisance in vulnerability testing, but false negatives are the ones you should worry about.
Anyone who has had a vulnerability scan or weak penetration test will know all about so-called false-positive vulnerabilities. The term is used to describe a test case reporting that a vulnerability is present, when in fact it isn't.
These are a real annoyance for security managers, as one cries “wolf” in order to have the vulnerability patched, then looks foolish when the infrastructure support or application development teams come back to say “it's not there”.
Any good penetration tester will go to great lengths to verify the presence of a vulnerability. Part of the problem is that many vulnerabilities require exploitation to verify their existence when testing remotely, and exploits have a habit of causing system instabilities.
The worst offenders for false positives are the many vulnerability scanners on the market. Comparatively cheap, they are good at providing ongoing assurance, but terrible at dealing with anything other than the most common, easily tested infrastructure vulnerabilities. Many readers will have had to go through the pain of a PCI approved scanning vendor (ASV) quarterly scan; for which misclassified vulnerabilities that create unnecessary PCI fails are all too frequent.
However, all of the above misses the point. The effect of crying wolf is well known: the threat is ignored when it finally occurs. But what if you didn't even know there was a wolf in the first place? This is the false negative; when a scanner or poor pen tester misses a vulnerability altogether. It's not found, you're not alerted and don't even know the problem exists.
Scanners are weak at detecting blind SQL injection accurately. They're also poor at most application layer vulnerabilities. Automated attacks don't care about that; they are excellent at trying the same attack on millions of websites. If a few work, the worm has done its job.
So how are you supposed to find out if the service you're using is missing things?
We recently set up a “honeypot” vulnerable web server and application, and a number of the PCI ASV scanning companies ran evaluation tests against it. The results were depressing: nearly all of the application layer vulnerabilities were missed, including the SQL injection and cross-site scripting vulnerabilities that result in PCI non-compliance if found.
Significant issues such as session stealing, privilege escalation and many others simply were not detected. Even worse, most had incorrectly implemented the new CVSS vulnerability scoring, therefore failing merchants for irrelevant vulnerabilities.
The result is a large number of online retailers perceive that they are PCI compliant because their quarterly scans say so. It's nice to have that “tick in the box”, but not so great when you find your customer database has been compromised. Given the list of high-profile websites that have been hit recently, there must be a few security managers out there wondering how they were hacked.
“The scanner report said we were secure” is a fairly thin defence, so cross-checking is the only solution, whether through sourcing multiple scanners or testing by hand during a penetration test. We also find numerous pen testers are just running scanners, which rather defeats the object. Look for accreditations such as CHECK and CREST as a minimum; at least then you have some assurance that the tester's ability has been checked by a third party.
Ken Munro is director of SecureTest, the penetration and security testing division of NCC Group.