Ignore the pentester at your peril
Ignore the pentester at your peril

The Black Report from Nuix is unlike most security research that comes our way, in that it's the result of talking to hackers.

To clarify, these are not black hats involved in criminal activities but rather professional hackers on the penetration testing frontline. What they manage to provide is a different perspective, that of attackers themselves.

So we learn that 88 percent of those questioned said they could infiltrate an organisation and exfiltrate target data within 12 hours, and 69 percent revealing they never get detected by the security teams that are meant to be stopping them.

Perhaps the most disturbing statistic to be included in this report is the one that relates to the clients that hire them, though. Three-quarters of clients, post-engagement, will perform 'some remediation' based upon the test report but this is usually restricted to critical and high-rated vulnerabilities. Worryingly, five percent did nothing at all, and only 10 percent put everything that was identified to rights.

The majority (64 percent) of penetration testers questioned admitted that the biggest frustration of their job was knowing that people don't fix the things that have been demonstrated to be broken.

Chris Pogue, CIO at Nuix which commissioned the Black Report, told SC Media UK that resource constraints can often lead to companies picking what they think are the most important holes to fix, based upon criticality rating levels. "This can often be misleading since many vulnerabilities by themselves have a lesser impact than when combined with other similar vulnerabilities," Pogue explains.

SC couldn't understand this application of a surface polish that leaves all the cracks underneath unfilled by organisations commissioning penetration tests. Who better to take our concerns to than the penetration testing industry itself?

Ilia Kolochenko, CEO of High-Tech Bridge, reckons that "without other security solutions and processes, penetration testing won't deliver much value" and emphasises the need to integrate such testing into all other business processes.

"Companies looking for professional penetration testing services," Kolochenko advises, "should have a look at reputable industry organisations such as CREST, whose member companies comply with technical, legal, financial and insurance regulations and best practices."

So we posed our “what's going wrong here, then” question to Ian Glover, president of CREST. He admits that "organisations often say that that they have had a penetration test without any description of the outcome or even simple pass or fail". For some, perhaps, the act of having a penetration test seems to be enough?

"We need to help businesses to understand the concept of a penetration test," Glover insists, but points to rarity of pass and fail metrics in this area. "The use of RAG (Red, Amber, Green) reports for describing vulnerabilities," he says, "can also be used to show progress where no real progress has been made to reduce risk to the business."

Glover concludes that this isn't a problem the penetration testing industry can solve, or at least not easily. "It must come from the buying community," he says. "CREST has produced procurement guides to help buyers select quality providers and has recently launched a guide to establishing a penetration testing programme and a maturity model, both free to use."

Matthew Gough, managing principal security consultant at Nettitude, also points to the variance in the severity ratings that appear in pentest reports. "Whilst the CVSS mechanism can assist in the scoring of vulnerabilities," Gough told SC, "the size and complexity of the attack surface can be difficult to assess entirely." Which leads to the client often having significant problems recasting vulnerability severities to match their own risk scoring matrices and environment.

Then there's the lack of executive management buy-in to consider. "In many organisations, security assessments are not driven from the executive level," BSI Espion security consultant Shane Ryan points out, "but are requested as part of a compliance process or the software development life-cycle." And without buy-in, those scarce resources mentioned earlier just aren't going to get signed off.

Tod Beardsley, research director at Rapid7 whose Metasploit tool is in most every pentesters' arsenal, believes that as most pentesters perform their attacks completely undetected, this should be the focus of the remediation. "This lack of useful detection capabilities," he says, "should be far more alarming than the particular vulnerabilities exploited or the network misconfigurations being abused."

Alex Mathews, lead security evangelist at vulnerability assessors Positive Technologies, spoke candidly to SC. "The main reason is money," he insisted. "Even a typical licensed software security update means stopping one or another critical business process." And the answer? "It's about clear risk presentation that highlights the concerns, in context," Mathews concludes.

Luke Potter, security practice director with SureCloud, is a penetration tester himself but in previous roles has been on the receiving end of pentest reports. He blames poor remediation on poor reporting, simple as. "Unless true impact can be demonstrated by the test company," he insists, "matters are often highlighted that are not correctly prioritised internally by clients."

Another pentester is Gemma Moore, director at Cyberis. She also looks to the reporting process, where individual systems or applications can often be tested in isolation. As a result, the issues highlighted in the report are in danger of reflecting the isolated technical impact of each individual vulnerability. "The aggregate risk of lower-risk issues when combined," Moore says, "may not, therefore, be fully understood by the customer."

Firms conducting penetration testing are in a strategic position not only to help their clients identify vulnerabilities and misconfigurations, but to help them best understand the business implications the presence of such weaknesses represent. "This sort of Business Impact Analysis goes beyond what has traditionally been labelled as a penetration test," Chris Pogue concludes, "and embodies a deeper trusted advisor relationship with the client."