Don't give away your brand
Don't give away your brand

What's the point of assessing system vulnerabilities at a single point in time? We need a more holistic approach.

A quick search on Google for "penetration testing" yields many, many more results than for any other testing method. From this it would seem that pen testing is the key method to assess information security.

But does it deserve that status? Today, penetration testing is well understood by those shopping for security testing, and is well served by the market. Key drivers include the requirements for testing set by bodies such as the Payment Card Industry (PCI). So, if you want to measure your IT security, you can't go wrong with a penetration test. Can you?

As typically practised, the process has a number of weaknesses. The results of a penetration test are often used to yield some objective assessment of the system's level of security - holding up a yard stick against which security can be measured. Which is fine, if you know that your stick is actually a yard long. And this is the key failing: the test is as much a measure of the abilities of the tester as it is of the security of the devices being tested.

If the test shows that your system has no vulnerabilities - is it just because the person doing the test has run nmap and an out-of-date version of nessus? Even the best security testers won't claim to be better than the hackers. There are a lot of people spending far longer developing ways to attack systems than the few hours a pen test takes.

Another problem is that a penetration test is usually run from one or two external IP addresses. This gives a very narrow focus, potentially missing out key attack vectors: VPNs, "special" address ranges given extra access and so on. The criminal won't be using a single IP address. An attack could come from an unsecured wireless access point on your WAN, from a partner or from a laptop brought into the network.

Run a vulnerability assessment tool against a server. Then turn that server off. Repeat the test a month later, the results will be different. More vulnerabilities will have been discovered. This is the third key failing of pen tests. They can only give an assessment of security at a single point in time. What about next week, when that test box is placed on the DMZ?

Finally, a penetration test is run on systems that are live, after they've been designed, developed and implemented. The test is often the final confirmation, provided by a third-party expert, that all is well. Hindsight will reveal that it would have been better to find the problems - and cheaper to fix them - before the system went live. Cost is usually the key reason given for deferring external security testing to the final production system. And security testing really does need to be done by a third party without assumptions about the way the system should behave.

So what's the solution? The first step is to spread your budget for third-party testing across the life cycle of the system. If you're developing code, involve the third party as early as possible, even at the design stage. If you bring them in to assess the code, it doesn't have to be a full review. A simple code audit - sampling a range of the code - will suffice; the issues raised will be generally applicable to the rest of the code. If you're not developing code, involve the third party at the systems integration phase.

For the penetration test, the issues described above can be reduced or eliminated by assessing the security architecture and matching testing to that. A static review of the actual configuration of the key security devices should be performed, in addition to traditional dynamic network testing.

"What-if" testing can also be conducted. This will determine vulnerabilities that would be apparent if a security device failed or its configuration changed. Finally, all of these approaches should be backed by a focus on identifying, quantifying and managing risk - rather than a simple assessment of vulnerabilities - forming a total approach to security testing.

I'm hoping that, one day, a search for "total security testing" will bring more hits than for "penetration testing". Ian Castle, CISSP, is a senior consultant at ECSC and heads the internet defence division.