Product Group Tests

Application vulnerability assessment (2007)

Group Summary

SPI Dynamics Assessment Management Platform (AMP) is a very solid offering which builds on the foundation of WebInspect, adding enterprise use and role management. We rate AMP as our Best Buy, despite its price, for its strong enterprise management of web vulnerabilities.

For any organisation that uses a system development life cycle, Ounce 4 should be a welcome addition. It is a great CASE program and we give it our Recommended rating

Scroll To Full Group Summary Below

Click for a side by side comparison of products
Click for a side by side comparison of products

Full Group Summary

Application vulnerability assessment tools scan for web-based or SQL database weaknesses that a simple network vulnerability assessment could easily miss. Justin Peltier reports.

Port 80 is often called the highway into networks. Port 443 for secure sockets layer (SSL) is often referred to as the UFBP - universal firewall bypass port. Today, many legacy applications are either already web-enabled or are in the process of becoming so. As a result, these applications, which were never designed to be used in this fashion, are now being exposed in new ways to larger and larger user communities as well as increasingly sophisticated attackers. The process of web-enabling an application often exposes critical assets, such as large databases with personal client information.

Even some non-legacy applications designed to be web-enabled can contain significant security vulnerabilities. There is often a disconnect between web programmers, auditors and infosec staff that allows these web applications to bypass many system development lifecycle controls such as code reviews and security testing.

In many organisations these issues have been brought to the forefront by the Payment Card Industry Data Security Standard (PCI DSS). Section 6.3.1 of the standard requires "testing of all security patches and system and software configuration changes before deployment".

This is typically part of a comprehensive system development lifecycle and the term vulnerability assessment is often applied to this testing.Section 6.5 of the PCI DSS standard lists several web-based vulnerabilities to be tested by the application provider, including unvalidated input, broken access control (for example, malicious use of user IDs), broken authentication and session management (use of account credentials and session, cookies) cross-site scripting (XSS) attacks, buffer overflows, injection flaws (for example, SQL injection), improper error handling, insecure storage and denial of service.

While this is not the complete list of common web vulnerabilities that need to be tested for, it is a good representative sample.

To mitigate these risks and comply with industry best practice standards, application vulnerability assessment must be performed. This is different from the more common network vulnerability assessment because it requires greater understanding of web-based vulnerabilities.

For example, the most commonly used network vulnerability assessment utility, Nessus, checks for XSS scripting errors. However, it does not check the hundreds of different permutations of these attacks. In order to scan for dynamic attacks such as XSS or SQL injection, a utility with greater understanding of the application environment is necessary.

The solutions in this group tested for either web-based vulnerabilities or those inside of a SQL database. All products had the additional intelligence to scan beyond the depth a traditional network vulnerability assessment utility could.

Products in this group broke down into one of two categories. The first assessed the web application itself, while the other tested the database manually. Pricing in this category ranged as much as the overall function. With products that started below £500 to those with a price tag upwards of £35,000, the range was truly surprising.

How we tested
We tested the applications by installing the utility on a Windows XP professional machine with an AMD 64 bit 4.0 Ghz processor, 1 GB of RAM and 100 GB hard drive. Next we ran the utility against a small PHP-base website with several minor vulnerabilities. The website used custom error pages that can throw off many of the spider features of application scanners by redirecting all bad web requests back to the site's home page. This is a common first step in securing many web servers and is deployed by most major organisations.

For a tool in the review that used a crawling engine to interpret the results correctly, the crawler had to distinguish between the returned custom error page of 302 - page moved as opposed to a 200 message for page found. Not all scanners were able to make this distinction.

All products were scored on ease of use, number of pages discovered, if vulnerabilities were sorted by class of vulnerability, an ability to report false positives to the manufacturer, the number of false positives found, the time the scan took to complete, the number of vulnerabilities uncovered, the types of reports offered, if remediation steps were included with the report, and if the product uninstalled cleanly.

- For details on how we test and score products, visit

All Products In This Group Test