Product Group Tests

Policy management (2008)

Group Summary

The Best Buy goes to the NetIQ Secure Configuration Manager 5.7 for its good risk-based approach for managing known weaknesses in configurations, patches and other host-level vulnerabilities.

The BigFix Enterprise Suite 7 represents great value given the number of platforms that it supports. We award it our Recommended award for its extensive feature set and competitive cost considering its capabilities.

Scroll To Full Group Summary Below

Click for a side by side comparison of products
Click for a side by side comparison of products

Full Group Summary

These tools can help organisations ensure compliance with regulations and best practices as we see continuing improvements in reporting capabilities. Nathan Ouellette reports

Like other tools that help manage security, policy management solutions have evolved into enterprise-class systems with centralised consoles, web-based tools, auditing, remediation, compliance reporting and a host of other features. The effectiveness for each offering within this group is based on how well they integrate into a specific environment and what types of tasks they can help with. Also, customers are demanding dashboard views of the enterprise, so reporting mechanisms, including scheduled and disseminated reports, are becoming more important within this space.

We examined two general classes of products; solutions that help manage workstations/servers and those aimed at network devices (firewalls, routers, VPNs, IPS etc). We did not review any tools that had the capabilities to perform both functions. Both classes can help with particular issues an organisation may be struggling with.

The tools in this group all performed well and used the same basic philosophy: decide which devices should be managed, add them to an inventory, create or use policy templates the devices need to be compared against and then monitor or actively push configurations and remediation to the devices while providing reports. There are subtle differences to how each product achieves the results, but the end result is mostly the same. Assets are held to a standard based on a series of regulations, compliance mandates or best practice items. It's up to the business to decide which devices can remain out of compliance or which devices need remediation.

How we tested
All of the products in our group review were installed on either Windows XP Professional SP2, Red Hat host machines or Windows 2003 SP2 servers with MS SQL 2005 and MySQL databases. Networking configuration testing was run against multiple vendors including Juniper Networks and Cisco. We tested our configuration management against firewalls, VPN devices, routers, switches and even some security devices.

We deployed agents to Windows and Linux devices for our workstation/server policy management tools and added devices to our network configuration management inventories. Surprisingly, we did not have any issues with installing agents and deploying configuration and policy for any of our Windows or Linux machines. In all fairness, we expected some Windows agent problems, but were pleasantly surprised when we didn't encounter any. Devices were all managed effectively and we were pleased with the capabilities of all products. The only noticeable performance hit was when we processed large amounts of configuration or policy change for many Windows devices at once. This is definitely a consideration for adding more horsepower to a dedicated policy management server.

From a reporting perspective, some products produced better reports than others. Those with a risk-based approach or compliance templates and reports scored highest. We feel these features are superior as opposed to simply focusing on a gap-analysis and whether you're in compliance or not.

All of the products were scored on our typical criteria of support, documentation and price. But we also considered ease of administration and configuration, ease with which the application creates and disseminate policy, support for a wide range of vendor operating systems or vendor models as well as reporting and remediation capabilities.

Even though the products performed as we would expect, decision makers will have their work cut out for them when deciding how many devices should be managed. Some product vendors differentiate between "managed" and "monitored" devices in their pricing schedules. Before spending money on a policy management system, buyers should ask detailed questions about the vendor's overall vision and commitment to the long-term roadmap of the product.

Unfortunately, we witnessed some disappointing and continuing trends with vendors and products that experience acquisition. This often bleeds into the overall confidence of the product, which we're not sure vendors are aware of. Some of our products suffered from a bit of an identity crisis in their branding and even product names. Customers may want to avoid solutions that have changed hands several times in the last few years.

All Products In This Group Test