It is conventional wisdom that corporate firewalls are fairly static, with well-defined rules that control what traffic may enter or exit an organisation's network. But in reality, firewall rules are constantly being changed. Enterprises typically process dozens to hundreds of firewall rule-change requests every week, totalling between 500 to 5000 changes annually. Large multinational organisations, and MSSPs (Managed Security Service Providers), that have hundreds or thousands of firewalls in their infrastructure, may process as many as 10,000+ firewall rule changes every year. Today's best firewall systems are anything but the static virtual bunker many perceive them to be. They are, rather, dynamic and flexible environments managed by multiple teams and stakeholders.
A serious challenge is that, in many organisations, the process of meeting firewall change requests is complex, slow, and expensive. It frequently involves numerous people in different organisations, requires a series of approvals and checks, and is subject to audit and regulation. For instance, both the Payment Card Industry Data Security Standard (PCI-DSS) and the international ISO 27001 standard require organisations to have controls over the firewall change process. In particular, there is the question of governance, of matching approved change requests to actual rule changes made to the firewalls. Organisations need to ensure that every rule change exactly matches an approved request: no more and no less.
The cost of a typical rule-change cycle
A typical change cycle starts with a business unit or project manager making a request. Next, the network architecture team needs to identify which devices will be affected. Then the information security team has to verify that the requested traffic does not contradict the organisation's security policy or regulatory requirements. Once approved, the changes to the firewall rules need to be planned, and then implemented, by the firewall administration team. Next, the implemented changes need to be tracked and matched to the original request, to ensure that every rule change came from an approved request, and every request was implemented exactly as approved. Finally, the whole process needs to be documented for audit readiness.
Handling one change request according to such a change cycle regularly requires four to five hours to complete – time spent by several team members, so it is has a significant cost associated with it. Table 1 shows some typical cost scenarios, under mild assumptions. Note that these work hours are usually spread over several days, and sometimes even two to three weeks, from request to implementation, since each team member typically has a queue of pending requests, and other unrelated tasks to take care of.
Table 1: Cost-saving opportunities in the firewall rule-change process
Work Hours (*)
Annual Budget (**)
(*) Assuming four work hours per end-to-end change cycle, going through the stages of: request, plan, approve, implement, verify, document, audit readiness.
(**) Budget amounts in US$, assuming a $100 hourly rate.
Cost-saving opportunities in the rule-change workflow
The high costs incurred by the firewall rule-change workflow can be reduced by adopting three major approaches:
1. Eliminating unnecessary work
2. Using intelligent decision-support algorithms to automate labour-intensive tasks
3. Streamlining and automating the information flow between the different teams
1 Eliminating unnecessary work
Information gathered from AlgoSec customers that manage large firewall infrastructures, shows that 20 per cent to 30 per cent of implemented rule changes are, in fact, unnecessary – the firewalls are already allowing the requested traffic.
Unfortunately, without proper tools it is very hard to check in advance whether a change is necessary. So firewall engineers simply write more (redundant) rules, wasting their time, and in the process, make the firewalls even harder to manage. A sophisticated firewall- and topology-aware workflow system that is able to identify such requests can eliminate all the redundant work that is normally done.
2 Using intelligent decision-support algorithms
Computer algorithms can reduce the time it takes to:
· Identify which firewalls and which rules block the requested traffic
· Provide ‘what if' risk-check analyses to ensure compliance with regulation and organisational policy
· Automatically produce detailed work-orders, indicating which existing rules to edit, which existing objects to reuse, and which to define as new
· Automatically match request tickets to detected changes and report mismatches and unauthorised changes
3 Streamlining information flow
A firewall-aware workflow system can save much of the time spent on forwarding the request between team members. Such a system knows the correct sequence of steps, so the next team member finds the request in their ‘in' queue immediately when their involvement is needed. In addition, the system stores the results of all the completed steps in a history database, for documentation and audit-readiness.
Technical requirements for a rule-change workflow system
Many change management systems, or ticketing systems, are currently available: from high-end systems such as BMC Remedy and HP ServiceCenter to free, open-source systems. Most of these offer web and email interfaces, and provide ticket queues with various levels of sophistication. In fact, many organisations already use such a system for a rudimentary rule-change request process. However, the real cost savings opportunities in the rule-change process require additional capabilities:
1. The baseline requirement for such a workflow system is that it be firewall-aware: it must integrate with the firewalls. This is necessary to extract the existing rules, and to detect any changes to the rules. In contrast, generic change management systems, that lack integration with the firewalls, have no insight into the domain-specific nature of the rule change process, and are unable to save much time for the technical team members.
2. The system must be able to support all the firewalls that the organisation uses. Large organisations typically rely on firewalls from multiple vendors, whether due to “defence in depth” considerations or due to mergers and acquisitions. Therefore, a good rule-change workflow system should provide multivendor support.
3. An intelligent rule-change system needs to be topology-aware: it needs to automatically understand how the organisation's IP address space is positioned in relation to the firewalls. Typically, very few of the organisations' devices are relevant to a particular request. A topology-aware system that automatically pinpoints these relevant devices saves significant effort that would otherwise be spent in the change planning stage.
4. A key technology in a rule-change workflow system is firewall simulation: the system needs to be able to simulate whether a given firewall will pass all the requested traffic, only some of it, or none of it – and which rules and objects are involved. This capability, in combination with the system's topology-awareness, lets the system save time by flagging unneeded changes: if the firewall simulations show that all the topologically-relevant firewalls already allow the requested traffic, then no change is needed.
5. The workflow system should automatically run a ‘what-if' risk check on every change request. This check should rely on a built-in, customisable, risk knowledgebase, incorporating general security best practice, regulatory requirements, and organisation-specific guidelines.
6. Based on its firewall awareness, the system should automatically detect all the rule changes made to the firewalls – and match them to the request tickets (and flag any mismatches and unauthorised changes). This provides audit readiness by offering the answer to questions such as “Do we (still) need rule 123? Who asked for it?” – even years after the rule was added.
7. The system must integrate with existing systems, to maximise the return on previously made investments. Moreover, if the organisation's business units are used to make IT change requests through an existing system, the new system should not require re-training of hundreds or thousands of users.
8. The system should be multilingual (even within a single organisation): rule change requests come from all the business units worldwide, so a multilingual user interface translates to greater convenience and fewer user mistakes – which all translate to reduced costs.
Discussion and conclusions
New technologies are emerging to meet these business challenges. An October/November 2009 policy management group test review by SC Magazine identifies relevant offerings from AlgoSec (it won the ‘Recommended' award), SecurePassage and others. Going beyond generic ticketing, these systems are integrated into the firewalls and are domain-specific. The more basic tools offer change alerting, possibly coupled with rule clean-up and audit support. The more sophisticated systems provide intelligent, multifirewall, multivendor decision support workflow systems.
We at AlgoSec estimate that an intelligent rule-change workflow system, that eliminates all the work caused by redundant tickets, and significantly reduces the time spent by the information security, network architecture and firewall administration teams, can save up to 60 per cent of the annual work hours spent on the firewall rule-change process. Using the estimates in Table 1, this could mean savings of $120,000 to $1,200,000.
About Avishai Wool
Prior to co-founding AlgoSec, Dr. Wool co-founded Lumeta Corporation in 2000 and was chief scientist until 2002. At Lumeta, Dr. Wool was responsible for transforming firewall analyzer technology he helped create while working at Bell Labs into a commercial product. Prior to Bell Labs spinning off the Lumeta Corporation, Dr. Wool was a member of Bell Lab's technical staff in the Secure Systems Research Department where he led a team of researchers who created the first research prototypes of the firewall analyzer. Dr. Wool is an associate editor of the ACM Transactions on Information and System Security. He has served on the program committee of the leading IEEE and ACM conferences on computer and network security. He has published more than 40 research papers and holds 10 US Patents with many more pending. He is an associate professor in the School of Electrical Engineering, Tel Aviv University. He holds a B.Sc. (Cum Laude) in Mathematics and Computer Science from Tel Aviv University, and a M.Sc. and Ph.D. in Computer Science from the Weizmann Institute of Science.