Log management may be seen as a purely on-premise solution, but following LogLogic's new partnership with VMware, its chief marketing officer Bill Roth looks at the potential future of logs in the cloud.

A well-known maxim proclaims that ‘knowledge is power', but where do we get our knowledge about information technology components such as computers, networking gear, application frameworks, SOA web infrastructure and the like?

The richest sources of such information that is always available, but often overlooked, are the logs and audit trails that are produced by these systems and applications. Through these, information systems often give signs that something is amiss, or an event logged in the log files provides insight into future problems.

Logs can also reveal larger weaknesses that may affect regulatory compliance and even IT governance and by extension, corporate governance. However, more often than not, it is difficult to extract information from log files and distil the data into useful and usable or actionable information.

Basically logs equal accountability. There are many other mechanisms for accountability in an organisation, but logs are the most common and most prevalent. To be clear, if your IT staff are not accountable, your business is not accountable.

Unless you take logs and the practice of managing them seriously, you may be sending out the message that your organisation shuns accountability. Along the same lines, logs are also immensely valuable for meeting regulatory compliance.

The inexorable trend toward server virtualisation makes it possible to combine multiple diverse systems onto a single hardware platform. Virtualisation also simplifies server provisioning. It all sounds good, but what happens to logs, logging and log management when IT environments are virtualised?

Virtualisation platforms present new sources of logs to manage. In addition to having new log information to collect and analyse, new challenges to logging and log analysis arise, such as the potential need to review access logs collected while virtual machine images were inactive.

In addition, new opportunities for log management are also present, such as ensuring new virtual images are pre-configured with central logging capabilities. There may be ways to use logs to solve new problems, such as monitoring health and uptime status of virtual platforms and application stacks. The ubiquitous nature of log management allows the development of new operational, security and compliance solutions for virtual infrastructures using the tools we already have.

First, let's review what stays the same. A virtual server is still a server, complete with operating system and applications and logs must be collected, retained and analysed, just as they do in ‘physical' environments.

IT infrastructure with virtual platforms, hosts systems and guest systems are largely the same as those with all physical elements; with all the usual logging that needs to be managed. Similarly, networking between guest systems running on a single virtual platform resembles networking between physical machines and needs to be monitored and audited just like on a physical network.

In a virtual environment, servers are still provisioned, modified and configured by system administrators and of course accessed and utilised by end-users. Such activities create audit trails that are collected and reviewed in just the same manner that physical environments are.

So the advent of virtualisation is not a reason to throw away tools that work for you in physical environments. They will continue to deliver value and help your IT and business to operate efficiently, be secure and compliant with relevant regulations, especially given the fact that the future belongs to a mix of physical and virtual environments.

On the other hand, virtualisation has brought a lot of new technologies, as well as new problems for IT departments to solve. Such problems might not have any equivalent in the physical world, where ‘a server' always meant ‘a piece of hardware' plus ‘an operating system' plus ‘one or more of user applications' running on it, a worldview that virtualisation is making obsolete.

Rogue virtual machines pose a unique security problem. If users provision their own virtual machines and their own guest systems, tracking such activities across the organisation presents a worthy challenge. For example, if an unauthorised application runs in its own virtual image, enforcing the security policy becomes harder since endpoint monitoring tools might not see through the virtualisation veil.

Rogue machines deployed in the cloud present the ultimate challenge of this type. If a system resides on somebody else's virtual platform in the cloud, the chances of getting evidence of activities on such systems becomes next to impossible.

At this point it should be clear that changes that IT staff must face as virtualisation becomes a reality in the data centre are indeed massive. For IT staff tasked with logging activity across the infrastructure, these changes can be good, bad or ugly:

  • They're good because it is easier to provision systems that centralised logging already enables. IT staff can also retrofit other systems by adding logging to the virtual image of that system.
  • They're bad because there are new logs to collect and analyse and new activities to track and monitor. Virtual machines must be closely watched for availability and security issues and to ensure they comply with policies and regulations.
  • They're ugly sometimes, because unmanaged virtual machines can pop up on the organisation's systems or even in the cloud, violating IT policies and presenting significant enforcement and investigation challenges.

In addition to being affected by it, logging and log management can also augment virtualisation projects, especially in the areas of security, compliance and manageability.

In security, logging creates a trail of accountability for users and especially, those privileged to access the underlying hypervisor. Tracking access to virtual machine hosts system and inactive guest images creates a trail that can be used for monitoring and auditing, as well as investigations for cyber crime or insider abuse. Perusing logs for security-relevant failures, such as missing controls, unauthorised access or unapproved changes is just as helpful in a virtual environment as it is in a physical environment.

Recent mandates such as PCI DSS require logging, log collection and retention, log analysis and review and log protection. For example, logging is requirement ten of the PCI DSS requirements, whether the environment is physical or virtual. Hence, logs from virtual machines must be given at least as much importance as logs from physical environments.

In manageability, administrators and system operators benefit from logging as well. Monitoring for failures and errors, as well as general virtual machine health is not possible without effective log management.

Along with all the promise and benefits of a virtual infrastructure comes significant change, requiring new ways for organisations to collect and manage logs. However existing tools and log management appliances can still be leveraged to address these new logging challenges and to optimise, secure and bring into compliance newly virtualised IT infrastructures.