Mark Fabro, Lofty Perch
Mark Fabro, Lofty Perch

Industrial control systems can be so fragile that the mere act of investigating a malware infection can be enough to crash the system and bring the supported system to a halt.

That was the message from Mark Fabro, a forensic investigation expert and president of Lofty Perch, speaking at the 4SICS industrial control system conference in Stockholm last week.

Fabro's company has specialised in investigations of ICS systems since its formation in 2005. He co-authored the US Department of Homeland Security Industrial Control System Computer and Emergency Response Team's recommended practice document along with another ICS cyber-security expert, Eric Cornelius, director of critical infrastructure and industrial control systems (ICS) at Cylance Inc.

Many industrial control systems were built at a time when there was no expectation that they would be connected to the internet, and before a time when malware was as active as it is now. Viruses that would otherwise be benign in the IT world can have extreme, catastrophic consequences in the control system domain, Fabro told

“Simple denial of service, zombying or botting – viruses that we can get in IT that are not much more than a nuisance, when you see the tactical implementation of those viruses and the kinetic impact they can have on these older systems, it can be very bad,” he said.

Fabro has extensive experience investigating malware on active production systems, and the biggest challenge he has found is the need to work on systems that cannot be taken out of production. As a result, many traditional IT forensic techniques can't be used because they require you to work on systems that can be isolated, imaged, bagged and tagged.

“We have to have a live analysis of the system at that time and do a comparative analysis against what we expect the norms to be in the systems,” he said.

ICS also have protective measures built into them to help prevent piracy. When these systems detect suspicious activity which could be an attempt to copy the software, they shut down.

They often have to work within times and dates set by the system owner, so it's not unusual to be working from 2am to 5am to avoid affecting vital systems, under the gaze of a system engineer to ensure it continues to function.

For instance, in a public transport environment – a typical 24/7 operation – it was impossible to take the system offline altogether but working in the wee hours of the morning meant that if a system were to shut down as a result of the investigation, it would impact a fewer travellers.

Every control system has its own characteristics and foibles, he said. ICS which run on newer operating systems tend to be more stable, but he said some of the systems he deals with are 25 to 30 years old.

“When you get into older systems on older platforms, there's an enormous amount of unpredictability on how the investigation process is going to be received by the system itself,” Fabro said. “Traditional investigation methods such as inserting USBs or trying to do memory captures – these are elements of investigation that these control systems are historically not used to. And it becomes very hard to predict what the very action of going to observe the state of the production system is going to have.

“The production system could hang, it could start to trip up, it could start to fail.”

A forensic investigation involves two steps, he said: firstly, getting to the assets themselves and identifying the indicators of compromise and, secondly, gaining access to the communications infrastructure.