Speaking in the latest SC Studio on ‘Cutting through the propaganda – software testing methodologies that work', Trend Micro's senior security advisor Rik Ferguson explained the vendor's reasons for pulling out of some high-profile anti-virus tests.
Ferguson said: “We didn't feel the tests were offering actionable information and a real insight into the effectiveness of the products to the people that were going to be doing the purchasing. We didn't feel that the testing methodology reflected the real world threat environment, so any scores that you derive from that become meaningless.”
Asked if new methods for testing were needed, Ferguson said that they were as anti-malware products have traditionally been based on a sample set in a lab environment and running a scan while unconnected from the internet.
“In the past when the threat was more file-centric that may have been a more viable testing methodology, but the fact is that the threat landscape has changed, threats have multiple vectors – it is not just about the file it is about how the file is delivered, where is the exposure level and also the rate of turnover has increased exponentially as everyone knows over the past two to three years and we see a malicious binary about every one and a half seconds now.
“So testing against a static set of files in an offline environment doesn't reflect the threat as regular users are exposed to it. So new methodologies need to take that into account, so we would say that the ideal testing method would be in an online environment, in a connected environment, exposing security products to live threats concurrently that are actually operating on the internet and looking at how rapidly we protect.”
In response to a question on correct testing, Ferguson said that there were two key threat vectors to address – on volume and the resource demands that the software places on the system.
View the latest SC Studio here - www.scmagazineuk.com/sc-studio