I have seen plenty of discussion in the blogosphere of late about the anti-malware testing standards organisation (AMTSO).
The discussion seemed to spin from the publication of new guidelines with added resources for software testers included. In a comment a couple of weeks later, David Harley, writing separately from his duties as senior research fellow at ESET and a director of AMTSO, claimed that he was ‘confused and not to mention exasperated, at the flurry of bad press that AMTSO is suddenly receiving'.
He pointed to a blog by Kevin Townsend to which he contributed some thoughts, and thought the time might be right for some healthy discussion about how the organisation might engage better with the general user population.
However the feedback received showed a general feeling of negativity towards AMTSO, with Harley saying that it had been described as ‘a self-serving group of anti-virus vendors (cantankerous vendors, even) trying to impose retrograde standards unilaterally on testers' to a ‘sinister cabal of vendors and testers and the organisation should be user driven'.
A joint statement from researchers at Kaspersky Lab, Panda Security, McAfee, Symantec and ESET aimed to clarify some of the complaints, but I asked Harley what the headlines have been about.
He claimed that he would not have made an investment of time and energy if he did not believe that there is a need for major improvements in testing and the public understanding of testing.
He said: “Nearly 20 years in the arena as a corporate customer, sometime tester, educationalist and anti-virus researcher and more recently within the anti-virus industry, suggests to me that it does. That is the sort of problem AMTSO was meant to solve. Of course, there's an element of self-interest in AMTSO's activities, but the thinking behind all that is that better testing benefits the population at large, not just the anti-virus industry.”
He highlighted three problems – firstly while AMTSO is not a profit-making organisation, the subscription fee is fairly hefty. It was heavier in the first year, because of the large setup costs, and was reduced accordingly for the second year, because many of those costs were one-off. Still, a subscription large enough to cover maintenance/administrative costs is still too large for most interested individuals.
He said that those costs are basically for professional services such as legal services; the board of directors and review analysis board, while the independent advisory board is made up of volunteers.
“That gives rise to another issue. Since we all have full-time jobs, we can't give AMTSO the time and attention some of us would like to. The best measure of what the organisation has achieved so far is probably its own repositories of papers, guidelines and other resources, put together in the course of several workshops and a lot of email. A lot more than that has been done behind the scenes, but there are no brownie points for setting up processes that are necessary, but don't directly impact on the public,” he said.
The second problem is that the group includes security vendors, as well as testers and product certification agencies. Harley admitted that while mainstream vendors and testers do not necessarily see that as a problem, most people do not see it that way, rather they see it as the foxes guarding the hen house.
He said: “Indeed, when a single vendor is behind an ‘independent' site, which does happen, you get some pretty biased testing, but AMTSO is slightly different. The group comprises (generally) researchers, not marketroids. It's difficult for a vendor or tester to cross the line when the eyes of so many competitors are on him.
“Still, there's a huge mistrust in the media and the population in general of security vendors, and that poses a significant PR problem. There's a subsidiary problem in that the organisation is not sufficiently engaged with the public at large. One approach I've been advocating is a much cheaper membership option with less privileges, something like the Anti-Phishing Working Group's basic membership, and something like that may happen sooner rather than later.”
Finally, he said that there is confusion over what AMTSO means by raising standards (rather than standards; despite the name, AMTSO is not equipped to impose standardised testing, even if it wanted to (it doesn't), and is not a suitable venue for formulating ISO/BSI-like standards.
He said: “While some labs are, quite rightly, stressing their achievements in meeting such standards, they're very generic. While I'd hope that a standards organisation addressing the need for standards specific to the IT security arena would want input from vendors and testers, AMTSO in its present form could not provide that kind of standard, in my opinion, because it would be seen as the AV industry imposing measures that suit the industry rather than its customers.”
So how could standards be raised in a more general sense? Harley said that this would be by improving the quality and availability of information about tests and testing, and by making testers more accountable for the accuracy and quality of their testing.
He said: “The AMTSO review analysis process is one step towards this, though its implementation to date has been far from smooth and not universally popular. A self-certification scheme is also in process, though excruciatingly slowly: that's the problem with relying on volunteer labour.”