For those of us in the anti-malware testing industry, this last month has had a lot of very interesting discussion.
To sum it up simply, there have been a lot of technology changes in the last five or so years due to a sea-change in the motivation of malware authoring, and testers and have been changing their methods as well.
Existing testing labs undertook methodology changes based on their own strengths and the need they perceived, and a number of new testers have cropped up to address a need they perceived in the industry. But depending on the expertise of those in the labs, and their understanding of how security software is intended to function, some testing changes were more useful than others.
At its best, testing is helpful not just for users but vendors also, as it points out strengths and weaknesses that they can improve within their product. Ideally, this is how tests should be – causing minimal risk to the general public during the testing, giving users an accurate picture of how the product will perform, and helping vendors to improve their products.
To analyse these new tests and to advise future tests, a body of researchers and experts have gotten together, calling themselves the Anti-Malware Testing Standards Organisation (AMTSO). Without getting into the controversial issues, I will say I see it as incredibly important that there be some guidance for those who're creating new tests.
There are a lot of organisations getting into testing, who do not understand the special needs of security software. One does not put anti-malware through the same sorts of paces as a toaster or a word-processor. It's seldom useful to know the history of egg-beaters when one is evaluating its effectiveness.
Conversely, if you don't know how to safely isolate a network, you can infect your virtual neighbours with those live viruses you're using or (heaven forefend!) have created. If you do not know what an anti-malware program is intended to do, while your test may be completely scientifically valid, your results will tell users nothing about what their day-to-day experience with a product will be.
Those with experience will understand how to create a scientifically valid and meaningful test, but not every group creating a test has the luxury of employing an experienced anti-malware expert. Publishing tests which are flawed in either area may lead to entirely sensationalistic results, and it makes not only the vendors but the whole testing industry look incompetent, as it becomes a public feud.
The new testers feel unfairly punished because they didn't know better, the vendors feel unfairly punished because of the flawed testing methodology.
Whatever else AMTSO has done, they have begun publishing papers that allow new testers to ‘know better'. What they then do with that information is up to them, as AMTSO is in the business of education, not enforcement. The ‘watchers' are still essentially under the honour system, but it is certainly a step in the right direction.
Lysa Myers is director of research at West Coast Labs