How to tell if security test results are useful, misleading or just rubbish?
Latest reports now online.
In security testing circles there is a theoretical test used to illustrate how misleading some test reports can be.
For this test you need three identical chairs, packaging for three anti-virus products (in the old days products came on discs in a cardboard box) and an open window on a high floor of a building.
The methodology of this test is as follows:
- Tape each of the boxes to a chair. Do so carefully, such that each is fixed in exactly the same way.
- Throw each of the chairs out of the window, using an identical technique.
- Examine the chairs for damage and write a comparative report, explaining the differences found.
- Conclude that the best product was the one attached to the least damaged chair.
The problem with this test is obvious: the conclusions are not based on any useful reality.
The good part about this test is that the tester created a methodology and tested each product in exactly the same way.* And at least this was an ‘apples to apples’ test, in which similar products were tested in the same manner. Hopefully any tester running the chair test publishes the methodology so that readers realise what a stupidly meaningless test has been performed, but that is not a given.
Sometimes test reports come with very vague statements about, “how we tested”.
When evaluating a test report of anything, not only security products, we advise that you check how the testing was performed and to check whether or not it has been found compliant with a testing Standard, such as the Anti-Malware Testing Standards Organization’s Standard (see below).
Headline-grabbing results (e.g. Anti-virus is Dead!) catch the eye, but we need to focus on the practical realities when trying to find out how best to protect our systems from cyber threats. And that means having enough information to be able to judge a test report’s value rather than simply trusting blindly that the test was conducted correctly.
*Although some pedants might require that each chair be released from the window at exactly the same time – possible from windows far enough apart that the chairs would not entangle mid-air and skew the results in some way.
UPDATE (10th June 2019): The tests were found to be compliant with AMTSO’s Standard.