SE Labs

Special Edition
Computer security testing comment and analysis from SE Labs

How reliable are security product tests?

And how can you tell?

Security product tests

If you are in charge of protecting an organisation, you need good data to help make buying decisions. The consequences of simply trusting internet reviews, vendor sales pitches and instinct are extremely serious. So which security product tests are the best?

This article first appeared on LinkedIn (17th November, 2020)

Testing computer security products and services comes with its own unique challenges and it is hard to assess the assessments. The industry is not known for its transparency in product effectiveness, and that extends to some testing.

Keeping security product tests (very) real

To test a security product properly, you have to behave like a real attacker. There are countless clever ways to simulate attacks, automate testing and so on, but at the end of the day nothing beats sitting down and manually hacking away at a target for realism.

But there’s no point in testing if the threats aren’t relevant – if they don’t relate to real-world situations. A case in point is applying software updates. Everyone knows they should keep their systems patched and up to date, and if they did then attackers would not still be using old exploits, because they wouldn’t work most of the time. But the reality is there’s always some server somewhere that, for a dozen different reasons, gets missed.

What you really want to know is, will your security pick it up?

Intentionally vulnerable

As it happens, last month, we got hacked. Or rather the network we’d set up to test was hacked and, you guessed it, we deliberately included an unpatched but well-known vulnerability. To us, this was a good result, as it proves our tests are as real and as relevant as they come.

I’m also pleased to say that the EDR product that we were testing at the time picked it up, albeit after the initial breach occurred.

Stamp of quality?

Until recently, there was no official way in which to demonstrate that tests of anti-malware and other security products were, in fact, even basically trustworthy. And that the tests were carried out in a way that could be validated and even repeated if necessary.

The Anti-Malware Standards Organization is a community of over 60 security and testing companies from around the world. In mid-2018 it approved and adopted the AMTSO Testing Protocol Standard. A test that complies to this Standard must demonstrate that the testing has been conducted fairly and transparently. In a nutshell, it means that you say what you’re going to do. Do it! Then be prepared to prove it.

A reliable tester states in advance what it’s going to do; follows its own rules; and then has the data to prove it has done what it said it would.

A reliable tester states in advance what it’s going to do; follows its own rules; and then has the data to prove it has done what it said it would.

Full disclosure: I am on the board of AMTSO, so it’s no surprise that SE Labs was the first testing lab to engage with the Standard. We ran both private and public pilots. We then complied with the Standard immediately, once it was official. You can read more about it in our annual report.

But AMTSO isn’t the only standard you should look out for. Alongside ISO standards (we comply with ISO 27001 and 9001), there are also others such as NetSecOPEN‘s to take into consideration. For example, this Standard helps to specify network traffic when testing performance levels to check if appliances such as firewalls or routers still operate correctly when under stress.

Transparency, transparency, transparency!

With so many published computer security product tests available on the internet sometimes it’s hard to know which ones to trust. However, a simple check of the transparency in its methodology, and the standards adhered to, should go a long way to proving its validity. In turn, this should increase your confidence in the results and help with that next massive IT security purchase.


SE Labs Ltd is a private, independently-owned and run testing company that assesses security products and services. The main laboratory is located in Wimbledon, South London. It has excellent local and international travel connections. The lab is open for prearranged client visits.


SE Labs Ltd
Hill Place House
55A High Street
SW19 5BA