In this blog post our CTO Stefan Dumitrascu explains some of the challenges behind our newly launched Breach Response testing, why things are now different (better) and the background on how we came to make some of our decisions.
One of our most exciting projects this year has been the Breach Response testing programme. In this article we explain what has changed since last year, and why.
Read more >
Have you ever been hacked? You, personally, or your business? If your answer is, “no” it would be interesting to know how you can be so sure.
I once spoke to a marketing manager at a global anti-malware company who claimed that his PC had never been infected by malware, despite him not using anti-virus. How would be know? Not all malware announces its presence as clearly as, say ransomware. “I just would,” he claimed.
Read more >
Well, 2020 was an interesting year! Some of the largest and newest security companies were bought and sold…
The world descended into chaos thanks to a biological virus (while digital viruses continue to wreak havoc)…
And SE Labs launched a new website and started using machine learning in its testing.
We appreciate some of these things are more important than others…
However, in the face of adversity we must all carry on and we are proud to announce our second annual report, in which we review the unprecedented year of 2020, announce our annual awards winners and discuss testing like hackers.
Read more >
Bad guys help SE Labs keep its testing up to date
Latest endpoint protection reports now online for enterprise, small business and home users.
For the first time in our endpoint protection tests we’ve seen a strong overall performance from both well-established anti-malware brands and newer entrants to the market.
Vendors such as FireEye and Crowdstrike are well-known and respected brands in the security world, but they are relatively new compared to Symantec, McAfee and even Microsoft. Microsoft has not promoted its anti-malware software until quite recently.
Read more >
A hacker mentality is keeping (computer) virus testing on track.
Latest endpoint protection reports now online for enterprise, small business and home users.
This is the first in our series of 2020 endpoint protection reports. And it is unique, for all the usual reasons but also a new one.
We would normally highlight the latest new threats that we’ve discovered on the internet and discuss how we test them against the security software you use in your business and at home in the most realistic ways possible. And we’ve done that. But these reports are different to any we’ve produced before, for another reason.
Bad guys evolve; defenders evolve; testing (should) evolve
Latest endpoint protection reports now online for enterprise, small business and home users.
These reports represent the state-of-the-art in computer security endpoint testing. If you want to see how the very best security products handle a range of threats, from everyday (but nevertheless very harmful) malware to targeted attacks, this is a great place to start.
Beefing up security advice with facts
Latest reports now online for enterprise, small business and home users.
At SE Labs we spend our time testing things that are supposed to protect you but we also understand that securing your business, or your home network, is never as simple as installing one or more security products.
The risks are many and varied, but the ways to mitigate them are often most successful with a good dose of common sense as well as the appropriate technology. You just need to think things through carefully and make sensible decisions.
SE Labs has been working at the core of the cyber security industry since its launch in 2016. We work with all of the major developers of IT security products as well as their main customers and even investors looking to increase their chances when betting on emerging technologies.
Why it’s important not to try to be too clever
Latest reports now online for enterprise, small business and home users.
Realism is important in testing, otherwise you end up with results that are theoretical and not a useful report that closely represents what is going on in the real world. One issue facing security testing that involves malware is whether or not you connect the test network to the internet.
The argument against this approach is that computer viruses can spread automatically and a test could potentially infect the real world, making life worse for computer users globally. One counter argument goes that if the tester is helping improve products then a few dozen extra infected systems on the internet is, on balance, worth it considering there are already millions out there. The benefits outweigh the downside.
Another counter argument is that viruses such as we understand them from the 90s are not the same as they are today. There are far fewer self-replicating worms and more targeted attacks that do not generally spread automatically, so the risk is lower.
Connecting to the internet brings more than a few advantages to a test, too. Firstly, the internet is where most threats reside. It would be hard to test realistically with a synthetic internet.
Secondly, for at least 10 years most endpoint security products have made connections back to management or update servers to get the latest information about current threats. So-called ‘cloud protection’ or ‘cloud updates’ would be disabled without an internet connection, effectively reducing the products’ protection abilities significantly. This then makes the test results much less accurate when running assessments.
There are cases in which turning off the internet is useful, though. Last year we ran a test to check whether or not artificial intelligence could predict future threats. We ran our Predictive Malware Response Test without an internet connection to see if a Cylance AI brain, which had been built and trained three years previously, could detect well-known threats that had come into existence since then. You can see the full report here.
But that was a special case. When assessing any security product or service for real-world, practical purposes, a live and unfiltered internet connection is probably a useful and even necessary part of the setup.
Naturally we have always used one in our testing, at one point even going as far as using consumer ADSL lines when testing home anti-malware products for extra realism. When reading security tests check that the tester has a live internet connection and allows the products to update themselves.
If you spot a detail in this report that you don’t understand, or would like to discuss, please contact us via our Twitter or Facebook accounts.
SE Labs uses current threat intelligence to make our tests as realistic as possible. To learn more about how we test, how we define ‘threat intelligence’ and how we use it to improve our tests please visit our website and follow us on Twitter.
This test report was funded by post-test consultation services provided by SE Labs to security vendors. Vendors of all products included in this report were able to request early access to results and the ability to dispute details for free. SE Labs has submitted the testing process behind this report for compliance with the AMTSO Testing Protocol Standard v1.0. To verify its compliance please check the AMTSO reference link at the bottom of page three of this report or here.
How to tell if security test results are useful, misleading or just rubbish?
Latest reports now online.
In security testing circles there is a theoretical test used to illustrate how misleading some test reports can be.
For this test you need three identical chairs, packaging for three anti-virus products (in the old days products came on discs in a cardboard box) and an open window on a high floor of a building.
The methodology of this test is as follows:
The problem with this test is obvious: the conclusions are not based on any useful reality.
The good part about this test is that the tester created a methodology and tested each product in exactly the same way.* And at least this was an ‘apples to apples’ test, in which similar products were tested in the same manner. Hopefully any tester running the chair test publishes the methodology so that readers realise what a stupidly meaningless test has been performed, but that is not a given.
Sometimes test reports come with very vague statements about, “how we tested”.
When evaluating a test report of anything, not only security products, we advise that you check how the testing was performed and to check whether or not it has been found compliant with a testing Standard, such as the Anti-Malware Testing Standards Organization’s Standard (see below).
Headline-grabbing results (e.g. Anti-virus is Dead!) catch the eye, but we need to focus on the practical realities when trying to find out how best to protect our systems from cyber threats. And that means having enough information to be able to judge a test report’s value rather than simply trusting blindly that the test was conducted correctly.
*Although some pedants might require that each chair be released from the window at exactly the same time – possible from windows far enough apart that the chairs would not entangle mid-air and skew the results in some way.
UPDATE (10th June 2019): The tests were found to be compliant with AMTSO’s Standard.
Archive of security product and service test results
