Over the last few years we have tested more than 50 different products using over 5,000 targeted attacks. And there’s news, both good and bad.
In this article we will look at the different tools available, how effective they are at helping attackers bypass anti-malware products and how security vendors have been handling this type of threat for over a year.
Why it’s important not to try to be too clever
Latest reports now online for enterprise, small business and home users.
Realism is important in testing, otherwise you end up with results that are theoretical and not a useful report that closely represents what is going on in the real world. One issue facing security testing that involves malware is whether or not you connect the test network to the internet.
The argument against this approach is that computer viruses can spread automatically and a test could potentially infect the real world, making life worse for computer users globally. One counter argument goes that if the tester is helping improve products then a few dozen extra infected systems on the internet is, on balance, worth it considering there are already millions out there. The benefits outweigh the downside.
Another counter argument is that viruses such as we understand them from the 90s are not the same as they are today. There are far fewer self-replicating worms and more targeted attacks that do not generally spread automatically, so the risk is lower.
Connecting to the internet brings more than a few advantages to a test, too. Firstly, the internet is where most threats reside. It would be hard to test realistically with a synthetic internet.
Secondly, for at least 10 years most endpoint security products have made connections back to management or update servers to get the latest information about current threats. So-called ‘cloud protection’ or ‘cloud updates’ would be disabled without an internet connection, effectively reducing the products’ protection abilities significantly. This then makes the test results much less accurate when running assessments.
There are cases in which turning off the internet is useful, though. Last year we ran a test to check whether or not artificial intelligence could predict future threats. We ran our Predictive Malware Response Test without an internet connection to see if a Cylance AI brain, which had been built and trained three years previously, could detect well-known threats that had come into existence since then. You can see the full report here.
But that was a special case. When assessing any security product or service for real-world, practical purposes, a live and unfiltered internet connection is probably a useful and even necessary part of the setup.
Naturally we have always used one in our testing, at one point even going as far as using consumer ADSL lines when testing home anti-malware products for extra realism. When reading security tests check that the tester has a live internet connection and allows the products to update themselves.
If you spot a detail in this report that you don’t understand, or would like to discuss, please contact us via our Twitter or Facebook accounts.
SE Labs uses current threat intelligence to make our tests as realistic as possible. To learn more about how we test, how we define ‘threat intelligence’ and how we use it to improve our tests please visit our website and follow us on Twitter.
This test report was funded by post-test consultation services provided by SE Labs to security vendors. Vendors of all products included in this report were able to request early access to results and the ability to dispute details for free. SE Labs has submitted the testing process behind this report for compliance with the AMTSO Testing Protocol Standard v1.0. To verify its compliance please check the AMTSO reference link at the bottom of page three of this report or here.
How to tell if security test results are useful, misleading or just rubbish?
Latest reports now online.
In security testing circles there is a theoretical test used to illustrate how misleading some test reports can be.
For this test you need three identical chairs, packaging for three anti-virus products (in the old days products came on discs in a cardboard box) and an open window on a high floor of a building.
The methodology of this test is as follows:
The problem with this test is obvious: the conclusions are not based on any useful reality.
The good part about this test is that the tester created a methodology and tested each product in exactly the same way.* And at least this was an ‘apples to apples’ test, in which similar products were tested in the same manner. Hopefully any tester running the chair test publishes the methodology so that readers realise what a stupidly meaningless test has been performed, but that is not a given.
Sometimes test reports come with very vague statements about, “how we tested”.
When evaluating a test report of anything, not only security products, we advise that you check how the testing was performed and to check whether or not it has been found compliant with a testing Standard, such as the Anti-Malware Testing Standards Organization’s Standard (see below).
Headline-grabbing results (e.g. Anti-virus is Dead!) catch the eye, but we need to focus on the practical realities when trying to find out how best to protect our systems from cyber threats. And that means having enough information to be able to judge a test report’s value rather than simply trusting blindly that the test was conducted correctly.
*Although some pedants might require that each chair be released from the window at exactly the same time – possible from windows far enough apart that the chairs would not entangle mid-air and skew the results in some way.
UPDATE (10th June 2019): The tests were found to be compliant with AMTSO’s Standard.
Detecting and preventing threats in real-time
Computer security products are designed to detect and protect against threats such as computer viruses, other malware and the actions of hackers.
A common approach is to identify existing threats and to create patterns of recognition, in much the same way as the pharmaceutical industry creates vaccinations against known biological viruses or police issue wanted notices with photographs of known offenders.
The downside to this approach is that the virus or criminal has to be known to be harmful, most likely after someone has become sick or a crime has already been committed. It would be better to detect new infections and crimes in real-time and to stop them in action before any damage is caused.
This approach is becoming increasingly popular in the cyber security world.
Deep Instinct claims that its D-Client software is capable of detecting not only known threats but those that have not yet hit computer systems in the real world. Determining the accuracy of these claims requires a realistic test that pits the product against known threats and those typically crafted by attackers who work in a more targeted way, identifying specific potential victims and moving against them with speed and accuracy.
This test report used a range of sophisticated, high-profile threat campaigns such as those believed to have been directed against the US Presidential election in 2016, in addition to directing more targeted attacks against the victim systems using techniques seen in well-known security breaches in recent months and years.
The results show that Deep Instinct D-Client provided a wide range of detection and threat blocking capability against well-known and customised targeted attacks, without interfering with regular use of the systems upon which it was deployed. The deep learning system was trained in August 2018, six months before the customised targeted threats were created.
Latest report now online.
Malware scanning is not enough. You have to hack, too.
Internal testing is necessary but inherently biased: ‘we test against what we know’. Thorough testing, including the full attack chains presented by threats, is needed to show not only detection and protection rates, but response capabilities.
Clear, open testing is needed and now available
Latest reports now online.
A year ago we decided to put our support behind a new testing Standard proposed by the Anti-Malware Testing Standards Organization (AMTSO). The goal behind the Standard is good for everyone: if testing is conducted openly then testers such as us can receive due credit for doing a thorough job; you the reader can gain confidence in the results; and the vendors under test can understand their failings and make improvements, which then creates stronger products that we can all enjoy.
The Standard does not dictate how testers should test. There are pages of detail, but I can best summarise it like this:
Say what you are going to do, then do it. And be prepared to prove it.
(Indeed, a poor test could still comply with the AMTSO Standard, but at least you would be able to understand how the test was conducted and could then judge its worth with clear information and not marketing hype!)
We don’t think that it’s unreasonable to ask testers to make some effort to prove their results. Whether you are spending £30 on a copy of a home anti-antivirus product or several million on a new endpoint upgrade project, if you are using a report to help with your buying decision you deserve to know how the test was run, whether or not some vendors were at a disadvantage and if anyone was willing and able to double-check the results.
Since the start of the year we put our endpoint reports through the public pilot and then, once the Standard was officially adopted, through the full public process. Our last reports were judged to comply with the AMTSO Standard and we’ve submitted these latest reports for similar assessment.
At the time of writing we didn’t know if the reports from this round of testing complied. We’re pleased to report today that they did. You can confirm this by checking the AMTSO reference link at the bottom of page three of this report or here.
If you spot a detail in this report that you don’t understand, or would like to discuss, please contact us via our Twitter or Facebook accounts.
SE Labs uses current threat intelligence to make our tests as realistic as possible. To learn more about how we test, how we define ‘threat intelligence’ and how we use it to improve our tests please visit our website and follow us on Twitter.
This test report was funded by post-test consultation services provided by SE Labs to security vendors. Vendors of all products included in this report were provided with early access to results and the ability to dispute details for free. SE Labs has submitted the testing process behind this report for compliance with the AMTSO Standard v1.0.
Our latest reports, for enterprise, small business and home users are now available for free from our website. Please download them and follow us on Twitter and/or Facebook to receive updates and future reports.
Email security test explores how and when services detect and stop threats.
Latest report now online.
This new email protection test shows a wide variation in the abilities of the services that we have assessed.
You might see the figures as being disappointing. Surely Microsoft Office 365 can’t be that bad? An eight per cent accuracy rating seems incredible.
Literally not credible. If it misses most threats then organisations relying on it for email security would be hacked to death (not literally).
But our results are subtler than just reflecting detection rates and it’s worth understanding exactly what we’re testing here to get the most value from the data. We’re not testing these services with live streams of real emails, in which massive percentages of messages are legitimate or basic spam. Depending on who you talk to, around 50 per cent of all email is spam. We don’t test anti-spam at all, in fact, but just the small percentage of email that comprises targeted attacks.
In other words, these results show what can happen when attackers apply themselves to specific targets. They do not reflect a “day in the life” of an average user’s email inbox.
We have also included some ‘commodity’ email threats, though – the kind of generic phishing and social engineering attacks that affect everyone. All services ought to stop every one of these. Similarly, we included some clean emails to ensure that the services were not too aggressively configured. All services ought to allow all these through to the inbox.
So when you see results that appear to be surprising, remember that we’re testing some very specific types of attacks that happen in real life, but not in vast numbers comparable to spam or more general threats.
The way that services handle threats are varied and effective to greater or lesser degrees. To best reflect how useful their responses are, we have a rating system that accounts for their different approaches. Essentially, services that keep threats as far as possible from users will win more points than those who let the message appear in or near the inbox. Conversely, those that allow the most legitimate messages through to the inbox rate higher than those which block them without the possibility of recovery from a junk folder or quarantine.
Hackers have spent well over 20 years stealing users’ passwords from internet companies.
They’ve almost certainly got yours.
The good news is it’s very easy to make your passwords useless to hackers. All you do is switch on Two-Factor Authentication (2FA).
2FA is a second login layer
It works much like the second lock on your front door. If someone’s stolen or copied your Yale key, that double-lock will keep them out.
A digital double-lock is now vital for protecting your online accounts – email, banking, cloud storage, business collaboration and the rest. It’s up there with anti-malware in the league of essential security measures. And it’s much easier to pick a 2FA method than choose the right anti-malware (our Anti-Malware Protection Reports can help you there).
So 2FA is essential, easy, and doesn’t have to cost a thing. It’s a security no-brainer. So how come hardly anyone uses it?
Join the one per cent elite!
Earlier this year, Google revealed that only 10 per cent of their users have ever bothered setting up 2FA. Just a fraction of those – we estimate around one per cent of all internet users – use the most secure type of 2FA, a USB security key.
In this article we’ll show you how to join that elite one per cent for less than £20. If you’d rather watch a step-by-step demo, here’s our YouTube video.
(This blog reflects the views and research of SE Labs, an independent security testing company. We never use affiliate links.)
Why everyone in your business should use 2FA
You’re not the only person who knows your usernames and passwords. Head over to Have I Been Pwned? and type in your email address to find out how many of your accounts have been hit by hacking attacks.
 |
| A quick (and scary) web search reveals how many times your passwords have fallen prey to hackers |
While you’re digesting those results, here’s a sobering statistic. More than 90 per cent of all login attempts on retail websites aren’t by actual customers, but by hackers using stolen credentials (Shape Security, July 2018).
Nearly everyone has had their passwords stolen. But hardly anyone protects their accounts using 2FA. We’re all leaving our front doors unlocked.
And as hackers plunder more and more big-name services (as well as all those services you’d forgotten you had accounts with), the more chance they have to steal the passwords you use everywhere.
This is why you must never using the same password twice. Don’t be tempted to use a pattern to help you remember them, either (‘123amazon’, ‘123google’ and so on). Hackers decode that stuff for breakfast. We’re also not keen on password managers. They’re Target Number One for hackers.
Instead, store your passwords where no-one can find them (not online!) and deadlock your accounts using 2FA. It’s the only way to make them hack-proof.
Why a USB key is the best way to lock your accounts
The ‘memorable information’ you have to enter when logging into your online bank account is a watered-down version of 2FA. Hackers can easily create spoof login pages that fool you into handing over all your info, as demonstrated in our NatWest phishing attack video.
Proper 2FA methods are much tougher to crack. They involve more than one device, so a hacker can’t simply ransack your computer and steal all pertinent data. Without the separate device, your passwords are useless to them.
Use more than one 2FA method if offered. This double-locks your double-locks – and also gives you another way into your account if one method fails. See our 2FA YouTube video for a step-by-step guide to doing this for your Google account.
Here’s a quick run-through of your options, starting with the most basic.
Google prompt
How it works: Tap your Android screen to confirm your identity.
Pros and cons: Very quick and easy, but only works with Google accounts and Android devices. Useful as a backup option.
SMS code
How it works: You’re texted (and/or voice-messaged) a PIN code to enter after your usual login.
Pros and cons: Authentication is split between two devices. It works on any mobile phone at no additional cost. But it can be slow, and the code may appear on your lock screen.
Authentication app
How it works: A free app, such as Google Authenticator, generates a unique numerical security code that you then enter on your PC.
Pros and cons: Faster and more reliable than SMS, and arguably more secure, but you’ll need a smartphone (Android or iOS).
 |
| Authenticate your logins with a code that’s sent to your phone (and only your phone) |
Backup codes
How it works: A set of numerical codes that you download and then print or write down – then keep in a safe place. Each code only works once.
Pros and cons: The perfect backup method. No need for a mobile phone. A piece of paper or locally-stored computer file (with disguised filename) is easier to hide from thieves than anything online.
And the most secure 2FA method of all…
USB security key
How it works: You ‘unlock’ your accounts by plugging a unique USB stick (such as this YubiKey) into your computer.
Pros and cons: A whole list of pros. USB keys are great for business security, because your accounts remain locked even if a hacker breaches your phone. They’re convenient: no need to wait for codes then type them in. And they cost very little considering how useful they are. One key costs from £18, and is all you need to deadlock all your accounts. Buy one for all your employees – and clients!
 |
| Give a USB security key to all your employees and clients – their security (and yours) will benefit |
Deadlock your Google account: a 2FA walk-through
Google lets you lock down your entire account, including Gmail and Google Drive, using multiple layers of 2FA (which it calls 2-Step Verification). It’s one of the most secure 2FA configurations you’ll find, and it’s easy to set up.
Here are the basic steps. For a more detailed step-by-step guide, see our YouTube video.
 |
| Double-lock your double-locks by choosing more than one 2FA method – and a backup |
Deadlock all your online accounts in minutes
All reputable online services now offer 2FA options. But, as you’ll discover from the searchable database Two Factor Auth, not all services offer the best 2FA options.
For example LinkedIn only offers 2FA via SMS, and doesn’t support authenticator apps or USB security keys – the most secure types of 2FA. Even Microsoft Office 365 doesn’t yet support security keys. We expect better from services aimed at business users.
What’s more, 2FA settings tend to be well buried in account settings. No wonder hardly anyone uses them. Here’s where to click:
UPDATE (29/10/2018): This set of reports are confirmed to be compliant with AMTSO Standard v1.0 by the Anti-Malware Testing Standards Organization.
It’s widely acknowledged that the cyber security workforce needs more talented young people to engage. Just as we, at SE Labs, want to help fix information technology security by testing products and services, we also want to encourage an interest among young people, hopefully igniting a passion for understanding and defending against hacking attacks.
We test next-gen security products AND encourage the gen-next!
Our attempts to enable youth from progressing from complete novice, through to getting their first job and then to reaching the top of industry, is an initiative to bring about the needed change and fill the gaps.
As part of our new corporate social responsibility programme we set up an event at Carshalton Boys Sports College to introduce the concept of cyber security and its career prospects to the students.
Around 15 participants ranged from year 10s to sixth formers (aged 16-18) attended the main presentation and all year groups approached us at the stand we set up.
We outlined various topics in the presentation including the different types of cybercrime and attacks; and institutions offering free and paid courses to certain age groups on cyber security, aimed at students.
We also addressed how to break into the cyber security sector; what positions are available in the industry; and how employees are in high demand in both public and private sectors, part- and full-time, in virtually every industry in countries around the world.
Then we went through a test run of a targeted attack to demonstrate what it looks like and what it means.
“Why do we use Kali Linux?”, “What should I do to get into cyber security?”, “What are the skills required?”, were a few curious questions asked by the students at the end of the presentation.
Those who came over to the stand wanted to know who we were, what we do and simply, “what is cyber security?”
They were interested in who are clients are (we gave limited answers due to NDAs), what do they need us and how did we manage to get this far. A lot of these were asked by the younger years who were inquisitive to learn more about this subject. Positive!
Feedback from the college:
On behalf of the Governors, Head Principle, students and parents of Carshalton Boys Sports College, I would like to thank you for your valued input, helping to make our Directions and Destinations Day a great success.
Our staff work tirelessly to open our students’ minds to the possibilities available to them, but without the support of partners like you, that job would be impossible. Together we had the school filled with a sense of purpose all day and responses we have had from students and parents have shown us that the day has inspired our students.
We have already started thinking about the future and would be grateful if you have any suggestions about how we might make things even better next year.
Thank you once again for giving your time, energy and expertise last week.
Well, yes! A career in cyber security is a journey for sure, but a worthwhile one. And in the end, it’s more about people than machines, as a mind’s software can be more powerful than any hardware.
Pooja Jain, March 2018
Archive of security product and service test results
