SE Labs

Posts filed under 'Test Results'

Testing deeper, wider and better

Testing deeper, wider and better

Bad guys evolve; defenders evolve; testing (should) evolve

Latest endpoint protection reports now online for enterprise, small business and home users.

These reports represent the state-of-the-art in computer security endpoint testing. If you want to see how the very best security products handle a range of threats, from everyday (but nevertheless very harmful) malware to targeted attacks, this is a great place to start.

Read more >

Breach Response Test: Kaspersky Anti Targeted Attack Platform

Kaspersky Anti Targeted Attack Platform

Testing anti-breach products needs the full chain of attack.

Kaspersky Lab should be congratulated, not only for engaging with this new and challenging test, but for submitting a product that performed so strongly against attacks that closely replicate advanced, nation-state level threats.

Its endpoint detection and response offering, Kaspersky Anti Targeted Attack Platform, is one of the very first to face our brand new Breach Response Test and it detected all of the attacks, while protecting against the vast majority of them.

Read more >

Anti-malware is just one part of the picture

Anti-malware is just one part of the picture

Beefing up security advice with facts

Latest reports now online for enterprise, small business and home users.

At SE Labs we spend our time testing things that are supposed to protect you but we also understand that securing your business, or your home network, is never as simple as installing one or more security products.

The risks are many and varied, but the ways to mitigate them are often most successful with a good dose of common sense as well as the appropriate technology. You just need to think things through carefully and make sensible decisions.

Read more >

Breach Response Test: Symantec Endpoint Security Complete

Symantec Endpoint Security Complete

Testing anti-breach products needs the full chain of attack. Symantec Endpoint Security Complete is the first endpoint detection and response offering to face our brand new Breach Response Test.

Report now online.

Read more >

SE Labs Annual Report 2019

SE Labs Annual Report 2019

We are proud to announce the SE Labs Annual Report 2019.

SE Labs has been working at the core of the cyber security industry since its launch in 2016. We work with all of the major developers of IT security products as well as their main customers and even investors looking to increase their chances when betting on emerging technologies.

Read more >

The best security tests keep it real

Best security tests

The best security tests are realistic. That’s why it’s important not to try to be too ‘clever’

Latest reports now online for enterprisesmall business and home users.

Realism is important in testing, otherwise you end up with results that are theoretical and not a useful report that closely represents what is going on in the real world. One issue facing security testing that involves malware is whether or not you connect the test network to the internet.

Read more >

How can you tell if a security test is useful or not?

security test is useful

How to tell if security test results are useful, misleading or just rubbish?

Latest reports now online.

In security testing circles there is a theoretical test used to illustrate how misleading some test reports can be.

The chair test

For this test you need three identical chairs, packaging for three anti-virus products (in the old days products came on discs in a cardboard box) and an open window on a high floor of a building.

The methodology of this test is as follows:

  1. Tape each of the boxes to a chair. Do so carefully, such that each is fixed in exactly the same way.
  2. Throw each of the chairs out of the window, using an identical technique.
  3. Examine the chairs for damage and write a comparative report, explaining the differences found.
  4. Conclude that the best product was the one attached to the least damaged chair.

The problem with this test is obvious: the conclusions are not based on any useful reality.

The good part about this test is that the tester created a methodology and tested each product in exactly the same way.* And at least this was an ‘apples to apples’ test, in which they tested similar products in the same manner. Hopefully any tester running the chair test publishes the methodology so that readers realise that they have carried out a stupidly meaningless test. But that is not a given.

How to tell if a security test is useful

Sometimes test reports make very vague statements about, “how we tested”.

When evaluating a test report of anything, not only security products, we advise that you check how the testing was performed. And check whether or not it complies with a testing Standard. The Anti-Malware Testing Standards Organization’s Standard (see below) is a good one.

Headline-grabbing results (e.g. Anti-virus is Dead!) catch the eye, but we need to focus on the practical realities when trying to find out how best to protect our systems from cyber threats. And that means having enough information to judge a test report’s value. Don’t simply trust blindly that the test was conducted correctly.

*Although some pedants might require that the tester release each chair from the window at exactly the same time. Possibly from windows far enough apart that the chairs would not entangle mid-air and skew the results in some way.

Find out more

If you spot a detail in this report that you don’t understand, or would like to discuss, please contact us via our Twitter or LinkedIn accounts.
 
SE Labs uses current threat intelligence to make our tests as realistic as possible. To learn more about how we test, how we define ‘threat intelligence’ and how we use it to improve our tests please visit our website and follow us on Twitter.
 
These test reports were funded by post-test consultation services provided by SE Labs to security vendors. Vendors of all products included in these reports were able to request early access to results and the ability to dispute details for free. SE Labs has submitted the testing process behind this report for compliance with the AMTSO Testing Protocol Standard v1.0. To verify its compliance please check the AMTSO reference link at the bottom of page three of each report or here.

UPDATE (10th June 2019): AMTSO found these test complied with AMTSO’s Standard.

Our latest reports, for enterprise, small business and home users are now available for free from our website. Please download them and follow us on Twitter and/or LinkedIn to receive updates and future reports.

Enemy Unknown: Handling Customised Targeted Attacks

Customised targeted  attacks

Detecting and preventing customised targeted attacks in real-time

Experts design computer security products to detect and protect against threats such as computer viruses, other malware and the actions of hackers.

A common approach is to identify existing threats and to create patterns of recognition. This is similar to the way the pharmaceutical industry creates vaccinations against known biological viruses. Or police issuing wanted notices with photographs of known offenders.

Detecting the unknown

The downside to this approach is that you have to know in advance that the virus or criminal is harmful. The most likely time to discover this is after someone has become sick or a crime has already been committed. It would be better to detect new infections and crimes in real-time and to stop them in action before any damage is caused.

The cyber security world is adopting this approach more frequently than before.

Deep Instinct claims that its D-Client software is capable of detecting not only known threats but those that have not yet hit computer systems in the real world. These claims require a realistic test that pits the product against known threats and those typically crafted by attackers. Attackers who work in a more targeted way. Attackers who identify specific potential victims and move against them with speed and accuracy.

Electioneering

This test report used a range of sophisticated, high-profile threat campaigns such as those directed against the US Presidential election in 2016. It also directed targeted attacks against victim systems using techniques seen in well-known security breaches in recent months and years.

The results show that Deep Instinct D-Client provided a wide range of detection and threat blocking capability against well-known and customised targeted attacks. It didn’t interfere with regular use of the systems upon which it was deployed.

The deep learning system was  trained in August 2018, six months before the customised targeted threats were created.

Latest report now online.

Assessing next-generation protection

Assessing next-generation protection

Malware scanning is not enough. You have to hack, too.

Latest report now online.
 
The amount of choice when trialling or buying endpoint security is at an all-time high. ‘Anti-virus’ first appeared 36 years ago and, in the last five years, the number of companies innovating and selling products designed to keep Windows systems secure has exploded.
 
And whereas once vendors of these products generally used non-technical terms to market their wares, now computer science is at the fore. No longer do security firms offer us ‘anti-virus’ or ‘hacker protection’ but artificial intelligence-based detection and response solutions. The choice has never been greater. Nor has the confusion among potential customers.
 
Assessing next-generation protection is not easy.
 
While marketing departments appear to have no doubt about the effectiveness of their product, the fact is that without in-depth testing no-one really knows whether or not an Endpoint Detection and Response (EDR) agent can do what it is intended.

Assessing next-generation protection

Internal testing is necessary but inherently biased: ‘we test against what we know’. We need through testing, including the full attack chains presented by threats. That’s how to show not only detection and protection rates, but response capabilities.

EventTracker asked SE Labs to conduct an independent test of its EDR agent, running the same tests as are used against some of the world’s most established endpoint security solutions available, as well as some of the newer ones.
 
This report shows EventTracker’s performance in this test. You can compare the results directly with the public SE Labs Enterprise Endpoint Protection (Oct – Dec 2018) report, available here.

Can you trust security tests?

trust security tests

Clear, open testing is needed and now available to help people trust security tests.

Latest reports now online.

A year ago we decided to put our support behind a new testing Standard proposed by the Anti-Malware Testing Standards Organization (AMTSO). The goal behind the Standard is good for everyone: if testing is conducted openly then testers such as us can receive due credit for doing a thorough job; you the reader can gain confidence in the results; and the vendors under test can understand their failings and make improvements, which then creates stronger products that we can all enjoy.

The Standard does not dictate how testers should test. There are pages of detail, but I can best summarise it like this:

Say what you are going to do, then do it. And be prepared to prove it.

(Indeed, a poor test could still comply with the AMTSO Standard, but at least you would be able to understand how the test was conducted and could then judge its worth with clear information and not marketing hype!)

Trust security tests

We don’t think that it’s unreasonable to ask testers to make some effort to prove their results. Whether you are spending £30 on a copy of a home anti-antivirus product or several million on a new endpoint upgrade project, if you are using a report to help with your buying decision you deserve to know how the test was run, whether or not some vendors were at a disadvantage and if anyone was willing and able to double-check the results.

Since the start of the year we put our endpoint reports through the public pilot and then, once the Standard was officially adopted, through the full public process. Our last reports were judged to comply with the AMTSO Standard and we’ve submitted these latest reports for similar assessment.

At the time of writing we didn’t know if the reports from this round of testing complied. We’re pleased to report today that they did. You can confirm this by checking the AMTSO reference link at the bottom of page three of this report or here. This helps people trust security tests.

Ask us

If you spot a detail in this report that you don’t understand, or would like to discuss, please contact us via our Twitter or Facebook accounts.

SE Labs uses current threat intelligence to make our tests as realistic as possible. To learn more about how we test, how we define ‘threat intelligence’ and how we use it to improve our tests please visit our website and follow us on Twitter.

This test report was funded by post-test consultation services provided by SE Labs to security vendors. Vendors of all products included in this report were provided with early access to results and the ability to dispute details for free. SE Labs has submitted the testing process behind this report for compliance with the AMTSO Standard v1.0.

Our latest reports, for enterprise, small business and home users are now available for free from our website. Please download them and follow us on Twitter and/or Facebook to receive updates and future reports.

About

SE Labs Ltd is a private, independently-owned and run testing company that assesses security products and services. The main laboratory is located in Wimbledon, South London. It has excellent local and international travel connections. The lab is open for prearranged client visits.

Contact

SE Labs Ltd
Hill Place House
55A High Street
Wimbledon
SW19 5BA

020 3875 5000

info@selabs.uk

Press