SE Labs

Posts filed under 'standards'

Strong protection in uncertain times

A hacker mentality is keeping (computer) virus testing on track.

Latest endpoint protection reports now online for enterprisesmall business and home users.

This is the first in our series of 2020 endpoint protection reports. And it is unique, for all the usual reasons but also a new one.

We would normally highlight the latest new threats that we’ve discovered on the internet and discuss how we test them against the security software you use in your business and at home in the most realistic ways possible. And we’ve done that. But these reports are different to any we’ve produced before, for another reason.

Read more >

Email security: Is it any good against hackers?

World’s first in-depth, public test of security services vs. targeted attacks.

This email security test report is the product of two years of advanced threat research. We have worked with the security companies themselves and with their customers.  We have monitored what the bad guys have been doing and identified and replicated real-world email threats that affect everyone generally, and also specific types of businesses.

There is no report like this anywhere in the public domain. We are extremely proud to present the results here.

Read more >

Securing a business from scratch

Building and launching a start-up company is a challenge in itself. Securing it when it is new, young and vulnerable is something else. It’s very necessary but also hard if you don’t know what you’re doing. And can you afford a consultant in the early days?

If your new business is IT-based and focused on security then you’re in a stronger position than, say, an organic make-up business or an ethical coffee brand.

Read more >

SE Labs Annual Report 2019

SE Labs has been working at the core of the cyber security industry since its launch in 2016. We work with all of the major developers of IT security products as well as their main customers and even investors looking to increase their chances when betting on emerging technologies.

Read more >

How can you tell if a security test is useful or not?

How to tell if security test results are useful, misleading or just rubbish?

Latest reports now online.

In security testing circles there is a theoretical test used to illustrate how misleading some test reports can be.

For this test you need three identical chairs, packaging for three anti-virus products (in the old days products came on discs in a cardboard box) and an open window on a high floor of a building.

The methodology of this test is as follows:

  1. Tape each of the boxes to a chair. Do so carefully, such that each is fixed in exactly the same way.
  2. Throw each of the chairs out of the window, using an identical technique.
  3. Examine the chairs for damage and write a comparative report, explaining the differences found.
  4. Conclude that the best product was the one attached to the least damaged chair.

The problem with this test is obvious: the conclusions are not based on any useful reality.

The good part about this test is that the tester created a methodology and tested each product in exactly the same way.* And at least this was an ‘apples to apples’ test, in which similar products were tested in the same manner. Hopefully any tester running the chair test publishes the methodology so that readers realise what a stupidly meaningless test has been performed, but that is not a given.

Sometimes test reports come with very vague statements about, “how we tested”.

When evaluating a test report of anything, not only security products, we advise that you check how the testing was performed and to check whether or not it has been found compliant with a testing Standard, such as the Anti-Malware Testing Standards Organization’s Standard (see below).

Headline-grabbing results (e.g. Anti-virus is Dead!) catch the eye, but we need to focus on the practical realities when trying to find out how best to protect our systems from cyber threats. And that means having enough information to be able to judge a test report’s value rather than simply trusting blindly that the test was conducted correctly.

*Although some pedants might require that each chair be released from the window at exactly the same time – possible from windows far enough apart that the chairs would not entangle mid-air and skew the results in some way.

If you spot a detail in this report that you don’t understand, or would like to discuss, please contact us via our Twitter or Facebook accounts.

SE Labs uses current threat intelligence to make our tests as realistic as possible. To learn more about how we test, how we define ‘threat intelligence’ and how we use it to improve our tests please visit our website and follow us on Twitter.

These test reports were funded by post-test consultation services provided by SE Labs to security vendors. Vendors of all products included in these reports were able to request early access to results and the ability to dispute details for free. SE Labs has submitted the testing process behind this report for compliance with the AMTSO Testing Protocol Standard v1.0. To verify its compliance please check the AMTSO reference link at the bottom of page three of each report or here.

UPDATE (10th June 2019): The tests were found to be compliant with AMTSO’s Standard.

Our latest reports, for enterprise, small business and home users are now available for free from our website. Please download them and follow us on Twitter and/or Facebook to receive updates and future reports.

Can you trust security tests?

Clear, open testing is needed and now available

Latest reports now online.

A year ago we decided to put our support behind a new testing Standard proposed by the Anti-Malware Testing Standards Organization (AMTSO). The goal behind the Standard is good for everyone: if testing is conducted openly then testers such as us can receive due credit for doing a thorough job; you the reader can gain confidence in the results; and the vendors under test can understand their failings and make improvements, which then creates stronger products that we can all enjoy.

The Standard does not dictate how testers should test. There are pages of detail, but I can best summarise it like this:

Say what you are going to do, then do it. And be prepared to prove it.

(Indeed, a poor test could still comply with the AMTSO Standard, but at least you would be able to understand how the test was conducted and could then judge its worth with clear information and not marketing hype!)

We don’t think that it’s unreasonable to ask testers to make some effort to prove their results. Whether you are spending £30 on a copy of a home anti-antivirus product or several million on a new endpoint upgrade project, if you are using a report to help with your buying decision you deserve to know how the test was run, whether or not some vendors were at a disadvantage and if anyone was willing and able to double-check the results.

Since the start of the year we put our endpoint reports through the public pilot and then, once the Standard was officially adopted, through the full public process. Our last reports were judged to comply with the AMTSO Standard and we’ve submitted these latest reports for similar assessment.

At the time of writing we didn’t know if the reports from this round of testing complied. We’re pleased to report today that they did. You can confirm this by checking the AMTSO reference link at the bottom of page three of this report or here.

If you spot a detail in this report that you don’t understand, or would like to discuss, please contact us via our Twitter or Facebook accounts.

SE Labs uses current threat intelligence to make our tests as realistic as possible. To learn more about how we test, how we define ‘threat intelligence’ and how we use it to improve our tests please visit our website and follow us on Twitter.

This test report was funded by post-test consultation services provided by SE Labs to security vendors. Vendors of all products included in this report were provided with early access to results and the ability to dispute details for free. SE Labs has submitted the testing process behind this report for compliance with the AMTSO Standard v1.0.

Our latest reports, for enterprise, small business and home users are now available for free from our website. Please download them and follow us on Twitter and/or Facebook to receive updates and future reports.

Are you buying solid protection or snake oil?

2018q1epp-9559512
Sometimes testers need to be tested too. We’re always up for a challenge!
Latest reports now online.
How do you know which security products to buy? Many rely on independent tests to help in the decision-making process. But how do you know if a test is any good or not?
The Anti-Malware Testing Standards Organization (AMTSO) has been working to create a Standard that will give you, the customer, some assurance that the test was conducted fairly.
Earlier this year AMTSO has been trying out its Standard, which it has been working on for many months. SE Labs is proud to be involved in this initiative and the testing for this report has been assessed for compliance with the Standard.
If that sounds a bit dry, what it means is that there are experimental rules about how a tester should behave and we have put ourselves up for judgment by AMTSO.
Did participating in this process change the way we worked? Yes, but not in the technical ways that we test. Instead we turned the testing world’s business model on its head.
Many testers charge vendors money to be tested. Some will test regardless, but charge money if the vendors want to see their results before publication (and have the opportunity to make requests for corrections).
We think that the dispute process should be free for all. SE Labs has not charged any vendor for its participation in this test and we provided a free dispute process to any vendor that requested it. In this way every vendor is treated as equally as possible, for the fairest possible test.

UPDATE (10th May 2018): We are extremely proud to announce that our 2018 Q1 reports have been judged compliant (PDF) with the AMTSO Draft Standard v6.1 – 2018-05-10.

If you spot a detail in this report that you don’t understand, or would like to discuss, please contact us via our Twitter or Facebook accounts.
SE Labs uses current threat intelligence to make our tests as realistic as possible. To learn more about how we test, how we define ‘threat intelligence’ and how we use it to improve our tests please visit our website and follow us on Twitter.
Our latest reports, for enterprise, small business and home users are now available for free from our website. Please download them and follow us on Twitter and/or Facebook to receive updates and future reports.

Who certifies the certifiers?

iso9001-7496453

At SE Labs we test security software and services methodically, realistically and in great detail. Or, at least, we claim to. But how does anyone really know?

Testing can be a very process-driven task. If you are going to be fair to every product undergoing a test you need to be consistent with how you run the test as a whole and how you test each individual product. It’s probably best carried out by well-qualified people, then?

You don’t need to be certified to work here…

We figured that as we certify, so should we be certified. As such, for the last few months we have worked towards having our business certified to an international level for providing consistent security testing services.

Another purpose of quality management is improvement. There is always room for improvement in testing, and we constantly strive to make things more realistic, useful and fair for everyone involved.

As such I am extremely proud to announce that SE Labs has now achieved compliance with the ISO 9001:2015 standard for quality management systems, specifically relating to “The Provision of IT Security Product Testing”.

That means we do what we say we do, and strive to improve.

About

SE Labs Ltd is a private, independently-owned and run testing company that assesses security products and services. The main laboratory is located in Wimbledon, South London. It has excellent local and international travel connections. The lab is open for prearranged client visits.

Contact

SE Labs Ltd
Hill Place House
55A High Street
Wimbledon
SW19 5BA

020 3875 5000

info@selabs.uk

Press