Endpoint Index
Home / Methodology

Reading lab results

Independent lab testing is the only evidence that lets you compare endpoint security products on the same samples, under the same conditions.

Contents
  1. AV-Test protection score
  2. MITRE ATT&CK analytic coverage
  3. AV-Comparatives false positives

Endpoint Index uses three labs: AV-Test, MITRE ATT&CK Evaluations and AV-Comparatives. This page explains what each result measures, how to read it, and where each lab publishes its method. The latest result for every tracked product is in the lab results scorecard, and MITRE's rounds are explained in full in MITRE ATT&CK Evaluations explained.

AV-Test protection score

AV-Test evaluates business endpoint products in rounds that cover two months, using two kinds of sample: a real-world test (new malware and malicious websites seen in the preceding weeks) and the AV-Test reference set (widespread malware from AV-Test's own collection). The protection result is given as a score from 0 to 6.

A 6 out of 6 is the top protection score. Sample counts change from round to round, so read the detection percentages in the lab's own report when two products are close. AV-Test documents its method in its published test procedures.

Why it matters

AV-Test's protection score is the most direct public measure of whether a product blocks malware. The real-world half matters most for a small business, because new threats are the ones a signature update has not caught yet.

Sources

  • AV-Test Institute. "Test Plan and Methodology for the Certification of IT Security Products." av-test.org

MITRE ATT&CK analytic coverage

MITRE ATT&CK Evaluations test endpoint products against emulated adversary campaigns (for example, ransomware operators or nation-state actors). The evaluation records whether each attack substep was detected and, if so, at what level: Technique (highest, maps to a specific ATT&CK technique), Tactic (maps to a broader category), General (detected but not classified), None, or Not Applicable.

Endpoint Index reports analytic coverage as the percentage of substeps that received any detection (Technique, Tactic, or General). A product that detects 80 out of 80 substeps has 100% analytic coverage. We count MITRE's default results view: the initial run, before any configuration change. MITRE also publishes results obtained after a vendor changed its configuration mid-test; vendors often quote those higher figures, and we mention them only alongside the default ones. The spread across levels matters too: a Technique-level detection tells an incident responder what the attacker did, while a General detection only says something happened.

Why it matters

MITRE ATT&CK Evaluations test a product against a full attack chain run by an emulated adversary, not against individual malware files. That makes them the closest public test of whether an EDR product sees an intrusion in progress. MITRE does not rank or score participants, so Endpoint Index reports coverage and leaves the ranking to the Fit Score formula.

Sources

  • MITRE Engenuity. "ATT&CK Evaluations Methodology." mitre.org
  • Strom, B.E., et al. (2020). "MITRE ATT&CK: Design and Philosophy." MITRE Technical Report MP180360R1, revised March 2020. mitre.org

AV-Comparatives false positives

AV-Comparatives tests business endpoint products in a real-world protection test and a malware protection test, each running for several months. The figure to watch alongside protection is false positives: how many legitimate files, URLs or applications the product wrongly blocks.

For a business, a false positive means a blocked installer, a quarantined internal tool or a website staff cannot reach. The AV-Comparatives test uses both common business software (where false positives are most disruptive) and other clean files as test sets.

A product with high protection (99% or higher) but many false positives (more than 10) can cause more day-to-day harm than one with slightly lower protection and zero false positives. Endpoint Index shows both figures side by side so you can make that trade-off yourself.

Why it matters

A small business rarely has anyone to investigate a false alarm. One false positive on a line-of-business application can stop work, and every exclusion added to get that work moving again is a gap in protection.

Sources