AV-Comparatives

AV-Comparatives
Name	AV-Comparatives
Type	Non-profit
Founded	2004
Headquarters	Innsbruck, Austria

AV-Comparatives is an independent Austrian testing laboratory specializing in antivirus software, endpoint security, and malware protection evaluations. Founded in the early 21st century amid rising computer virus outbreaks and the aftermath of incidents like the I Love You and Code Red worms, the organization conducts comparative tests used by vendors, researchers, and regulators including entities related to Microsoft, Google, Apple Inc., and Cisco Systems. Its reports are cited by media outlets such as the BBC, The New York Times, and specialized publications like Wired and Ars Technica.

History

AV-Comparatives emerged after a series of high-profile incidents including Nimda, Melissa, Sasser, and concerns highlighted by institutions such as the European Commission and agencies like CERT/CC. The lab developed testing programs parallel to historical efforts by groups like the Istanbul Technical University security teams and the Virus Bulletin comparative work, while interacting with vendors including Symantec, McAfee, Kaspersky Lab, Trend Micro, ESET, and Avast. Over time it expanded testing regimes to reflect shifts following events like the rise of ransomware campaigns exemplified by WannaCry and NotPetya, and regulatory scrutiny informed by rulings from courts connected to entities such as the European Court of Justice.

Organization and Funding

The organization operates from Innsbruck and collaborates with academic partners including faculties at the University of Innsbruck and industry stakeholders such as Intel, AMD, IBM, Dell Technologies, and cloud platforms like Amazon Web Services and Microsoft Azure. Funding sources combine membership fees from vendors including Bitdefender, Panda Security, Fortinet, and Sophos with grants and project-based support related to initiatives involving ENISA, NATO CCDCOE, and national research councils such as the Austrian Science Fund. Governance involves a board and testing teams composed of researchers with backgrounds at institutions like G DATA Software, F-Secure, ESET Research, and collaborations with analysts from Krebs on Security and think tanks such as the Center for Strategic and International Studies.

Testing Methodologies

AV-Comparatives publishes test methodologies covering on-access protection, performance, false positive rates, and real-world protection based on representative malicious software samples sourced from feeds used by organisations like VirusTotal, Hybrid Analysis, Any.Run, and partners such as Palo Alto Networks and FireEye. Methodologies cite standards and practices related to testing used by groups like ISO committees, and incorporate techniques developed alongside researchers at Carnegie Mellon University, Massachusetts Institute of Technology, Stanford University, and security conferences such as Black Hat USA, DEF CON, and RSA Conference. Tests include reproducible procedures for sample selection, environment setup with virtualization platforms from VMware and VirtualBox, and telemetry analysis compatible with tooling from Splunk, Elasticsearch, and Wireshark.

Product and Certification Programs

AV-Comparatives runs certification and award programs such as the yearly "Product of the Year" and "Advanced+" rankings, engaging vendors like Microsoft Defender, Kaspersky Endpoint Security, NortonLifeLock, andMcAfee Enterprise. Certification criteria are published for functional areas including anti-phishing, mobile protection for platforms like Android and iOS, and business endpoint suites used by enterprises such as SAP, Oracle Corporation, and Salesforce. The lab’s performance benchmarks are used by procurement units in organizations like Deutsche Bank, HSBC, United Nations, and governmental agencies in countries including Austria, Germany, and United Kingdom.

Reception and Impact

Security researchers and journalists from outlets including The Guardian, Bloomberg, Forbes, ZDNet, and TechCrunch reference AV-Comparatives’ reports when discussing events like SolarWinds and supply chain attacks disclosed by FireEye and Mandiant. Its comparative data feeds into vendor marketing and product roadmaps at companies such as Trend Micro, ESET, Bitdefender, and Avira, and informs procurement by institutions including European Commission bodies, NATO, Interpol, and large enterprises such as Siemens and General Electric. Policymakers and standards bodies including ENISA and national cybersecurity centers cite the lab’s methodologies when shaping guidance influenced by incidents like Equifax data breach.

Controversies and Criticism

AV-Comparatives has faced criticism and debate common to testing organizations, involving disputes similar to past controversies around Virus Bulletin and independent labs, with vendors such as McAfee and Kaspersky Lab occasionally contesting test conditions comparable to criticisms leveled during AV-Test comparisons. Critics from companies and commentators in publications like SC Magazine and Help Net Security have argued about sample selection, testing transparency, and potential conflicts of interest tied to vendor-funded membership models seen in other industries involving firms like Gartner and Forrester Research. The lab has responded by publishing detailed methodologies and by engaging with stakeholders including academics from ETH Zurich and TU Wien to improve reproducibility and address concerns raised in forums like HELDred and panels at RSA Conference.

Category:Computer security organizations