Automated accessibility testing plays an important role in identifying problems early in the development cycle. Organizations often use multiple tools to conduct automated testing leading to a confusion on what results are accurate. In addition, these tools produce different results, even when scanning the same webpage. This raises important questions for teams working toward accessibility:
- Which tool detect the most issues?
- Which tool have the fewest false positives?
- Which tool identifies the issues that matter?
To answer this, we conducted a comparison using BarrierBreak A11yInspect alongside several widely used automated accessibility testing tools. Each tool scanned the same pages for Web Content Accessibility Guideline (WCAG) 2.2 Level AA to ensure a fair and meaningful evaluation.
The goal of this study was to benchmark the accessibility testing tools.
Accessibility Tools in the Comparative Study
Total of 8 tools were compared in this study, namely:
- A11yInspect
- WAVE
- axe DevTools
- Silktide
- ARC Toolkit
- Siteimprove
- Microsoft Accessibility Insights
- Lighthouse
Pages Tested
We evaluated all tools using three sample pages containing accessibility issues created by Ted Drake. These intentionally problematic pages represent common failures across structure, content, and interaction patterns. The pages are publicly available in the following GitHub repository: https://github.com/7mary4/a11y-control-test/tree/master
You can find the list of pages along with their URLs in the table below.
| Page Name | URL |
|---|---|
| Product – Bad | https://fyvr.net/data-bad.html |
| About – Bad | https://fyvr.net/content-bad.html |
| Apply – Bad | https://fyvr.net/apply-bad.html |
Test Results Across Automated Testing Tools
The table below shows valid WCAG 2.2 Level AA failures detected on each page and in total.
| Page | A11yInspect | Tool 1 | Tool 2 | Tool 3 | Tool 4 | Tool 5 | Tool 6 | Tool 7 |
|---|---|---|---|---|---|---|---|---|
| Product – Bad | 2 | 0 | 0 | 1 | 1 | 0 | 0 | 0 |
| About – Bad | 20 | 10 | 18 | 8 | 16 | 13 | 18 | 17 |
| Apply – Bad | 37 | 22 | 29 | 28 | 37 | 21 | 29 | 25 |
| Total | 59 | 32 | 47 | 37 | 54 | 34 | 47 | 42 |
Key Takeaways
- A11yInspect detected the highest number of confirmed WCAG failures across the tested pages followed by Tool 4 and Tools 2 & 6 have a tie.
- 5 tools didn’t identify any failure on Product – Bad.
- On About – Bad, most tools average between 13 to 18.
- On Apply – Bad, A11yInspect and Tool 4 were the only tools that identified more than 30 issues.
Analysis
There is a clear differences in detection depth across tools. BarrierBreak A11yInspect consistently identified more meaningful WCAG failures, demonstrating stronger coverage of structural, semantic, and programmatic issues compared to competitors.
- Product – Bad: Most tools found little or nothing, with some of the tools reporting zero issues. A11yInspect led the page with two confirmed failures, while Tools 3 and 4 detected one valid issue each. Tool 3 also reported four additional items that were determined to be invalid or false positives on table heading scope, indicating differences in detection accuracy.
- About – Bad: Several tools detected common content issues, yet A11yInspect reported the highest number of failures on this page, reflecting broader coverage across both structural and metadata-related accessibility failures. Tool 1 & Tool 3 had a false positive.
- Apply – Bad: The gap became decisive on this complex page. A11yInspect identified the highest number of failures, tied with Tool 4, highlighting the importance of deep analysis capabilities for detecting programmatic accessibility issues that many tools struggle to capture. Tool 3 identified 3 false positives.
Methodology
- Automated Accessibility tools report findings using different terminology and severity levels, such as alerts, recommendations, advisories, errors, or critical issues, we standardized the assessment by considering only issues classified as confirmed WCAG failures or errors.
- Informational alerts and best practices were excluded to avoid inflating the results.
- Competitor tools data has been anonymized as Tool 1 to Tool 7.
- To ensure accuracy, we verified issues identified by A11yInspect and competitor tools to confirm that they were valid WCAG failures and not false positives. Focusing only on verified failures ensures the comparison reflects genuine accessibility barriers and provides a more accurate measure of each tool’s effectiveness.
Why This Matters for Organizations?
Over the last 10+ years, the industry has been focusing on pre-dominantly one open source ruleset engine. There are numerous tools who are just wrappers using this ruleset at the back of them. The progress has been slow.
As organizations who are focusing on accessibility, it is important to:
- Select the appropriate tool with the maximum coverage while ensuring a low number of false positives.
- Ensure that the tool is easy for developers to understand and not have to interpret accessibility.
- Using multiple tools is not the answer, it only leads to confusion for the developers.
If we want to truly shift accessibility left, giving the right tools to developers should be our focus.
Reducing technical debt should be one of our strategies to implement accessibility, thereby reducing rework & lowering remediation cost.
Conclusion
Automated accessibility testing tools are the starting point in your process. A11yInpect empower developers to push a build that implements the most basic accessibility challenges.
Thereafter doing manual accessibility testing will ensure that we build better products that are inclusive and accessible.
Some of the tools in the industry claim 80% or more WCAG coverage but the data shows otherwuse. As seen in the study, A11yInspect detected more issues than any other tool.
About BarrierBreak A11yInspect
In 2025, BarrierBreak released A11yInspect, a browser extension for accessibility testing that helps teams detect and resolve accessibility issues quickly and accurately. It combines deep automated scanning with actionable diagnostics, including element highlighting, code insights, and annotated screenshots to streamline remediation. With more than 530+ automated checks aligned with WCAG 2.2 Level AA and other accessibility standards, A11yInspect identifies complex issues in forms, ARIA implementation, structure, labelling, and metadata, enabling organisations to build more inclusive digital experiences.