If website accessibility is part of your work, chances are you rely on automated testing. Click a button and let it do its thing, right?
But have you ever wondered what goes into creating the rules that govern accessibility testing? Or who makes the rules?
As a long-standing participant in, and former co-chair of, the Accessibility Conformance Testing Rules Community Group (or as we call it, ACT-R), I can tell you that the rules don’t come out of nowhere. Each one takes hours to write, review, and tinker with, by multiple people, often over the course of days. By my estimate, development of each new testing rule takes a total of about 50-200 hours and requires the expertise of 4-5 people.
Speaking for myself, I dedicated a day per week to reviewing and writing rules. As co-chair, I also hosted a bi-monthly call with the group and synced each week with my fellow co-chair, Carlos Duarte, an expert in Human-Computer Interaction (HCI) from the University of Lisbon.
It does take a lot of time, but it’s also incredibly gratifying: Reliable testing rules are key to detecting failures and prevent false positives (alerts that there’s a problem when there really isn’t), which means there are digital barriers to people with disabilities, and they go a long way toward saving time for web ops experts. A win for inclusivity and business!
When I wrote this, I was deep in the weeds of WCAG 2.2, the latest version of the Web Content Accessibility Guidelines, which was released in October 2023 and required hard work from accessibility experts like me, Carlos, and others to pore over the success criteria and do our part to develop the testing rules that uncover issues with meeting those criteria.
Why ACT exists
Right out of the gate, it’s important to know that people can interpret WCAG criteria differently, and that’s why Accessibility Conformance Testing exists. For example, ambiguity in the success criterion WCAG 1.4.3 for color contrast, which ensures that there’s sufficient contrast between the text and the background colors so people with color vision deficits can read the text, can lead to designers to calculate contrast ratios differently; this inconsistency naturally creates problems for users who are colorblind.
ACT ensures that we all agree on the same methods to assess if a criterion, such as color contrast, is met or not.
An ACT rule is a precise test for one part of a WCAG success criterion. It defines which elements the rule applies to, what each of them must satisfy to pass, and a set of test cases that let anyone check whether a testing engine implements the rule correctly. Every rule follows the ACT Rules Format, whose version 1.1 became a W3C Recommendation in February 2026.
Rules written by the Community Group have no formal standing at W3C until the Accessibility Guidelines Working Group approves them. W3C keeps a list of ACT rules that shows which are approved and which are still proposed.
How a new accessibility testing rule gets made
So, how are new testing rules created? And how does everyone in the ACT-R Community Group agree on a rule?
It’s typically a matter of intensive review, tinkering, and building consensus. Once anyone from this group drafts a rule, three participants from the group who come from organizations ranging from Siteimprove or other accessibility testing providers like Deque to companies as large as IBM, CVS, and Airbnb adjust, tinker, debate, agree, and disagree before arriving at consensus and moving it along to the W3C’s ACT Task Force for review. The Accessibility Guidelines Working Group, which maintains WCAG, then decides whether to approve it.
Along the way, anyone else in the group beyond those three can jump in with their feedback and sniff out any problems with it. If one of the participants does find an issue, the rule goes back to the start of the process.
Once the Working Group approves the rule, W3C publishes it and links it from the relevant Understanding WCAG page.
What we look for
These W3C-approved testing rules form the core of automated accessibility checks. Siteimprove’s checks have been built on ACT rules since Siteimprove started helping write them in 2017, and we take great care in making sure that the rules are robust and reliable because there are certainly scenarios in which automated testing fails: It doesn’t flag issues that should be flagged.
There are two main blockers the group considers when assessing if a success criterion can’t be automated:
- Heading off false positives: We can easily detect some valid patterns but can't ensure the page doesn't implement a more tricky but still valid pattern. So, we can never be 100% sure of a failure (automatically), and thus do not want a rule with a lot of false positives.
- Valid reproduction: Sometimes criterion simply can’t be tested by software because the actions aren’t ones that can be reproduced. For example, an automated tool can’t mimic dragging and dropping with a mouse.
False positives are our primary concern: If a test produces too many false positives, we can’t in good conscience recommend automated testing. After all, reviewing content issues that don’t actually exist wastes enormous amounts of time, including for our customers. (Just to give you an idea of how needlessly messy it can get, I’ve seen a single faulty testing rule create tens of millions of false positives across our customer base. Obviously, we were quick to fix it in an update).
After the WCAG 2.2 release, we gave the green light to automating just one new testing rule, for 2.5.8 Target Size (Minimum), because there was only one that the ACT-R Community Group determined could produce adequate results. The other new success criteria still need manual testing.
Unfortunately, some other companies that offer accessibility testing claim that their tools automate more rules than is actually possible from ACT’s collective point of view. It might seem harmless enough, but this practice inevitably yields poor results that create unnecessary work, and it risks souring teams against accessibility in general, which is never a good thing.
It’s not unlike the buyer beware situation with claims that overlays can magically make websites compliant: Those might promise the world, but companies that use them are still at risk of being sued.
A common estimate in the accessibility community is that only about one-third of WCAG success criteria can be tested fully automatically. That figure counts success criteria, not individual issues. All the other criteria require some level of manual testing by a person who knows what to look for and what disabilities to replicate.
Some testing engines now add AI-assisted checks, or guided reviews in which a person confirms a flagged item, on top of their ACT-based rules. Those can extend coverage, but they aren’t ACT rules. So when a vendor quotes a coverage figure, ask which kind of check it’s counting.
How to avoid shortcuts that short-circuit
Automated testing certainly saves a lot of time and money, when it’s appropriate, but if you rely on it too much, you won’t be protected from all compliance issues, nor will you be assured of providing all your users with an accessible experience.
Your best bet is to go with a vendor that puts its money where its mouth is by being active in the accessibility testing community. Also, be sure to ask questions about how aligned the vendor’s testing engine is with ACT rules and note that W3C, the governing body for WCAG, provides implementation reports that check different testing engines against implementation of ACT rules. (Because Siteimprove’s testing engine, Alfa, is open source, you can easily see what we’ve implemented.)
If your web accessibility vendor doesn’t follow ACT rules, you run the risk of inconsistent testing that’s not in line with the W3C industry standard of accessibility testing, which in turn puts you at risk of running afoul of requirements like those of the ADA.
It’s not, however, just a matter of making sure you’re following the rules: It’s a matter of making sure that at every digital juncture, you’re truly serving — not alienating — the people who the rules were made to protect and include.
Updated by the Siteimprove editorial team in November 2026 to reflect changes in the ACT process.