The Limits of Automation in Website Accessibility: How Scans Help and Overlay Widgets Fail

There is still confusion when it comes to website accessibility and what can be done through automation. So let’s start with all work must be performed manually, and that is the case when we are auditing a website, so finding accessibility issues and remediating a website, fixing accessibility issues.

And we can’t combine automation with manual work, so we can’t install an overlay widget and say that overlay widget fixes 70% of issues, and then we just need to fix the remaining 30%.

And similarly with the scan, we can’t run a scan and say that scan has caught 30% of accessibility issues, and then we just have to refine the remaining 70%, that’s not how this works.

So let’s start with scans, scans are very helpful, but they are limited. With a scan, you might be able to flag or partially flag 12-15 accessibility issues. It really depends on who you ask and what accessibility scan you’re using. But think of a scan as being able to flag or partially flag about 25% of accessibility issues under WCAG 2.1 conformance level AA.

So in 2.1 AA, there are 50 success criteria or requirements, and the scan is going to flag or partially flag about 12-15 of those, so that’s very helpful. It reduces incidence of error,
it speeds up the process. But we can’t solely rely upon that scan to find accessibility issues, and a big reason why is because scans can have false negatives.

So that means a scan will not return an error or alert when an accessibility issue exists. And the reason that is so is because scans are using rulesets, and that’s all that’s happening, is someone has programmed to look for x, and if x is there, then return y result, and that does not- there can be many times where an accessibility issue exists, but a scan’s rulesets will not catch that accessibility issue.

And so for that reason, we still have to look through everything manually. But we can use the scan to help with again reducing error and speeding up the process, but we can’t rely upon that scan, remember, I said flag or partially flag. What that means is it’s raising our attention- it’s bringing our attention to a potential accessibility issue.

But it doesn’t mean that the issue necessarily exists, so there could, in rare instances there, are false positives, but again, I’m more concerned with false negatives where a scan does potentially flag an accessibility issue, but it hasn’t done so in the case of our website.

So even though scans are really helpful, we still need to manually review the entire website. And now, when it comes to overlays, they don’t help at all there may be marginal benefit, because what’s happening with an overlay is there is a script that’s literally laying over the website, and so that script is rendering adjustments that are literally laying over the website.

So it’s not the script that’s a laying over the website, but the script is rendering adjustments that lay over the website, but the website itself, the code and the content has not been remediated, so it remains inaccessible.

And this is not to say that even the adjustment rendered has made the website accessible even when it’s activated. This is just to say that there are automatic adjustments that are being made and rendered on the website.

But they don’t make the website accessible, and in fact, overlays can actually introduce accessibility issues, and that has been the case time and again, or that has been shown through many demonstrations. So any marginal benefit to using an overlay is outweighed by the problems it introduces, and we have to keep in mind that it’s not making our website accessible.

It’s just making some adjustments that lay over our website. So keep in mind there may be a pop up blocker or an ad blocker that exists that blocks the overlay widget from even showing, or someone may not notice the overlay widget on the website, or someone may not want to use the overlay widget, or again, it doesn’t make your website accessible anyway.

And if you’re concerned with litigation, plaintiffs’ lawyers have repeatedly shown that they disregard overlay widgets and they don’t think the website is made accessible by them. So there are really nothing, you shouldn’t use them; you should avoid overlay widgets altogether.

But the point of this video is that automation can only take you so far, and the way to think of automation in terms of website accessibility is there are website accessibility scans, or they’re commonly referred to as checkers.

These scans, through automated rulesets, help you flag certain accessibility issues. But it’s important to remember that they are limited in the accessibility issues they flag, so there are many accessibility issues that remain unchecked by scans. But even the accessibility issues that are flagged by scans, we still need to manually review them because the scans are not conclusive.

And importantly, there may be a false negative or a scan has not detected an accessibility issue that remains on your website, so if you solely rely upon a scan, you could still have a perfect score on the scan returning zero errors, and accessibility issues can remain.

So when it comes to automation, just remember that everything must be- all work must be performed manually.