Automated and manual accessibility testing work best together

In accessibility conversations, there’s often a divide between automated and manual accessibility testing. Organizations looking to “check off accessibility” invest in automated testing incorrectly thinking it is all they need to do. While manual testing is often thought of as the golden standard most organizations can’t realistically meet.

Like most classic debates (queue Apple vs. Android or Xbox vs. Playstation), there are strengths and limitations on both sides. Unlike classic debates, it shouldn’t be a debate at all. It’s not a battle of automated vs manual testing because automated and manual accessibility testing work best together.

In this article, we’ll cover:

• Two camps: Automated and manual accessibility testing
• The role of automated accessibility testing
• The role of manual accessibility testing
• Your new automated and manual testing accessibility strategy
• How to start combining them

Two camps: Automated and manual accessibility testing

Let’s get the perceived issues people have with both automated and manual testing out and in the open.

The beef with automated testing

Automated testing is when software scans code to detect accessibility issues. It gets a bad rap because it can’t catch everything AND beginners often rely on it like it does. The reality is that automated testing is one accessibility tool that helps detect some WCAG failures or aid in manual testing.

Adrian Roseli compares free automated tools with manual testing and puts it this way: “This does not mean you should avoid automated tools. Like any tool, it means they have a place in your toolbox. When used correctly they can be extremely helpful. When used in the hands of a novice they can result in a sense of complacency. When not used at all they can be a missed opportunity.”

The beef with manual testing

Manual accessibility testing is when human testers evaluate digital content for accessibility barriers typically using assistive technology like a screen reader. Web accessibility is a human-centered issue, so it needs humans (manual testing) to find issues and determine their impact.

The problem for most organizations is manual testing can require a lot of time and training to do successfully. There’s training in accessibility standards and best practices plus training in tools and assistive technologies to actually do the testing. This means most organizations won’t even try to include manual testing – it’s simply too much on top of everything else.

Bringing automated and manual testing together

Automated testing can’t find everything and for most organizations, continuous manual testing of every edited page and component isn’t realistic. But, the moral of the story is not accessibility isn’t possible for your websites.

The moral of the story is accessibility becomes very possible when we use both automated and manual together.

The path to a more accessible web is a better understanding of the strengths and limitations of automated and manual accessibility testing and how they can complement each other to make up for limitations and create a reliable accessibility strategy.

Back to top

The role of automated accessibility testing

Even though automated testing is limited by how many issues it can detect, it is very good at testing thousands of pages quickly and repeatedly. It is the breadth of your accessibility strategy taking a snapshot of your entire digital content environment to inform your accessibility health. It is also an aide to help quickly manually test potential issues.

Let’s review automated testing’s specific strengths and limitations.

Automated testing strengths

• Speed and volume. Automated testing can test thousands of web pages quickly.
• Consistency. Applies rules uniformly every time for the issues it can detect.
• Scheduling. Allows recurring, scheduled reports to stay on top of issues and keep accessibility top of mind.
• Makes manual testing easier. Using automated testing means finding less issues when you do manual testing. This means the time spent manual testing can be used for more nuanced issues that require more human interpretation.
• Manual testing aide. Automated tools can help people complete manual testing by finding instances of images, PDFs, videos, ARIA, labels, etc. for a human tester to review.
• Quick feedback. Immediate or quick results help more people get involved in accessibility since it’s an easier way to start incorporating accessibility into their processes. (Then, you can make the case to learn some manual testing methods).
• Reinforces training. Learning about alternative text, descriptive links, ARIA, heading structures, etc., and then being able to quickly review those results with the help of an automated tool starts to bring training into their real work in a manageable way.

Automated testing limitations

• Can’t find everything. There are debates about how many issues automated tools can find. Unless automated tools can find 100% of issues 100% of the time, which they can’t, the exact amount of what they can and can’t do is less important. Automated tools are meant to be one accessibility tool to help find issues – they aren’t meant to determine if a page is accessible or not.
• Lacks context. Automated testing often can’t determine if an accessibility strategy is applied correctly or the impact of an issue. For example, it can’t determine if alternative text for an image is equivalent or if incorrect alternative text is a small or large barrier for someone with a disability.

Back to top

The role of manual accessibility testing

Manual testing does take time and training, but it is possible to incorporate into your strategy on a smaller scale. You can do this by testing a smaller sample of pages for very specific accessibility concerns that only manual testing can find. Even on this smaller scale, manual testing is the necessary depth your accessibility strategy needs to understand your content’s accessibility health.

Let’s review manual testing’s specific strengths and limitations.

Manual testing strengths

• Determine impact. Web content is a human-centered interaction and only a human can evaluate the impact and experience other people may have. (Keep in mind, a person without a disability testing and determining the impact for someone with a disability is still not completely authentic.)
• Test assistive technology compatibility. A person can test with assistive technologies to find out first-hand how the content or interactions works with it.
• Potentially find every accessibility issue. With the right training and time, you can test every WCAG success criteria and other accessibility issues on the sample of pages you test.

Manual testing limitation

• Time and resource intensive. Only doing manual testing to find every accessibility issue on every page of a website would be impossible for most organizations. Unless you’re managing a small web site, there isn’t the time and resources needed to manually test every issue on potentially thousands of pages. For this reason, manual testing actually can’t test everything for a lot of organizations.

Back to top

Your new automated and manual testing accessibility strategy

In an accessibility strategy that combines automated and manual accessibility testing you get the breadth and depth needed to truly understand your web content’s accessibility state. Automated and manual testing’s strengths and limitations complement each other giving you a more reliable snapshot of your web content’s overall accessibility.

Here is the accessibility testing strategy that includes both automated and manual testing that your organization can realistically implement:

• Run automated accessibility testing on all web pages every month.
• Manually test up to four pages (homepage, critical page, critical page, and random page) with assistive technologies and the help of an automated tool each year.
• Run automated tests on all changes and manual tests on any critical template/component updates before they’re released.

Back to top

How to start combining them

Here are two ways to start combining automated and manual testing.

WAVE extension and manual tests

If you’re only testing a few pages, you could use the WAVE extension to run an automated test on individual pages and to help start manual testing on individual pages.

You can also start learning web accessibility strategies and how to test with assistive technologies to help you perform manual tests on your content.

Pope Tech platform with built-in manual testing support

If you have a website with several pages or multiple websites, you need something scalable.

Pope Tech is a web accessibility platform that combines automated and manual testing for a more reliable accessibility metric with the AIM score.

Our automated testing engine quickly finds issues and presents them in an actionable way on the dashboard while the AIM score wizard takes you through a manageable manual test step-by-step. This makes your automated and manual accessibility testing strategy more second-nature.

You can try Pope Tech’s Platform Free Plan – free for life and no credit card required – to start exploring how a comprehensive web accessibility solution can support your accessibility goals and strategy.

Back to top