Improve effectiveness and accuracy of web accessibility testing with sampling

Effective and accurate web accessibility testing uses both manual and automated testing.

Automated testing quickly finds several accessibility issues while manual testing finds issues automated testing can’t find. Together, you have a good sense of how accessible your content is.

Finding the majority of accessibility issues sounds great, but some of the first questions you’ll have are:

• Do I need to manually test my entire website?
• How often should I do manual and automated testing?
• How do I handle testing updated content?

The answers to these questions come with sampling (or picking parts of your content) for automated and manual testing. Using sampling in your accessibility strategy means you’ll get the most out of your accessibility testing without spending all your time and resources continually testing content.

In this article, we’ll give sampling strategies for manual and automated accessibility testing:

• What sampling is and how it applies to web accessibility
• Regular cadence testing sample strategy
• Critical content updates sample strategy
• Getting started

What sampling is and how it applies to web accessibility

At its core, sampling is picking a subset of something from the larger group. This subset is what you’ll test. The idea is this subset is a good representation of the larger group.

When it comes to web accessibility, the subset is web content – it could be pages, specific components like a form, or a template piece like a header or navigation. The larger group is the entire website.

Both automated and manual testing benefit from the idea of sampling.

Automated testing finds a subset of accessibility issues (the ones a computer can detect). But, it can quickly find these issues on every page.

Manual testing is done on a subset of pages or user flows making it realistic to actually do regular manual testing. Ideally, this subset is a good representation of the entire website because it includes components and template pieces that are reused throughout your website. This means you’ll find issues that impact more than just the subset of pages.

When orgs use sampling in their accessibility strategy it means more accessibility testing isn’t always better. Now, it’s about picking the right test, subset of content, and frequency for your org’s needs.

Each strategy and example we’ll go over below is built by answering these three questions:

1. What test are you doing?
2. Which web pages, user flows, or content get that test?
3. What is the frequency of the test?

Back to top

Regular cadence sample strategy

Both automated and manual testing should be done regularly to keep up-to-date on any issues. But, the test, content subset, and frequency are different for each.

Monthly automated accessibility testing

Let’s start by answering the three questions to build the regular automated accessibility testing sample:

1. What test are you doing: Automated accessibility testing
2. Which web pages and user flows get the test: All web pages and user flows
3. What is the frequency of the test: Monthly

Automated accessibility testing is already a subset of accessibility issues, so the power of automated testing comes from being able to test all your content each month.

If you’re not able to test all your content, pick the most important web pages and user flows first then add other pages from there.

Annual manual accessibility testing

The regular manual accessibility testing sample is:

1. What test are you doing: Manual accessibility testing
2. Which web pages and user flows get the test: Critical pages and user flows
3. What is the frequency of the test: Yearly

When you pair manual and automated testing, it means your manual tests can focus on the issues you can only find with manual testing. So, even though you could find every issue with manual testing, ideally most issues are found with the monthly automated testing.

Since manual testing can take a lot of time, we suggest doing it once a year on a subset of content. This makes it realistic for organizations to actually do.

Our suggestion is a sample of at least 4 pages and/or userflows. More complicated websites and apps might need more to create a representative sample.

There are two goals when picking your sample:

1. Some of these 4 pages and user flows have components and template pieces that are on all pages. That way, it’s a good representation of the whole website meaning you’ll find issues that affect every page even though you’re only manually testing 4 pages.
2. Choose high-impact pages and user flows. Pages and user flows that are used the most and align with your org’s goals should be included in the sample.

Example

For our example, we’ll say we have an e-commerce website. Every month automated testing is done on every page – including the checkout user flow.

Every year, these 3 pages and 1 user flow are manually tested:

1. Home page
2. Contact us page
3. The entire checkout user flow (from product page to checkout)
4. Random page

Together, each of these pages includes components like signing up for emails and product listings. They also catch each template piece like the header, navigation, content area, and footer.

The most important user flow for users and the business is the checkout process. So, that process is manually tested each year. It starts on a product listing page, goes to the product page, and then through checkout pages that include forms. It also includes error flows users can experience during checkout.

Since the organization has regular automated and manual testing, they find accessibility issues quickly, which is the first step to a more accessible website.

Back to top

Critical content updates sample strategy

In addition to monthly automated testing and yearly manual testing, organizations should also do automated and manual testing whenever there’s a critical update.

“A critical update includes website template updates, new interactive components, or updates to user flows. A good rule of thumb is if it triggers any sort of change management process to users or employees and it’s a change to web content, then it’s a critical update.” – tweak this to be more suggestive, so orgs need to define this themselves – its possible some orgs trigger this with all updates

Your organization will define what types of updates trigger automated and manual testing. But, some examples of critical updates include website template updates, new interactive components, updates to user flows, or updates that trigger any sort of change management process to users or employees.

It doesn’t include every time a new page is published or minor bug fixes in the website’s code. The content writer or developer can use the WAVE extension to check individual pages as part of their workflow.

This sampling method could be used as part of the change management process for critical content updates:

1. What test are you doing: Automated and manual accessibility testing
2. Which content gets the test: Updated content or page with updated component or template piece on it
3. What is the frequency of the test: Whenever there’s a critical update

The automated and manual testing is only done on the content that was updated. So, you’d run an automated testing scan on those pages that include the change and could manually test only that component or template piece on that page.

Example

As an example, let’s say we have software that tracks employees’ time. We updated the dialogue box users fill out to manually enter time. This is critical content because it’s part of a key user flow.

We’ll run an automated scan on the page where the dialogue box is to find detectable issues. Then, we’ll manually test that dialogue box. We don’t need to test every other page or other components because they weren’t updated.

Since the organization runs accessibility tests whenever critical content is updated, they avoid introducing barriers for people with disabilities meaning a better experience for users from the start and fewer issues to fix later.

Back to top

Getting started

Finding and tracking all the issues isn’t very helpful if nothing gets fixed. So, you can always start small. You could start with one page or one accessibility strategy.

For example, if you start with one page, your sample strategy might be:

1. What test are you doing: Automated and accessibility testing
2. Which web pages and user flows get the test: Homepage
3. What is the frequency of the test: Monthly automated testing and yearly manual testing

If you start with one accessibility strategy, your sample strategy might be:

1. What test are you doing: Testing images for alternative text (Use an automated testing tool to find all images and manual testing to review alternative text)
2. What content gets the test: All images (could make this smaller with just critical content)
3. What is the frequency of the test: Every March when the accessibility focus topic is alternative text

Actually fixing one thing is better than testing 100s of pages without fixing anything. It’s then easy to build on the small start into something bigger.

Back to top