Updated:

Speech recognition and complex tables with fields

by Whitney Lewis

This might seem like an oddly specific blog article. We’re talking about speech recognition, an assistive technology that doesn’t get talked about nearly as much as screen readers. And not just any tables, but complex tables with input fields or links baked into the cells.

If you’re designing or developing tables like this, you’re in the right place. You’ll get specific considerations for building complex tables with speech recognition in mind.

Even if you’re not building complex tables, this article gives an interesting look at the process of learning about AT, testing assumptions, and rethinking. That’s the part worth taking with you, whether or not you ever touch a table again.

What is speech recognition software?

A common example of speech recognition software is using speech-to-text to dictate a text message. You might also be familiar with the frustration of it getting what you said wrong and then having to go and correct it.

The speech recognition software we’re demoing and considering today is similar. But instead of just dictating words, it also lets people use their voice to navigate. Users can say things like, “Click here,” or “Scroll down.”

Much like dictating a text message that’s supposed to say “See you tonight” only to find it says “Sea you fright” is frustrating. It can be frustrating for a speech recognition user to say “Click [button name]” and have nothing happen. Or, not know what to say because there is no visual label.

Back to top

Mac’s Voice Control demo

There are several speech recognition software options: Windows Voice Access and Mac’s Voice Control are free software built into Windows and Mac computers. Dragon is a paid option that’s historically been popular, though it’s no longer available for Mac.

In this video, we briefly demo Mac’s Voice Control selecting text and clicking a link:

For more on how to setup Voice Control, watch our Speech Recognition Basics: Setup and a Quick Table Demo video.

Back to top

How I tested Voice Control and Voice Access

As promised, this article is more than “here’s how to handle complex tables with buttons and links accessibly.” It’s also, “here’s how you can go through a similar process yourself.”

This matters because your complex table, application’s custom components, or website’s design is unique. The web is getting more complicated.

WebAIM found that homepage complexity has nearly doubled in the last seven years, with more elements packed onto pages every year. Testing with AT to find issues and create better experiences is critical with this increased complexity.

This is how it went for me:

Starting point

I had a specific custom component. A complex table with row actions, form fields, and links.

I researched accessibility concerns with speech recognition. I looked into what already existed on the topic, what potential considerations and issues to start testing, and how tools like Voice Control and Voice Access work.

Testing

I spent five minutes getting familiar with Mac’s Voice Control. I hadn’t used it before, so I turned it on and poked around. I said “Click” on a few things, tried “Show numbers,” and got a feel for how it behaves.

I tested what I researched in controlled environments. I built demo tables that isolated specific issues I wanted to test, then tested with Mac’s Voice Control and Windows Voice Access. That let me get a feel for correct and incorrect behavior. I learned that some things I thought would be an issue weren’t. I also learned that some things I thought would work didn’t. Going beyond research and doing human testing is critical.

I tested in Pope Tech’s app. Testing only a demo helped me understand what is desired and broken behavior. I then tried the same commands against a real complex table in our own app to see if there was code and design we could improve.

Logging issues

I logged a ticket with developers. Testing without acting on it doesn’t help any of your users. I found ways we could improve our tables, so I logged a bug to our developers’ project management tool.

Back to top

Complex tables and speech recognition considerations

These are the four considerations I found and confirmed through testing.

1. When every button says the same thing

If every row on your table has an “Edit” button, you’ve probably got a lot of buttons that all say the same thing: “Edit.” Same goes for “Remove,” “View,” or any other repeated row action.

That adds an extra step for speech recognition users. When you say “Click Edit” and there are ten buttons named “Edit” on the screen, the software can’t tell which one you mean. So it falls back to showing a number next to every matching option, and you say the number instead.

This video demos multiple buttons with the same name and that number fall back:

This isn’t necessarily a problem. My research flagged the number fallback as a real issue for speech recognition users, but testing told a fuller story: It’s a visually noisy and probably worse experience for other visual users to have uniquely named buttons in every row (now every button says Edit [website name] instead of just Edit).

Falling back to numbers is a reasonable cost of a data-dense UI, not a failure state.

2. The visual label doesn’t match the accessible name

Speech recognition users say what they see to select an element. So the visible text on a button or field needs to be part of what assistive technology interprets as that field’s name (also called the accessible name).

This is WCAG Success Criterion 2.5.3: Label in Name, and when the accessible name doesn’t have any of the visible text as part of it, it fails this criterion.

For example, let’s imagine a column named Details with arrow buttons. The only visual label is Details, so a speech recognition user would say “Click Details” and land in the number fallback, since more than one button shares that name. That’s an expected user flow.

But if the arrow buttons got an aria-label or a CSS hidden text label that is Edit, then saying “Click Details” won’t do anything because the buttons aren’t named Details. They’re named Edit. This is a straightforward failure of 2.5.3: the accessible name (Edit) doesn’t contain the visible label (Details) at all.

When I tested this, Voice Control usually failed silently, meaning I got no feedback and would repeat the action several times before trying something else. I ended up saying “Show names” to find out what the buttons were actually called.

This video demos Voice Control’s Show Names navigation method:

The fix is simple: make sure your visual label is part of the field’s or button’s accessible name and is not something completely different.

3. Visual label isn’t there

The other issue with visual labels is having none at all.

For example, let’s take an Actions column with two buttons in it: an edit pencil icon and a delete trash can icon. There are only icons with no visual label, and the visual label for the column isn’t helpful since there are two actions they could take.

As long as there is an accessible name on the icon, this technically won’t fail WCAG, but it’s still a design consideration for speech recognition users.

The user could make a good guess based on the icons. But with no visual label, they could be guessing between “Edit,” “Modify,” or “Update.” For the trash can, they could be guessing between “Delete” or “Remove.”

When I tested this, I had to say “Show names” to find out what the buttons were actually called. The name ended up being long, and I had to repeat it a few times before it registered. In practice, I’d probably use “Show numbers” instead to select it quickly rather than trying to say the full name each time.

Watch this video for a demo of this example:

Again, the fix can be simple. Instead of a single Actions column, adjust it to two Edit and Delete columns. Now, there is no guessing about what the speech recognition user should say.

If splitting columns isn’t practical for your table, adding visible text next to each icon works too. It keeps a single Actions column, but gives the user something real to look at and say instead of guessing based on the icon alone.

If none of these are practical for your table, at the very least, make sure your naming is consistent throughout the table and in other tables. That way, the user can learn what they are called once through fallback navigation and use them when they encounter the icons again.

4. Differences between Voice Control and Voice Access

There is a difference between how Mac’s Voice Control and Windows Voice Access listen for commands. Mac’s Voice Control has to be an exact match. For example, if the accessible name is, “Edit User 1,” the user has to say that exact phrase to trigger the command.

Windows Voice Access accepts a partial match. If the accessible name is “Edit User 1,” the user could say “Click Edit” or “Click User 1” and it would trigger an action.

When I tested this, Voice Access was easier in practice: a partial match just falls back to numbers for everything with that word in the accessible name, so I didn’t need to know the exact phrase to get moving.

Mac was a different story. I noticed I had to rely on “Show names” a lot more when the visual label wasn’t clear, since a partial guess doesn’t trigger the numbers fallback the way it does on Windows.

Watch this video to see a demo of using exact phrasing with Voice Control versus a partial match with Voice Access:

The most helpful choice a designer or developer can make to help with this is keeping their accessible names consistent.

In the next section, we cover a suggested pattern for naming table fields. If you use this pattern consistently, Mac’s Voice Control users can learn it and know how to interact with your complex tables.

Back to top

The suggestion: Give interactive fields in the table a unique name

So how should you label your tables with form fields, links, and buttons?

Our suggestion is to use Adrian Roselli’s Uniquely Labeling Fields in a Table technique.

Here’s the pattern:

The video in the Differences between Voice Control and Voice Access section above demos this pattern in action with Voice Control and Voice Access.

Limitations

This isn’t a perfect fix, and there are some trade-offs.

Giving every field a full, unique name that comes from visual labels solves the guessing problem for speech recognition users. Especially those using Voice Access, where a partial match works. Voice Control users might have to learn the pattern.

This naming convention can get repetitive for screen reader users navigating with table navigation shortcuts. As they move from column to column, a screen reader announces the column header on its own, then announces the field’s full accessible name right after, so you end up hearing the same words twice in a row. Roselli found the same limitation in his testing.

One way to mitigate a noisy screen reader output is keep your row and column headers short and predictable. Avoid pulling in user-generated content as a header, and instead use clear, consistent labels over anything long or variable.

Disability needs can sometimes overlap and even conflict like this, so it’s about finding the best balance between them.

Back to top

Takeaways

Here are the takeaways:

  1. Testing with assistive technology like screen readers and speech recognition is possible. There are free versions of both on Mac and Windows. It’s also critical because the web is getting more complex. Even a little testing can lead to small tweaks that drastically improve the experience.
  2. Use visual labels and make sure they match the field’s accessible name.
  3. Label your table cells with fields in them with arialabelledby and reference the ids of the column and row headers. This has trade-offs but makes it more robust for more users.