Results of Quick Testing of Documentation Tools' Accessibility

When writing my blog post about what backend devs can do for accessibility, I did some quick accessibility tests for different documentation tools. Initially, I thought I'd write about the results in that blog post, but there was just too much to report. So, I ended up writing this blog post.

I chose four tools that I know many projects are using. These tools are:

Swagger
Read the Docs
Docusaurus
GitBook

First, I ran an automated test with the aXe browser extension and dug deeper into the DOM based on the test results. After that, I tested the site with a keyboard.

In this blog post, I'll share the test results and some thoughts about them. As you can see from the steps described above, the testing was not exhaustive. I just wanted to get an overview of the issues. I did not, for example, test with a screen reader or with narrower viewports (or zooming).

This was a conscious decision, partly because I didn't have time and partly because this blog post would've been super long. And I think this already provides some overview of the state of accessibility of these tools.

Let's get started.

Swagger

URL used for testing: https://petstore3.swagger.io/

Swagger is a set of tools used for API documentation. The Swagger API project started in 2011. Initially, it was a specification, but it was later named OpenAPI Specification. So it has a long history, and many projects use it.

One of its tools is Swagger UI, a user interface for browsing the API endpoints. That's the one I'm testing for this blog post.

Testing Swagger UI with Axe

Runnin Axe-browser extension showed 64 issues, two of which are critical, 35 serious, 18 moderate, and 7 minor. Most of the issues (37) were related to color contrast issues. The contrast ratio for the problematic color combinations ranged between 2.03 and 3.75.

Another problem was form input without programmatically associated labels. Some of them were flagged, but when I started digging deeper into the site's code, I noticed that not every instance was flagged. So, there were more problems with form labels than Axe was showing.

The third problem I want to raise is the lack of landmark regions. Not having these regions means that user, who needs these landmark regions for navigating, can't use them. For example, screen reader users and keyboard users who use tools to navigate using landmark regions would need to either listen to a lot more items and/or press a tab many more times to be able to navigate to certain parts of the website.

Testing Swagger UI with Keyboard

The next test was how well I could navigate Swagger UI with a keyboard. From this quick testing, it seems that Swagger UI is working somewhat ok with a keyboard. I could reach every element and interact with them.

Read the Docs

URL used for testing: https://docs.readthedocs.io/en/stable/tutorial/index.html

Read the Docs is an open-source documentation platform that was created in 2010. On their website, they say they host over 800 000 open source projects and support over 100 000 users.

Testing Read the Docs with Axe

Running Axe-browser extension showed 206 issues, 6 critical, 132 serious, and 3 moderate. For Read the Docs, too, the color contrast was the most significant cause of problems. The contrast ratio for the problematic color combinations ranged between 2.36 and 4.19.

Read the Docs uses ARIA-attributes - but not correctly. For example, some headings are annotated as headings with ARIA but without the level of the heading. Here's an example code from the site:

<p class="caption" role="heading">
  <span class="caption-text">First steps</span>
</p>

Another problem this automated testing revealed is a scrollable area, which is not keyboard accessible. In this case, it's a code-block visualizing warning messages. Lines are longer than the code block, adding horizontal scrolling to the element. And when a user, who uses a keyboard, comes to that particular element and wants to see what's at the end of the line (hidden by scrolling), they can't because that area is not keyboard accessible.

Read the Docs is also missing landmarks, so the same problems as for Swagger UI are present.

Testing Read the Docs with Keyboard

The search field is one of the first focusable items when navigating the page. When the user focuses on it, a modal opens. The "Close"-"button" is actually made of a div without a possibility to focus on it. Fortunately pressing Esc closes the modal.

When the modal is visible, the focus is set to the modal's first focusable element - the search bar. However, the focus is not trapped inside the modal, it's only moved there, and the modal is the last item on the DOM. The user can still reach the underlying links and other focusable elements if they navigate backward (which is definitely not optimal).

After closing the search modal, the focus is not managed. At least on my tests on Mac, Chrome, and Firefox, if I close the modal and then press Tab, the focus ends up in the first tab of my browser.

Combined with the on-focus behavior of the modal, this creates a sort of focus trap - the user can't reach the rest of the page by tabbing forward. Yes, they can do that by navigating backward with Shift and Tab, but that's problematic. And if the user can survive without the information on the page, they probably leave instead of trying to find all the ways they could navigate the site.

Another thing I wanted to mention from the keyboard navigation point of view is that there's no "Skip to content"-link. If the user somehow gets past the search modal, they need to tab through all of the links in the left side panel to reach the main content. On the example page, that means a lot of links to tab through.

Docusaurus

URLs used for testing:

https://docusaurus.io/docs and https://docusaurus.io/docs/styling-layout for automated testing
https://docusaurus.io/docs/styling-layout for other tests

Docusaurus is a static site generator created with React. On the Introduction page, they say that

It provides out-of-the-box documentation features but can be used to create any kind of site (personal website, product, blog, marketing landing pages, etc.).

The team behind the Docusaurus is from Meta, as it's one of its Open Source projects.

I'm testing with Docusaurus version 2. At the time of testing, it was still in beta. However, now it's published. I chose two pages to test so that I'd get a bit more components to experiment with.

Testing Docusaurus with Axe

Testing Docusaurus with Axe revealed 51 issues on the "Styling and Layout" page and 15 on the "Introduction"-page. From the first, 8 were "Needs review", 1 critical, 41 serious, and 1 moderate. On the "Introduction" page, 14 needed review, and 1 was moderate.

Most of the issues are about color contrast - on the "Styling and Layout"-page, 46 of the issues were about that. And some of the problems were because of examples of failing color contrast, so that's expected. Others were mostly from the code snippets' styles.

Other problems consisted of missing labels on a color input and non-unique landmarks.

Testing Docusaurus with Keyboard

Most of the problems I encountered when testing Docusaurus with the keyboard were related to focus management. For example, when I opened the search modal and left it without typing anything on the search field, the focus was not managed and ended up at the beginning of the page.

Another possible problem (although not a WCAG violation) is that Docusaurus uses the browser's default focus styles. In some cases, the focus is hard to see, especially if the theme of the documents is close to the default focus indicator's color.

One thing I noticed is the menu's dropdown-pattern Docusaurus uses. You can navigate the items on the dropdowns with Tab, but that's not the correct pattern for dropdowns in the menubar. Per ARIA Authoring Practices, the correct pattern would be that navigation happens with up and down arrows, and Tab gets you to the next widget.

I also found good things: Docusaurus has a "Skip to main content" link. It works - this is not always the case with skip-to-content links. Another nice thing is that the color picker in the "Styling and layout"-page works with a keyboard.

GitBook

URL used for testing: https://docs.gitbook.com/editing-content/rich-content/with-command-palette

GitBook is a documentation platform with lots of different features. It started as a simple open-source tool and grew into what it is today. It's free for open-source projects and priced for everyone else.

Testing GitBook with Axe

When I ran aXe on GitBook, there were 237 issues, from which 27 needed review, 2 were critical, 65 serious, and 143 moderate.

There were many color contrast issues (61), and the non-passing elements had contrast ratios between 2,53 and 4.48. Also, there were problems with HTML semantics; aria-labels on divs, lists with incorrect markup, and lang-attribute missing.

It also had missing landmarks and some missing alt-texts. And it had zooming disabled; this means that if someone needs to zoom in to see better, they can't. It's a terrible practice, making the site unusable for some users.

Testing GitBook with Keyboard

Continuing with the theme "Unusable for some users", GitBook doesn't have a focus indicator for most elements, so it's frustrating to try navigating with a keyboard. Because of that, I couldn't test the site much with a keyboard.

Final Thoughts

In this blog post, I've shared results from quick tests I've made on four documentation tools: Swagger UI, Read the Docs, Docusaurus, and GitBook. Each has accessibility problems, but I would definitely pick Docusaurus for documentation from these four.

After testing all the tools, I researched how they have reacted to accessibility. I wanted to know if they have accessibility statements on their pages, how they track accessibility issues, etc.

Research on Swagger led me to Github. Swagger tracks the accessibility issues with a cat: a11y-tag, but there seems to have been no activity this year. Search with "accessibility" on Swagger's site did not find anything, so I'm assuming they don't have any statements.

Read the Docs has no policies or statements on accessibility. They have some Github issues when searching for accessibility in the theme's repository. Read the Docs has a theming system, so in theory, the user can fix some of the problems found with the search. And with "some", I mean the ones related to styles (so, color, for example). If I've understood correctly, these themes are about styles, not the underlying HTML.

In Docusaurus' introduction, they say that Docusaurus has "attention to accessibility, making your site equally accessible to all users." They also track accessibility issues with the domain: a11y-tag in Github. However, I couldn't find any separate accessibility statement for Docusaurus.

Last, Gitbook mentions accessibility only in "Space customisation"'s "Primary color" section, where there is a mention about accessibility being important when selecting colors. When searching for accessibility and Gitbook, I found a couple of pages using Gitbook and writing about accessibility. That's a bit ironic - as the Gitbook itself is not that accessible.

What's your favorite documentation tool? Have you thought about or tested its accessibility?

Blog