Summary: An accessibility analysis of web pages of colleges and universities in the United States (nearly 4,000 higher education institutions) from 2019-2023. The 4k Project considers the automatically detectable errors that can be identified by WAVE.
Purpose
The main purpose of the 4k Project is to document the progress of United States Higher Education Institutions’ web accessibility.
Interpreting the results
All automated tools, including WAVE, have limitations—only about a third of possible conformance failures can be automatically detected. The absence of detectable errors does not indicate that a site is accessible or compliant. Only a human can determine true accessibility. WAVE is a suite of tools to help in this process and not the end goal.
The data presented in this project provide a meaningful representation of the state of web accessibility in Higher Education. “Errors” are WAVE-detected, accessibility barriers that have notable end-user impact and are likely WCAG 2 Level A/AA conformance failures.
Background on the Project
The Higher Ed in 4k Project was inspired by the WebAIM Million Project. The WebAIM Million focuses on the top 1 Million Websites from all sectors and industries and limits itself to each websites’ home page. We asked ourselves: What would we learn if we went deeper than the home page and if we focused on one sector of websites (Higher Education in the US)? How would results change over time?
To answer these questions we launched the 4k Project to track the nearly 4,000 degree-granting higher education websites.
In November 2019, November 2020, and November 2023 Pope Tech conducted an accessibility evaluation of every top-level .edu domain in the United States listed in the IPEDS database.
In 2019, we used the IPEDs list of all degree-granting, higher-education institutions in the United States (nearly 4,000 schools) to obtain the domain for each instituions flagship website. We then crawled up to 100 pages from these domains and their subdomains limiting the crawler to remain within 4 link levels deep from the main websites’ home page. These pages were then tested for accessibility using the Pope Tech platform powered by the WAVE testing engine. The average errors and contrast errors detected were then calculated for each higher education intuition and used in the analyses.
The full description of the sampling and results for 2019 and 2020 results can be found the respective reports.
Annual Reports
- Higher Ed in 4k: Nov 2023 report (current report)
- Higher Ed in 4k: Nov 2020 report
- Higher Ed in 4k: Nov 2019 report
2023 Updated Methodology
As the list of institutions and their websites are not static (e.g. a 2-year school becoming a 4-year school and getting a new .edu domain) and pages on websites change, we updated our IPEDS list in 2023. We used the same selection criteria for schools used in 2019 (degree-granting, had .edu URL, allowed us to crawl the website).
During the 2023 crawl to obtain the sample of pages for each school, we made the following 2 changes from what we did in 2019:
- Limited the crawler to the flagship domain for the school and did not include sub-domains
- We limited the crawler to up to 20 pages from each website within a 2-link depth from the homepage (previously we had allowed for up to a 4-link depth from the homepage and up to 100 pages)
We made these changes to provide greater consistency for comparisions within schools and across schools overtime.
Summary of the 2023 Methods
- 3,661 higher education institutions from the IPEDS list were included in the final sample/analysis
- Up to 20 pages (all within 2 links from the main home page) were sampled from each institution
- The accessibility evaluation was performed by Pope Tech using the WAVE engine
- Automatically detectable WAVE errors (Errors and Contrast Errors) were used to calculate the average errors per page for each institution
- Each institution’s average errors per page were used for the analyses
2023 Results
Overall, Higher Education has historically done better than most industries with web accessibility. While many industries and business sectors have been slower to respond to improving web accessibility, higher education continues to improve.
Our 4k Project results below documents the progress made by higher education institutions in the United States between 2019 and 2023 with regards to automatically detected accessibility issued.
Key 4k Project findings (Overall)
- From 2019-2023 there has been an overall 33% decline in detectable errors*
- In 2023, we detected an average of 16 errors per page (in 2019, the average number of errors/page was 24)
- Alerts increased between 2019 to 2023 by an average of 11% increase in alerts detected (27 alerts/page in 2019 to 30 alerts/page in 2023)
- In 2023, 47 institutions (1.28% of the sample) had 0 detectable errors on the pages we sampled (compared to only 7 institutions in 2020, and 2 institutions in 2019)
*Comparisons across time seem to be consistent with other data, such as the WebAIM Millions higher education results. However, it is important to remember we changed sampling methodologies limiting the pages crawled to 20 instead of 100. A review of our results compared to .edu sites in the WebAIM Million indicate similar trends giving us confidence the change in methodology didn’t affect validity of the results for the level of comparisons we are doing in this project.
WebAIM Million vs. WebAIM Million (.edu) vs. 4k Project by Year
Using the WebAIM Million data between 2019-2023, higher education tends to have fewer errors/page on their homepages than other industries. The average decline in errors per page is also slightly higher for higher education than all industries. All industries decreased detectable errors on their home page by 9.6 errors/homepage over the 4 years (an overall 16% decline). During the same time period higher education’s errors/page declined by 20%.
The 4k Project results expands on this decline by going beyond just the homepage of the higher education institutions’ flagship websites. By doing so we are able to document greater progress made on websites in higher education. The first notable result is a difference in average errors per page when comparing the WebAIM .edu results with the 4k Project results at each year. Overall the 4k Project results show fewer errors per page. In both 2019 and 2023, the difference in errors per page between only homepage (WebAIM Million .edu) and a sample with homepage + other pages (4k Project) was roughly 13-14 errors/page at each year. This is not surprising as homepages tend to have more complexity and consequently more detectable errors than other website pages.
Focusing on more than homepage also allows for a better understanding of progress being made over time. Comparing 2019 to 2023 results, the 4k Project found a 33.6% decline in average errors/page over that 4 year period of time (compared to a 20% decline found by WebAIM Million .edu over those same years.
| Project type | 2019 | 2020 | 2021 | 2022 | 2023 |
|---|---|---|---|---|---|
| WebAIM Million (all industries) | 59.6 | 60.9 | 51.4 | 50.8 | 50.0 |
| WebAIM Million .edu | 37.1 | 34.9 | 30.0 | 30.1 | 29.7 |
| Pope Tech 4K Project | 24.1 | 19.8 | (no data) | (no data) | 16.0 |
Rankings by classifications
Using the IPED List data, and other higher education institution databases, we also were able to compare institutions based on different classifications beyond just overall rankings.
Below are some of the interesting comparisons we found.
Private vs. public institutions
| Type | 2019 Errors per page | 2020 Errors per page | 2023 Errors per page | % change from 2019-2023 |
|---|---|---|---|---|
| Public | 20.21 | 15.4 | 12.4 | 38.6% decline |
| Private not-for-profit | 25.3 | 21.2 | 17.4 | 31.0% decline |
| Private for-profit | 30.1 | 26.2 | 23.8 | 20.1% decline |
We observed declines in detectable errors across the across the board, with public institutions having the fewest errors and having the largest declines in errors overtime. In 2023, Public institutions averaged 12.4 errors per page, and have fewer accessible errors per page than private institutions. Similarly, Private not-for-profit institutions have fewer detectable errors (with 17.4 errors per page) than private for-profit institutions which have 23.8 errors per page.
It would make sense that public institutions would be better as they have additional laws beyond the Americans with Disability Act including Section 508.
Student enrollment
| Enrollment | 2019 Errors per page | 2020 Errors per page | 2023 Errors per page | % change |
|---|---|---|---|---|
| under 1,000 | 28.3 | 24.1 | 21.2 | 25.1% |
| 1,000 – 4,999 | 24.1 | 19.2 | 16.6 | 31.1% |
| 5,000 – 9,999 | 19.0 | 15.4 | 11.8 | 37.9% |
| 10,000 – 19,999 | 19.0 | 14.3 | 11.3 | 40.5% |
| 20,000 and above | 14.0 | 11.2 | 8.1 | 42.1% |
There was a direct correlation between student enrollment numbers and accessibility errors per page. The more students, the fewer detectable errors. Also, schools with higher enrollments saw a great decline in errors per page between 2019 and 2023. For example, institutions with over 20,000 students enrolled had only 8.1 detectable accessibility errors per page and saw a 42.1% decline in errors. Whereas schools with between 1,000 to 5,000 students had 16.6 detectable errors and only saw a 31.1% decline over the same 4 year period.
This could be because the larger institutions have more resources and budgets.
The 5 most common errors
The 5 most common accessibility errors detected in 2023 represented 86% of the total errors we detected in our scan. Collectively, the top 5 errors represent a smaller % of the total errors than they did in years past. For example, in 2019 the top 5 errors represent 91% of detected accessiblity errors.
| Error (sorted by 2023 rankings) | 2019 ranking (% of detected errors) | 2020 ranking (% of detected errors) | 2023 ranking (% of detected errors) |
|---|---|---|---|
| Very Low Contrast | #1 (64%) | #1 (66.1%) | #1 (57.5%) |
| Empty Links | #2 (14.6%) | #2 (11.9%) | #2 (12.6%) |
| Missing Form Label | #3 (4.3%) | #4 (3.9%) | #3 (6.8%) |
| Linked Image Missing Alternative Text | #4 (4.2%) | (not in top 5 in 2020) | #4 (5.1%) |
| Missing Alternative Text | #5 (3.6%) | #3 (8.4%) | #5 (4.1%) |
| Empty Button | (not in top 5 in 2019) | #5 (2.6%) | (not in top 5 in 2023) |
It is important to realize that if the 5 most common errors detected in this project were resolved, it would remove 86% of the detectable errors from the pages we scanned.
In 2020, we released an article on the most common errors detected in this study, as well as well as suggestions on how to fix them. The 2023 result not found in most common errors article (2020) is linked image missing alternative text.
What about alerts?
WAVE also detects alerts – items that are likely an accessibility issue but might depend on context (e.g., redundant links) or require human verification (e.g. checking the captions on a video). Between 2019 to 2023, alerts increased from 27 alerts per page to 29.96 (an 11% increase). A substantial number of the alerts are issues that do need to be fixed and would go away once fixed. As the scope of accessibility work matures in higher education we would expect and hope to see a decline in alerts in many of the alert categories (e.g., skipped heading levels). Currently, the upward trend in alerts likely indicates that schools are primarily focusing on fully detectable errors.
Some other interesting accessibility tidbits
During our analysis of the 2023 data, we found some other interesting tidbits that we thought we would pass along:
Skip Links
We found 4,231 skip links on 432 websites that didn’t have a target. This means that someone went through the effort to add a skip link but then either never tested it or it was broken with a template update.
Marquee tags
There were 28 marquee tags still around. These were located on just 5 of the websites we tested. We have noted a substantial decline in the usage of these from 170 we found in 2019.
Layout tables
We detected 1 or more layout tables on 53.4% of the websites. It is likely that many of these “layout tables” are actually data tables without the proper structuring (making them difficult for accessibility). A data table is classified as a data table if it is properly structured as a data table. Realistically, we suspect that there are relatively few true layout tables.
PDFs
We found an average of 2.5 PDFs per page scanned. With 86.3% of the websites having at least 1 PDF or more on pages we scanned. These PDFs may or may not be accessible. But as we know from the WebAIM screen reader survey, 74% of screen reader users are either Very Likely or Somewhat Likely to encounter significant issues accessing a PDF document.
Conclusion
While there is still significant work to be done to ensure Higher Education websites are accessible to everyone, we are encouraged by the improvements in web accessibility. Collectively, between 2019-2023, the schools in our 4k Project saw an average decline of 33% of detectable errors on the pages we analyzed. Improved were limited to just public, large-student body, schools (although they saw the largest improvements). This suggests that the majority of higher education institutions are making efforts to improve web accessibility.
We are hopeful that this project and other endeavors by the Higher Education web accessibility community will continue to bring more awareness to web accessibility. We are optimistic that over time we will see additional improvement.