Skip to main content

Inclusive League Table methodology

The Inclusive League Table is a tool to compare the most popular UK video streaming services and the services they provide for their disabled customers. We looked at both the web accessibility standards of each streaming service and the percentage of accessible content available.

We requested up-to-date figures from each service provider with the percentage of video content available on their platform with audio description and closed captions.

Our league table tests against three main metrics:

  • Percentage of content available with audio description
  • Percentage of content available with closed captions
  • Accessibility standards of the streaming service website when accessed in a web browser

Wherever possible, we’ve used Ofcom’s statutory targets for UK broadcasters as a benchmark for comparison. These targets are expressed as percentages of the service available with closed captions and audio description.

How we calculated an overall score for each service

Each streaming service has been awarded a score out of 10 to reflect how inclusive their services are for disabled customers.

The maximum marks for each section are as follows:

  • Closed captions = 3 marks
  • Audio description = 3 marks
  • Web accessibility = 4 marks

How we assessed each video streaming service:

For closed captions

Closed captions (CC) provide a text alternative to dialogue in the same way subtitles do, but they include other relevant sounds (like “alarm ringing”) in the video. They are essential for people who are deaf and hearing-impaired but help a range of users understand video content better.

We contacted each video on demand streaming service in our league table to request the total percentage of content in their catalogue available with closed captions (or subtitles). This overview is based on self-reported percentages from each service, where providers have been able to supply figures.

Ofcom’s statutory target for subtitling (closed captions) for UK broadcasters is 80%. For a service provider to be described as “doing well” in our CC analysis, more than 75% of its content must be available with closed captions.

Any streaming service with less than 75% content available with closed captions is described as “needing improvement.”

That percentage is then turned into a score out of 3 and rounded to one decimal place.

For example, if a streaming service has 80% of content available with closed captions, it would score 2.4 out of 3 possible marks.

There are three total marks available for closed captions.

Not all streaming service providers responded to our requests for a percentage breakdown of content. Streaming services that did not respond or were unable to provide us with figures for closed captions, received a score of 0 for this section of our analysis.

For audio description

Audio description (AD) is additional commentary on a video that allows a person who is blind or visually impaired to understand what is going on between spoken dialogue.

We contacted each video on demand streaming service in our league table to request the total percentage of content in their catalogue available with audio description. This overview is based on self-reported percentages from each service, where providers have been able to supply figures.

Ofcom’s statutory target for audio description for UK broadcasters is 10%. We feel this target is low given the limited choice it offers for people who are blind and visually impaired that depend on this service. For a service provider to be described as “doing well” in our AD analysis, more than 20% of its content must be available with audio description.

Any streaming service with less than 20% of its content available with audio description is described as “needing improvement.”

That percentage is then turned into a score out of 3 and rounded to one decimal place.

For example, if a streaming service has 20% of content available with audio description, it would score 0.6 out of 3 possible marks.

There are three total marks available for audio description.

Not all streaming service providers responded to our requests for a percentage breakdown of content. Streaming services that did not respond or were unable to provide us with figures for audio description, received a score of 0 for this section of our analysis.

For website accessibility

We carried out a series of website accessibility audits for key pages on each streaming platform. These key pages included the landing homepage, the player page and the account sign-in page.

How we conducted our accessibility audits

Each accessibility audit was carried out on a Windows computer, with each streaming service assessed using Google Chrome, Internet Explorer, Firefox and an NVDA screenreader.

The following browser versions were used:

  • Google Chrome desktop browser [version 83]
  • Firefox [version 76]
  • Windows desktop computer [Windows 7 and 10]
  • Internet Explorer [version 11]

Assistive technology specialists at Scope carried out manual accessibility audits against WCAG 2.1 web accessibility standards.

Each streaming service has been tested manually for web accessibility by a Scope Digital Inclusion specialist. These manual accessibility audits go beyond automated accessibility testing tools, like WebAIM’s WAVE Web Accessibility Evaluation Tool.

Each web page has been tested for accessibility using an accessibility fundamentals methodology similar to the one that Scope uses for itself and consultancy clients.

We carried out a total of 25 tests per web page. Each test has an accessibility impact score of either ‘high’ or ‘medium.’

These are basic accessibility elements that will impact the highest number of users.

The 25 tests are comprised of 18 high impact tests and 7 medium impact tests. High impact failures are then separated into a further category of ‘critical’ impact failures.

  • An example of a critical impact failure is when a website cannot be navigated with only the keyboard.
  • An example of a high impact failure is when the focus border (used to highlight an interactive element on the screen to a keyboard-only user) is difficult to see.
  • An example of a medium impact failure is when a website loads with an auto-playing video that cannot be paused, stopped or hidden.

Critical impact failures are so severe that they directly impact functionality for the user. In most cases, these failures mean the website is inoperable to certain users.

Each web page starts with a score of 100 and is deducted points for test failures. The more serious the accessibility issue, the more points are deducted.

  • -10 points for a critical impact failure
  • -7 points for a high impact failure
  • -5 points for a medium impact failure

For a service provider to be described as “doing well” in our web accessibility analysis, it must achieve higher than 75% in the accessibility audit.

Any streaming service that received less than 75% in our accessibility audit is described as “needing improvement.”

H4: How we calculated a standardised score

We turned the results of the individual accessibility audits (out of 100) into a score out of 4.

So if a streaming service received a score of 75 out of 100 in the web accessibility audit, it would be awarded a score of 3 out of 4:

(75% of 4 = 3)

As the number of key pages for each streaming service varies, the scoring is applied to each page separately, then averaged between them to deliver a final score out of 4.

There are 4 total marks available for website accessibility.

Frequently Asked Questions

Here you can find some answers to our commonly asked questions.

FAQ: What is the Inclusive League Table?

The Inclusive League Table is part of our broader research into video streaming services and accessibility. It is intended as a tool to compare the most popular video streaming services in the UK against basic accessibility standards.

The purpose of this analysis is to highlight that all online streaming service providers could be doing more for their disabled customers.

The table compares the web accessibility standards of popular UK streaming services. It also compares the percentage of content available (on each streaming service) with closed captions and audio description.

The league table compares video streaming services for accessibility only. It does not consider additional factors like each streaming provider’s business, content and subscription model. If and how Ofcom regulates individual service providers also differs.

We created the Inclusive League Table in response to our Big Hack video streaming survey of more than 3,300 disabled people. The purpose of both is to learn about the accessibility issues that prevent disabled people from accessing the same service as non-disabled people.

Our campaign builds on the work of Ofcom’s 2018 recommendations to government “Making on-demand services accessible” (PDF) and the Action On Hearing Loss Subtitle it! campaign.

FAQ: How did you choose which video streaming services to analyse?

We chose the video on demand streaming services that are the most popular with disabled UK viewers.

The 10 most popular video streaming services in the UK are Netflix at number one, followed by BBC iPlayer, Amazon Prime Video, Youtube*, ITV Hub, All 4, NOW TV, Sky Go, Google Play and Sky Store Player, according to Finder.com. These streaming services informed the multiple-choice options for our Big Hack video streaming survey.

Feedback from our survey of more than 3,300 disabled people suggests these are the most popular video streaming platforms with disabled viewers.

Video on-demand streaming service Percentage of respondents who use or have used this service in the past
BBC iPlayer 15.4%
Youtube* 15.0%
Netflix 14.3%
ITV Hub 11.9%
Amazon Prime Video 11.9%
All 4 10%
My 5 7.2%
NOW TV 4.3%
Sky Go 3.6%
Google Play 2.8%
Disney Plus 1.4%
Apple TV Plus* 0.8%
Other 0.8%
Britbox 0.5%

These results helped us choose the streaming services to analyse for the accessibility audits.

*YouTube was removed from our analysis because of its radically different content model. YouTube relies on user-generated content and offers automated closed caption functionality. We were unable to conduct an accessibility audit on Apple TV Plus as the streaming service is only accessible through an Apple device.

FAQ: How can you compare TV catch-up services like BBC iPlayer with paid subscription services like Netflix?

The video streaming services we have chosen to analyse all vary in their pricing, subscription and content models. It is intended as an overview to see how each service compares in its offering for disabled viewers with specific access requirements.

We decided to assess a mix of video on-demand service providers, including broadcast catch-up services (like BBC iPlayer and ITV Hub) and streaming video on-demand providers (like Netflix) as most UK viewers use a combination of these services.

For disabled viewers, how the streaming service provider is categorised makes little difference to their viewing or user experience.

We would also argue that discrepancies in business, content and subscription models between streaming service providers are not an excuse for overlooking accessibility or dropping access services.

FAQ: Why did you carry out the accessibility audits on a computer and not a mobile phone?

There are many ways people can watch and stream their favourite TV shows.

Many people use video streaming services on their smart phone or tablet device. Or using a third-party platform like a Smart TV, games console or Amazon Firestick to stream video content. Some will access video streaming services through their laptop and then cast it to their television using a device like a Google Chromecast.

The provision of access services and accessibility standards, therefore vary across different platforms. To control for these factors, and work within our resource limitations, we tested all services within a computer web browser.

Using a laptop and web browser to test for accessibility meant we could control for inconsistencies across platforms. These technical challenges have allowed both service providers and platforms to shift the responsibility or providing access services onto the other.

The web browser version of each streaming service is the instance in which the service provider has the most control over the output.

In theory, it should be the best and most accessible version. This is because all native functionality and access services are controlled by the service provider.

FAQ: Many of the streaming services in the league table are not required to report on their percentage of access services to Ofcom. Why are you measuring them against Ofcom standards?

Ofcom regulates on-demand programme services (ODPS) as part of the Communications Act 2003. This includes UK broadcaster catch-up services (like ITV Hub) and certain subscription services. It does not include on-demand services owned and operated outside of UK jurisdiction, like Netflix. Ofcom access service regulations do not apply to the BBC iPlayer in the same way either, as these are regulated under separate agreements with Ofcom due to the nature of its UK licensing agreement.

There is currently no robust, comprehensive, independent industry regulator for assessing streaming video on-demand service providers. Ofcom’s statutory targets for broadcasters are therefore the closest bench mark we can use for comparison.

The statutory targets for broadcasters are expressed as percentages of the service. They rise from a low level to the ten-year targets prescribed by the Communications Act 2003.

These targets are:

  • 80% for subtitling
  • 5% for signing
  • 10% for audio description.

There are exceptions to this. For Channel 3 (ITV and STV) and Channel 4, the target for closed captions is 90% and for BBC channels, it’s 100%. (excluding BBC Parliament and BBC Scotland, which are exempted on the basis of the size of their audience) it is 100%.

FAQ: Why did some services receive a score of “0” for audio description and closed captions?

Not all streaming service providers responded to our request for figures. Others could not give us an accurate breakdown of content available with specific access services.

FAQ: What defines a ‘key page’ for each streaming service?

These are the pages that a typical user interacts with the most. It includes the content catalogue homepage, the individual video player page and any required account sign-in pages.

FAQ: Why did we audit three key pages for some services and only one web page for others?

Many of the streaming service websites we audited use the same design and layout for all web pages. This means the results of any accessibility audit will be the same for these pages.

To avoid having to audit thousands of identical pages and creating duplicate tests. The best solution is to audit each of the primary layout templates and apply the audit results to the rest of the website

The majority of streaming services were audited on their homepage and account sign-in page. For video streaming services that used an identical code template throughout the website, we only performed one audit.

Some streaming services had critical impact failures that meant no further tests could be carried out.

The number of web pages is taken into account when calculating the standardised score.

Limitations of this analysis

Measuring the percentage of, rather than the quality of, access services

Measuring services on the percentage of content available with closed captions and audio description does not take into account the quality of those services. Poor quality closed captions was the second most reported accessibility issue in our Big Hack video streaming survey. Disabled people have told us about issues with the appearance of captions like font size, colour and position on the screen. Others have complained about the inaccuracy of the words being transcribed.

The quality of access services provided is clearly an area that warrants further investigation. For now, we are using the percentage of access services as a basic minimum standard of accessibility.

Reliance on self-reported data from services

The league table relies on individual companies to report on their percentage scores for audio description and closed captions. This self-reported data is by its nature, limited as it cannot be independently verified.

This is an issue that Ofcom experience too, who require on-demand programme services (ODPS providers) to submit twice-yearly data on the extent to which they make their services accessible to people with sight and hearing impairments. Ofcom admits that “the number of providers who respond can vary. This can make it difficult to compare data.”

Content catalogues change all the time

TV and film titles are added and removed every day. Video streaming services, whether they require paid subscriptions or not, change all the time. Much BBC iPlayer content is only available to watch for 30 days. Other services like Amazon Prime Video add new titles every day.

The percentage figures for audio description and closed captions are therefore likely to be inaccurate. They are useful, however, in communicating each company’s broader commitment to accessibility.

Accessibility audits are not the same as lived experience

Advanced web accessibility audits are helpful for pointing out immediate accessibility issues. But audits do not compare with the lived experience of disabled people who interact with these services every day.

See what disabled people are saying about video streaming services in our roundup of Big Hack video streaming survey feedback.

Sources

Statement by Ofcom: Making on-demand services accessible

Ofcom’s TV and on-demand access services report Jan to Dec 2019

The Big Hack video streaming survey analytics

How Ofcom reports on on-demand programme services

Office for National Statistics

If you have any further questions about our video streaming campaign or the way the inclusive league table was created, please feel free to email us at bighack@scope.org.uk.