How AI is improving the web for the visually impaired

We’re excited to bring Transform 2022 back to life on July 19th and virtually July 20-28. Join AI and data leaders for sensible conversations and exciting networking opportunities. Register today!

There are approximately 350 million people worldwide with blindness or some other form of visual impairment who need to use the Internet and mobile apps just like anyone else. However, they can only do so if websites and mobile applications are built with accessibility in mind – and not as a later idea.


Consider these two sample buttons that you can find on a web page or mobile application. Everyone’s background is simple, so it looks the same.

In fact, they are different from the rest of the world when it comes to accessibility.

It is a question of contradiction. The text on the light blue button has less contrast, so for someone with a visual impairment such as color blindness or Stargard disease, the word “hello” may be completely invisible. It turns out that there is a standard mathematical formula that defines the proper relationship between the color of the text and its background. Good designers know this and use an online calculator to calculate the ratio for any component in the design.

So far, so good. But when it comes to text Complex Against a background like image or gradient, things start to get complicated and helpful tools are scarce. Before today, accessibility testers had to manually examine these cases by taking background samples of the text at specific points and calculating the contrast ratio for each sample. In addition to being cumbersome, measurements are also inherently subjective, as different testers can sample different points within the same area and come up with different measurements. This problem – cumbersome, subjective measurement – has been hampering digital accessibility efforts for years.

Accessibility: AI for rescue

Artificial intelligence algorithms, it turns out, can also be trained to solve problems like this and to automatically improve when exposed to more data.

For example, AI can be trained to summarize text, which is helpful for users with cognitive impairments; Or doing image and facial recognition, which helps people with visual impairments; Or real-time captioning, which helps people with hearing loss. Apple’s VoiceOver integration on the iPhone, whose main use is to pronounce emails or text messages, also uses AI to describe app icons and report battery levels.

Guiding Principles for Accessibility

Wise companies are rushing to comply with the Americans with Disabilities Act (ADA) and give everyone equal access to technology. In our experience, the right technology tools can help make it much easier, even for today’s modern websites with their thousands of components. For example, site design can be scanned and analyzed by machine learning. It can then improve its accessibility through face and speech recognition, keyboard navigation, audio translation of descriptions, and dynamic redesign of image elements.

In our work, we have found three guiding principles that, I believe, are important for digital accessibility. I will explain to them here with reference how our team, led by Asya Frumkin, the leader of our data science team, has solved the problem of text on a complex background.

Example of complex background. Image by author

Divide the big problem into small problems

If we look at the text in the image below we see that there is some kind of legible problem, but overall it is difficult to quantify, just by looking at the whole phrase. On the other hand, if our algorithm examines each of the letters in a sentence separately – for example, “e” on the left and “o” on the right – we can more easily tell for each one whether it is legible or not. Or not.

If our algorithm continues to pass through all the characters in the text in this way, we will be able to count the number of legible characters and the total number of characters in the text. In our case, four out of eight are legible characters. The next fraction, with the number of legible characters as a fraction, gives us the legibility ratio for the overall text. We can then use the agreed-on pre-set threshold, for example, 0.6, below which the text is considered unreadable. But the point is, we got there by operating on everyone Piece Texting and then telling from there.

Example of complex background solution. Image by author

Reuse existing tools wherever possible

We all remember the Optical Character Recognition (“OCR”) of the 1970s and ’80s. Those tools had a promise but ended up being too complex for their original purpose.

But it was part of a tool called the CRAFT (Character-Region Awareness for Text) model that promised AI and accessibility. CRAFT maps each pixel in the image to the center of the character. Based on this calculation, it is possible to create a heat map in which high probability areas will be painted red and low probability areas will be painted blue. From this heat map, you can count the bounding boxes of letters and cut them out of the image. Using this tool, we can extract individual characters from long text and run a binary taxonomy model on each of them (as in # 1 above).

Craft example. Image by author

Find the right balance in the dataset

The problem model classifies individual characters directly binary – at least in theory. In practice, there will always be challenging real-world examples that are difficult to quantify. What makes this matter even more complicated is the fact that everyone has different perceptions of what is legible, whether they have a visual impairment or not.

Here, one solution (and one we’ve taken) is to enrich the dataset by adding objective tags to each component. For example, each image can be stamped with a reference piece of text on a fixed background before analysis. That way, when the algorithm runs, it will have an objective basis for comparison.

For the future, for the better

As the world evolves, every website and mobile application needs to be built with accessibility in mind from the beginning. AI for accessibility is a technological capability, an opportunity to exit and engage on the one hand and to create a world where people’s difficulties are understood and taken into account. In our opinion, the solution to inaccessible technology is simply better technology. As such, making websites and apps accessible is a part and parcel of creating working websites and apps – but this time, for everyone.

Naveen Thadani is the co-founder and CEO of Evinsd.


Welcome to the VentureBeat community!

DataDecisionMakers is a place where experts, including tech people working on data, can share data-related insights and innovations.

If you would like to read about the latest ideas and latest information, best practices and the future of data and data tech, join us at DataDecisionMakers.

You might even consider contributing to your own article!

Read more from DataDecisionMakers

Similar Posts

Leave a Reply

Your email address will not be published.