At present, Android already offers a host of options to make the entire user interface easier to navigate for those with certain disabilities, and by extension, its rival Apple's iOS offers similar considerations as well. But Google thinks it can do more.
At its I/O developer conference, the Mountain View company demonstrated perhaps one of the most impressive feature on the forthcoming Android Q upgrade. Google calls it simply as Live Caption, and when enabled, it'll transcribe any video or audio in real time.
Live Captions On Android Q
Closed captions aren't new, of course, but Google's implementation is extremely accurate, according to The Verge. Captions or subtitles are overlaid on top of whatever app is running, whether it's YouTube, Instagram, or anything else. It also supports video chat platforms including Skype and Google Duo. It'll even transcribe video and audio the user records themselves.
"For 466 million deaf and hard of hearing people around the world, captions are more than a convenience — they make content more accessible," Google wrote in a blog post. "We worked closely with the Deaf community to develop a feature that would improve access to digital media,"
Onstage, CEO Sundar Pichai said that building for everyone entails making Google's products easy to use even for those with certain impairments.
"We believe technology can help us be more inclusive, and AI is providing us with new tools to dramatically improve the experience for people with disabilities," he said.
How Live Captions Work
Live Captions make use of on-device machine learning instead of relying on an active connection, which means it can work even when offline, and it doesn't require sending data about the user's activity to the cloud. As for how this'll blend into the interface, Google is putting live transcriptions inside a black box onscreen, which can then be moved freely around. The feature will still work if the phone's volume is toned down or muted — it'll still be able to analyze audio. The only downside is that live captions cannot be saved for later reference.
Although primarily for hearing-impaired users, the feature could also prove useful in certain situations even for people without disabilities. Say, for example, a person wants to watch a video in public but they forgot their headphones and don't want to disturb anyone within earshot. They can just turn Live Captions on and still understand the content.
Users will be able to trigger Live Captions by clicking either the up or down volume button on their phone. It will appear as an extra icon when the volume UI surfaces.
