Currently, those who are Deaf or hard of hearing can experience trouble completing basic daily tasks such as interacting with others. The two methods being used to deal with this are lipreading and sign language interpretation; however, both of these solutions are flawed. It is estimated that the most accurate lipreaders can understand just 30% of the words being spoken; furthermore, there is a critical lack of sign language interpreters, and this is only expected to get worse as Deaf individuals enter public situations more often. These issues in part contribute to the high unemployment rate and high prevalence of problems such as anxiety in the Deaf community.
As a solution, an attachment to a pair of glasses was developed that provides a real-time transcription of the words being spoken to the wearer of the glasses, as well as a warning to the wearer if there are sounds nearby that indicate danger (for example, if there is a car honking or a dog barking). The device uses the Python programming language combined with the Google Speech Recognition API to transcribe speech and a custom Convolutional Neural Network to identify other sounds of significance. This information is displayed on the lens of the device, which the wearer can see. As the lens is transparent, it allows for the wearer to make eye contact with the person they are speaking to while understanding what they are saying, and it does not interfere with their daily life.
What inspired you (or your team)?
The development of VADAR was fueled by a vision of a world in which there would be virtually no barrier between those who are Deaf or hard of hearing and those who are not. While previously, it would have been unthinkable to account for the loss of a sense as important as hearing, modern-day technology can be used to empower these individuals. Recent advancements in machine learning allow for the democratization of information gathered from sound by making it accessible to everyone, regardless of the differences they may have, and augmented reality provides a platform to relay this information back to these individuals in a visual manner. It is evident that the intersection of these technologies will continue to provide solutions to the problems our world faces today and in the future.
I was lucky to present VADAR at the Wood Buffalo Regional Science Fair (bronze medal) as well as the international INVENTURE$ 2019 Student Pitch Competition, winning 3rd place overall. I am very passionate about how machine learning will combine with other disruptive technologies to make the world a better place, and plan to pursue this topic in my future.