Eye2Voice is a real-time, gaze-to-speech tool for people who cannot communicate verbally. A phone camera reads the user's gaze direction (up/down, left/right) to make a selection.
Eye2Voice received the Hal R. Varian MIDS Capstone Award, given to top projects each semester and named in honor of Hal R. Varian, founding dean of the Berkeley School of Information and former chief economist at Google. Projects are judged on originality, potential impact, completeness, evaluation, presentation and overall quality.
Each year in the United States, 5.4 million people experience physical paralysis. Among children with autism, 1 in 4 are non-verbal or minimally verbal. For people who cannot speak, the tools that help are out of reach for most. Dedicated communication devices cost around $12,000 and take weeks to set up. They also depend on hardware that can break or wear out.
Eye2Voice addresses that gap. It runs in the browser on a device the user already owns, with little to no setup. It offers two modes. In Conversation mode, someone poses a question to the user by voice or text, a language model generates a set of likely responses and the user answers by looking Up, Down, Left, or Right to choose one. In Quick Chat mode, the user moves through a guided tree of common phrases, which is faster for everyday exchanges and needs no question first.
The interface adapts to each person. A caregiver can switch between two and three block layouts, so users with more limited movement get fewer, easier targets and can adjust how long a gaze must hold before it registers. Selections are confirmed by a long gaze or a double blink. Color schemes for text and background can be personalized for easier reading and a read aloud option announces each choice for users who cannot read the screen.
We built a multi-input convolutional neural network, since CNNs handle image data well. Rather than predicting exact gaze coordinates, it classifies the direction the user is looking, which suits people who cannot hold a steady gaze and removes the need for special hardware or calibration.
It takes three image crops from each frame — the face and each eye — plus seven geometric measurements from facial landmarks, such as iris position and head angle, fed as numbers rather than pixels.
Each input feeds its own branch: a shared network reads both eyes for gaze direction and pupil state, a second reads the face for head orientation and a third handles the geometric measurements. The branches merge into a single vector and a final classifier scores each direction, taking the highest as the prediction.
We trained it on two public datasets:
Two challenges shaped the work. The first was persistent overfitting, a gap between training and validation accuracy that indicated the model was memorizing rather than generalizing. Uneven participant data was one suspected factor, since some people contributed as few as 18 frames and others several thousand. Balancing those frame counts narrowed the gap only partway.
The second was inconsistent class identification. The Up direction was recognized far less reliably than the others, as it made up only about 3% of the training data. Several measures helped, including augmentation, loss-level adjustments and supplemental Up samples.
The most effective single change was removing the Straight (center) class, a persistent source of error and unnecessary for a four-direction interface. That one change improved both problems at once, raising Up recall and narrowing the overfitting gap.
A model can score well on data and still fail a real person, so we tested in three ways: on data, in live use and with real users.
First, we checked whether the model learned to read gaze correctly. We held back a portion of the GazeCapture data that the model never trained on, then measured how often it predicted the right direction. We looked at accuracy overall and at each direction on its own, since a model can do well on average while consistently missing one direction, which would make that part of the interface unusable.
Next, we checked whether it held up in actual use, which a data score alone cannot tell you. We built a small test app that asked a person to look in a given direction across 24 trials, then recorded how often the model read the direction correctly and how long it took to respond. We ran this across different devices, seating positions and camera setups to mimic real conditions.
Finally, we put the tool in front of real users, including a speech-language pathologist and a caregiver of someone who relies on assistive communication. Their feedback told us whether Eye2Voice actually worked for the people it is meant to serve, the test that matters most.
Overfitting peaked at roughly 17%, too high for assistive technology. By the final version, the model improved across the board:
*Eye2Voice is a live product under active development, so the code is private.