Georgia Tech Research Horizons
Fall 2003
COVER STORY – Computing at the Boundaries"
Bees, Ants, Monkeys & Robots
Hey, Coach, What's Next?
Face-to-Face Meetings
Reality, Only Better
Behind the Scenes
Navigating Real & Virtual Worlds
Back to School
GVU Center Research Highlights

Cover Story
Face-to-Face Meetings

Researchers augment conversation
with translator and listening agents.

IN THIS AGE OF VIRTUAL COMMUNICATION, one team of researchers hasn't forgotten the importance and difficulty of face-to-face conversation.
photo by Gary Meek

For sign language translation, Assistant Professor Thad Starner, left, his Ph.D. student Helene Brashear, pictured, and sign linguist Danielle Ross have developed a system in which a deaf person – equipped with a wearable computer, computer vision technology and two accelerometer wristbands – signs questions. (300-dpi JPEG version -720k)

Thad Starner, an assistant professor of computing in the GVU Center, and his students are developing two interfaces to augment such encounters. One will assist people who communicate through sign language, and the other will help PDA owners use their devices more effectively.

The researchers' sign language translator, intended for on-the-go communication, is modeled after a one-way foreign language translator being used by U.S. soldiers in Iraq. A soldier speaks in English, and the translator matches the English to Arabic phrases. Then the device speaks it in Arabic. The person to whom the soldier is speaking can then respond with universal gestures, such as nodding or pointing.

For sign language translation, Starner, his Ph.D. student Helene Brashear and sign linguist Danielle Ross have developed an apartment-hunting scenario in which a deaf person – equipped with a wearable computer, computer vision technology and two accelerometer wristbands – signs questions such as, "How many bedrooms?" The system recognizes the signer's hand gestures, and its head-up display shows the top five possible matching phrases in English.The signer selects the correct phrase, and the device speaks the phrase. Then the apartment manager can respond with universal gestures, such as holding up fingers or pointing.

"This is a first step at developing a useful translator for sign language," Starner says. "The more we interact with the deaf community, the more we realize this is a reasonable approach." Funded by a GVU seed grant, the researchers hope to have a working prototype in 2004.
photo by Gary Meek

Computer vision technology is one component of a wearable computing system for sign language translation. (300-dpi JPEG version -596k)

Meanwhile, Starner and his team are developing computational support for wearable computer users to employ when scheduling appointments during serendipitous conversations at water coolers or elsewhere. PDA and paper calendar users seldom pause their conversations for the 30 seconds it takes to formally note appointments. Typically, they scribble them on scrap paper and add to their calendars later.

Starner and students Niels Snoeck and Ben Wong want to eliminate this hassle with the Calendar Navigator Agent, which listens in on conversations when the user directs it with socially acceptable keywords. The program then navigates the wearable computer's interface to enter an appointment almost instantly.

As a backup listening agent, Starner and his students created Dialog Tabs, which remembers snippets of conversation at the user's direction and puts a reminder icon on the wearable computer user's head-up display.

"This system facilitates natural behavior," Starner says. "It remembers appointments and then tries to do speech recognition on them. You click on the screen icon, and it pulls up what it thinks is the right parsing of the conversation. It enters the appointment into your calendar if it's correct, or you click on the translation and correct the appointment time.... It's all there on your computer. You don't have to keep it in your head."

Starner and his team have developed lab prototypes of both systems, which they will test in their daily lives. "Preliminary tests show the Calendar Navigator Agent is significantly faster than PDAs or paper-based Day Planners," he adds.    – JMS

For more information, contact Thad Starner at 404-385-0816 or

Contents    Research Horizons    GT Research News    GTRI    Georgia Tech

Send questions and comments regarding these pages to
Last updated: Dec.11, 2003