Research Horizons Magazine
December 2, 2004

One Size Doesn't Fit All: New Software Would Customize Graphics-Based Computer Interaction for People with Low Vision

In the prime of her life, Virginia Jacko began gradually losing her vision to a disease called retinitis pigmentosa. The disease forced Virginia, now 62 and completely blind, to leave her high-profile job as the director of fiscal affairs for the president and provost of Purdue University several years ago.

In their experiments, researchers use a tracking apparatus that captures eye movements at a rapid rate. Subjects may wear a head-mounted optical apparatus or be monitored by a remote optical device that allows researchers to capture pupil and corneal reflection and pupil diameter. Seated are graduate students Katie Emery and Kevin Maloney. Standing are Professors Francois Sanfort and Julie Jacko. Georgia Tech Photo: Gary Meek Download 300 dpi version

But with considerable determination and energy, she has learned to adapt with help from her yellow Labrador retriever guide dog “Tracker” and her innovative daughter, who has made a career out of improving the quality of life for people with uncorrectable visual impairments.

Virginia’s daughter is Julie Jacko, an associate professor in the Georgia Institute of Technology’s School of Industrial and Systems Engineering (ISYE). An expert in human-computer interaction, Jacko was moved by her mother’s experience to create software that measures the capabilities of computer users with low vision and automatically customizes computer graphical user interfaces (e.g., file and folder icons, drop-down menus) to provide greater user accessibility to the tens of millions of people who need it.

Virginia’s condition is too advanced to benefit from the software Jacko and her colleagues are developing; she must use a speech interface to interact with her computer. But for others suffering from disease-related vision loss, the researchers hope to provide new options within the next couple of years.

They are developing customizable software that will provide visually challenged computer users with multiple feedback mechanisms, such as auditory and haptic (touch-related) cues. This information will allow them to interact with traditional and handheld computers using the graphical interfaces that most people use.

Researchers have constructed a prototype software program based on clinical and experimental test data from about 200 subjects with uncorrectable visual impairments, as well as normal vision. Here, test subject Bess Kronowitz participates in an experiment. Georgia Tech Photo: Gary Meek Download 300 dpi version

“Generally, most feedback from a computer is visual,” Jacko explains. “But people who are visually impaired need other types of feedback to supplement this. For example, a haptic mouse provides vibratory cues when a user moves closer to a target on the screen. Or the user might hear a sound and feel a vibration as they move the cursor closer to a file.”

Both auditory and haptic feedback may increase or decrease in intensity depending on the user’s movements around the screen. “In our experiments, we are manipulating the variables to determine which combination is best for people with different types of capabilities,” she says.

Jacko began this research with a National Science Foundation (NSF) Career grant in 1997 while at the University of Wisconsin-Madison. Subsequently, a prestigious Presidential Early Career Award for Scientists and Engineers (PECASE) grant from NSF extended Jacko’s research and involved her colleague and husband ISYE Professor François Sainfort as a collaborator on the project in 1999. Sainfort, also an associate dean in the Georgia Tech College of Engineering, is an expert in health and decision support systems.

The couple brought the project to Georgia Tech when they joined the faculty in 2000. Since then, they have received additional NSF support, as well as funding from the Intel Corporation. Their research has been published in the American Journal of Ophthalmology, Behaviour and Information Technology, and the International Journal of Human-Computer Interaction, among other journals.

Clinical image shows a diseased retina affected by retinitis pigmentosa. Photo courtesy John A. Moran Eye Center, University of Utah

The researchers initially focused on age-related visual deficits, specifically age-related macular degeneration (AMD), which is characterized by central vision loss, while peripheral vision remains relatively intact. AMD is the leading cause of legal blindness in people age 50 and older. The researchers have since expanded their study to include people with other eye diseases, including retinitis pigmentosa, which usually begins with peripheral vision loss followed by central vision loss, and diabetic retinopathy, which affects both young and older people. Diabetes-related vision loss is variable and unpredictable, which presents unique challenges in adapting computer interfaces, Jacko notes.

Jacko and Sainfort have also broadened their research beyond a study of just manipulative computer tasks. Their experiments now capture subjects’ eye movements so they can understand the behavior and strategies people employ when using a desktop or laptop computer, or even a personal digital assistant (PDA).

“We want to know how a patient’s eye movements vary depending upon the type of ocular pathology they have and their resulting functional capabilities,” Jacko explains. “We’re able to sample eye gaze at a rate of about 60 to 100 Hertz so we know at any point in time where they’re looking on the screen and how long they spend at various regions of the screen. This gives us a lot of insight into the strategies they’re using.”

In their experiments, researchers use an eye-movement-tracking apparatus manufactured by ASL Inc.; it captures eye movements at a rapid rate. Subjects may wear a head-mounted optical apparatus or be monitored by a remote optical device that allows researchers to capture pupil and corneal reflection and pupil diameter.

Researchers make audio and video recordings of each experiment with visually impaired patients. In one session that Jacko uses to explain the process, an 86-year-old woman with AMD struggles to drag a file and drop it in a folder on the screen. The woman is a concert pianist, so her high level of manual dexterity is an asset for the task. Still, it takes her numerous attempts before she succeeds. As she approaches the target folder on the black screen background, it turns purple, she feels a vibration from the mouse in her hand and then a sound cues her about her progress.

Meanwhile, researchers track the subject’s pupil diameter, which is important because research elsewhere has shown that it is a physiological indicator of cognitive workload in people with normal vision. Pilots, for example, exhibit changes in pupil diameter in response to the complex operations they perform.

“We want to understand pupil behavior in people with visual pathologies,” Jacko says. “For example, people with diabetic retinopathy sometimes exhibit aberrant pupil behavior. People with AMD exhibit relatively normal pupil behavior, but compared to people without AMD, you see clear trends with respect to the way the pupil responds to information on a computer screen.”

With the statistical expertise of ISYE Professor Brani Vidakovic, Jacko and Sainfort are analyzing the pupil data they have collected in their experiments. “This data is ‘noisy’ due to eye blinks, and it’s captured at a high rate,” Jacko explains. “But Brani’s analysis is giving us a very accurate and informative data.”

The researchers have successfully demonstrated new tools for analysis of complex pupil response data from people who are aging and/or have visual impairments.

The researchers’ goal is to create a commercially available software program that analyzes pupil and other data in real time and then adapts computer interfaces as needed by the user. They have constructed a prototype based on clinical and experimental test data from about 200 subjects – including Jacko’s mother – with uncorrectable visual impairments, as well as normal vision, ranging in age from 60 to 90.

They conducted these experiments with patients in the University of Miami School of Medicine and the Department of Optometry at Nova Southeastern University in Ft. Lauderdale, Fla. The research team will assess about 50 more subjects in experiments this winter.

In addition to directing the software design, these experiments have yielded guidelines for the interaction of visually healthy older adults and their computers.

“We’ve learned we could supplement their display with specialized auditory feedback that may enhance their computer performance,” Jacko notes. “We can address age-related deficits in manual dexterity, cognitive function and short-term memory. For example, a sound may cue a user’s memory about what they have just seen on the screen.”

The researchers are also considering input mechanisms in their software development. But they have found that the patients they are studying have enough residual vision that they resist the addition of a speech input interface. “They want to use computers like we use them,” Jacko explains.

But the final software product will likely contain both input and output interfaces as user options, she adds. Users will begin with an on-screen diagnostic test, and then the software will customize itself for the user depending on the test results, the researchers explain.

A grant from the NSF Integrative Graduate Education and Research Training (IGERT) program to Professor Marie Thursby in the College of Management has enabled Jacko’s Ph.D. student Katie Emery to work with Georgia Tech management students and Emory University law students on a business proposal for a commercially viable software product.

This has led to a proposal for a Small Business Innovation Research grant. If funded, this work could lead to patent applications and the establishment of a start-up company within two years, the researchers say. The final product could be integrated into the Microsoft Windows operating system.

“It will be wonderful someday if people with low vision can sit down and their computers adjust to their level of vision,” Virginia says. “The programming technology is there…. Now, Julie’s work is using low-vision subjects to guide the interface design work and test it.”

RESEARCH NEWS & PUBLICATIONS OFFICE
Georgia Institute of Technology
75 Fifth Street, N.W., Suite 100
Atlanta, Georgia 30308 USA

MEDIA RELATIONS CONTACTS: Jane Sanders (404-894-2214); E-mail: (jane.sanders@edi.gatech.edu); Fax: (404-894-4545) or John Toon (404-894-6986); E-mail: (john.toon@edi.gatech.edu).

TECHNICAL CONTACTS: Julie Jacko (404-894-2342); E-mail: (julie.jacko@isye.gatech.edu) or François Sainfort (404-385-2905); E-mail: (francois.sainfort@isye.gatech.edu).

WRITER: Jane Sanders