Research

Goal

The overall goal of the research we conduct is to understand how people understand spoken language. One of the pre-requisites to understanding a sentence is identifying the words that compose the sentence. The recognition of a spoken word is more complex than it appears. A chunk of the spoken signal has to be matched with one of the 100,000 words that an adult knows. Because speech is a continuous, highly complex, and rapidly changing signal, processing must occur as soon as acoustic information becomes available and in an incremental fashion.

Incremental processing is rendered difficult by the fact that the interpretation of an incomplete fragment of speech is almost always ambiguous. Consider the word candle. By the end of the fragment can, possible interpretations include candle, but also candy, canvas, can, etc. Thus, the listener must perform incremental interpretation of the speech signal in the face of substantial ambiguity.

Questions

The questions we address are the following:

What are the processes by which recognition takes place on-line, as the speech signal is heard?

How is ambiguity between different interpretations resolved?

What are the lexical alternatives that listeners entertain as speech information becomes available and how is each alternative weighted?


In order to address these questions, we need a measure that reflects people’s ongoing interpretation of the speech signal. We use people’s eye gaze to a visually present referent of the spoken word.

In the so called “visual world” paradigm--at least our version of it, participants see a set of pictured objects on a computer screen and listen to spoken instructions asking them to click on and move one of the objects using the computer mouse. Participants’ eye movements to each of the objects are monitored as the spoken instruction is heard and processed. The time it takes people to fixate the referred object, as well as fixations to the other objects of the display, are informative about the mapping of the speech signal onto lexical representations.

For more information about the procedure and measures used to infer lexical processing, click here.


For more pictures from our lab, click here.