Combining Speech and Sketch Modalities to Interpret Unconstrained Descriptions of Mechanical Devices

David Tyler Bischel
Department of Mechanical Engineering
Advisor: Professor Thomas Stahovich

Abstract: Traditional CAD tools hinder creativity in the conceptual phase of design, in part because they require over-specification of details before they are relevant, and because they typically have clumsy interfaces. As a remedy, we envision design tools that can understand the same sorts of crude sketches and informal speech engineers use to communicate design ideas to one another. In natural conversation, sketching and speech serve complementary roles, and when one of these modalities is unavailable, communication is substantially less effective. The specific aim of this research is to create techniques that enable a computer to use speech and sketch modalities to distinguish pen strokes representing device geometry from those representing gestures - a critical first step in understanding a sketch of a device. To accomplish this, we developed a classifier that combines features extracted from the sketch with those extracted from the speech. The speech features were derived from statistical properties of word groups temporally aligned with the stroke to be classified. The sketch features were constructed from the geometric properties of the strokes, and the spatial and temporal relationships between them. The classifier was trained and tested on a corpus of unconstrained speech and sketches produced by mechanical engineers. Experimental evaluation of our techniques indicates that combining both speech and sketch modalities resulted in higher interpretation accuracy than using a single modality, although surprisingly, speech was the more important modality.