Research Overview

Today’s graphical user interfaces limit our use of computers in several important ways: 1) They restrict who can use computers, 2) they constrain where computers can be used, 3) they do not let users communicate in ways that they naturally do with other human beings, and 4) they leverage very few of our innate perceptual, motor, and cognitive abilities. Our group is focussed on overcoming these limitations through informal user interfaces.

Informal user interfaces are interfaces designed to support natural, ambiguous forms of human-computer interaction without much up front recognition or transformation of the input. They are a reaction to the "overly restrictive computational formats" of current applications.

These restrictive interfaces are often a result of tradition. Computers have found their greatest success in applications where they have replaced humans in tasks at which humans are poor, such as performing complex, redundant mathematical calculations, or storing and searching large amounts of data. We use the term formal user interfaces to describe these precision-oriented interfaces as well as perceptual interfaces that try to recognize human input immediately. Immediate recognition structures the data in a form that is best for the computer, but can often get in the way during creative or communications-oriented tasks, such as brainstorming or design.

As computers grow more powerful, less expensive, and more widespread, we expect them to assist us in tasks that humans do well, such as writing, drawing, and designing. Unfortunately, the historical strengths of computers have led to a design bias towards support of precise computation, and away from the more human properties of ambiguity, creativity, and communication. Consequently, interfaces are designed to facilitate structured input rather than natural human communication, which consists of the imprecise modes of speaking, writing, gesturing, and sketching.

The primary tenet of our group’s work in informal user interfaces is to bend computers to people’s mode of interaction, not the other way around. In many domains the formal approach clearly presents an obstacle to effective task performance. One such domain that we have investigated is user interface design itself, and this led to our work on SILK, SILK II, and SILKWeb. Any application that requires interaction between humans and computers, and in which absolute precision is unnecessary, could benefit from an informal approach.

One such application is in multimodal user interface design. Multimodal interfaces can take advantage of a wide variety of human perceptual and motor abilities, such as speech or gesture, to achieve a more natural human-computer communication. Unfortunately, multimodal interfaces are still very hard to build. We are addressing this difficulty by creating new interactive tools for designing multimodal user interfaces targeted at a range of devices, from the palmtop to the desktop to wall-sized displays.

Our tools will support an iterative design methodology. The key idea is to support the informal techniques that designers currently use in the early stages of design, from sketching and storyboarding to using "Wizard of Oz" techniques. We will also use programming by demonstration for specifying events that are not screen-based, such as speech input or output, or body gesture input. The tools we produce will not handle all of the necessary cases for a final product. Instead, our tools will allow designers to rapidly create "rough cuts" of their ideas, which can be quickly tested and modified. When ready, the designer can use these tools to generate user interface code for a variety of devices. This code can then be extended with the final interface details and application code.