CS789 - Spring, 2005
Activity instructions, Week 6
This week's activity: The Multi-media Kitchen
What is unique about multi-media as part of a user interface? First, what
is it? We say that an interface provides a multi-media experience when is
provides several streams of output, which are processed separately and merged
internally by the user. Usually these streams are perceived by the user
through different sensory modalities.
In understanding the potential for multi-media in the user interface it's
important to understand that all life is a multi-media experience. Perceiving
a single external object through two or more distinct input streams,
especially when they are attached to different modalities, powerfully
enhances the reality of that object. What's obvious, then, is that
multi-media perception is, for users, both natural and significant; what's
not obvious, however, is how to harness these user abilities in the
interface. To get an idea of what might be possible, we'll look this week, at
a typical multi-media experience that exists in everyday life.
Probably the aspect of your daily activity that is the most multi-media is
cooking. While you are doing it you see, hear, touch, smell and taste the
food with which you are working, and every one of these interactions is
necessary if you are to create a successful result. You could think of remote
cooking as the ultimate multi-media interface challenge. This week's activity
is designed to get you focussed.
What you should do
This week's activity is to cook a meal. When I say `cook' I mean `make
something starting from basic ingredients', not `heating something in the
microwave'. Ideally, your cooking will be done from a recipe, and will
include some cutting or chopping, and some actual cooking on the stove-top.
While cooking make sure that there is a good source of light shining on the
area where you are working which, among other things, makes it much easier to
While cooking observe that you provide a large amount of input, mostly
with your hands, and that cooking provides a large amount of feedback through
most sensory modalities.
- Touch (Taction).
- Smell (Olfaction). Volatile chemicals from food are carried on air
currents to dissolve on liquid surfaces within the nose, where they are
sensed. There are 1000s of different receptors: over 1% of the rat genome
codes for olfactory proteins. Make sure that you drink water while you
cook to keep the olfactory sensors working.
- Taste (Gustation). To be tasted something must be put on the tongue
where are the gustatory (taste) receptors. There are four types of taste
receptor: salty, sour (acidic), sweet, bitter. Taste is an insensitive,
simple sensory mechanism, designed to identify certian elements in food;
most of what we call taste is actually olfactory, caused by volatile
substances in our food passing into our nose via the throat while food is
in the mouth.
- Common chemical sense. Receptors, concentrated in eyes, nose, mouth and
other liquid-covered areas of the body, detect a number of noxious
chemicals, such as soap.
- Proprioception, which is the body's sense of where its parts are.
What you should observe and think about
- Each item of your cooking, each action you perform, provides very rich
feedback through several sensory modalities simultaneously.
- The time course of each modality is different; the localization of each
modality is different; the impact on your perception of each modularity
is different. How to they combine to form a unitary perception of the
state of what you are cooking?
- When you shut out one or more of the modularities, by closing your
eyes, ears or nose, for example, how does the perception change in the
other modularities change?
- Think about which modularities dominate which others, and in what
respect. For example, when watching a movie, the sound and picture come
from different places, so either your eyes are wrong about where the lips
are, or your ears are wrong about where the sound is coming from. For
most people vision strongly dominates hearing in this example. Think
about how the senses interact while cooking: for example, would you eat
something that looks good but smells bad?
- Sometimes modularities fall apart. For example, when watching a movie
if the sound and image are desynchronized, there seems to be two
different things going on, a lip-moving appearance and a talking sound,
and the two are not connected. Under what conditions might you observe
such a thing when cooking?
- Sometimes modularities combine inappropriately. Once, while I was
watching a movie about a couple with a young baby, I heard a baby in the
audience cry and wondered why the couple on the screen didn't respond.
Suppose you put on different types of music while cooking - fast, loud
and lively; or tranquil and relaxed; or ...? How does the music affect