CS789, Spring 2005
Activity instructions, Week 9
Extend the direct manipulation dialogue style
Outside of graphical interfaces, which we will discuss separately, the
direct manipulation dialogue style has been pretty well restricted to select,
open and drag-and-drop. How do they work?
The result is a fairly limited set of things to do. How might we do more
within the direct manipulation metaphor?
- Select -- point at the thing you want to select and single-click the
select button on the mouse; this is called "point-and-click". The analogy
is to pointing at one of several objects and saying, "That one." This is
not the only selection mechanism. Selecting a group of objects can often
be done differently. Selecting a menu item can be point-and-click, but
there are also other techniques. Selecting the window or the field of a
form to receive input can be point-and-click, but there are also other
ways. How do different selection methods co-exist?
- Open -- point to the thing you want to open and do an operation
different from single-clicking the select button. Most often this
operation is double-clicking, but there are other possibilities. Think of
a few examples. The "open" operation has been generalized to a group of
operations clustered around the open/start idea. The precise meaning is
determined by context and users seem not to have any difficulty with the
- Drag-and-drop -- unlike select and open an operation like move requires
two operands, the thing to move (source) and the place to which it should
be moved (destination). It's very natural to represent this operation
using actual movement of the icon representing the source. The movement
starts at the source with a mousedown event (select?) and ends at the
destination with a mouseup event. If CPU and graphics resources are
available there is usually real-time motion of the source, or its shadow,
as the tracker moves. Drag-and-drop can be generalized without leaving
the metaphor of motion. If the destination is not a location, but an
object that does something, then dropping the source into it represents
moving the source to the object and having it processed. Is this
Let's start by considering interaction with a shovel as a real-world
analogue of direct manipulation. "Shovel" is both a noun and a verb, a tool
that you can use on something and an action that you can perform.
Furthermore, when you use a real-world shovel you do it by directly handling
the shovel in some way.
Now, before going any further, find a shovel and use it
for digging a hole. Observe as you dig how the shovel transfers the force
exerted by your muscles to the earth in which you are digging. And observe
how the shovel tranfers the resistance of the earth back to your hands and
arms so that you can adjust the direction and magnitude of the forces you are
applying. Fill in the hole again, smoothing and packing the earth with your
shovel. Note how you hold the shovel differently when doing different
With this experience under your belt you are ready to think about what
would be involved in creating a direct manipulation interface to a virtual
shovel, which would allow a user to use the shovelling metaphor as a way of
doing things with a shovel.
Try the following analogies to see if they make sense.
Once you can move the shovel arbitrarily it's possible to do any shovelling
operation. There are still a couple of gotchas.
- To get a shovel that is among a collection of tools we reach in with a
hand, and pull it out. You can do this with a standard selection
- To move the shovel to a place you want to dig, carry it there. A
drag-and-drop interface should do this fine.
- Opening the shovel seems like a natural way of starting to dig with it.
Of course, allowing it to shovel randomly is not likely to be very
useful. Possibly opening the shovel puts the interface into a mode where
the mouse controls the physical position of the shovel, allowing the user
to move it arbitrarily.
This is starting to sound a lot like virtual reality shovelling: hand, arm
and foot motions, visual and force feedback very like the ones I experience
while doing real shovelling. With just the right scaling I might exactly
duplicate the shovelling work-out. Venting could provide fresh air controlled
by biological sensors that monitor the rate of my metabolism. But perhaps
this is direct manipulation taken a little too far.
- How strong should the shovel be? Perhaps the strength should vary,
under user control. Force feedback is likely to be useful. And when the
force of my hand isn't enough, might I bring my feet into play?
- How do we match the precision of mouse motions to the precision of
shovel motions? I can position a real shovel for digging to within a
couple of millimetres, which is comparable to the accuracy with which I
can see the object of my shovelling. The scale of visual feedback needs
to be close to one-to-one; the scale of hand motion to shovel motion
needs to be close to one-to-one.
Another approach is to focus on the task being undertaken. Mostly shovels
are used to move things. The operation encompasses
This sounds more like it: a drag-and-drop interface will be fine. We need to
be a little more clever, however. When we want to make a hole we want to move
everything within a particular volume, regardless of what it is. Think of a
variety of ways of indicating what is to be moved.
- a thing to move, and
- a place to move it to.
And what about constraints on the destination? I can use a shovel to fill
a barrel with sand, where the termination condition is based on the
destination rather than the source.
In reality a shovel is a lot more flexible than an interface like this one
allows, and has a lot wider range of uses. You can use it to pound in a
stake, to level a pile of sand, or to cut a piece of sod. And you do these
operations by holding the handle and/or pushing on the blade with your foot.
What this means is that we can "operate" a shovel in more than one way.
Is it possible that there is a description of shovel manipulation
intermediate between between "duplicate real shovel control", which gives
maximum flexibility, and "select from pre-defined goals", which restricts the
user to objectives conceived by the shovel designer? Try to think of one.
- We can swing it.
- We can slide it.
- We can cut with it
Here are a few ideas on which you can test your ideas.
Try and think up a few other ways of doing things with a shovel, and consider
different ways of activating a shovel icon with the mouse that would express
the differences in what you might want to do. (This is a potential interface
to the computer-shovel which digs in your garden in the cold rain while you
operate it from inside using your PC.)
- Sometimes there is only one thing you can reasonably do with a
particular operand. If a burglar is the operand what is the shovel doing?
If a wheelbarrow is the operand what is the shovel doing?
- You can operate on different parts of the shovel: the blade for digging
or cutting, the handle for throwing.
- You can operate the shovel in different directions: up to empty a hole,
down to dig into the bottom of a hole, sideways to smooth the top of a
hole you've just filled.
- You can operate the shovel with different operands: foot for cutting,
hands for lifting.
- You can operate the shovel at different speeds: slowly for heavy loads
of gravel, quickly for light loads of peat moss.
Think of the above set of solutions as something like middleware. The
shovel itself is activiated using primitive degrees of freedom operated by a
hardware driver. The middleware this interface controls encapsulates
parametrized collections of low-level actions, considered in terms of higher
level descriptions: digging, pounding, cutting, moving, loading, and so
Real automated shovels - highhoes, backhoes, loaders - don't have
middleware. A skilled operator directly manipulates low level degrees of
freedom to put the shovel where it's needed. What is manipulated is a
kinematic description of the shovel. For a conventional shovel there are
actually only a few kinematic degrees of freedom:
Imagine a very crude interface: five sliders on a screen. As you drag each
slider it moves the shovel in that direction with a velocity corresponding to
the speed of slider motion. Could you do work like this? (Hint. Feedback.)
- the position of the centre of mass,
- direction cosines of angular displacements from a standard
- five velocities associated with the five positional degrees of freedom
What about integrating control and feedback using virtual reality gloves
that both sense and resist the movements of our hands and arms? We can even
allow the user to get the strength and aerobic exercise associated with using
a real shovel. Haven't we been here before?
Thinking about a shovel interface is a fantasy. Shovelling is too much fun
to leave to a computer! But because a shovel is very concrete it gives you a
push in the direction of thinking about concrete actions that have natural
analogues in a mouse/icon interface. Now think about things that actually
happen when you are using a word processor or spread sheet or database.
Formulate actions that can be implemented in the interface to give you direct
manipulation interfaces to these operations. Try to more imaginative than
pulling a tab or dragging to the trash. Remember to think in terms of
operation, signified by action and operand signified by icon. There is lots
of freedom in how you do this. For example, is printing or editting an action
or an operand?