CS789, Spring 2005

Activity instructions, Week 9

Extend the direct manipulation dialogue style

Outside of graphical interfaces, which we will discuss separately, the direct manipulation dialogue style has been pretty well restricted to select, open and drag-and-drop. How do they work?

1. Select -- point at the thing you want to select and single-click the select button on the mouse; this is called "point-and-click". The analogy is to pointing at one of several objects and saying, "That one." This is not the only selection mechanism. Selecting a group of objects can often be done differently. Selecting a menu item can be point-and-click, but there are also other techniques. Selecting the window or the field of a form to receive input can be point-and-click, but there are also other ways. How do different selection methods co-exist?
2. Open -- point to the thing you want to open and do an operation different from single-clicking the select button. Most often this operation is double-clicking, but there are other possibilities. Think of a few examples. The "open" operation has been generalized to a group of operations clustered around the open/start idea. The precise meaning is determined by context and users seem not to have any difficulty with the overloading/generalization.
3. Drag-and-drop -- unlike select and open an operation like move requires two operands, the thing to move (source) and the place to which it should be moved (destination). It's very natural to represent this operation using actual movement of the icon representing the source. The movement starts at the source with a mousedown event (select?) and ends at the destination with a mouseup event. If CPU and graphics resources are available there is usually real-time motion of the source, or its shadow, as the tracker moves. Drag-and-drop can be generalized without leaving the metaphor of motion. If the destination is not a location, but an object that does something, then dropping the source into it represents moving the source to the object and having it processed. Is this logical?

Let's start by considering interaction with a shovel as a real-world analogue of direct manipulation. "Shovel" is both a noun and a verb, a tool that you can use on something and an action that you can perform. Furthermore, when you use a real-world shovel you do it by directly handling the shovel in some way.

Now, before going any further, find a shovel and use it for digging a hole. Observe as you dig how the shovel transfers the force exerted by your muscles to the earth in which you are digging. And observe how the shovel tranfers the resistance of the earth back to your hands and arms so that you can adjust the direction and magnitude of the forces you are applying. Fill in the hole again, smoothing and packing the earth with your shovel. Note how you hold the shovel differently when doing different tasks.

With this experience under your belt you are ready to think about what would be involved in creating a direct manipulation interface to a virtual shovel, which would allow a user to use the shovelling metaphor as a way of doing things with a shovel.

Try the following analogies to see if they make sense.

1. To get a shovel that is among a collection of tools we reach in with a hand, and pull it out. You can do this with a standard selection interface.
2. To move the shovel to a place you want to dig, carry it there. A drag-and-drop interface should do this fine.
3. Opening the shovel seems like a natural way of starting to dig with it. Of course, allowing it to shovel randomly is not likely to be very useful. Possibly opening the shovel puts the interface into a mode where the mouse controls the physical position of the shovel, allowing the user to move it arbitrarily.

• How strong should the shovel be? Perhaps the strength should vary, under user control. Force feedback is likely to be useful. And when the force of my hand isn't enough, might I bring my feet into play?
• How do we match the precision of mouse motions to the precision of shovel motions? I can position a real shovel for digging to within a couple of millimetres, which is comparable to the accuracy with which I can see the object of my shovelling. The scale of visual feedback needs to be close to one-to-one; the scale of hand motion to shovel motion needs to be close to one-to-one.

Another approach is to focus on the task being undertaken. Mostly shovels are used to move things. The operation encompasses

1. a thing to move, and
2. a place to move it to.

And what about constraints on the destination? I can use a shovel to fill a barrel with sand, where the termination condition is based on the destination rather than the source.

In reality a shovel is a lot more flexible than an interface like this one allows, and has a lot wider range of uses. You can use it to pound in a stake, to level a pile of sand, or to cut a piece of sod. And you do these operations by holding the handle and/or pushing on the blade with your foot. What this means is that we can "operate" a shovel in more than one way.

• We can swing it.
• We can slide it.
• We can cut with it

Here are a few ideas on which you can test your ideas.

• Sometimes there is only one thing you can reasonably do with a particular operand. If a burglar is the operand what is the shovel doing? If a wheelbarrow is the operand what is the shovel doing?
• You can operate on different parts of the shovel: the blade for digging or cutting, the handle for throwing.
• You can operate the shovel in different directions: up to empty a hole, down to dig into the bottom of a hole, sideways to smooth the top of a hole you've just filled.
• You can operate the shovel with different operands: foot for cutting, hands for lifting.
• You can operate the shovel at different speeds: slowly for heavy loads of gravel, quickly for light loads of peat moss.

Think of the above set of solutions as something like middleware. The shovel itself is activiated using primitive degrees of freedom operated by a hardware driver. The middleware this interface controls encapsulates parametrized collections of low-level actions, considered in terms of higher level descriptions: digging, pounding, cutting, moving, loading, and so on.

Real automated shovels - highhoes, backhoes, loaders - don't have middleware. A skilled operator directly manipulates low level degrees of freedom to put the shovel where it's needed. What is manipulated is a kinematic description of the shovel. For a conventional shovel there are actually only a few kinematic degrees of freedom:

• the position of the centre of mass,
• direction cosines of angular displacements from a standard orientation,
• five velocities associated with the five positional degrees of freedom described above.0

What about integrating control and feedback using virtual reality gloves that both sense and resist the movements of our hands and arms? We can even allow the user to get the strength and aerobic exercise associated with using a real shovel. Haven't we been here before?

Thinking about a shovel interface is a fantasy. Shovelling is too much fun to leave to a computer! But because a shovel is very concrete it gives you a push in the direction of thinking about concrete actions that have natural analogues in a mouse/icon interface. Now think about things that actually happen when you are using a word processor or spread sheet or database. Formulate actions that can be implemented in the interface to give you direct manipulation interfaces to these operations. Try to more imaginative than pulling a tab or dragging to the trash. Remember to think in terms of operation, signified by action and operand signified by icon. There is lots of freedom in how you do this. For example, is printing or editting an action or an operand?

Return to:

• Bill Cowan's home page.
• Bill Cowan's teaching page.
• Bill Cowan's CS789 page.
• Bill Cowan's CS789 page for Spring 2005.
• Bill Cowan's CS789 page for Spring 2005 : Activities