CS789, Spring 2005

Activity instructions, Week 9

Extend the direct manipulation dialogue style

Outside of graphical interfaces, which we will discuss separately, the direct manipulation dialogue style has been pretty well restricted to select, open and drag-and-drop. How do they work?

  1. Select -- point at the thing you want to select and single-click the select button on the mouse; this is called "point-and-click". The analogy is to pointing at one of several objects and saying, "That one." This is not the only selection mechanism. Selecting a group of objects can often be done differently. Selecting a menu item can be point-and-click, but there are also other techniques. Selecting the window or the field of a form to receive input can be point-and-click, but there are also other ways. How do different selection methods co-exist?
  2. Open -- point to the thing you want to open and do an operation different from single-clicking the select button. Most often this operation is double-clicking, but there are other possibilities. Think of a few examples. The "open" operation has been generalized to a group of operations clustered around the open/start idea. The precise meaning is determined by context and users seem not to have any difficulty with the overloading/generalization.
  3. Drag-and-drop -- unlike select and open an operation like move requires two operands, the thing to move (source) and the place to which it should be moved (destination). It's very natural to represent this operation using actual movement of the icon representing the source. The movement starts at the source with a mousedown event (select?) and ends at the destination with a mouseup event. If CPU and graphics resources are available there is usually real-time motion of the source, or its shadow, as the tracker moves. Drag-and-drop can be generalized without leaving the metaphor of motion. If the destination is not a location, but an object that does something, then dropping the source into it represents moving the source to the object and having it processed. Is this logical?
The result is a fairly limited set of things to do. How might we do more within the direct manipulation metaphor?

Let's start by considering interaction with a shovel as a real-world analogue of direct manipulation. "Shovel" is both a noun and a verb, a tool that you can use on something and an action that you can perform. Furthermore, when you use a real-world shovel you do it by directly handling the shovel in some way.

Now, before going any further, find a shovel and use it for digging a hole. Observe as you dig how the shovel transfers the force exerted by your muscles to the earth in which you are digging. And observe how the shovel tranfers the resistance of the earth back to your hands and arms so that you can adjust the direction and magnitude of the forces you are applying. Fill in the hole again, smoothing and packing the earth with your shovel. Note how you hold the shovel differently when doing different tasks.

With this experience under your belt you are ready to think about what would be involved in creating a direct manipulation interface to a virtual shovel, which would allow a user to use the shovelling metaphor as a way of doing things with a shovel.

Try the following analogies to see if they make sense.

  1. To get a shovel that is among a collection of tools we reach in with a hand, and pull it out. You can do this with a standard selection interface.
  2. To move the shovel to a place you want to dig, carry it there. A drag-and-drop interface should do this fine.
  3. Opening the shovel seems like a natural way of starting to dig with it. Of course, allowing it to shovel randomly is not likely to be very useful. Possibly opening the shovel puts the interface into a mode where the mouse controls the physical position of the shovel, allowing the user to move it arbitrarily.
Once you can move the shovel arbitrarily it's possible to do any shovelling operation. There are still a couple of gotchas. This is starting to sound a lot like virtual reality shovelling: hand, arm and foot motions, visual and force feedback very like the ones I experience while doing real shovelling. With just the right scaling I might exactly duplicate the shovelling work-out. Venting could provide fresh air controlled by biological sensors that monitor the rate of my metabolism. But perhaps this is direct manipulation taken a little too far.

Another approach is to focus on the task being undertaken. Mostly shovels are used to move things. The operation encompasses

  1. a thing to move, and
  2. a place to move it to.
This sounds more like it: a drag-and-drop interface will be fine. We need to be a little more clever, however. When we want to make a hole we want to move everything within a particular volume, regardless of what it is. Think of a variety of ways of indicating what is to be moved.

And what about constraints on the destination? I can use a shovel to fill a barrel with sand, where the termination condition is based on the destination rather than the source.

In reality a shovel is a lot more flexible than an interface like this one allows, and has a lot wider range of uses. You can use it to pound in a stake, to level a pile of sand, or to cut a piece of sod. And you do these operations by holding the handle and/or pushing on the blade with your foot. What this means is that we can "operate" a shovel in more than one way.

Is it possible that there is a description of shovel manipulation intermediate between between "duplicate real shovel control", which gives maximum flexibility, and "select from pre-defined goals", which restricts the user to objectives conceived by the shovel designer? Try to think of one.

Here are a few ideas on which you can test your ideas.

Try and think up a few other ways of doing things with a shovel, and consider different ways of activating a shovel icon with the mouse that would express the differences in what you might want to do. (This is a potential interface to the computer-shovel which digs in your garden in the cold rain while you operate it from inside using your PC.)

Think of the above set of solutions as something like middleware. The shovel itself is activiated using primitive degrees of freedom operated by a hardware driver. The middleware this interface controls encapsulates parametrized collections of low-level actions, considered in terms of higher level descriptions: digging, pounding, cutting, moving, loading, and so on.

Real automated shovels - highhoes, backhoes, loaders - don't have middleware. A skilled operator directly manipulates low level degrees of freedom to put the shovel where it's needed. What is manipulated is a kinematic description of the shovel. For a conventional shovel there are actually only a few kinematic degrees of freedom:

Imagine a very crude interface: five sliders on a screen. As you drag each slider it moves the shovel in that direction with a velocity corresponding to the speed of slider motion. Could you do work like this? (Hint. Feedback.)

What about integrating control and feedback using virtual reality gloves that both sense and resist the movements of our hands and arms? We can even allow the user to get the strength and aerobic exercise associated with using a real shovel. Haven't we been here before?

Thinking about a shovel interface is a fantasy. Shovelling is too much fun to leave to a computer! But because a shovel is very concrete it gives you a push in the direction of thinking about concrete actions that have natural analogues in a mouse/icon interface. Now think about things that actually happen when you are using a word processor or spread sheet or database. Formulate actions that can be implemented in the interface to give you direct manipulation interfaces to these operations. Try to more imaginative than pulling a tab or dragging to the trash. Remember to think in terms of operation, signified by action and operand signified by icon. There is lots of freedom in how you do this. For example, is printing or editting an action or an operand?

Return to: