CS349 - Implementing User Interfaces - Spring 2010

Public Service Annoucements

1. Assignment 3.

Lecture 26 - Direct Manipulation

How direct manipulation works

1. Grab something with the mouse
2. Move around the mouse to torment it a little
3. Drop it when you like the result

The very simplest direct manipulation interface

1. Locations on the screen have special functionality associated with them
   • usually indicated by little drawings, called icons
   • e.g. wastebasket.
2. Move something
   • usually indicated by a little drawing
   to the special location.
3. The special functionality is applied it it
   • if appropriate

Such interfaces are usually best when there are lots of special locations.

• Programs that manipulate geometry are the obvious candidates.
• Every location is special, meaning `put me here.'

In fact, geometry manipulation programs often want different interpretations for a single location

• `Select what's here.'
• `Move whatever's selected to here.'
• `Make here the centre of rotation.'
• and so on.

Modes are used to create separate contexts that disambiguate the meanings.

What we need to think hard about

When we manipulate graphical objects

• What is an object?
  • We click on a single pixel.
  • The click selects a collection of pixels.
  • How does the user know what's in and out of the selection?
• Manipulation techniques for defining objects.
  • a few basic objects; user builds composite objects: temporary or permanent
  • everything in an area
    • lasso
    • drag out the area

    only things completely inside the area or things partially inside the area

The Future of Direct Manipulation

Lasso is distinctly different from anything we have seen so far.

• How?

Gesture

Multitouch

Overloading

Learning The Principles of Direct Manipulation

1. Geometry manipulation applications have the richest repertoire of direct manipulation.
2. Direct manipulation requires a more powerful model of geometry.

One Preliminary - Points and vectors

A point is a location in space; what is the difference between two points?

• a vector

Both are designated by two real numbers (in the plane); what's the difference?

• consider translations in the plane
  • points change
  • vectors don't
• Computer example - addresses and offsets
  • offsets are the difference between addresses
  • create position independent (relocatable) code by using only offsets.

Thus, we have two kinds of entities, both represented by the same notation. To differentiate

• (x, y, 1) is a point.
• (x, y, 0) is a vector.

Does this make sense?

1. P - P -> V
2. P + V -> P
3. V + V -> V
4. P + P doesn't make sense: it gives (x + x', y + y', 2), which isn't defined
5. aV -> V
6. aP doesn't make sense: it gives (ax, ay, a), which is only defined if a=1..

Important finicky (and practical) point

• The interface we create is, in theory, position independent, but
  • to appear on a screen it must be attached to a particular position.
• Thus, when we write
  • DrawRectangle( window, x, y, width, height );

  window - indicates the origin to use

  x, y - is a point at which to start drawing

  width, height - define vectors that are combined with the starting point to make new points

Graphical User Interfaces

What does this tell us about graphical user interfaces (more precisely, interfaces that use direct manipulation)?

• doing graphics
  • separate DrawRectangle into two parts.
    1. a point part to place the first vertex
    2. a vector part to get from the first vertex to each subsequent vertex
  • separate `click and drag' into two parts
    1. a point part to locate the object to be dragged
    2. a vector part to update geometry while dragging occurs

We (for some value of we) know a powerful formalism for manipulating the vector part

• linear algebra
  1. rotation
  2. scaling
  3. reflection

  But, remember that these transformations all occur with respect to the origin, which you can consider to be the tail of the vector

• represented as 2x2 matrices, which operate on two-dimensional objects. But we are writing a vector not as (x, y), but as (x, y, 0), so we need to operate on three dimension objects.
  • We must extend the matrices to 3x3 in order to handle the third coordinate correctly.
  • Consider the structure of matrix multiplication. The upper left 2x2 part is just a two-dimensional matrix operating on a two-dimensional vector (x, y).
  • The first two elements of the third row just multiply the third component, which is zero, so they can be anything.
  • All vectors (x, y, 0) must have a scalar product of zero with the third column, which must be (0, 0, ?).

  So, getting the right result when we transform a vector determines the first two rows of the transformation matrix. The remainder of the matrix is determined by getting points right.

How do we extend these to points?

1. Rotate about the origin.
2. Reflect about the y axis.
3. Scale with respect to the origin.

To get these correct in 3x3 matrices

• bottom row must be (0, 0, 1)
• For example, the effect of rotating the point (x, y, 1) about the origin by the angle t is

          (  cos(t)  sin(t)  0 )
(x, y, 1) ( -sin(t)  cos(t)  0 ) = (x cos(t) - y sin(t), x sin(t) + y cos(t), 1) 
          (    0       0     1 )

Now we just need to add in translation.

1. Leave vectors unchanged (x, y, 0) -> (x, y, 0)
   • 2x2 identity matrix
   • rightmost column must be (0, 0, anything)
2. Translate points (x, y, 1) -> (x+dx, y+dy, 1)
   • bottommost row must be (dx, dy, 1)

Now we can get any geometric figure specified as sets of points and vectors anywhere and any shape, just by multiplying matrices, which computers are very good at.

Matrix multiplication is just a technology for implementing an interface, and not an interface. We also need a framework for putting a human-understandable interface onto matrix multiplication. Here's how it's done.

1. Implement an interface that returns a geometric operation that you want to perform.
2. Interpret the geometric operation in terms of a sequence of basic transformations
   • rotating around the origin
   • scaling with respect to the origin, along the x-axis
   • reflecting in the y axis
3. Write each basic transformation as a matrix.
4. Multiply the matrices together, and apply the result to the thing you want to transform

Redundancy check

There are two ways to rotate a rectangle which is away from the origin.

1. Consider it as a collection of points
   1. Translate to the origin
   2. Rotate at the origin
   3. Translate back to where it started
2. Consider it as one point plus a collection of vectors
   1. Rotate the vectors

They `obviously' give the same answer. (Exercise left to the reader.)

You will find both representations in PostScript because one is sometimes more convenient than the other.

But for making a graphical interface it's usually better to stick to one or the other.

Graphical User Interfaces

Last lecture we talked about points and vectors, how they transform.

How does this help us make graphical user interfaces (more precisely, interfaces that use direct manipulation)?

Let's think for a minute about assignment 5. What needs to be done?

1. drawing
   • using painter's algorithm and Xlib - need to know parameters like x, y, width, height
2. selecting
   • depends on an inside test
3. transforming
   • translate
   • rotate
   • rescale
   • reflect

Here's how we do it.

1. Base all things you draw on a small number of basic figures, such as
   • a standard line running from (0, 0) to (1, 0)
   • a standard rectangle with corners at (0, 0), (0, 1), (1, 1), (1, 0)
   • a standard ellipse with zero eccentricity (a circle), centred at (0, 0), of radius 1.
   • and so on.
2. Use transformations to turn this into any quadrilateral shape
   • represent transformations by matrices 3x3
     1. translation by dx, dy
        • point: (x, y, 1) -> (x + dx, y + dy, 1)
        • vector: (x, y, 0) -> (x, y, 0)
        • Matrix that does this must do its thing in the bottom row
     2. rotation by t
        • point: (x, y, 1) -> (x cos(t) - y sin(t), x sin(t) + y cos(t), 1)
        • vector: (x, y, 0) -> (x cos(t) - y sin(t), x sin(t) + y cos(t), 0)
        • Matrix that does this must do its thing in the 2x2 part.
     3. reflect about y-axis
        • point: (x, y, 1) -> (-x, y, 1)
        • vector: (x, y, 0) -> (-x, y, 0)
        • Matrix that does this must do its thing on the diagonal of the 2x2 part
     4. rescale by a along the x-axis
        • point: (x, y, 1) -> (ax, y, 1)
        • vector: (x, y, 0) -> (a, y, 0)

        Matrix that does this must do its thing on the diagonal of the 2x2 part

Matrices

The matrices you need (transposed if you like column vectors rather than row vectors)

1. Translation (points only)

   ( 1.0    0.0   0.0 )
   ( 0.0    1.0   0.0 )
   ( dx     dy    1.0 )

2. Rotation about the origin

   (  cos(t)   sin(t)   0.0 )
   ( -sin(t)   cos(t)   0.0 )
   (    0.0      0.0    1.0 )

3. Scale along the x-axis

   (   a    0.0    0.0 )
   ( 0.0    1.0    0.0 )
   ( 0.0    0.0    1.0 )

4. Reflect about the y-axis

   ( -1.0    0.0    0.0 )
   (  0.0    1.0    0.0 )
   (  0.0    0.0    1.0 )
   
       

These are standard transformations. How do we do more interesting and useful work?

• Make the transformations we want as compositions of the basic transformations
• Examples.
  1. Rotation about an arbitrary point.
  2. Scaling along an arbitrary direction.
  3. Scaling about an arbitrary point and along an arbitrary direction.
  4. Reflection in an arbitrary line.

These matrices transform points and vectors. Let's see how we do drawing.

1. Store the standard shape in terms of points and vectors.
2. Know how to draw the standard shape in terms of its points and vectors.
3. Store with it a transformation matrix that turns it into the actual shape.
4. Draw the actual shape using the transformed points and vectors.

And let's see how we do the inside test. (We could just use the transformed points, but there's an easier way to do it.)

1. Fact. The transformations above - when applied to a point and a shape - do not affect whether or not the point is inside the shape.
2. This gives us two options for doing the inside test
   1. Transform the standard shape into the point's world, using the transformation matrix stored with the point.
   2. Transform the point into the standard shape's world, using the inverse of the transformation matrix.

   We prefer the second, because we can choose the standard object so that the inside test is easy.

Therefore store the inverse matrix with the shape, along with the transformation matrix.

Exercises.

1. What is the inverse matrix of each of the basic matrices?
2. How do we multiply inverse matrices?

Return to:

• CS349 - Spring 2010 - Lecture schedule
• CS349 - Spring 2010
• CS349 page
• Bill Cowan's home page.