CS789, Spring 2005

Lecture notes, Week 6

Synchronization

Why are context and time related like this? Because human ability to handle different contexts simultaneously is limited. Often, when an interface offers several contexts simultaneously the user operates within only one at a time.

As an example, I can imitate your facial expressions. How?

• One mechanism watches you and analyses your expressions.
• Another mechanism formulates target expressions to assume, and releases appropriate motor programs.
• Both mechanisms run simultaneously.

• Each person hears two voices, but can use stream segregation to pay attention to only one of them.
• Each person only has to control one mouth.
• A process interprets what is coming in and formulates things to say. (This process runs coincidently with listening and interpreting when we interchange conversational control.)

A simple example: the prompt

• Return/enter is the trigger, transferring control from user to computer. Before the trigger
  • what has been typed continues to be owned by the user, and
  • it can still be editted by the user.
  • Note the existence of a second level trigger within the interaction. (Typing a character is revocable until a particular instant when the switch contacts meet.)
  After the trigger
  • what has been typed is owned by the computer.
• At the prompt the computer relinquishes control, returning it to the user.
• Where do prompts and triggers exist outside computer interfaces?
• What is type-ahead? What makes echoing type-ahead problematic?

What makes a good trigger?

• Something that can be used in many contexts. We want the trigger to be overlearned.
• Something that is atomic.
• Something that gives feedback. (What actually provides the signal to stop when you mistype?)

• "Enter", but key-down or key-up?
• Mouse button release
• #, in most telephone interfaces
• Pause, of a particular duration.

What makes a good prompt?

• Something that says "I'm a prompt."
• Information about what input is legal: the range of possible actions
• Information about relevant system state, to help the user predict the result of subsequent actions.

• %
• Menu items displayed when a pop-up menu is summoned
• (~/courses/cs689/w99)[34]%
• "Uh-huh."
• Rising intonation.
• Change in the tracker icon, such as watch changing to arrow..

Sequential synchronization (synchronization protocols)

There are three conceptual modes of human/system synchronization

1. Request mode, where the system requests input from the user. E.g., Unix.
   • Prompt(% ): "Start giving me a request."
   • Trigger(<ENTER>): "Here's the request you asked for."
   The process is explicitly sequential.
2. Event mode, where the user provides the system with things to which it must respond. E.g., typing to a word processor.
   • The trigger (key press): "Here's an event to respond to. (Add this character to the document.)"
   • The prompt (insertion point) says, "Here's the context in which I will evaluate the next event. (Here's the location where I will place the next character.)"
   • Usually it is possible for the prompt to change to something like: "Slow down the events, please. (I can't format as fast as you can type.)"
   The user doesn't have to wait for the processing of a previous event to finish before sending a new one. Maintaining and showing the context in which the new event will be interpreted is not necessarily simple.
3. Sample mode, where the system tries to follow the user. E.g., free-hand drawing with controls points appearing, as in Illustrator.
   • The trigger is generated by the system using its own rules
   • The prompt usually shows the mode in which the interface will handle the input. The representation may be implicit, explicit, or both.
   Time is normally an essential part of the system's interpretation of the input.

Note that while the trigger is a useful concept in the first two cases it's less useful in the third. We can usefully make the following distinction between two input styles:

• triggered: The user specifies explicitly when the input is valid. In event mode the user adjusts the value to be measured, with no guarantee that it has useful intermediate values. The user guarantees that the measure is useful when the trigger is presented.
• measured: The user makes sure that the input is valid all the time, because he or she doesn't know when it will be taken. In sample mode the user provides a measure as a value defined at all times; the system takes it when convenient.

We may conceptualize synchronization in terms of handshaking, but it is important to note that the prompt concept actually elides three separate actions that users may wish to distinguish. Separating them out an interface transaction can follow the sequence:

1. prompt -- the system can accept new input,
2. trigger -- input is ready to be accepted,
3. echo -- here is the input that was accepted,
4. acknowledge -- input has been processed.

In Unix the four stages are easily distinguishable in the Cshell when the command !! is issued.

% !!                                # prompt and trigger
/usr/sbin/shutdown -y -i5        # echo
<blank screen>                        # acknowledgement

One reason for differentiating the three components is that we might want to reorder them. Why? (Hint. "&" in Unix, which gives a new prompt before acknowledge.)

Now, what about context?

When the system provides output to the user,

• The system provides content (local).
• The user assesses the output in context, which includes
  • other available information (the rest of the screen),
  • what happened recently (the user's memory),
  • how things work in the world (the user's knowledge).

When the system takes input from the user,

• The user provides content (local).
• The system assesses the input in context, which is the state of the system, including
  • earlier elements of the stream of which the input is a part,
  • the state of other computations with which the system interacts.

Slippage between context and content, (The ATM thinks I'm putting in the amount of money I want; I think I'm putting in my PIN number.)

• the right content in the wrong context, e.g. Netscape has a File menu containing "Close"; what does it do? Immediately above Netscape's File menu is the window manager's window title menu, containing "Close"; what does it do?
• the wrong content in the right context, giving vi commands to an emacs session

Solutions depend on continuity, which is takes advantage of the user's memory. How is continuity broken? The user's continuity is broken by interruptions (which are, by definition, unexpected),

• The telephone rings.
• A child calls for attention.
• Grep returns nothing.

• Things lying around that remind the user of system/application/interface state are important. These depend on contiguity.
• Cues to change context sometimes should be processed voluntarily so they can be bypassed when the user already has the context. E.g., "Pull up, pull up."
• The lower the level at which the cues are processed, the less effortful they are to the user. But, note the effect of practice on the "Do you really want to remove this file?" prompt.

• Provide specific stimuli to indicate significant changes in state. These are normally orienting cues, to which the user cannot avoid responding. E.g., sirens.
• Require the user to initiate any significant change in state. Thus any change takes place as a result of explicit attention from the user. (An example is the ubiquitous confirmation box. But what about automatization of user responses?)

Multi-stream synchronization (windowing systems)

• Window frames and other decoration are provided to allow the user visually to parse the screen into conceptual objects.
• Content within a single object is taken as related; content in different objects is taken as unrelated.
• Actions are requested from different conceptual objects, and responses must be directed to the appropriate object.
• One object is attended by the user; the others are monitored.
• Monitored objects must be able to request user attention. There are three types of cues:
  1. ones that compel attention (sirens),
  2. ones that attract attention from suitably primed users (signs),
  3. ones that retain freely given attention but do not explicitly attract attention (sentences).

Real-time Interfaces (multi-media)

Multi-media concepts

• attribute of users:
  • sight, hearing, touch, smell, taste, proprioception for input
  • what for output?
• Temporal modalities are sequential, but remember the existence of simultaneous streams.
• Spatial modalities have limited parallel capabilities. Assemble percepts at different locations into a single object.
• Attention is unitary: every place in time, sooner or later; a single place in space.

Medium

• attribute of systems:
  • CRTs, LCDs, speakers, touch sensors for output
  • keyboards, mice, trackballs, eye-trackers for input
• Temporal media are sequential
• Spatial media have limited parallel capabilities, depending on the ability to provide integral percepts to the user.
• Inter-medium synchronization is very hard to implement, and is almost always ad hoc. (This is a matter of what is easy and hard to program.)

Streams are

• channels through which information flows
• attributes of human-system interaction:
  • CRT/vision, speaker/audition for computer/human information flow
  • hand/mouse, head position/3D sensor, speech/microphone for human/computer information flow
• Streams have well-defined temporal properties
• Streams contain information that is related.
• Inter-stream correlation is powerful for human input, wholeness, causality
• Inter-stream correlation is difficult to control in output directed at humans, unless it's easy, of course.

Obviously modality and medium, while useful as general terms, are not very precise. They are useful when attaching characteristics to capabilities: CRTs have different output characteristics than LCDs; eyes have different input characteristics than ears. On the other hand they are less useful when we try to describe interface architecture. For this purpose stream is a much more useful basic concept.

Using the above terminology it is obvious that windowing systems are multi-stream systems. They usually have a single output medium, a CRT, connected to a single modality, vision, and making available several output streams (windows), interacting with a single attentive (human-)input stream that is time-multiplexed between windows. They usually have two (computer-)input streams, mouse and keyboard, both associated with the same user modality, hand/finger. Control of the (human-)output modality is time-multiplexed over the two input streams.

Multi-media systems, as the name suggests, are systems that employ more than one medium for communication with the user. From the user's point of view it is most significant that different streams contain correlated input. Users seek out correlation among the many input streams they process. Correlation is based on simultaneity.

Simultaneity

• defined by the user (<100 milliseconds)
• inter-stream concept
• defines the granularity of time
• humans appear not to have much flexibility

Multi-media architecture

Return to:

• Bill Cowan's home page.
• Bill Cowan's teaching page.
• Bill Cowan's CS789 page.
• Bill Cowan's CS789 page for Spring 2005.
• Bill Cowan's CS789 page for Spring 2005 : Notes