CS452 - Real-Time Programming - Spring 2011

Lecture 18 - Administrator/Detective

Pubilic Service Announcement

1. Convocation

Servers and Attendant Tasks

1. Proprietor with a Notifier

2. Using a Courier

3. Using a Warehouse

4. The Receptionist

Note

This structure clears up most problems when a burst of requests to the server would leave the notifier waiting in a long sendQ..

• Warehouse and proprietor share the work.
  • Server's Tid is public; Warehouse's Tid is private.
• This is far from the only way to share the work. For example,
  • The server could be guarded by a receptionist (assistant) who ensures that another client request occurs only when the previous request is complete. Then the warehouse is unnecessary.
• Handles bottlenecks of all sizes.
  1. But at the price of holding off waiting clients
     • It's important that the computation of waiting clients is less important to the overall application than the work being done by the proprietor, warehouse and notifier.
  2. Give a precise and quantitative definition of `bottleneck'.
  3. Server could be buffered on the other side
  4. Often called a guard.

What this amounts to is that servers should be lean and hungry

• do as little as possible
• almost always be receive blocked

5. Administrator, Worker

Administrator is a proprietor who does no work but only assigns work to others

• Tasks are given to workers
• If Create is fast and you have a method for reclaiming resources you can Create and Destroy workers on demand.
• Otherwise workers are created at initialization and the administrator maintains a pool of free workers
• each free worker is a REPLY-BLOCKED task

Real administrators manage workers

• Static ordanizations hire a workforce of employees who are assigned tasks as they come up.
• Dynamic organizations hire workers after the need for work appears, and lay them off when the work is done. These are called contract workers or consultants.

Most workers prefer employee status to consultant status.

Worker code

  Send( administrator, nil, workOrder );
  FOREVER {
    Send( administrator, workResult, workOrder );
    doWork( workOrder );
  }

Administrator code

Initialization

Administrator( ) {
  for ( i = 0; i < NUM_WORKERS; i++ ) worker[i] = Create( workerCode );
  for ( i = 0; i < NUM_WORKERS; i++ ) {
    Receive( *worker[i].tid, ... );
    enqueue( employeeQ, worker[i1 );
  }
    
  FOREVER {
    Receive( requester, request/result );
    switch( request.type ) {
    case CLIENT:
      enqueue( orderQ, {client = requester, request} );
      if ( !empty( employeeQ ) ) Reply( dequeue( employeeQ ) ), dequeue( orderQ ) ); 
      break;
    case WORKER:
      Reply( result.client, result );
      enqueue( employeeQ, requester );
      if ( ! empty( orderQ ) ) Reply( dequeue( employeeQ ) ), dequeue( orderQ ) ); 
      break;
  }

Note: Tid of the client is included in the workorder to the administrator does not have to maintain and search a list of jobs being done. (Administrators are by nature lazy!)

Alternative Worker/Administrator Model

1. As above, Administrator includes Tid of the client in the order.
2. Worker replies to client with result and to administrator with request for another order to process.

Comments

1. The administrator can add a little more value.
   • Suppose that there is data required for processing each order.
   • The administrator could receive it from a notifier or courier.
   • It would be maintained internally and added to the appropriate order before the order is despatched to the worker.
   • Another model: the worker queries a detective.

8. The Detective

Simple Events

The notifier is a task that waits on events.

• AwaitEvent is like Send

  It has two features

  • a simple, kernel-defined event that it waits on
  • hardware/kernel is like Receive/Reply
  • can only serve one master
• The notifier needs to pass on that the event has happened
  • which it does using Send

    FOREVER {
      AwaitEvent( eventId );
      Send( server );
    }

You could call a notifier a detective,

• who looks around on your behalf,
• and let's you know when something you care about happens,
• but really it is a detective's worker.

Complex Events

In an application there is likely to be lots of waiting on combinations of events.

• form the combinations using Boolean operators
• for example, Respond `okay' when this sensor is triggered or `time-out' when Delay returns.

We use the detective to discover that a complex event has occurred.

• How does the detective work?

Conjunction

Code could be

FOREVER {
  Send( part1 );
  Send( part2 );
  ...
  Send( master );
}

Disjunction

Code above doesn't even pretend to work!

Try instead

FOREVER {
  Receive( *requester, request );
  switch( request.type ) {
  case CLIENT:
    enqueue( requestQ, {requester, request} );
    if ( happened( request, DB ) ) Reply( dequeue( requestQ ), ... );
    else foreach clue in missingInfo( request, DB ) {
      if ( ! empty( employeeQ ) Reply( dequeue( employeeQ ), clue );
      else enqueue( clueQ, clue );
    break;
  case WORKER:
    enqueue( employeeQ, requester );
    updateDB ( request );
    while ( ! empty( clueQ ) {
      if ( ! empty( employeeQ ) Reply( dequeue( employeeQ ), dequeue( clueQ ) );
      else break;
    }
    foreach request in requestQ
      if ( happened( parsedRequest, DB ) ) Reply( request.client, ... );
    break;
}

This is the code of a detective.

Not

We can say that an event has not happened yet.

Only at the end of the universe can we say that an event simply has not happened.

Time-outs are needed for NOT

• OR with Delay on the clock server.

Who is the client of the detective

1. Initiator of a normal action

Housekeeper of the system who will clean up pathologies (Idletask?))

Debugging Real-time Programs

Breakpoint

What does it do?

• snapshot of the system
  • This means that computation, including respose to interrupts, must stop, or it isn't a snapshot.
• provides interactive tools for examining kernel data structures, such as
  • task descriptors
  • lists and queues
  • stacks, including the program counter and local variables, of individual tasks
• restart system immediately afterwards
  • If you want to continue where processing stopped you must make certain that all state is saved when you enter Beakpoint and restored when you leave it. What about pending interrupts? You can't stop the entire universe!
  • Otherwise you can re-enter RedBoot.

How do you get it started?

• function call, which you insert in your code when compiling.
  • The easiest and fastest form to implement.
  • Having the call as part of ASSERT is common.
  • Has to exit to RedBoot. (Jump to x00.)
• system call instead of function call, which respects the kernel/user distinction.
• an exception triggered externally
  • at initialization
    1. Set up the system so that the external event will generate an exception
    2. E.g. attach a button to PDIO on the third connector, set up ICU.
  • at run-time
    1. Trigger the interrupt
    2. Switch to Breakpoint in the event handler
    3. Either exit to RedBoot,
    4. Or clean up pending interrupts and resume execution.

Breakpoint is a special case of a particular sort of tool that is very common.

• condition occurs => information is made available
• printf provides the information in `real-time'.
• breakpoint provides the information interactively (`interactively' = `on the time scale of the user')
  • You really would like to examine a shapshot, seeing the system at a moment in time.
  • You can stop the system completely. How?
  • but you have limited ability to stop the real world
    • i.e., it hides some bugs

Getting information closer to real-time.

Gossip

Circular buffer somewhere

• You write into it as things happen
• This gives a history of recent events that have occurred in the system

How do you write it?

1. Write it in English
2. Write it in code

What are the advantages and disadvantages of each?

How do you read it?

1. From the corpse of a dead execution of the application
2. From Breakpoint, which is probably the same thing

Use Bits

Gossip encodes its content as semantic content

• possibly translated to English on output

One weakness of Gossip is that to go back in time very far you need a very big buffer

• Any non-trivial application generates a lot of separate interactions with the kernel
• Side comment. There has been a huge amount of research into methods for acquiring and displaying the real-time activity of real-time applications..
  1. It is designed and tested on toy programs, often called test-beds.
  2. It doesn't scale. Therefore it isn't used.
  3. Most real-time debugging is done using logs, which seriously limits what can be debugged.
  4. These facts are well-known, and are considered to be a serious problem.
  5. But nobody has any very good ideas about how to proceed.

Use bits encode the information differently.

How do they work?

1. Set aside a large-enough block of memory: 1 Megabyte contains 8,000,000 bits.
2. Assign each bit to an event that interests you.
3. Write the bit when the event occurs. (If it occurs several times assign more than one bit.)
4. Examine the bits in a dead execution to see how far individual tasks have gotten.

Return to:

• Bill Cowan's lecture notes for CS452 in s11
• Bill Cowan's Spring 2011 CS452 page
• Bill Cowan's CS452 page
• Bill Cowan's teaching page
• Bill Cowan's home page