CS452 - Real-Time Programming - Spring 2012

Lecture 28 - Demos, Reservations

Public Service Annoucements

1. Final exam date: 9.00 August 7 to 11.30 August 9

Pathologies

As we go down this list both pathology detection and the length of the edit-compile-test cycle grow without bound.

1. Deadlock

One or more tasks will never run again. For example

1. Task sends to itself (local: rest of system keeps running, task itself will never run)
2. Every task does Receive( ) (global: nothing is running)
3. Cycle of tasks sending around the cycle (local: other tasks keep running)

Kernel can detect such things

• What does it do?

Potential deadlock can be detected at compile time

• cycle in the send graph of all sends that could happen
  • remember implicit sends
• doesn't necessarily occur at run-time
  • that is, it's a necessary but not sufficient condition.
• It's worse,
  • It doesn't happen when tests are short
  • It appears near the end when tests run for longer
  • Changes in a critical race can make a potential deadlock reveal itself.

Solutions

• Gating
  • Most common example is initialization, where the send/receive pattern may be different than FOREVER
  • Gate the end of initialization
• Define four types of task
  • Administatrer (A), including servers of all kinds: only receives
  • Worker (W), including notifiers: only sends
  • Client (C): only sends
  • Notifier (N): only sends to its administrator
  • Two A tasks cannot communicate directly; two W/C tasks cannot communicate directly.
  • For W/C/N tasksSend appears in two flavours
    • C tasks

      FOREVER {
          Send( A, request, result )
          ...
      }

    • W tasks

      FOREVER {
          Send( A, result, request )
          ...
      }

    • N tasks

      FOREVER {
          Send( A, result, request )
          result.data = AwaitEvent( request.event )
          ...
      }

    • The Receives corresponding to W, C & N tasks are normally the same.
      • N is effectively a W task
        • The important difference is that while W & C tasks are lower priority than the A task, N tasks are higher priority.
      • C, W & N requests and results must have compatible data types.
      • The request might be effectively a union
        • The payload is interpreted differently at run-time,
        • using different cases of the switch on message type.
    • A courier is W-type to one A-task, C-type to the other

      FOREVER {
          Send( A1, request, result )
          Send( A2, result, request )
      }

    • Occasionally, but not often, two A-tasks, are synchronized in a way that makes it possible to communicate in two directions through one server.
      • Then two couriers are used

2. Livelock (Deadly Embrace)

Definition

Two or more tasks are READY. For each task, the state of the other tasks prevents progress being made regardless of which task is ACTIVE.

A higher level of coordination is required.

Two types of livelock exist

1. Ones that are the result of bad coding
   • Find bug and remove it.
2. Ones that are inherent in the application definition
   • Detect livelock and work around it.

Looking for solutions we prefer ones that avoid a central planner. Why?

• In the twentieth century there were a collection of political systems relying on central planners
  • Most no longer exist

Livelock usually occurs in the context of resource contention

Livelock that's Really Deadlock

• client1 needs resource1 & resource2;
  • obtains resource1 from proprietor1;
  • asks proprietor2 for resource2
• client2 needs resource1 & resource2;
  • obtains resource2 from proprietor2;
  • asks proprietor1 for resource1
• possible code
  • Client 1

    Send( prop1, getres1, ... );
    Send( prop2, getres2, ... );
    // Use the resources and release them

  • Client 2

    Send( prop2, getres2, ... );
    Send( prop1, getres1, ... );
    // Use the resources and release them

  • Proprietor

    FOREVER {
       Receive( &clientTid, req, ... );
       switch ( req-type ) {
       case REQUEST:
          if( available ) { 
             Reply( clientTid, use-it, ... );
             available = false;
          }
          else enqueue( clientTid );
       case RELEASE:
          available = true;
          Reply( clientTid, "thanks", ... );
          if( !empty( Q ) ) {
             available = false;
             Reply( dequeue( ), use-it, ... );
          }
       }
    }

• state:
  • client1, client2: REPLY-BLOCKED - can't release resources
  • proprietor1, proprietor2: SEND-BLOCKED - waiting for release
  • this is a true deadlock -- none of the four tasks will ever run again -- even though there are no cycles in the call graph.
  • The dependencies lie elsewhere. Where?
  • (You can find on the internet arguments about terminology just as intense as anything you will ever see in vi vs emacs or Apple vs Microsoft.)

Solutions

1. Make a single compound resourse, BUT
   • all clients may not need both
   • some resources simply cannot be compounded
2. Impose a global order on resource requests that all clients must follow.
   • unsafe against malicious or incompetent programmers
   • some resources don't admit strong enough ordering, e.g. pieces of track in the train set
3. Create a mega-server that handles all resource requests
   • clients request all at once, mega-server provides an optimal solution to resource use in the presence of hundreds of deadlines.
   • client might not know that A is needed until processing with B is well-advanced

Real Livelock

Proprietor1 & proprietor2 fail the requests

• Proprietor

  FOREVER {
     Receive( &clientTid, req, ... );
     switch ( req-type ) {
     case REQUEST:
        if( available ) { 
           Reply( clientTid, use-it, ... );
           available = false;
        }
        else Reply( clientTid, "sorry", ...);
     case RELEASE:
        available = true;
        Reply( clientTid, "thanks", ... );
     }
  }

• Polling is the most likely result. Typical client code.

  while ( Send( prop1, getr1, ... ) != GotIt ) ; 
  while ( Send( prop2, getr2, ... ) != GotIt ) ; 
  // Use the resources

• And the problem is that this code usually works as long as the tests are relatively short

Livelock that's Really a Critical Race

We could try to make the clients a little more considerate

While ( no resources ) {
   Send( prop1, getres1, result );
   while ( result == "sorry" ) {
   if ( result == "sorry" ) {
      Delay( ... );
      Send( prop1, getres1, result );
   }
   Send( prop2, getres2, result );
   if ( result == "sorry" ) {
      Send( prop1, relres1, ... );
      Delay( ... );
   } else {
      break; 
   }
}

Inherent Livelock

Remember the example where two trains come face to face, each waiting for the other to move. They will wait facing each other until the demo is over, probably polling.

What's hard about solving this problem?

• Neither driver knows what the other driver is trying to do.

In real life,

• the drivers would communicate, but
• in your software that's neither easy
  • How many different `conversations' might need to be available?

  nor desirable

  • What is the effect on other trains of the two drivers' special arrangement?

What's most easy for you to do is to programme each driver with

1. detection, e.g.,
   • Delay a random time
   • Request again
   • If turned down, work around
2. work around, e.g.,
   • Recommence working on goal as though track is blocked.

3. Critical Races

Example

1. Two tasks, A & B, at the same priority
2. A is doing a lot of debugging IO
3. B always reserves a section of track before A, and all is fine.
4. Debugging IO is removed
5. A reserves the section before B can get it, and execution collapses.
6. Lower priority of A to the same level as C.
7. Now C executes faster and gets a resource before D .
8. You shuffle priorities forever, eventually reverting, to put back in the debugging IO.

Definition

The order in which computation is done is an important factor in determining whether or not it is successful.

Critical races, like Livelock can be

• internal to the program, like the one above, or
• external to the program but inherent in the application domain

Symptoms

1. Small changes in priorities change execution unpredictably, and drastically.
2. Debugging output changes execution drastically.
3. Changes in train speeds change execution drastically.
   • Example from several terms ago

`Drastically' usually means chaos in both senses of the term

1. Sense one: a small change in the initial conditions produces an exponentially growing divergence in the execution.
2. Sense two: exercise for the reader.

Solutions

1. Explicit synchronization
   • but you then have to know the orders in which things are permitted to occur
   • e.g. Try listing all the orders in which events can occur in your system
     • and then notice that just arriving in the same order is often not enough
2. Gating is a technique of global synchronization
   • which can be provided by a detective/coordinator

4. Performance

Changes in performance of one task with respect to another often give rise to critical races

The hardest problem to solve

• You just don't know what is possible
• Ask a question like:
  • Is my kernel code at the limit of what is possible in terms of performance?
  • We can compare the performance on message passing, etc., because two kernels are pretty much the same.
    • Compare a lot of kernels and you should be able to find a lower limit
  • Can't do the same thing for train applications

In practice, how do you know you have performance problems? Problems I have seen

Priority

The hardest thing to get right

• NP-hard for the human brain
• Practical method starts with all priorities the same, then adjusts
• symptoms of good priority assignment
  • The higher the priority, the more likely the ready queue is to be empty
  • The shorter the run time in practice the higher the priority

Problems with priority

1. Priority inversion
2. One resource, many clients
3. Tasks try to do too much

Congestion

1. Too many tasks
   • blocked tasks don't count,
   • lowest priority tasks almost don't count

Layered abstraction are costly

e.g. Notifier -> SerialServer -> InputAccumulater -> Parser -> TrackServer

Output

1. Too much terminal output interferes with train controller communication
   • Don't redraw the entire screen
2. Requests to poll the sensors get backed up in the serial server, or whoever provides output buffering.

Hardware

1. Turn on optimization, but be careful
   • There are places where you have done register allocation by hand
2. Turn on caches
   • locking is possible

Size & align calibration tables by size & alignment of cache lines

• linker command script
• I think that this is stretching it.

Return to:

• Bill Cowan's lecture notes for CS452 in s12
• Bill Cowan's Spring 2012 CS452 page
• Bill Cowan's CS452 page
• Bill Cowan's teaching page
• Bill Cowan's home page