CS452 - Real-Time Programming - Spring 2010

Lecture 24 - Pathologies

Public Interest Announcements

1. Demos
2. How to give a demo

Pathologies

1. Deadlock

2. Livelock (Deadly Embrace)

3. Critical Races

Example

1. Two tasks, A & B, at the same priority
2. A is doing a lot of debugging IO
3. B always reserves a section of track before A, and all is fine.
4. Debugging IO is removed
5. A reserves the section before B can get it, and execution collapses.
6. Lower priority of A to the same level as C.
7. Now C executes more slowly, and D gets a different resource before C .
8. You shuffle priorities forever, eventually reverting to leave in the debugging IO.

Theory of relativity and the event horizon.

Symptoms

1. Small changes in priorities change execution unpredictably, and drastically.
2. Debugging output changes execution drastically.
3. Changes in train speeds change execution drastically.
   • Example from two terms ago

`Drastically' means chaos in both senses of the term

1. Sense one: a small change in the initial conditions produces an exponentially growing change in the system
2. Sense two: exercise for the reader.

Solutions

1. Knowing what order you expect things to happen
   • Think a lot about pre-conditions
2. Explicit synchronization
   • but you then have to know the orders in which things are permitted to occur
3. Gating is a technique of global synchronization
   • which can be provided by a detective/coordinator

4. Performance

The hardest problem to solve

• You just don't know what is possible
• Ask a question like:
  • Is my kernel code at the limit of what is possible in terms of performance?
  • We can compare the performance on message passing, etc., because two kernels are pretty much the same.
    • Compare a lot of kernels and you should be able to find a lower limit
  • Can't do the same thing for train applications

Priority

The hardest thing to get right

• Sizing stacks used to be harder, but now we have lots of memory
  • But last term, ...
• NP-hard for the human brain
• Practical method starts with all priorities the same, then adjusts
• symptoms of good priority assignment
  • The higher priority, the more likely the ready queue is to be empty
  • The shorter the run time in practice the higher the priority

Problems with priority

1. Priority inversion
2. One resource, many clients
3. Tasks try to do too much

Congestion

1. Too many tasks
   • blocked tasks don't count,
   • lowest priority tasks almost don't count
2. Layered abstraction are costly
   • e.g. Notifier -> SerialServer -> InputAccumulater -> Parser -> TrackServer
   • With the hardware you have this occurs only when you start passing around big messages.
3. Access to the train controller

Hardware

1. Turn on optimization, but be careful
   • There are places where you have done register allocation by hand
2. Turn on caches
   • locking is possible
3. Size & align calibration tables by size & alignment of cache lines
   • linker command script
   • I think that this is stretching it.

Reservations

Somebody has been doing something right for the last century. The answer is reservations.

Two Level Train Control

The two levels are completely independent of one another.

• On heavily used sections of track the lower level is done completely by hardware with no possibility (almost) of human intervention

Upper Level

1. Train asks despatcher for a route
2. Despatcher provides a route that he/she thinks to be conflict free
3. Train follows the route, reporting back to the despatcher as landmarks (sensors) are passed.
   • The despatcher gets two reports
     1. One from the hardware
     2. One from the engineer
   • It is up to the despatcher to make certain that they do not conflict

Lower Level

The lower level is encoded in the coloured lights you see along the track

• Everything is rigidly enforced by hardware
• The human enters the loop only in that the lights tell the engineer what he/she is allowed to do
  • The engineer loses his licence, FOREVER, if he/she ever goes through a red light.
• If the system ever gets something it doesn't understand, it enters a failsafe mode
  • All lights go red.

Here's how it works

1. Train asks reservation system for several blocks of track
   • Blocks usually end at switches and/or sensors
   • If you are ending blocks at switches you must know well enough how enough where the train is to be confident it is completely clear of the switch.
2. Reservation system grants the blocks if they are available
   • Grants include the condition that the train must travel so that it can come to a complete stop without leaving any reserved block.

In practice this works as follows.

• The reservation includes two distinct sets of blocks
  1. In the first set the train can travel at full speed
  2. In the second set the train must travel at reduced speed so that it can stop by the end of the second set.
• Before arriving at the end of the first set, the train requests another reservation

An Interesting Tit-bit That Doesn't Concern You

Real trains are long and sometimes occupy several blocks at once. How does the engineer know that the end of a train is clear of a switch?

• When an engineer takes over a train he/she is given printed orders that include the length of the train.
• There is a resetable counter in the locomotive that count the revolutions of the locomotives wheels
• When the locomotive passes a switch the engineer resets the counter.

Something Essential that You Must Do

Design your reservation system before coding it.

Before coding your reservation system work it out on paper and make sure that it works for all the generic cases you can think of

1. One train following another
2. Two trains on a collision course
3. There are one or more switches in the path

Return to:

• Bill Cowan's lecture notes for CS452 in s10
• Bill Cowan's Spring 2010 CS452 page
• Bill Cowan's CS452 page
• Bill Cowan's teaching page
• Bill Cowan's home page