CS452 - Real-Time Programming - Fall 2011

Lecture 18 - Servers, Courier, Warehouse

Public Service Annoucements

1. Due date of kernel 4 (Friday, 28 October)
2. Saturday November 5, 11.00 to 13.00. Open house for high school students.

Anthropomorphic Programming

We all, even most programmers (!), have effective intuitions about human relations

• We use them to `understand' pets, which means attributing to them
  • goals
  • knowledge
  • capability
  • emotions
• Why not programs?
  • apply them to intertask relationships

Tasks are independent entities

• Understand them by thinking about them as if they have capabilities and goals.
• When you are developing something like the train application you are defining roles and relationships.

Servers and Attendant Tasks

Why do servers need attendant tasks?

• What happens if a server calls AwaitEvent?

1. Proprietor with a Notifier

Proprietor `owns' a service, which usually means a resource. Proprietor actually does the work.

• Think of the owner at the counter of an old-fashioned store
  • `store' means where things are stored;
  • it's in the back and only the proprietor can access it.
  • Many clients come to the front and are processed one by one.
  • Comment. The modern `store' is considered by many to be the most important innovation of the 20th century. (Yes, including the transistor, the computer, quantum mechanics, antibiotics, etc.) A whole lot of work that was previously done by store personnel is now done by the client. This is possible only because extensive codes of conduct have been internalized by clients. (That is, a large collection of new behaviour norms have been created and propagated.)
• Somebody has to sit out back waiting for the truck and bringing its load to the proprietor
• Proprietor operates at the priority of the service he provides
  • not at the priority of the client, which is what happens when the proprietor is implemented as a function call
    • good when client is a consumer of the service
    • not so good when the client is a notifier

Kernel is handling hardware in this example

Notifier

1. Running

       AwaitEvent
       Send

2. Initializing
   • Gets in contact with server, cluding synchronization
   • Turns on interrupt(s) in ICU, device
     • not IRQ
3. Can only handle one event. Package several events into one
   • In the kernel, separated by the notifier
   • Several notifiers report to one proprietor

Proprietor

1. Notifier is usually of higher priority than proprietor
   • Use early reply in the proprietor
2. When, and how, do interrupts get turned on and/or cleared?
3. Who coordinates hardware ownership?
4. We have made proprietor code
   • easy to break into parts
   • easy to extend

   Why? It's something you might end up doing during your project

2. Using a Courier

Simplest is best

Notifier

Initializing

• Gets in contact with Courier, including synchronization
• Turns on interrupt(s) in ICU, device
  • not IRQ

Work

• Receive    /from Courier
  AwaitEvent
  Reply      /to Courier

Courier

Initializing

• Initialize

  Receive( &serverTid, notifierTid );
  Send( notifierTid, ... );
  Reply( serverTid );

• Work

  FOREVER {
    Send( notifierTid, {data} );
    Send( serverTid, {req}, {data} );
  }

Transmit Proprietor Code

• Initialize

  // queues & fifos
  notifierTid = Create( notifier );
  courierTid = Create( courier );
  Send( courierTid, notifierTid, ... ); // On return courier & notifier are known to be okay
  RegisterAs( );                        //On return client requests will begin.
          

• Work

  FOREVER {
    requesterTid = Receive( request, {request-type, data} );
    switch ( request-type ) {
    case NOT_XMIT:
      enqueue( requesterTid, xmitQ )
      if ( ! empty( xmitFifo ) ) Reply( dequeue( xmitQ ), dequeue( xmitFifo ) );
      break;
    case CLIENT_XMIT:
      Reply( requesterTid, ... );
      enqueue ( xmitFifo, data );
      if ( ! empty( xmitQ ) ) Reply( dequeue( xmitQ ), dequeue( xmitFifo ) );
      break;
    default:
      ASSERT( "..." );
    }
  }

Notes

This gets you through a bottleneck where no more than two events come too fast.

Remember that all the calls provide error returns. You can/should use them for error recovery

• static error recovery: debugging
• dynamic error recovery: at run time

Another possible arrangement for task creation

• Server creates the courier
• Couier creates the notifier

Another possible arrangement for initialization

• Server Receives
• Courier sends to its parentTid
• Notifier sends to its parentTid

Distributed gating

I am showing you collections of tasks implemented together because sets of related tasks is a level of organization above the individual task.

E.g., the decision to add a courier requires revision of code within the group, but not outside it.

3. Using a Warehouse

Add a warehouse between the courier and the notifier.

Notifier Code

• Initialize

  Receive( &warhouseTid, ... );
  Reply( warhouseTid, ... );
      

• Work

  FOREVER {
    data = AwaitEvent( eventid );  // data includes event-type and volatile data
    switch( event-type ) {
    case XMT_INT:
      // test transmitter?
      Send( warehouseTid, NOT_XMIT, byte );  // byte is to be transmitted
      Write( UART, byte );
      break;
    default:  // 
      ASSERT( "This didn't happen because my kernel is bug-free." );
  }

Warehouse Code

• Initialize

  // data structures
  Receive( &courierTid, notifierTid, ... );
  Send( notifierTid, ... );
  Reply( courierTid, ... );

• Work

  FOREVER {
     Receive( &requester, {req-type, data}, );
     switch( req-type ) {
     case NOT_XMIT:
        enqueue( xmitQ, requester );
        if ( !empty( xmitFifo ) ) { Reply( extract( extract( xmitQ ), extract( xmitFifo ) };
        break;
     case COUR_XMIT:
        enqueue( courQ, requester );
        insert( xmitFifo, unpack( data ) );
        if( !empty( xmitQ ) ) Reply( extract( xmitQ ), extract( xmitFifo ) );
        if( empty( xmitFifo ) ) Reply( extract( courQ ), "Send more." )
        break;
     default:
       ASSERT( "This didn't happen because my kernel is bug-free." );
     }
  }

Transmit Courier Code

• Initialize

  Receive( &serverTid, {notifierTid, warehouseTid} ... );
  Send( warehouseTid, notifierTid, ... );
  Reply( serverTid );

• Work

  FOREVER {
    Send( warehouseTid, {req, data} );
    Send( serverTid, {req, data} );
  }

Proprietor Code

• Initialize

  // queues & fifos
  notifierTid = Create( notifier );
  warehouseTid = Create( warehouse );
  courierTid = Create( courier );
  Send( courierTid, notifierTid, ... ); // On return courier, warehouse & notifier are known to be okay
  RegisterAs( );                        // On return client requests can begin.
          

• Work

  FOREVER {
    Receive( &requesterTid, {request-type, data} );
    switch ( request-type ) {
    case COUR_XMIT:
      enqueue( xmitQ, requesterTid );
      if ( !empty( xmitfifo ) ) Reply( dequeue( xmitQ ), dequeue( xmitfifo ) );
      break;
    case CLIENT_XMIT:
      Reply( requesterTid, ... );
      enqueue( xmitfifo, data );
      if ( !empty( xmitQ ) ) Reply( dequeue( xmitQ ), dequeue( xmitfifo ) );
      break;
    default:
      ASSERT( "This didn't happen because my kernel is bug-free." );
    }
  }

Note

This structure clears up most problems when a burst of requests to the server would leave the notifier waiting in a long sendQ..

• Warehouse and proprietor share the work.
  • Server's Tid is public; Warehouse's Tid is private.
• This is far from the only way to share the work. For example,
  • The server could be guarded by a receptionist (assistant) who ensures that another client request occurs only when the previous request is complete. Then the warehouse is unnecessary.

Two issues:

1. Handles bottlenecks of all sizes.

   Give a precise and quantitative definition of `bottleneck'.

2. Server could be buffered on the other side

   Called a guard.

What this amounts to is

Server should be lean and hungry

• do as little as possible

always be receive blocked

Debugging Real-time Programs

The most common set of debugging tools used by experienced programmers is the oldest: printf, grep & stack trace.

• The power of these tools is greatly enhanced by strong conventions in code formatting.

Debugging real-time programs, at its base, is just the same as any other debugging, and just the same as empirical science.

1. Gather data.
2. Create a model that explains the data
3. Test the model
4. If the model is not correct, go to 1.
5. Remember that the model is ALWAYS provisional: data collected later may invalidate it, no matter how much data has confirmed it.

But real-time programs are harder to debug. Very few programs are entirely free of critical races, which are the worst type of bug, lurking for weeks months or years in seemingly correct code, the appearing when innocuous, unconnected changes occur.

RedBoot

The memory contents are not wiped by reset. Some of the most difficult errors can be detected only by using the contents of memory after a reset.

On some types of exceptions RedBoot will attempt to connect with gdb. In such cases it writes a bunch of gibberish on the bottom of the monitor screen. Among that gibberish is the address of the instruction that caused the exception. Using the load map generated by the linker you can find

• the function that your application was running when the exception occurred, and
• the assembly language instruction where the exception occurred.

It is usually pretty easy to figure out which line of C source was responsible for the instruction.

Stack Trace

In single-threaded programs this is often the most useful tool.

• Anything that terminates execution abnormally prints the set of active stack frames
• Minimal version
  • name of calling function
  • line number of call
• Extreme version
  • values of arguments
  • values of local variables

What is the equivalent of a stack trace in a real-time multi-tasking environment?

• Most would say it is the set of tasks, and the run state of each.
• That information is available starting at the set of task descriptors, but how do you get it in an understandable way?

Breakpoint

What does it do?

• snapshot of the system
  • This means that computation, including respose to interrupts, must stop, or it isn't a snapshot.
• provides interactive tools for examining kernel data structures, such as
  • task descriptors
  • lists and queues
  • stacks, including the program counter and local variables, of individual tasks
• restart system immediately afterwards
  • If you want to continue where processing stopped you must make certain that all state is saved when you enter Beakpoint and restored when you leave it. What about pending interrupts? You can't stop the entire universe!
  • Otherwise you can re-enter RedBoot.

How do you get it started?

• function call, which you insert in your code when compiling.
  • The easiest and fastest form to implement.
  • Having the call as part of ASSERT is common.
  • Has to exit to RedBoot. (Jump to x00.)
• system call instead of function call, which respects the kernel/user distinction.
• an exception triggered externally
  • at initialization
    1. Set up the system so that the external event will generate an exception
    2. E.g. attach a button to PDIO on the third connector, set up ICU.
  • at run-time
    1. Trigger the interrupt
    2. Switch to Breakpoint in the event handler
    3. Either exit to RedBoot,
    4. Or clean up pending interrupts and resume execution.

Breakpoint is a special case of a particular sort of tool that is very common.

• condition occurs => information is made available
• breakpoint provides the information interactively (`interactively' = `on the time scale of the user')
  • it can stop the system completely. How?
  • but it has limited ability to stop the real world
    • i.e., it hides some bugs

We need methods of getting information closer to real-time

Return to:

• Bill Cowan's Fall 2011 CS452 page
• Bill Cowan's CS452 page
• Bill Cowan's teaching page
• Bill Cowan's home page