CS452 - Real-Time Programming - Winter 2015
Lecture 28 - Pathologies III
Public Service Annoucements
-
Exam: 16.00 14 April to 18.30 15 April
-
Milestone 2: 25 March, 2015
-
Progress in the state of the train sets?
-
Final demos. 6,7 April, 2015. Choice of format?
-
Menu of possible lectures.
-
-
-
Implementing a server in Go.
-
Cyclic computation.
-
Admission control.
-
What happens when a computer powers on.
-
Ada rendez-vous (aka type-safe message passing).
-
Remote delay.
-
-
Course evaluation.
Pathologies
1. Deadlock
2. Livelock (Deadly Embrace)
3. Critical Races
Solutions
- Explicit synchronization
-
but you then have to know the orders in which things are
permitted to occur
-
e.g. Try listing all the orders in which events can occur
in your system
-
and then notice that just arriving in the same order
is often not enough
-
How long is the list?
-
Gating is a technique of global synchronization
-
which can be provided by a detective/coordinator
-
Think about execution order when you are programming
-
Which tasks are on the ready queues?
-
What is the sequence of active tasks?
-
Look at what happens when only the idle task is running and
an interrupt (clock, terminal UART, train controller UART)
occurs.
-
For the UARTs, consider receive, transmit and modem control.
Changes in performance of one task with respect to another often
give rise to critical races.
4. Performance
The hardest problem to solve
-
You just don't know what is possible
-
Ask a question like:
-
Is my kernel code at the limit of what is possible in terms
of performance?
-
We can compare the performance on message passing, etc.,
because two kernels are pretty much the same.
-
Compare a lot of kernels and you should be able to find
a lower limit
-
Can't do the same thing for train applications
In practice, how do you know you have performance problems? Here
are a few sources of performance problems
Priority
The hardest thing to get right
-
NP-hard for the human brain
-
Practical method starts with a few priorities, then adjusts
-
symptoms of good priority assignment
-
The higher the priority, the more likely the ready queue
is to be empty
-
In practice, the higher the priority, the smaller the
share of CPU usage
-
The standard usage pattern should be highly correlated with
priority
- Idle task
- Notifier
- Server
- Cascade to servers, couriers, workers and clients
- Server(s)
- Notifier(s)
- Idle task
-
What happens when two of these patterns occur at once?
-
or three or four ...
-
You should be able to estimate how often near-simultaneous
external events occur.
Problems with priority
Priority changes the order of execution of your tasks, sometimes
exposing a critical race. When this happens remember that the
critical race was there all along, and will pop up later just
when it's most incomvenience. Priority is useful for overall
sensible design; it's a poor tool for trouble-shooting and
bug-fixing.
-
Priority inversion occurs when a high priority task blocks
on a lower priority one. It can occur, for example, when a
high priority server cannot proceed until it acquires information
from a lower priority one. The higher priority server is
then effectively at the lower priority.
-
One resource, many clients. Our send queues are first-in,
first-out. If the send queues are long a high priority client
(a notifier, for example) waits in the send queue behind lower
priority clients. (Of course, a long send queue is a sign of
trouble ahead.)
-
Tasks try to do too much. It is tempting to add functionality
to a task rather than creating a new task. Each time you do
this you increase the longest path from entering the task
until leaving it. In fact, you can put your whole project
into a single task -- it's happened before. Such a project
does not get very high marks for design!
Congestion
-
Too many tasks ready at the same time.
-
blocked tasks don't count,
-
lowest priority tasks (e.g. idle task) almost don't count
-
Layered abstractions are costly
Limited resources
- Bandwidth to the train controller
-
Can you use it more effectively?
-
Requests to poll the sensors get backed up in the serial
server, or whoever provides output buffering.
-
Bandwidth to the terminal
-
Can you use it more effectively?
-
Too much terminal output interferes with train controller
communication
- Don't redraw the entire screen
- Formatting and parsing are ridiculously expensive
-
Any server with clients, workers, notifiers or couriers in its
sendQ
-
Can you find mixes of high and low priority CPU usage?
Hardware
-
Turn on optimization, but be careful
-
There are places where you have done register allocation
by hand
-
Turn on caches
Size & align calibration tables by size & alignment of
cache lines
I think that this is stretching it.
Symptoms of performance problems
When you have a performance problem the hard part is recognizing
that it is a performance problem because, when performance is bad
the project crashes for a concrete reason, such as trains colliding.
Looking at system performance is usually the last thing that is
done.
Every one of the suggestions below has a problem: you must know
what you expect to find in order to recognize that you are seeing
something anomalous. Use your brain to estimate.
-
Internal queues and buffers reach their capacity. Students
usually just increase the memory allocated for any one that
fills up, but there is almost always a deeper problem.
-
Delay gets occasional negative arguments.
-
Arguments to Delay decrasing.
-
More tasks on ready queues.
-
Occasional misses when switching turn-outs.
-
Idle task running less than 90% of the time. You can be
successful down to 80%, but with a hit to performance. 75%
needs to be fixed. There is almost always some code that is
doing something stupid.
-
Any task monotonically increasing its memory use.
-
Any task monotonically increasing its use of CPU.
-
Sensor-reading falls behind.
Return to: