CS457 - System Performance Evaluation - Winter 2008

Questions and Comments

1. Office hours for A1.

Lecture 10

Reprise of Little's law

Response Time

1. R >= S, equality when U < 1.
2. R = N/X - Z = (NS)/U - Z >= NS - Z, equality when U = 1

Throughput

1. U = XS <= 1, which implies X = U/S <= 1/S.
2. X(R + Z) = N, which implies X = N / (R + Z) >= N / (S + Z).

Now draw some pictures to see what's actually going on.

Measuring Performance

Think of the following scenario

1. The boss calls: The performance our our system is lousy. What do you respond? Give me the details.
   • Now you know the details. What do you do?
2. Look at the logs
   • What is a log? There are usually system logs and application logs
   • 2/3 of performance problems are understood at this point.
3. Turn on one or more logs that were turned off
   • If all the logs are running all the time the system is overwhelmed with data
   • 2/3 of the remaining performance problems are understood at this point.
4. You now know whether the problem is where the boss thinks it is: i.e., inside a particular application.
   • Turn on the monitoring (debugging) code in the application, and look at the logs it produces
   • If the application is well-designed 2/3 of the remaining problems are understood at this point
5. All this time you have been examining the application while it is `in service', because that's where the problem was noticed. Now you are going to start doing things that you can't try while the application is in service.
   • Prepare a clean system with the same hardware resources, operating system, and only the application.
   • Use one of your logs to prepare a set of input requests.
   • Feed the set of requests to the system using a load generator, and look for the problem in the output logs
   • What do you if has disappeared?
6. Add instrumentation to the application by altering its source code.
   • If the problem is in the application this will allow you to see it.
   • But the problem may be in the interaction with the operating system.
7. Find a more-instrumented version of the operating system
   • Run the application under it. Look at the logs.
   • But the problem may be in the way hardware responds to operating system requests.
8. Find a hardware monitor and put it on the system in the critical place
   • Run the application. Compare the logs of the hardware monitor to those of the operating system

If you have identified the problem in steps 6-8 you will almost certainly want to test performance on `in service' and synthetic workloads during testing.

Once you think the problem is solved you work your way back up this lists until you get to 1, which changes to `You call the boss.'

Traces

What I have called a log is usually called a trace, which is

• a time-ordered sequence of events from a real system, as opposed to a simulation
• useful for assessing performance and generating workloads

Monitors

The code inserted to generate traces.

• often include capability to analyse and display results

Measurement

What other reasons might you have for wanting to measure performance?

• Identify heavily used components
  • hardware or software modules
• Locate bottlenecks
• Characterize workloads
• Validation of simulations or models

What do we try to measure

1. Arrival and departure times
2. Processor activity, which can give service times
3. Other resource activity
4. Failures

Levels of Measurement

1. Application code
2. Profiling tools
3. Operating system
4. Kernel
5. Hardware

The Big Question

How much does the introduction of monitoring software influence the performance of the application.

1. Event-driven monitors
2. Sampling monitors

Return to:

• Bill Cowan's lecture notes for CS457 in W08
• Bill Cowan's Winter 2008 CS457 page
• Bill Cowan's CS457 page
• Bill Cowan's teaching page
• Bill Cowan's home page