CS457 - System Performance Evaluation - Winter 2008

1. Textbook

Lecture 2 - Importance of Performance Evaluation

Performance Evaluation

Where it Occurs

• CPU design
  1. Petterson & Hennesey: remember CS251
  2. Pipelining
  3. Cache size
• System configuration
  1. Databases for banks and insurance companies
  2. Your own computer purchasing: memory versus CPU versus GPU
• Network performance
  1. Protocols
  2. Configuration: CS employment bust in 2001
  3. Virutal reality

Why it Matters

Ultimately for one reason, and one reason only,

• the service provided to human users, who have hard limits
  1. time
  2. money

Relationship between performance evaluation and quality of service (QoS)

Its Goals

• Response time
  1. Average case
  2. Worst case
  3. Maybe even best case
• Throughput
• Financial cost
• Power consumption
• Memory footprint
  1. Small scale
  2. Large scale
• Clock speed
  1. related to power consumption

It's possible to take this too far. We restrict ourslelves to aspects of performance that are easily quantified, measured and modelled.

Topics in Performance Evaluation

1. Abstract: based on objectives
   • Abstractions are good or bad
   • But the good ness and badness is not measurable, at least not ex ante
2. Design: what to measure
   • More measurements are only sometimes better
   • recursive performance analysis

   Design also applies to simulation and to modelling.

3. Measure: good measurements produce robust data
   • `robust' means `If you did it again holding only the abstraction constant you would get the same results.'
4. Analyse: data requires objective analysis to produce results. Because
   • You are human, and
   • Humans are half-blind. (All too often we see what we want to see.)
5. Simulate: results require surface explanations
   • Surface explanations solve immediate problems
6. Model: surface explanations sometimes demand deep explanations
   • Deep explanations actually make you, and those you work with, smarter.
   • Remember that models must be tested.
   • There is no `waterfall model' in performance evaluation.

Return to:

• Bill Cowan's lecture notes for CS457 in W08
• Bill Cowan's Winter 2008 CS457 page
• Bill Cowan's CS457 page
• Bill Cowan's teaching page
• Bill Cowan's home page