CS457 - System Performance Evaluation - Winter 2010

Public Service Announcements

Textbook sections in list of lecture notes
Assignment deadlines

Lecture 3 - Abstraction for Performance Evaluation

Performance Evaluation

Example of Abstraction: Amazon.com (not really!)

Diagram

Topics in Performance Evaluation

Abstract: based on objectives
- Abstractions are good or bad
- But the goodness and badness is not measurable, at least not ex ante
- A good abstraction just keeps on giving
Design: what to measure
- More measurements are only sometimes better
  - measuring bricks
  - measuring atoms
- What to measure comes back to what the system is designed to accomplish
Design also applies to simulation and to modelling.
Measure: good measurements produce robust data
- `robust' means `If you did it again holding only the abstraction constant you would get the same results.'
- types of measurement
  - in the wild
  - in an abtracted environment
Analyse: data requires objective analysis to produce results. Because
- You are human, and
- Humans are half-blind. (All too often we see what we want to see.)
- Analysis transforms data into useful facts.
Simulate: results require surface explanations
- Vary system properties and observe responses.
- On completion you have a bunch of correlations,
  - which you would like to turn into causes.
- Surface explanations solve immediate problems,
  - but you shold want to limit and generalize
Model: surface explanations sometimes demand deep explanations
- Deep explanations actually make you, and those you work with, smarter.
- Remember that models must be tested.
- There is no `waterfall model' in performance evaluation

Setting up an Evaluation

Two example systems

RPC versus Remote pipes
- Simple, but easy to understand
E-commerce server
- Realistic, but easy to get swamped by the details


Stage	RPC	E-commerce
1. Define Goals, System goals first minimize system size	Write a paper comparing RPC to RPs System should include overhead. System should not include computation	Ensure that a system architecture is scalable. Find the right size for an expected workload
2. List services (from 1) possible outcomes for each service including failures	Data transfer (why?) large transfers small transfers failures omitted (why?)	Client requests too many to list make generic categories Bank Bricks & mortar
3. List metrics (from 2) speed accuracy utilization & availability	Speed is everything. (The measurements are performed on an unloaded system with errors discarded.) throughput average response time maximum request rate	speed average response time maximum response time accuracy incorrect transactions inconsistencies
4. List parameters (from 2, 3) that affect performance as defined by metrics system parameters h/w & s/w resources resource characteristics system configuration workload parameters determined by client objectives	System client speed network speed server speed Workload request rate request size	System distribution of resources across subsystems organization of subsystems internal failures Workload rate of requests number of simultaneous clients response of bank response of bricks & mortar
5. Select factors to study (from 1,4) determined by study goals What about the other parameters?	System network delay Workload request size	System distribution of resources down-time failures Workload rate of requests of each type number of clients response of bank response of bricks & mortar
6(7). Select workload (from 2) measurement: scripted, usually with randomness, could be a trace or log simulation: scripted or trace model: probability distribution

Metrics

Sequence of events for a single request

System receives request, ai
System starts executing the request - reaction time
System starts responding to request - response time 1
System completes response to request, di - response time 2, ri = di - ai

Types of metrics

Speed
- response time
  - best case
  - worst case
  - average case
  - variability
- throughput
  - mean response time = L/n = 1/throughput
Accuracy
- error probability
Availability
- uptime/downtime
- utilization

Always remember Goodhart's law.

Return to: