CS457 - System Performance Evaluation - Winter 2010

Public Service Announcements

1. This lecture and the previous one are based on this example.
2. Assignment 1 deadline moved to 20 January.

Lecture 5 - Performance Metrics

The Textbook's Categories

1. Goal(s)

• Provide merchandise information and process sales.. Assume
  • Separate data system(s) for
    • fulfilment
    • payment processing
    • catalogue maintenance
    • etc.
  • Undifferentiated user requests
• Zero down time
• Easily scalable
• Vary response user to user
• Accommodate dynamic catalogue

An Example System

Description

2. Services

External requests

Requests to secondary servers

• user requests
  • browse
  • add/remove from cart
  • purchase
  all these requests have exogenous arrival rates
• primary server assignment

Requests to primary server, all the above plus

• external requests, such as catalogue updates

Requests to load balancer

• user requests
  • login to system
• system requests
  • redirect user
  • (I am assuming that these requests, which are the result of load imbalance or server failure, are exogenous.)

Internal requests

Requests to secondary servers

• as home servers
  • catalogue updates from primary
  • assignment and re-assignment from load balancer
• as server set servers
  • shopping cart updates from primary

Requests to primary servers, all the above plus

• shopping cart updates from home servers
• home and server set updates from load balancer

3. Metrics

Possible metrics

• Load balancing
  • response time to user requests
    • mean
    • variability, such as standard deviation
    • percentiles
    • tails
    The above list applies to all response times.
  • response time for home server re-assignment
  • system utilization
• Consistency
  • response time to catalogue updates
• Uptime
  • response time for primary server re-assignment
  • availability: uptime / ( uptime + downtime )

Always remember Goodhart's law.

Response Time, Throughput and Utilization

For request i:

• arrival time, a(i)
• departure time, d(i)
• response time, r(i) = d(i) - a(i)

Draw a time line

• number in system at the beginning of the time interval, n(0).
• number in system at the end of the time interval, n(L).
• number arriving between 0 and L, a(0,L)
• number departing between 0 and L, d(0,L)
• identity: n(0) + a(0,L) = n(L) + d(0,L)
  • Let L -> infinity, either
    • n(L) -> infinity & system is unstable (r(i) -> infinity)
    or
    • n(L) stays finite & system is stable
    • a(0,L) - d(0,L) -> 0

Throughput is the number of requests processed per unit time. For stable systems,

• throughput = a(0,L) / L = average arrival rate = 1 / \lambda
• mean response time = (1/a(0,L)) sum_i r(i)
• total work done = total busy time = sum_i s(i) ~ S * a(0,L)
  • s(i) is the amount of time the server takes to do request i on an unloaded system
• system capacity = L
• average system utilization: U = (S * a(0,L)) / L = S * throughput

4. Parameters

System Parameters

Things that could change in the system and that would change measured values:

1. service capacity
   • ability of one server to process requests, service time per request
     • on an unloaded system
   • number of servers
2. network delays
3. resource management strategy
4. scheduling algorithm
   • transmission frequency of system state to secondary servers
   • transmission frequency of user state to server set
5. load balancing algorithm
6. time between failure for servers
   • downtime per failure for servers

Workload Parameters

Things that might change in the workload and that would change measured values:

1. arrival rate of requests
   • possibly broken down by request type
2. service requirement per request
   • Note that service requirement = service time * service capacity (See system parameters.)

5. Factors

Things that we will actually study

• variables determined by goals, metrics and parameters
• independent variables (we choose their values)
  • from parameters
• dependent variables have measured values
  • from metrics

Metrics

1. Response time to requests
   • average
   • 99th percentile
2. Catalogue freshness
   • percentage of time a user's catalogue view is up to date
3. System utilization
4. System downtime

System factors

1. service capacity per server
2. number of servers
3. time between failures for servers
4. downtime per failure for servers

Workload factors

1. Arrival rate of user requests
2. Arrival rate of catalogue requests

Return to:

• Bill Cowan's lecture notes for CS457 in W10
• Bill Cowan's Winter 2010 CS457 page
• Bill Cowan's CS457 page
• Bill Cowan's teaching page
• Bill Cowan's home page