1

PERF EVAL (CONT’D)PERF EVAL (CONT’D)There are many other “tools of the

trade” used in performance evaluationOnly a few will be mentioned here:

– queueing theory– verification and validation– statistical analysis– multi-variate analysis– presentation of results

2

Queueing TheoryQueueing Theory

A mathematical technique that specializes in the analysis of queues (e.g., customer arrivals at a bank, jobs arriving at CPU, I/O requests arriving at a disk subsystem)

General diagram:CustomerArrivals Departures

Buffer Server

3

Queueing Theory (cont’d)Queueing Theory (cont’d)

The queueing system is characterized by:– Arrival process (M, G)– Service time (M, D, G)– Number of servers (1 to infinity)– Number of buffers (infinite or finite)

Example notation: M/M/1, M/D/1Example notation: M/M/ , M/G/1/k8

4

Queueing Theory (cont’d)Queueing Theory (cont’d)

There are well-known mathematical results for the mean waiting time and the number of customers in the system for several simple queueing models

E.g., M/M/1, M/D/1, M/G/1Example: M/M/1

– q = rho/ (1 - rho) where rho = lambda/mu < 1

5

Queueing Theory (cont’d)Queueing Theory (cont’d)These simple models can be cascaded in

series and in parallel to create arbitrarily large complicated queueing network models

Two main types:– closed queueing network model (finite pop.)– open queueing network model (infinite pop.)

Software packages exist for solving these types of models to determine steady-state performance (e.g., delay, throughput, util.)

6

Verification and ValidationVerification and ValidationAn important step in any modeling work

(simulation or analytical) is convincing others that the model is “correct”

Verification: develop simple test cases with known inputs; compare to expected outputs

Validation: the “reality check” to see if model predictions agree with real world

Sanity checks (e.g., Little’s Law: N = T)This V&V process is often overlooked!!!

7

Statistical AnalysisStatistical Analysis “Math and stats can be your friends!!!” CWThere are lots of “standard” techniques from

mathematics, probability, and statistics that are of immense value in performance work:– confidence intervals, null hypotheses, F-tests,

T-tests, linear regression, least-squares fit, maximum likelihood estimation, correlation, time series analysis, transforms, Q-Q, EM...

– working knowledge of commonly-observed statistical distributions

8

Multi-Variate AnalysisMulti-Variate Analysis For in-depth and really messy data analysis,

there are multi-variate techniques that can be immensely helpful

In many cases, good data visualization tools will tell you a lot (e.g., plotting graphs), but in other cases you might try things like:– multi-variate regression: find out which

parameters are relevant or not for curve fitting– ANOVA: analysis of variance can show the

parameters with greatest impact on results

9

Presentation of ResultsPresentation of ResultsGraphs and tables are the two most common

ways of illustrating and/or summarizing data– graphs can show you the trends– tables provide the details

There are good ways and bad ways to do each of these

Again, it is a bit of an “art”, but there are lots of good tips and guidelines as well

10

Table TipsTable TipsDecide if a table is really needed; if so, should it

be part of main paper, or just an appendix?Choose formatting software with which you are

familiar; easy to import data, export tablesTable caption goes at the topClearly delineate rows and columns (lines)Logically organize rows and columnsReport results to several significant digitsBe consistent in formatting wherever possible

11

Graphing TipsGraphing TipsChoose a good software package, preferably

one with which you are familiar, and one for which it is easy to import data, export graphs

Title at top; caption below (informative)Labels on each axis, including unitsLogical step sizes along axes (10’s, 100’s…)Make sure choice of scale is clear for each

axis (linear, log-linear, log-log)Graph should start from origin (zero) unless

there is a compelling reason not to do so

12

Graphing Tips (cont’d)Graphing Tips (cont’d)Make judicious choice of type of plot

– scatter plot, line graph, bar chart, histogramMake judicious choice of line types

– solid, dashed, dotted, lines and points, colours If multiple lines on a plot, then use a key,

which should be well-placed and informative If graph is “well-behaved”, then organize the

key to match the lines on the graph (try it!)Be consistent from one graph to the next

wherever possible (size, scale, key, colours)