Personal tools
You are here: Home Events ICSA Colloquium Talk

ICSA Colloquium Talk

— filed under:

Statistical Performance Comparisons of Computers - Dr Tianshi Chen, Institute of Computing Technology, Chinese Academy of Sciences

  • Colloquium Series
When Jul 30, 2013
from 02:00 PM to 03:00 PM
Where IF 4.31/4.33
Add event to calendar vCal

Dr Tianshi Chen
Institute of Computing Technology,
Chinese Academy of Sciences

Statistical Performance Comparisons of Computers

As a fundamental task in computer system research, performance comparison has been continuously hampered by the variability of computer performance. In traditional performance comparisons, the impact of performance variability is usually ignored (i.e., the means of performance observations are compared regardless of the variability), or in the few cases directly addressed with t-statistics without checking the number and normality of performance observations. In this study, we formulate performance comparison as a statistical task, and empirically illustrate why and how common practices can lead to incorrect comparisons.

We suggest a non-parametric Hierarchical Performance Testing (HPT) framework for performance comparison, which is significantly more practical than standard t-statistics because it does not require to collect a large number of performance observations in order to achieve a normal distribution of the sample mean. This HPT framework has beenimplemented as an open-source software, and integrated in the PARSEC 3.0 benchmark suite.

Dr. Tianshi Chen  is an assistant professor at Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China. He received the Ph.D. degree in computer science from University of Science and Technology of China in 2010. His research interests include performance evaluation and modeling of processors
(using statistical/learning techniques), hardware implementation of novel learning systems, and computational intelligence. Dr. Chen is the recipient of China Computer Federation (CCF) Distinguished Doctoral Dissertation Award (2011) for his Ph.D. work on computational complexity analysis of evolutionary algorithms.

Document Actions