Benchmarking Techniques

Michael Hope <michael.hope@linaro.org>

bzr branch lp:~michaelh1/+junk/benchmarking-techniques r6

2

Why benchmark?

3

Issues are

RelevanceAccuracy

Repeatability

4

Picking relevant benchmarks:

ProfileWorkloadFeatures

We use SPEC 2000 and EEMBC

We'd like shareable benchmarks

5

Test Platform

Build / test / benchmark via Linux / web / commodity hardware

6

Measuring

7

Realtime timers

See 'man clock_gettime'

8

<comparison of different timers>

See https://wiki.linaro.org/WorkingGroups/ToolChain/Benchmarks/TimerAccuracy

9

10

11

External influences

12

Other features to be wary of

schedulergovernor

cpuidle, power management, thermal limiting

SMPbugs, like core lockdown

NEON startup

See “Understanding the Linux Kernel” ch10:http://oreilly.com/catalog/linuxkernel/chapter/ch10.html

13

Putting things together and running

We use:timer built into the app

run five timescollect everything

post process

14

Statistics

15

16

17

Standard deviationDispersion / coefficient of variance

t-scorest-test

18

t=X̄ 1− X̄ 2

√ s12

N 1

+s22

N 2

See http://en.wikipedia.org/wiki/Welch%27s_t_test

19

Compiler Mean Std CVgcc-4.6.2 1.00 134u 134ugcc-linaro-4.6-2011.11 4.25 5035u 1178u

t = 30,300

21

Variant Mean Std CVPlain 1.00 320u 320uWith SMS 1.01 430u 426u

t = 118

22

Variant Mean Std CVPlain 1.00 320u 320uWith vectoriser 1.001 404u 404u

t = 8.27 - significant

23

Our tools

perfdifftestbetterstabulatePython

LibreOffice!

Other statistical tools like scipy.stat, R, Judge, and ministat

24http://help.libreoffice.org/Calc/Applying_AutoFilter

25

www.linaro.org / wiki.linaro.org

people.linaro.org/~michaelh/presentations