Date post: | 28-Nov-2014 |
Category: |
Technology |
Upload: | linaro |
View: | 299 times |
Download: | 0 times |
Benchmarking Techniques
Michael Hope <[email protected]>
bzr branch lp:~michaelh1/+junk/benchmarking-techniques r6
2
Why benchmark?
3
Issues are
RelevanceAccuracy
Repeatability
4
Picking relevant benchmarks:
ProfileWorkloadFeatures
We use SPEC 2000 and EEMBC
We'd like shareable benchmarks
5
Test Platform
Build / test / benchmark via Linux / web / commodity hardware
6
Measuring
7
Realtime timers
See 'man clock_gettime'
8
<comparison of different timers>
See https://wiki.linaro.org/WorkingGroups/ToolChain/Benchmarks/TimerAccuracy
9
10
11
External influences
12
Other features to be wary of
schedulergovernor
cpuidle, power management, thermal limiting
SMPbugs, like core lockdown
NEON startup
See “Understanding the Linux Kernel” ch10:http://oreilly.com/catalog/linuxkernel/chapter/ch10.html
13
Putting things together and running
We use:timer built into the app
run five timescollect everything
post process
14
Statistics
15
16
17
Standard deviationDispersion / coefficient of variance
t-scorest-test
18
t=X̄ 1− X̄ 2
√ s12
N 1
+s22
N 2
See http://en.wikipedia.org/wiki/Welch%27s_t_test
19
Compiler Mean Std CVgcc-4.6.2 1.00 134u 134ugcc-linaro-4.6-2011.11 4.25 5035u 1178u
t = 30,300
21
Variant Mean Std CVPlain 1.00 320u 320uWith SMS 1.01 430u 426u
t = 118
22
Variant Mean Std CVPlain 1.00 320u 320uWith vectoriser 1.001 404u 404u
t = 8.27 - significant
23
Our tools
perfdifftestbetterstabulatePython
LibreOffice!
Other statistical tools like scipy.stat, R, Judge, and ministat
24http://help.libreoffice.org/Calc/Applying_AutoFilter
25
www.linaro.org / wiki.linaro.org
people.linaro.org/~michaelh/presentations