Presented by

Date

Event

SFO15-303: LKP on ARMChase Qi, Milosz WasilewskiChase Qi

Wednesday 23 September 2015

SFO15

Overview- LKP Project- Enable LKP on ARM- Steps to run LKP- Run LKP in LAVA- Result Parsing in LAVA- Result comparison with LAVA- Error Margin- Next steps- Q&A

lkp-tests Projectlkp-tests git repo: https://git.kernel.org/pub/scm/linux/kernel/git/wfg/lkp-tests.git Linux Kernel Performance: https://01.org/lkp

■ Avoid kernel performance regression■ Enhance the Linux kernel performance

Contact via email: lkp@lists.01.orgMail list: https://lists.01.org/mailman/listinfo/lkp

Problems with initial implementation■ Test run setup failures

○ root file system dependencies○ package dependencies○ Some features are x86 specific○ Lack of arm/arm64 support○ Other compile, setup issues

■ Designed for high-performance system○ Powerful processor○ Fast memory architecture○ Massive storage systems

Enable LKP on ARMAuthor Q4 2014 Q1 2015 Q2 2015 Q3 2015 TotalFengguang Wu 172 192 245 8 617Wanlong Gao 49 47 78 7 181Philip Li 0 0 74 55 129Huang Ying 0 17 58 0 75Yuanhan Liu 16 12 39 3 70Zexi Li 0 4 36 5 45Yao Weiqi 0 13 27 0 40Huang, Ying 0 0 22 8 30Riku Voipio 0 0 9 0 9huangqh 0 1 5 2 8Helia Correia 3 4 0 0 7Aaron Lu 0 4 2 0 6Tian Ye 3 0 0 0 3Li, Zexi 0 2 0 0 2Wenzhong Sun 0 0 0 2 2Chase Qi 0 0 0 1 1weiqi.yao 1 0 0 0 1Total 244 296 595 91 1226

Commits per author per quarter to LKP upstream

lkp-tests StructureSteps to write a test case; Tools for test run(in blue)lkp-tests/├── allot├── bin # b. setup and test run├── cluster├── daemon├── distro├── doc├── etc├── filters├── hosts├── jobs # 4. job files in yaml format to define test parameters├── lib├── lkp-exec├── monitors├── pack├── params├── pkg # 1. package maker script├── plot├── repo├── rootfs # 2. define package dependence├── sbin # a. job split├── setup├── stats # 5. result paser├── tests # 3. main test case script├── tmp└── tools

Steps to run LKPSet PATH: cd lkp-tests

export LKP_SRC=$(pwd)export PATH=$PATH:$LKP_SRC/bin

Set-up a job: lkp install $LKP_SRC/jobs/hackbench.yaml

Split the job: lkp split-job $LKP_SRC/jobs/hackbench.yamloptions:

-o, --output PATH output path-c, --config CONFIG test kernel config-k, --kernel COMMIT test kernel commit

--no-defaults do not load the defaults headers

Run the job: lkp run ./hackbench-50%-threads-socket.yamloptions:

-o, --output RESULT_ROOT dir for storing all results-s, --set 'KEY: VALUE' add YAML hash to job

Check Results: lkp result hackbench

Run LKP in LAVATest definition yaml file https://git.linaro.org/qa/test-definitions.git/blob/HEAD:/ubuntu/lkp.yaml test script https://git.linaro.org/qa/test-definitions.git/blob/HEAD:/common/scripts/lkp.py Supported params in the test definition

# Used to specify jobs located in 'lkp-tests/jobs/'.JOB: 'hackbench'# Used to specify all 40 commit hash characters.COMMIT: 'commit'# Set LOOPS to the times to run in the same lava_test_shell.LOOPS: 1# Set MONITORS to '--no-defaults' to disable the default monitors.MONITORS: 'default'

Main steps of the test script# lkp-tests/bin/setup-local ./jobs/hackbench.yaml# lkp-tests/sbin/split-job MONITORS --kernel COMMIT --output $(pwd)/hackbench ./jobs/hackbench.yaml# lkp-tests/bin/run-local --set 'compiler: CompilerVersion' $(pwd)/hackbench_sub_test.yamlParse test results.

Result Parsing in LAVAstructure of result_root: /result/ \hackbench/ \ # benchmark name50%-process-socket/ \ # sub-testcase namelinaro-developer/ \ # hostnameubuntu/ \ # rootfsdefconfig/ \ # configgcc-4.9/ \ # compiler version1efdb5f0a9243ca8f3460a5ce1b407b06a021f02/ \ # commit hash0/ # test run

hostname, rootfs, config, compiler version and commit are changeable with run-local or split-job options

Tree view of the first test run

Results in LAVA -single runDecode job.json, collect results with lava-test-case. In Results Bundle, it looks like:

Attach raw data for post comparison between commits or test runs.lkp-result-hackbench.tar.xz

Results in LAVA -multiple runs{ "command": "lava_test_shell", "parameters": { "testdef_repos": [ { "git-repo": "url/test-definitions", "parameters": { "COMMIT": "commit1", "JOB": "hackbench", "LOOPS": "13" }, "testdef": "ubuntu/lkp.yaml" }], "timeout": 1080000 }},

LKP comparison with LAVAManual comparison

■ Download test results from LAVA■ Compare a specific field with lkp compare tool

# lkp-tests/sbin/compare -d commit -f hackbench.throughput \commit1_result_root commit2_result_root

7d9071a095023cd1 1efdb5f0a9243ca8f3460a5ce1 testbox/testcase/testparams---------------- -------------------------- --------------------------- %stddev %change %stddev \ | \ 21801 -2.6% 21236 hackbench/50%-process-pipe/linaro-developer 21801 -2.6% 21236 GEO-MEAN hackbench.throughput

Auto LKP comparison toolshttps://git.linaro.org/people/chase.qi/lkp-comparison.git Tools: lavadownload.py resultcompare.pyLAVA result download: lavadownload.py -n lkp-result -l https://validation.linaro.org/RPC2/ -i lava_job_idCompare scores produced by benchmark: python resultcompare.py --job1 jobID1 --job2 jobID2Compare a specific field: python resultcompare.py --job1 jobID1 --job2 jobID2 -f time.system_time

LKP Comparison Outputv4.2-9700-g7d9071a09502 VS v4.2-rc6-176-g1efdb5f0a9247d9071a0/1efdb5f0 measurement-testcase-29.5% hackbench.throughput-1600%-process-pipe+4.3% hackbench.throughput-1600%-process-socket-25.7% hackbench.throughput-1600%-threads-pipe+8.2% hackbench.throughput-1600%-threads-socket-2.6% hackbench.throughput-50%-process-pipe-27.8% hackbench.throughput-50%-process-socket-3.9% hackbench.throughput-50%-threads-pipe-24.2% hackbench.throughput-50%-threads-socket-30.1% netpipe.bigger_5M_Mbps.avg-tcp+4.0% netpipe.less_8K_usec.avg-tcp+0.7% pbzip2.throughput-100%-500K+0.4% pbzip2.throughput-100%-900K+0.6% pbzip2.throughput-25%-500K-0.5% pbzip2.throughput-25%-900K+18.0% perf-bench-sched-pipe.ops_per_sec-10000000ops-process+23.3% perf-bench-sched-pipe.ops_per_sec-10000000ops-threads+0.1% pigz.throughput-100%-128K+0.1% pigz.throughput-100%-512K-0.2% pigz.throughput-25%-128K+2.7% pigz.throughput-25%-512K-0.1% pixz.throughput-100%+0.3% pixz.throughput-25%+0.1% plzip.throughput-100%-0.1% plzip.throughput-25%-0.3% pxz.throughput-100%-0.2% pxz.throughput-25%+20.4% unixbench.score-context1+0.1% unixbench.score-dhry2reg-1.8% unixbench.score-execl-10.1% unixbench.score-fsbuffer-9.2% unixbench.score-fsdisk-8.7% unixbench.score-fstime-16.4% unixbench.score-pipe+0.1% unixbench.score-shell1+9.8% unixbench.score-shell8+0.1% unixbench.score-spawn+2.6% unixbench.score-syscall+0.0% unixbench.score-whetstone-double

Error Marginv4.2-rc6-176-g1efdb5f0a924, two test runs on the same board.1efdb5f0/1efdb5f0 measurement-testcase+0.3% hackbench.throughput-1600%-process-pipe-0.3% hackbench.throughput-1600%-process-socket+0.5% hackbench.throughput-1600%-threads-pipe+0.0% hackbench.throughput-1600%-threads-socket-0.4% hackbench.throughput-50%-process-pipe-0.5% hackbench.throughput-50%-process-socket+0.0% hackbench.throughput-50%-threads-pipe+0.1% hackbench.throughput-50%-threads-socket-0.0% netpipe.bigger_5M_Mbps.avg-tcp-0.1% netpipe.less_8K_usec.avg-tcp-0.0% pbzip2.throughput-100%-500K+0.9% pbzip2.throughput-100%-900K+0.4% pbzip2.throughput-25%-500K+1.2% pbzip2.throughput-25%-900K-1.5% perf-bench-sched-pipe.ops_per_sec-10000000ops-process-0.2% perf-bench-sched-pipe.ops_per_sec-10000000ops-threads-0.1% pigz.throughput-100%-128K+0.1% pigz.throughput-100%-512K+0.3% pigz.throughput-25%-128K+1.8% pigz.throughput-25%-512K-0.1% pixz.throughput-100%+0.5% pixz.throughput-25%-0.0% plzip.throughput-100%-0.0% plzip.throughput-25%+0.0% pxz.throughput-100%+0.2% pxz.throughput-25%+0.3% unixbench.score-context1+0.0% unixbench.score-dhry2reg-1.1% unixbench.score-execl+0.5% unixbench.score-fsbuffer+0.3% unixbench.score-fsdisk+0.8% unixbench.score-fstime+0.4% unixbench.score-pipe+0.9% unixbench.score-shell1+0.3% unixbench.score-shell8-0.1% unixbench.score-spawn+0.1% unixbench.score-syscall-0.1% unixbench.score-whetstone-double

1efdb5f0/1efdb5f0 measurement-testcase+0.0% dbench.clients-100%-15.8% dbench.max_latency-100%+0.0% dbench.procs-100%-0.8% dbench.throughput-MB/sec-100%

Next steps■ Store results to database (kernelci.org)

■ Polish comparison tools

■ Add benchmarks to kernel (LSK) test plan

■ Investigate issues in lkp-tests

Q&A

Any questions?