Cyclic Tests Unleashed:Large-Scale RT Analysis with
JitterdebuggerOpen Source Summit Japan 2019
Prof. Dr. Wolfgang MauererTechnical University of Applied Sciences RegensburgSiemens AG, Corporate Research
Daniel WagnerSUSE Linux GmbH(Work done while at Siemens AG)
W. Mauerer and D. Wagner 17. July 2019 1 / 31
About: WMI Siemens Corporate Technology: Corporate Competence Centre Embedded Linux
I Civil Infrastructure PlatformI Technical University of Applied Science Regensburg
I Theoretical Computer ScienceI Head of Digitalisation Laboratory
About: DWI SUSE Linux GmbHI Primary Author of JitterdebugerI Stable-RT Maintainer
W. Mauerer and D. Wagner 17. July 2019 2 / 31
Introduction & Overview
About: WMI Siemens Corporate Technology: Corporate Competence Centre Embedded Linux
I Civil Infrastructure PlatformI Technical University of Applied Science Regensburg
I Theoretical Computer ScienceI Head of Digitalisation Laboratory
About: DW
W. Mauerer and D. Wagner 17. July 2019 2 / 31
Introduction & Overview
1. Measuring Real-Time Systems2. Jitterdebugger: Cyclictest for Dummies
2.1 Measuring2.2 Archiving
3. Analysis Examples3.1 Comparing Distributions3.2 Time-Resolved Analysis3.3 Estimating Upper Bounds/WCET
W. Mauerer and D. Wagner 17. July 2019 3 / 31
1. Measuring Real-Time Systems
Outline
Why Measure RT Systems?I CPUs & systems: Effectively indeterministic these daysI Development/Debugging vs. Deployment Guarantees
Debugging/DevelopmentI Functional Correctness
I Locking etc.I Odd use of system functionalitiesI Functional correctness
I Eliminate large outliersI Triggers and Tracing
Verification/DeploymentI Characterising System Behaviour in
fieldI Reference Distributions (regression
testing)I Satisfy Certification Criteria
W. Mauerer and D. Wagner 17. July 2019 4 / 31
1. Measuring Real-Time Systems
Measuring Real-Time Systems I
Why Measure RT Systems?I CPUs & systems: Effectively indeterministic these daysI Development/Debugging vs. Deployment Guarantees
Debugging/DevelopmentI Functional Correctness
I Locking etc.I Odd use of system functionalitiesI Functional correctness
I Eliminate large outliersI Triggers and Tracing
Verification/DeploymentI Characterising System Behaviour in
fieldI Reference Distributions (regression
testing)I Satisfy Certification Criteria
W. Mauerer and D. Wagner 17. July 2019 4 / 31
1. Measuring Real-Time Systems
Measuring Real-Time Systems I
Why Measure RT Systems?I CPUs & systems: Effectively indeterministic these daysI Development/Debugging vs. Deployment Guarantees
Debugging/DevelopmentI Functional Correctness
I Locking etc.I Odd use of system functionalitiesI Functional correctness
I Eliminate large outliersI Triggers and Tracing
Verification/DeploymentI Characterising System Behaviour in
fieldI Reference Distributions (regression
testing)I Satisfy Certification Criteria
W. Mauerer and D. Wagner 17. July 2019 4 / 31
1. Measuring Real-Time Systems
Measuring Real-Time Systems I
1. Measuring Real-Time Systems2. Jitterdebugger: Cyclictest for Dummies
2.1 Measuring2.2 Archiving
3. Analysis Examples3.1 Comparing Distributions3.2 Time-Resolved Analysis3.3 Estimating Upper Bounds/WCET
W. Mauerer and D. Wagner 17. July 2019 5 / 31
2. Jitterdebugger: Cyclictest for Dummies
Outline
Key PointsI Few tuneable knobs (deliberately)I PostprocessingI Control load/stress generationI Mass deployments (network)
W. Mauerer and D. Wagner 17. July 2019 6 / 31
2. Jitterdebugger: Cyclictest for Dummies
Jitterdebugger: Cyclictest for Dummies I
Do one thing, and do that wellJitterdebugger→ Jittersamples→ Statistical Software
OutputCPUID;Timestamp;Latency
W. Mauerer and D. Wagner 17. July 2019 7 / 31
2.1 Measuring
Jitterdebugger II
Do two things, and do/dispatch that wellJitterdebugger + Stress→ Jittersamples→ Statistical Software
OutputCPUID;Timestamp;Latency
W. Mauerer and D. Wagner 17. July 2019 7 / 31
2.1 Measuring
Jitterdebugger II
Do two things, and do/dispatch that wellJitterdebugger + Stress→ Archive
OutputCPUID;Timestamp;Latency
W. Mauerer and D. Wagner 17. July 2019 7 / 31
2.1 Measuring
Jitterdebugger II
Do two things, and do/dispatch that wellJitterdebugger + Stress→ Send to Host→ Jittersamples→ StatisticalSoftware
OutputCPUID;Timestamp;Latency
W. Mauerer and D. Wagner 17. July 2019 7 / 31
2.1 Measuring
Jitterdebugger II
Data HandlingI Reproducible & systematic approachI Regression & comparisonI CertificationI Decouple measurement from statistical
methods
Time ResolutionI Advanced statistical analysisI Machine Learning! AI!I Improve worst-case latency analysis
W. Mauerer and D. Wagner 17. July 2019 8 / 31
2.1 Measuring
Jitterdebugger III: Main Advantages
Measuring ProperlyI Reproducibility – can others reproduce/interpret results?I Sufficient Duration – when is certainty achieved?I Traceability – do we understand what’s going on?
First rule of data analysisI 80% of effort: cleaning up dataI 20% of effort: analysis
W. Mauerer and D. Wagner 17. July 2019 9 / 31
2.2 Archiving
Archiving I
Reproducibility & Cleanun: Tidy Data1. Each variable forms a column2. Each observation forms a row3. Each type of observational unit forms a table
I Separate file (CSV)I Separate Entity (HDF5)
W. Mauerer and D. Wagner 17. July 2019 10 / 31
2.2 Archiving
Archiving II
Jitersamples FormatsI CSV
I You may have heard that beforeI UniversalI Building structures: FS level
I HDF5I Hierarchical Data FormatI 1987: AEHOOI Comprehensive support in analysis software
(R, Octave, Python, Mathematica, Julia, . . . )I Embedded structures in single file
Image Source: www.desy.de/web/mosaic/hdf-browsing.html
W. Mauerer and D. Wagner 17. July 2019 11 / 31
2.2 Archiving
Archiving III
Jitersamples FormatsI CSV
I You may have heard that beforeI UniversalI Building structures: FS level
I HDF5I Hierarchical Data FormatI 1987: AEHOOI Comprehensive support in analysis software
(R, Octave, Python, Mathematica, Julia, . . . )I Embedded structures in single file
Image Source: www.desy.de/web/mosaic/hdf-browsing.html
W. Mauerer and D. Wagner 17. July 2019 11 / 31
2.2 Archiving
Archiving III
Jitersamples FormatsI CSV
I You may have heard that beforeI UniversalI Building structures: FS level
I HDF5I Hierarchical Data FormatI 1987: AEHOOI Comprehensive support in analysis software
(R, Octave, Python, Mathematica, Julia, . . . )I Embedded structures in single file
Organisation: Best PractiseI Identical filename for each
measurementI Active parameters: directoriesI Derived parameters: file(s)I Reproducibility: Keep microcode
binaries, non-upstream patches etc. aspart of measurement results
Image Source: www.desy.de/web/mosaic/hdf-browsing.html
W. Mauerer and D. Wagner 17. July 2019 11 / 31
2.2 Archiving
Archiving III
Recorded VariablesI Kernel version + parameters (activated fixes etc.)I Microcode version(s)I Online CPUsI Usual suspects: Included by Jitterdebugger by default
W. Mauerer and D. Wagner 17. July 2019 12 / 31
2.2 Archiving
Archiving IV
1. Measuring Real-Time Systems2. Jitterdebugger: Cyclictest for Dummies
2.1 Measuring2.2 Archiving
3. Analysis Examples3.1 Comparing Distributions3.2 Time-Resolved Analysis3.3 Estimating Upper Bounds/WCET
W. Mauerer and D. Wagner 17. July 2019 13 / 31
3. Analysis Examples
Outline
Three ways of understanding data1. Descriptive analysis2. Exploratory analysis3. Confirmatory analysis
Order mattersI Simple analyses before formal testI Proper visual understanding often more important than explicit calculations
W. Mauerer and D. Wagner 17. July 2019 14 / 31
3. Analysis Examples
Visualising and Understanding Data
1. Fine-grained detail analysis2. Improving/quantifying trustworthiness of WCET estimates
W. Mauerer and D. Wagner 17. July 2019 15 / 31
3. Analysis Examples
Analysis Examples
New 2 New 3
Base New 1
0 50 100 150 0 50 100 150
50000
100000
150000
50000
100000
150000
Latency
Cou
nt
W. Mauerer and D. Wagner 17. July 2019 16 / 31
3.1 Comparing Distributions
Comparing Distributions I
Why?I Track behavioural changes after
system changesI Load vs. idle behaviourI Evaluate alternative choices
How?I Comparing summaries 7
I Visual/explorative inspection 3
I Formal methods/tests 37
W. Mauerer and D. Wagner 17. July 2019 17 / 31
3.1 Comparing Distributions
Comparing Distributions II
(Empirical) Cumulative Distribution FunctionI Sums up fraction of values that fall into [0, x] at position xI Parameter free!I Interpretation requires trained eye
W. Mauerer and D. Wagner 17. July 2019 18 / 31
3.1 Comparing Distributions
Comparing Distributions II
New 2 New 3
Base New 1
50 75 100 125 50 75 100 125
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Latency
Cou
nt
ggplot(dat, aes(x=latency)) + stat_ecdf() + facet_wrap(~type)
W. Mauerer and D. Wagner 17. July 2019 19 / 31
3.1 Comparing Distributions
Comparing Distributions III
Point of viewWhich one is better?
Formal TestsI t-test: Check with identical mean values (Gaussian distribution/large sample size)I Wilcoxon signed-rank test: Test for identical distributions
Visual TestsI Quantile-Quantile plot 37
I Facetted(!) histograms 37
I Empirical cumulative distribution functions (ecdf) 3
W. Mauerer and D. Wagner 17. July 2019 20 / 31
3.1 Comparing Distributions
Comparing Distributions IV
ECDF & Probabilities for Worst-Case Latencies
New 2 New 3
Base New 1
60 80 100 120 60 80 100 120
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Latency
ED
CF
I pe: Probability of exceeding WCET wI w = F−1(1− pe)
W. Mauerer and D. Wagner 17. July 2019 21 / 31
3.1 Comparing Distributions
Comparing Distributions V
9 10
5 6 7 8
1 2 3 4
0 20 40 60 0 20 40 60
0 20 40 60 0 20 40 60
2e−044e−046e−048e−04
0.000250.000500.00075
5e−04
1e−03
5e−04
1e−03
1e−042e−043e−044e−04
0.000250.000500.000750.00100
0.002
0.0040.006
0.001
0.0020.003
1e−042e−043e−044e−045e−04
0.00050.00100.00150.00200.0025
Latency[us]
Den
sity
[log
, a.u
.]
ggplot(dat, aes(x=Latency)) + geom_density() + facet_wrap(~range) + scale_y_sqrt()
W. Mauerer and D. Wagner 17. July 2019 22 / 31
3.2 Time-Resolved Analysis
Time-Resolved Analysis
9 10
5 6 7 8
1 2 3 4
0 20 40 60 0 20 40 60
0 20 40 60 0 20 40 60
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Latency[us]
y
ggplot(dat, aes(x=Latency)) + stat_ecdf() + facet_wrap(~range)
W. Mauerer and D. Wagner 17. July 2019 22 / 31
3.2 Time-Resolved Analysis
Time-Resolved Analysis
1 2 3 4 5 6 7 8 9 100
12
3
0 20 40 60 0 20 40 60 0 20 40 60 0 20 40 60 0 20 40 60 0 20 40 60 0 20 40 60 0 20 40 60 0 20 40 60 0 20 40 60
0.000.250.500.751.00
0.000.250.500.751.00
0.000.250.500.751.00
0.000.250.500.751.00
Latency[us]
y
ggplot(dat, aes(x=Latency)) + geom_density() + facet_wrap(CPU~range)
W. Mauerer and D. Wagner 17. July 2019 22 / 31
3.2 Time-Resolved Analysis
Time-Resolved Analysis
1 2 3 4 5 6 7 8 9 100
12
3
0 20 40 60 0 20 40 60 0 20 40 60 0 20 40 60 0 20 40 60 0 20 40 60 0 20 40 60 0 20 40 60 0 20 40 60 0 20 40 60
1e+052e+053e+05
1e+052e+053e+05
1e+052e+053e+05
1e+052e+053e+05
Latency[us]
coun
t
ggplot(dat, aes(x=Latency)) + geom_histogram(bins=100) + facet_wrap(CPU~range) + scale_y_sqrt()
W. Mauerer and D. Wagner 17. July 2019 22 / 31
3.2 Time-Resolved Analysis
Time-Resolved Analysis
01
23
0 20 40 60
2.5
5.0
7.5
10.0
2.5
5.0
7.5
10.0
2.5
5.0
7.5
10.0
2.5
5.0
7.5
10.0
Maximal Latency[us]
Tim
e R
ange
ggplot(dat.max, aes(x=Latency)) + geom_bar(stat=’identity’) + facet_wrap(CPU~.)
W. Mauerer and D. Wagner 17. July 2019 22 / 31
3.2 Time-Resolved Analysis
Time-Resolved Analysis
9 10
7 8
5 6
3 4
1 2
0 20 40 60 0 20 40 60
0123
0123
0123
0123
0123
0123
0123
0123
0123
0123
Maximal Latency[us]
Cor
e
ggplot(dat.max, aes(x=Latency)) +geom_bar(stat=’identity’) + facet_wrap(range~.)
W. Mauerer and D. Wagner 17. July 2019 22 / 31
3.2 Time-Resolved Analysis
Time-Resolved Analysis
Accuracy: ProblemsI Correct tail classification: Huge number of samplesI Values larger than sample cannot appearI How reliable is a measurement?
W. Mauerer and D. Wagner 17. July 2019 23 / 31
3.2 Time-Resolved Analysis
Comparing Distributions VI
StatementsI “The highest threshold we’ve seen is XYZ”
I “After XYZ hours of measuring, the highest threshold we’ve seen is XYZ”
while TimBird was standing in front of the system
I “At a confidence level of ABC, the probability to exceed threshold XYZ is at most DEFpercent”
W. Mauerer and D. Wagner 17. July 2019 24 / 31
3.3 Estimating Upper Bounds/WCET
WCET Estimation I
StatementsI “The highest threshold we’ve seen is XYZ”I “After XYZ hours of measuring, the highest threshold we’ve seen is XYZ”
while TimBird was standing in front of the system
I “At a confidence level of ABC, the probability to exceed threshold XYZ is at most DEFpercent”
W. Mauerer and D. Wagner 17. July 2019 24 / 31
3.3 Estimating Upper Bounds/WCET
WCET Estimation I
StatementsI “The highest threshold we’ve seen is XYZ”I “After XYZ hours of measuring, the highest threshold we’ve seen is XYZ” while Tim
Bird was standing in front of the system
I “At a confidence level of ABC, the probability to exceed threshold XYZ is at most DEFpercent”
W. Mauerer and D. Wagner 17. July 2019 24 / 31
3.3 Estimating Upper Bounds/WCET
WCET Estimation I
StatementsI “The highest threshold we’ve seen is XYZ”I “After XYZ hours of measuring, the highest threshold we’ve seen is XYZ” while Tim
Bird was standing in front of the systemI “At a confidence level of ABC, the probability to exceed threshold XYZ is at most DEF
percent”
W. Mauerer and D. Wagner 17. July 2019 24 / 31
3.3 Estimating Upper Bounds/WCET
WCET Estimation I
Traditional: Two-Step Approach1. Measure2. Determine Maximum
New: Three-Step Approach1. Measure2. Determine model from measurement3. Infer WCET and uncertainty/credibility from model
W. Mauerer and D. Wagner 17. July 2019 25 / 31
3.3 Estimating Upper Bounds/WCET
WCET Estimation II
Model PropertiesI Generality
I Applicability to multiple real-world situationsI Realism
I Accurately represent real-world phenomenaI Precision
I Minimise errors compared to real-world results
W. Mauerer and D. Wagner 17. July 2019 26 / 31
3.3 Estimating Upper Bounds/WCET
WCET Estimation III
Model PropertiesI Generality 3
I Applicability to multiple real-world situationsI Realism 7
I Accurately represent real-world phenomenaI Precision 3
I Minimise errors compared to real-world results
W. Mauerer and D. Wagner 17. July 2019 26 / 31
3.3 Estimating Upper Bounds/WCET
WCET Estimation III
Fundamental: Random VariablesI Random Variable XI Measurements/observations x1, x2, . . . , xN sampled from X
Random Variable 6= Random Properties
1 2 3 25
1 2 3 4 5 2.5 5.0 7.5 10.0 4 8 12 50 60 70 80 900
20
40
60
0
50
100
150
0
50
100
150
200
0
50
100
150
200
Sum of Dice
Cou
nt
W. Mauerer and D. Wagner 17. July 2019 27 / 31
3.3 Estimating Upper Bounds/WCET
WCET Estimation IV
Fundamental: Random VariablesI Random Variable XI Measurements/observations x1, x2, . . . , xN sampled from X
Random Variable 6= Random Properties
1 2 3 25
1 2 3 4 5 2.5 5.0 7.5 10.0 4 8 12 50 60 70 80 900
20
40
60
0
50
100
150
0
50
100
150
200
0
50
100
150
200
Sum of Dice
Cou
nt
W. Mauerer and D. Wagner 17. July 2019 27 / 31
3.3 Estimating Upper Bounds/WCET
WCET Estimation IV
Latencies: Maxima matter!I Latency is a random variable XI Measurements x1, x2, . . . , xN sampled from X
I Quantity of interest: max(x1, x2, . . . , xN )
W. Mauerer and D. Wagner 17. July 2019 28 / 31
3.3 Estimating Upper Bounds/WCET
WCET Estimation V
What can we say about maxima?I Independent and identically distributed (iid) random variables {X1, X2, . . . , Xn}I Maxima: max({X1, X2, . . . , Xn})
I Jitterdebugger: Postprocess results to obtain block maxima!
W. Mauerer and D. Wagner 17. July 2019 28 / 31
3.3 Estimating Upper Bounds/WCET
WCET Estimation V
What can we say about maxima?I Independent and identically distributed (iid) random variables {X1, X2, . . . , Xn}I Maxima: max({X1, X2, . . . , Xn})
I Jitterdebugger: Postprocess results to obtain block maxima!
Generalised Extreme Value Distribution
G(z) = exp
(−(1 + ξ
(z − µσ
)) 1ξ
)I Distribution function for (extremely) rare eventsI Determine parameters from measured maxima
I ξ: Distribution shapeI σ: ScaleI µ: Location
W. Mauerer and D. Wagner 17. July 2019 28 / 31
3.3 Estimating Upper Bounds/WCET
WCET Estimation V
What can we say about maxima?I Independent and identically distributed (iid) random variables {X1, X2, . . . , Xn}I Maxima: max({X1, X2, . . . , Xn})I Jitterdebugger: Postprocess results to obtain block maxima!
Generalised Extreme Value Distribution
G(z) = exp
(−(1 + ξ
(z − µσ
)) 1ξ
)I Distribution function for (extremely) rare eventsI Determine parameters from measured maxima
I ξ: Distribution shapeI σ: ScaleI µ: Location
W. Mauerer and D. Wagner 17. July 2019 28 / 31
3.3 Estimating Upper Bounds/WCET
WCET Estimation V
Fit ParametersI gev(dat.measured$max)I ξ = 0.229± 0.027, σ = 5025.82± 147.45, µ = 9340.03± 182.53
But Wait! Model Assumptions + Quality!I Carefully inspect diagnostic plotsI Prediction Quality: Usual ML approach
I Split dataI Model building/testing
W. Mauerer and D. Wagner 17. July 2019 29 / 31
3.3 Estimating Upper Bounds/WCET
WCET Estimation V
ProblemsI IID assumption
I Caches, Branch Prediction: No independence between runsI Continuous vs. discrete distribution
I Only a discrete set of runtimes is possible
Don’t Forget: Models and/vs. RealityI No model is correctI Some models may be useful
W. Mauerer and D. Wagner 17. July 2019 30 / 31
3.3 Estimating Upper Bounds/WCET
WCET Estimation VI
Thanks for your interest!
W. Mauerer and D. Wagner 17. July 2019 31 / 31
3.3 Estimating Upper Bounds/WCET