Bootstrapping Skynet:Calibration and Autonomic Self-Control of

Structured Peer-to-Peer Networks

Timo Klerx and Kalman Graffi

Department of Computer ScienceUniversity of Paderborn

Research Group Knowledge-Based SystemsHans Kleine Büning

September 11, 2013

Knowledge-Based SystemsUNIVERSITY OF PADERBORN

Motivation Approach Evaluation Conclusion

Outline

1 Motivation

2 Approach

3 Evaluation

4 Conclusion & Future Work

Bootstrapping Skynet Klerx and Graffi 1/17

Motivation Approach Evaluation Conclusion

Outline

1 Motivation

2 Approach

3 Evaluation

4 Conclusion & Future Work

Bootstrapping Skynet Klerx and Graffi 2/17

Motivation Approach Evaluation Conclusion

Bootstrapping SkyNetTowards self-optimization

SkyNet: Management layer in PeerfactSim.KOM(P2P-)Systems become more and more complex

ApplicationsParametersLayers. . .

Ideally, systems manage themselvesChoose parametersDefend attacksRestore network structure. . .

Bootstrapping Skynet Klerx and Graffi 3/17

Motivation Approach Evaluation Conclusion

MAPEHow to achieve self-management?

Monitor

Analyze

Plan

Execute

Systems implementing the MAPE circuit are autonomous.Everything except Planning is already implemented.

Bootstrapping Skynet Klerx and Graffi 4/17

Motivation Approach Evaluation Conclusion

Outline

1 Motivation

2 Approach

3 Evaluation

4 Conclusion & Future Work

Bootstrapping Skynet Klerx and Graffi 5/17

Motivation Approach Evaluation Conclusion

Plan PhaseIdea

OfflineGather data by simulationLearn the interdependencies in the dataConstruct a regressor with goal as input to compute parametervalues

OnlineDefine a desired goalAsk the regressor for optimal parameter valuesChange parameter values on every node

Bootstrapping Skynet Klerx and Graffi 6/17

Motivation Approach Evaluation Conclusion

Plan PhaseIdea

OfflineGather data by simulationLearn the interdependencies in the dataConstruct a regressor with goal as input to compute parametervalues

OnlineDefine a desired goalAsk the regressor for optimal parameter valuesChange parameter values on every node

Bootstrapping Skynet Klerx and Graffi 6/17

Motivation Approach Evaluation Conclusion

Neural NetworksBasics

Classification and regression(Often) supervised learning – need labeled training dataLearn effects of parametersInput must be specified preciselyCan approximate arbitrary functions with arbitrary precision

Bootstrapping Skynet Klerx and Graffi 7/17

Motivation Approach Evaluation Conclusion

Data GenerationData characteristics

Three types of figuresEnvironment Parameters (E , |E | = 5) – Changed by all usersnode count, churn, . . .Overlay Parameters (O, |O| = 8) – Changeable by single nodesmessage timeout, max hop count, . . .Metrics (M, |M| = 18) – Performance valuesavg. hop count, avg. network messages in, . . .

View as function f : E × O → M

Bootstrapping Skynet Klerx and Graffi 8/17

Motivation Approach Evaluation Conclusion

Data GenerationCombination approaches

Full factorial designAll possible combinations of parameters∏n

i=1 |pi |Takes too much time

One factorial designOnly one parameter varied at a timeRest set to default values∑n

i=1 |pi |Few data points

Mixed factorial designTradeoff between one and full factorial designSome parameters (E ) in full factorial design, others (O) set todefault values∏s

j=1 |ej | ·∑t

k=1 |ok |

Bootstrapping Skynet Klerx and Graffi 9/17

Motivation Approach Evaluation Conclusion

Data GenerationCombination approaches

Full factorial designAll possible combinations of parameters∏n

i=1 |pi |Takes too much time

One factorial designOnly one parameter varied at a timeRest set to default values∑n

i=1 |pi |Few data points

Mixed factorial designTradeoff between one and full factorial designSome parameters (E ) in full factorial design, others (O) set todefault values∏s

j=1 |ej | ·∑t

k=1 |ok |

Bootstrapping Skynet Klerx and Graffi 9/17

Motivation Approach Evaluation Conclusion

Data GenerationCombination approaches

Full factorial designAll possible combinations of parameters∏n

i=1 |pi |Takes too much time

One factorial designOnly one parameter varied at a timeRest set to default values∑n

i=1 |pi |Few data points

Mixed factorial designTradeoff between one and full factorial designSome parameters (E ) in full factorial design, others (O) set todefault values∏s

j=1 |ej | ·∑t

k=1 |ok |

Bootstrapping Skynet Klerx and Graffi 9/17

Motivation Approach Evaluation Conclusion

Neural NetworksLearn the data characteristics

Remember function f : E × O → MReorder f to f̂ : M × E → O

M: The preferred stateE : The current environment statev ∈ f̂ : (m1, ..., mr , e1, ..., es , o1, ..., ot)

Approximate f̂ : Predict the overlay parameter values when givenenvironment state and a goalOnly realistic goals as inputTrain with resilient backpropagationOne neural network for each overlay parameterSplit data in three disjoint sets: training, validation, prediction

EnvironmentParameters

Overlay Parameter

Metrics

Hidden Layer(s)

...

...

Bootstrapping Skynet Klerx and Graffi 10/17

Motivation Approach Evaluation Conclusion

Neural NetworksLearn the data characteristics

Remember function f : E × O → MReorder f to f̂ : M × E → O

M: The preferred stateE : The current environment statev ∈ f̂ : (m1, ..., mr , e1, ..., es , o1, ..., ot)

Approximate f̂ : Predict the overlay parameter values when givenenvironment state and a goalOnly realistic goals as inputTrain with resilient backpropagationOne neural network for each overlay parameterSplit data in three disjoint sets: training, validation, prediction

EnvironmentParameters

Overlay Parameter

Metrics

Hidden Layer(s)

...

...

Bootstrapping Skynet Klerx and Graffi 10/17

Motivation Approach Evaluation Conclusion

Neural NetworksLearn the data characteristics

Remember function f : E × O → MReorder f to f̂ : M × E → O

M: The preferred stateE : The current environment statev ∈ f̂ : (m1, ..., mr , e1, ..., es , o1, ..., ot)

Approximate f̂ : Predict the overlay parameter values when givenenvironment state and a goalOnly realistic goals as inputTrain with resilient backpropagationOne neural network for each overlay parameterSplit data in three disjoint sets: training, validation, prediction

EnvironmentParameters

Overlay Parameter

Metrics

Hidden Layer(s)

...

...

Bootstrapping Skynet Klerx and Graffi 10/17

Motivation Approach Evaluation Conclusion

Outline

1 Motivation

2 Approach

3 Evaluation

4 Conclusion & Future Work

Bootstrapping Skynet Klerx and Graffi 11/17

Motivation Approach Evaluation Conclusion

OverviewWhich generation approach leads to good results?

Mixed factorial design (65,100 combinations)Most of the overlay parameter values are the default valueAlways predict "default" results in small errors

One factorial design (80 combinations)Use feature selection (CFS and PCA)Not all parameters predicted successfully

Bootstrapping Skynet Klerx and Graffi 12/17

Motivation Approach Evaluation Conclusion

Prediction QualityError while/after training

o1 =message timeouto6 =update fingertable intervalo7 =update neighbors intervalo8 =update successor interval

o2 =message resendo3 =operation timeouto4 =operation max. redoso5 =max hop count

0

20

40

60

80

100

120

o1 o2 o3 o4 o5 o6 o7 o8

Err

or in

Per

cent

Parameter

cfspca

no fs

(a) validation

0

20

40

60

80

100

120

o1 o2 o3 o4 o5 o6 o7 o8

Err

or in

Per

cent

Parameter

cfspca

no fs

(b) prediction

Figure: Error on prediction and validation setBootstrapping Skynet Klerx and Graffi 13/17

Motivation Approach Evaluation Conclusion

Prediction QualityError in the MAPE circuit without feature selection

0.12 0.14 0.16 0.18

0.2 0.22 0.24 0.26 0.28

0.3

w1 w1 w1 w2 w2 w2 w3 w3 w3

Avg

. Dur

atio

n (m

16)

Message Timeout (o1)

optimalpredicted

0 20 40 60 80

100 120 140 160 180 200

w1 w1 w1 w2 w2 w2 w3 w3 w3

Avg

. Net

Mes

sage

s O

ut (m

2)

Update Fingertable Interval (o6)

optimalpredicted

20

30

40

50

60

70

80

90

w1 w1 w1 w2 w2 w2 w3 w3 w3

Avg

. Net

Mes

sage

s O

ut (m

2)

Update Neighbors Interval (o7)

optimalpredicted

30

35

40

45

50

55

60

65

w1 w1 w1 w2 w2 w2 w3 w3 w3

Avg

. Net

Mes

sage

s O

ut (m

2)

Update Successor Interval (o8)

optimalpredicted

Bootstrapping Skynet Klerx and Graffi 14/17

Motivation Approach Evaluation Conclusion

Outline

1 Motivation

2 Approach

3 Evaluation

4 Conclusion & Future Work

Bootstrapping Skynet Klerx and Graffi 15/17

Motivation Approach Evaluation Conclusion

Conclusion

Mixed factorial design not suitableMAPE circuit closed in a proof-of-conceptFeature selection not beneficialEvaluation results are ambiguous

Good results for some parameters, bad for others

Bootstrapping Skynet Klerx and Graffi 16/17

Motivation Approach Evaluation Conclusion

Future Work

Investigate the other parametersTry full factorial design with less parameters, but more granularDesign more metricsEmbed the implemented MAPE circuit in a real systemDecentralize the neural network(s) – use local view

Thank you for your attention!

Bootstrapping Skynet Klerx and Graffi 17/17

Motivation Approach Evaluation Conclusion

Future Work

Investigate the other parametersTry full factorial design with less parameters, but more granularDesign more metricsEmbed the implemented MAPE circuit in a real systemDecentralize the neural network(s) – use local view

Thank you for your attention!

Bootstrapping Skynet Klerx and Graffi 17/17

Parameter values Evaluation

Outline

5 Parameter values

6 Evaluation

Bootstrapping Skynet Klerx and Graffi A-1

Parameter values Evaluation

Overlay Parameters

Code Overlay Parameter Unit default

o1 Message Timeout s 10o2 Message Resend # 3o3 Operation Timeout s 120o4 Operation Max. Redos # 3o5 Max Hop Count # 50o6 Upd. Finger Table Intv. ms 30o7 Upd. Neighbors Intv. ms 30o8 Upd. Successor Intv. ms 30

Bootstrapping Skynet Klerx and Graffi A-2

Parameter values Evaluation

Environment Parameters

Code Env. Parameter Unit default

e1 Node Count # 1000e2 Churn Factor # 0e3 Mean Session Length s ∞e4 Bandwidth MB/s OECDe5 Random Lookup Rate 1/h 30

Bootstrapping Skynet Klerx and Graffi A-3

Parameter values Evaluation

MetricsCode Metrics Unit

Messagesm1 Avg. Network Message In #m2 Avg. Network Message Out #m3 Avg. Transport Message In #m4 Avg. Transport Message Out #m5 Avg. Forwarded Queries #m6 Avg. Service Message Throughput #/sm7 St. Dev. Service Message Throughput #/sm8 Avg. Service Message Count #m9 St. Dev. Service Message Count #Trafficm10 Avg. Network Bytes Sent kBm11 Avg. Transport Bytes Sent kBm12 Avg. Free Upload Bandwidth kB/sPerformancem13 Avg. Hop Count #m14 Avg. Lookup Hops #m15 St. Dev. Lookup Hops #m16 Avg. Lookup Duration sm17 St. Dev. Lookup Duration sm18 Avg. Operation Duration s

Bootstrapping Skynet Klerx and Graffi A-4

Parameter values Evaluation

Parameter Variations

Code Mixed Factorial One Factorial

o1 5, 10, 20 2,3,4,5,8,10,12,15,18,20o2 0, 1, 3, 10 0,1,2,3,4,5,7,8,9o3 60, 120, 300 60,90,120,150,180,

210,240,270,285,300o4 0, 1, 3, 10 0,1,2,3,4,5,7,8,9,10o5 5, 10, 25, 50, 100 3,5,7,10,17,35,50,75,100o6 3, 10, 30, 60 3,5,7,10,15,20,30,40,50,60o7 3, 10, 30, 60 3,5,7,10,15,20,30,40,50,60o8 3, 10, 30, 60 3,5,7,10,15,20,30,40,50,60

e1 10, 33, 100, 330, 10001000, 3300, 10000

e2 0, 110 , 3

10 0e3 30, 60; 180,∞ ∞e4 OECD, random:

1-2, 5-10, 10-30, 1-30

OECD

e5 0, 1, 3, 6, 30 30

Bootstrapping Skynet Klerx and Graffi A-5

Parameter values Evaluation

Outline

5 Parameter values

6 Evaluation

Bootstrapping Skynet Klerx and Graffi A-6

Parameter values Evaluation

Process of Evaluation

ParameterValue

X

ParameterValueX‘

MetricVector

M

MetricVectorM‘

Simulation SimulationNeural

Network

compare compare

Bootstrapping Skynet Klerx and Graffi A-7

Parameter values Evaluation

Results of Evaluation

Table: Comparison of X , X ′, M and M ′

Timestamp X X′ M M′ |1− MM′ | |1− X

X′ |

For o1 and m16w1 6s 6s 0.14 0.15 0.07 0.00w2 11s 11s 0.18 0.20 0.10 0.00w3 19s 18s 0.24 0.27 0.11 0.06

For o6 and m2w1 4s 4s 175.00 173.66 0.01 0.00w2 25s 26s 39.60 38.34 0.03 0.04w3 55s 54s 24.50 25.51 0.04 0.02

For o7 and m2w1 6s 4.6s 65.80 77.27 0.15 0.30w2 25s 27.5s 36.60 35.98 0.02 0.09w3 51s 55s 31.70 31.63 0.00 0.07

For o8 and m2w1 4s 3.9s 60.21 60.90 0.01 0.03w2 37s 36s 34.27 34.10 0.00 0.03w3 55s 52s 33.21 33.00 0.01 0.06

Bootstrapping Skynet Klerx and Graffi A-8