Design of Experimentspolaris.imag.fr/.../2015.AISSE.WSM.lecture6.case-sudy.pdf0 50 100 150 200 250...

Wireless example Guidelines References

Design of ExperimentsOption PDES

Jean-Marc Vincent1

Laboratoire LIG{Jean-Marc.Vincent}@imag.fr

1Université Joseph FourierÉquipe-Projet MESCAL

Some elements for the design of experiments : an example

1 / 21Design of Experiments


Wireless environment

Environment

Wireless DevicesLaptop computers,

Personal digital assistants (PDAs),

Mobile phones, ...

Wireless Ad-Hoc NetworkWIFI,

Bluetooth, ...

PrincipleDevices share services and collaborate to maintain the community.

General distributed system problem→ Dynamical control of the architecture.→ State of the system observation and distributed decision process.



Wireless environment

Environment

Wireless DevicesLaptop computers,

Personal digital assistants (PDAs),

Mobile phones, ...

Wireless Ad-Hoc NetworkWIFI,

Bluetooth, ...

PrincipleDevices share services and collaborate to maintain the community.

General distributed system problem→ Dynamical control of the architecture.→ State of the system observation and distributed decision process.



Operating systems on a wireless network

Dynamic Architecture

Heterogeneity of devices

Behavior of wireless devices→ connections / disconnectionsBehavior of wireless network

Unreliability of communicationsVariability of latencies

Solution to maintain the consistency of the community

Design and adapt distributed algorithms ;

to make some distributed decisions.

(consensus, election, atomic broadcast, group membership,...)



The Consensus Problem

The impossibility of Fisher, Lynch & Paterson [Fischer-Lynch-Paterson 85]

Some approaches to circumvent this impossibility result :

Probabilistic algorithms [Canetti-Rabin 93]

Self-Stabilizing algorithms [Tixeuil 00]

An approach with partial synchrony [Dwork-Lynch-Stockmeyer 88]

....

⇒ The selected approach :

Consensus + unreliable failure detectors [Chandra-Toueg 96]

Interest :Dynamicity analysis of the environment is concentrated only inside failure detectors.

Objective :Guarantee a quality of service for failure detectors.



The Consensus Problem

The impossibility of Fisher, Lynch & Paterson [Fischer-Lynch-Paterson 85]

Some approaches to circumvent this impossibility result :

Probabilistic algorithms [Canetti-Rabin 93]

Self-Stabilizing algorithms [Tixeuil 00]

An approach with partial synchrony [Dwork-Lynch-Stockmeyer 88]

....

⇒ The selected approach :

Consensus + unreliable failure detectors [Chandra-Toueg 96]

Interest :Dynamicity analysis of the environment is concentrated only inside failure detectors.

Objective :Guarantee a quality of service for failure detectors.



Unreliable Failure Detectors

Principle :For each remote device, build an estimation of the global state.

Local view of the global system.→ List of suspected devices

Properties

Accuracy : a correct process should not be suspected

Completeness : an incorrect process should be suspected

Quality of serviceQuality of information and reactivity

false suspicion rate = function(reactivity)

- false suspicion type 1 : correct process suspected- false suspicion type 2 : crashed process not suspectedRisk analysis

Implementation



Unreliable Failure Detectors

Principle :For each remote device, build an estimation of the global state.

Local view of the global system.→ List of suspected devices

Properties

Accuracy : a correct process should not be suspected

Completeness : an incorrect process should be suspected

Quality of serviceQuality of information and reactivity

false suspicion rate = function(reactivity)

- false suspicion type 1 : correct process suspected- false suspicion type 2 : crashed process not suspectedRisk analysis

Implementation

SuspicionSuspicion

No

ReceiptHeartbeat

Timeout



Failure Detectors Implementation

Need of information on remote devices

Export local data

Collect and analyse data coming from remote devices

Informations Export

→ Broadcast of information by anticipation (Heartbeatmecanism)

ConsultationInfo publication

Information

parameters )&

Middleware Interface

moduleExport

( policyMW control

Suspicion requestNetwork Interface

Information

List of suspects

Network interface Suspicion request

Importmodule

Consensus interface

parameters )

( policy&

MW control

MiddlewareInterface

List to query

Informations Import

→ Collect information

→ Estimate the state of remote devices



Failure Detectors Parameters

The running principle :Implemented mecanism : "heartbeat".

� � ��

� ��

FD ExportProcess

SenderDevice

DeviceReceiver

ProcessFD Import

FD sending delay

FD receipt delayθ

θ θθ

θ

Parameters

Heartbeat sending period.

Estimate function of suspicions (timeout). [Bertier-Marin-Sens 03]

GoalEstimate the timeout values according to the expected quality of service.



Failure Detectors Parameters

The running principle :Implemented mecanism : "heartbeat".

� � ��

� ��

FD ExportProcess

SenderDevice

DeviceReceiver

ProcessFD Import

FD sending delay

FD receipt delayθ

θ θθ

θ

Parameters

Heartbeat sending period.

Estimate function of suspicions (timeout). [Bertier-Marin-Sens 03]

GoalEstimate the timeout values according to the expected quality of service.



Statistical Description

Network

Network

FD ExportProcess

FD ImportProcess

FD sending delay

Network sending delay

Network receipt delay

FD receipt delay

SenderDevice

ReceiverDeviceHB interval (measured)

HB Interval (given)

Variability of HB arrivals

λ0 = emission beat rate

Xi = Heartbeat inter-arrivals.

λ = 1HB period (assumption : few losses λ = λ0.(1− loss probability)



Statistical Description

False Detection Probabilityθ = suspicion threshold (timeout)φI (θ) = asymptotic false suspicion rate

φI (θ) = λ limn→∞

1

n

n∑i=1

(Xi − θ)+

If the inter-arrivals {Xi} of beats are independent and identically distributed, then :

φI (θ) = λEπ [X − θ]+

where π is the distribution of Xi . (renewal process)



Independent assumption Model

Variable Sending Delay

Hypothesis : {Xi} : renewal process (iid)

Model suspicion rate

no information on variance Exponential φI (θ) = e−λθ

low variation coefficient Erlang(k,kλ) φI (θ) = e−kλθPk (λθ)

high variation coefficient Pareto(α) φI (θ) = 1(1+ θ

α−2 )α−2

Suspicion probability related to reactivity

Erlang model : Pareto model :



Independent assumption Model

Variable Sending Delay

Hypothesis : {Xi} : renewal process (iid)

Model suspicion rate

no information on variance Exponential φI (θ) = e−λθ

low variation coefficient Erlang(k,kλ) φI (θ) = e−kλθPk (λθ)

high variation coefficient Pareto(α) φI (θ) = 1(1+ θ

α−2 )α−2

Suspicion probability related to reactivity

Erlang model :

k=1 (exponential model)

k=2

k=3

k=4k=5

False suspicion rate

1e−05

1e−04

0.001

0.01

0.1

0 1 2 3 4 5 6 θ

1

Pareto model :

σ=

σ=2

3

σ=1.1 0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

0.5

2 4 6 8 10 12 14 16 18 θ



Contention on receiver (1)

Variability of Heartbeat Arrivals

Depends on the type of receiver (Laptop or PDA)

+ Correlation between inter-beats arrival periods

Network

FD ImportProcess

ReceiverDevice

variability of delivery

⇒ HB contention on the receiver

Buffer

Network

Service

Delivery to

upper layer

Heart−beats Failure detector

input process : {An}n∈N

service model : {Sn}n∈N

hypothesis : deterministic arrivals

→ GI/M/1 queue




Output process of a D/M/1 queue → Suspicion rate computation

Inter-output period : A = 1λ

; β unique solution of β = LA(µ(1− β)) = e−Aµ(1−β)

fZ (x) =

{µ

2−β e−µ(1−β)A((1− β)eµ(1−β)x + e−µx ) if x < A;µ

2−β e−µx (e−µ(1−β)A + (1− β)eµA) if x ≥ A.

Rate of false suspicion : θ > A

φI(θ) =1

A(2− β)µe−µθ(e−µ(1−β)A + (1− β)eµA) θ ≥ A

False suspicion probability, D/M/1 model




Output process of a D/M/1 queue → Suspicion rate computation

Inter-output period : A = 1λ

; β unique solution of β = LA(µ(1− β)) = e−Aµ(1−β)

fZ (x) =

{µ

2−β e−µ(1−β)A((1− β)eµ(1−β)x + e−µx ) if x < A;µ

2−β e−µx (e−µ(1−β)A + (1− β)eµA) if x ≥ A.

Rate of false suspicion : θ > A

φI(θ) =1

A(2− β)µe−µθ(e−µ(1−β)A + (1− β)eµA) θ ≥ A

False suspicion probability, D/M/1 model

µ=2 µ=10

A=1 A=1 1e−07

1e−06

1e−05

1e−04

0.001

0.01

0.1

1

10

0 1 2 3 4 5 6 7

False probability suspicion

θ 1e−35

1e−30

1e−25

1e−20

1e−15

1e−10

1e−05

1

100000

0 1 2 3 4 5 6 7 θ

False probability suspicion



Experimental environment

Contexte Expérimental - Latences (2)

Pocket PCs en mode ad-hoc

0

200

400

600

800

1000

1200

1400

0 50 100 150 200 250 300 350 400 450 500latence en ms

sans economie d’energieeconomie d’energie

Taille échantillon :5000 mesures! ping (délai : 1 s)

{Corine.Marchand, Jean-Marc.Vincent}@imag.fr – CFSE’2003 – Detecteurs de Defaillances et Qualite de Service dans un Reseau Ad-Hoc Heterogene – 10







! Grand nombre de paramètres

Facteurs influentsretenus :

Distance

Nb obstacles

Nb entités

Charge réseau

Type émetteur

Type récepteur

Economie NRJ





! Grand nombre de paramètres

" Plan d’expérience :! Méthode de Taguchi(à 2 niveaux)


Distance

Nb obstacles

Nb entités

Charge réseau

Type émetteur

Type récepteur

Economie NRJ






Distance

Nb obstacles

Nb entités

Charge réseau

Type émetteur

Type récepteur

Economie NRJ





Facteursprépondérants:

Type récepteur

Charge réseau

Economie NRJ

Type émetteur

Interactions





Experimental Design :

Devices (same OS, Java) :Architecture 1 : 4 devices (2 Laptops + 2 PDAs)Architecture 2 : 6 devices (2 Laptops + 3 PDAs + 1 sensor

Interconnection : 802.11b ad-hoc network

Experimental duration : 15 min (→ about 10,000 measurements)

HB parameter Settings :

Architecture 1 Architecture 2 Architecture 2

Highly loaded Ideal Setting Perturbed Environment

HB emission period 100 ms 500 ms 500 ms

Timeout none none none

Reception process analysisDensity of the delivery process⇒ timeout tuning




Experimental Design :

Devices (same OS, Java) :Architecture 1 : 4 devices (2 Laptops + 2 PDAs)Architecture 2 : 6 devices (2 Laptops + 3 PDAs + 1 sensor

Interconnection : 802.11b ad-hoc network

Experimental duration : 15 min (→ about 10,000 measurements)

HB parameter Settings :

Architecture 1 Architecture 2 Architecture 2

Highly loaded Ideal Setting Perturbed Environment

HB emission period 100 ms 500 ms 500 ms

Timeout none none none

Reception process analysisDensity of the delivery process⇒ timeout tuning



Highly loaded system

Distribution of the update times :

0

0.002

0.004

0.006

0.008

0.01

0 200 400 600 800 1000

sender: PDA1 - receiver: PDA2

Times between 2 heartbeat receipts

Prob

abili

ty

250 300 350 400 450 500

sender: PDA1 - receiver: laptop2

Prob

abili

ty

0.02

0.015

0.01

0.005

00 50 100 150 200Times between 2 heartbeat receipts

00.0010.0020.0030.0040.0050.0060.0070.0080.009

0 200 400 600 800 1000Times between 2 heartbeat receipts

sender: laptop1 - receiver: PDA2

Prob

abili

ty

0

0.01

0.02

0.03

0.04

0.05

0.06

0.07

0.08

0 50 100 150 200Pr

obab

ility


sender: laptop1 - receiver: laptop2

If timeout value = 200 ms→ Quality of service highly depends on the type of the receiver



Highly loaded system

Distribution of the update times :

0

0.002

0.004

0.006

0.008

0.01

0 200 400 600 800 1000

sender: PDA1 - receiver: PDA2


Prob

abili

ty

250 300 350 400 450 500

sender: PDA1 - receiver: laptop2

Prob

abili

ty

0.02

0.015

0.01

0.005

00 50 100 150 200Times between 2 heartbeat receipts

00.0010.0020.0030.0040.0050.0060.0070.0080.009

0 200 400 600 800 1000Times between 2 heartbeat receipts

sender: laptop1 - receiver: PDA2

Prob

abili

ty

0

0.01

0.02

0.03

0.04

0.05

0.06

0.07

0.08

0 50 100 150 200Pr

obab

ility


sender: laptop1 - receiver: laptop2

If timeout value = 200 ms→ Quality of service highly depends on the type of the receiver



"Ideal Setting" Experimentation

Heartbeat Reception Analysis :

0

0.02

0.04

0.06

0.08

0.1

0.12

400 450 500 550 600

Prob

abili

ty

Elapsed times between 2 receipts

sender: laptop1 − receiver: laptop2sender: laptop1 − receiver: pda2sender: pda1 − receiver: pda2

0.14sender: pda1 − receiver: laptop2

Timeout value :

If the timeout value = 2∗ (HB period time)Then, the suspicion rate is around

10−3 if the receiver is a laptop10−2 if the receiver is a PDA



"Ideal Setting" Experimentation


0

0.02

0.04

0.06

0.08

0.1

0.12

400 450 500 550 600

Prob

abili

ty

Elapsed times between 2 receipts

sender: laptop1 − receiver: laptop2sender: laptop1 − receiver: pda2sender: pda1 − receiver: pda2

0.14sender: pda1 − receiver: laptop2

Timeout value :

If the timeout value = 2∗ (HB period time)Then, the suspicion rate is around

10−3 if the receiver is a laptop10−2 if the receiver is a PDA



Experimentation in perturbed environment

Perturbation control :An external device is used to generate an external load (ping with 200kbytes/s)


0

0.02

0.04

0.06

0.08

0.1

450 500 550 600 650 700Elapsed Times between 2 heartbeat receipts

sender: Laptop1 − receiver: Laptop2sender: Laptop1 − receiver: PDA2sender: PDA1 − receiver: Laptop2

Prob

abili

tysender: PDA1 − receiver: PDA2

Results :

Long non receiving period for PDA

Some very small delays between HB receipts (after a long waiting time)

⇒ Correlation between succesive waiting times of two HB (bursty receptions)



Experimentation in perturbed environment

Perturbation control :An external device is used to generate an external load (ping with 200kbytes/s)


0

0.02

0.04

0.06

0.08

0.1

450 500 550 600 650 700Elapsed Times between 2 heartbeat receipts

sender: Laptop1 − receiver: Laptop2sender: Laptop1 − receiver: PDA2sender: PDA1 − receiver: Laptop2

Prob

abili

tysender: PDA1 − receiver: PDA2

Results :

Long non receiving period for PDA

Some very small delays between HB receipts (after a long waiting time)

⇒ Correlation between succesive waiting times of two HB (bursty receptions)



Guideline (Montgomery)

1 Recognition of and statement of the problemseries of small experimentsfactors screening

2 Selection of the response variable3 Choice of factors, levels, and ranges potential design factors (nuisance factors)

factors controllabilitycause and effects diagram (fishbone diagram)

4 Choice of experimental designrandomization, replication, blockingfactorial planning

5 Performing the experiment6 Statistical analysis of the data7 Conclusions and recommendations



Références I

Jain, R. (1991), Art of Computer Systems Performance Analysis, JOHNWILEY and SONS.

Lawson, J., ed. (2014), Design and Analysis of Experiments with R,Chapman and Hall/CRC.

Lilja, D. J. (2009), Measuring Computer Performance : A Practitioner’s Guide,Cambridge University Press.

Linder, R. (n.d.), Les plans d’expériences (French Edition), PRESSESECOLE NATIONALE PONTS CHAUSSEES.

Montgomery, D. C. (2009), Design and Analysis of Experiments, 7EInternational Student Version, 7th international student edition edn,John Wiley and Sons Ltd.


Date post:	31-Jul-2020
Category:	Documents
Upload:	others
View:	0 times
Download:	0 times

Design of Experimentspolaris.imag.fr/.../2015.AISSE.WSM.lecture6.case-sudy.pdf0 50 100 150 200 250...

Documents