1
Wi-Fi Conferencing System: Locating Array Efficient experiment parameter screeningIngrid Moerman, Eli De Poorter, Michael Mehari (iMinds)Violet Syrotiuk, Randy Compton, Charles Colbourn (Arizona state university)
2
Overview
• Theoretical background• Applying these concepts in FIRE
3
Screen Experiments
• A wireless system is characterized by a large amount of input parameters (factors) each with different configurable levels or settings.
• The purpose of a screening experiment is to identify the significant factors and interactions between them that influence the overall performance
4
Screen Experiments
• An experiment is given by an N × k array• The k columns correspond to the factors
• Each factor Fi, 1 ≤ i ≤ k has a set of levels Li
• Each of the N rows corresponds to a test in which each factor Fi is set to a level in Li
• Example• 2 factors F1 & F2
• Tx power, 20 levels (0 -> 20 dBm)
• Transmission rate, 5 levels (6, 9, 12, 24, 36 Mbps)
• 5 experiments
(2 columns, k=2)
(5 rows, N=5)
5
Coverage
• To measure a t-way interaction it must be covered• For example, if t=2, then all pairs of (factor, level)
combinations must occur in the experiment
• Covering arrays (CA) are designed to provide coverage
• A CA(N;t,k,v) is an N × k array where each N × t subarray contains all ordered t-sets
6
Covering Arrays
• An example covering array in which each 6 × 2 subarray contains all ordered (binary) pairs
0 1 1 1 1
1 0 1 0 0
0 1 0 0 0
1 0 0 1 1
0 0 0 0 1
1 1 0 1 0
CA(6;2,5,2)
7
Location is more than Coverage• However, the ability to locate factors and
interactions is more than just coverage• A locating array must uniquely identify the rows in
which significant t-way interactions are covered• The number of weak or faulty interactions must
be limited (not known a priori)
8
Locating Arrays – How Large?
• How does the array size N grow as a function of k for fixed v, t, and d?
• Number of factors: k
• Number of levels: v
• “Size” of interaction: t
• Maximum number of faulty interactions d
• There are many many sets of selection tuples to distinguish
When t and d are fixed, and the number of factors k increases, the number of tests grows as a logarithmic function of k
9
Overview
• Theoretical background• Applying these concepts in FIRE
10
From theory to experiment
• The locating arrays have been used successfully in simulations [1] to identify parameter interactions
• Reducing the design points from 1043 to only 421
• But is it also feasible on realistic systems?
[1] Abraham N. Aldaco, Charles J. Colbourn, Violet R. Syrotiuk: Locating Arrays: A New Experimental Design for Screening Complex Engineered Systems. ACM SIGOPS Operating Systems Review - Special Issue on Repeatability and Sharing of Experimental Artifacts, 49(1): 31-40 (2015)
11
1 transmitter42 receivers
Experiment set-up
13
# Factor Description Range
1 band Frequency band (2.4, 5) GHz
2 channel Frequency channel 1, 6, 11, 36, 40, 44
3 rate Transmission bitrate (6, 9, 12, 24, 36) Mbps
4 txpower Transmission power 20 dBm
5 mtu Maximum Transmission unit (256, 512, 1024, 1280, 1500) bytes
6 txqueuelen Transmission Queue length (10, 50, 100, 500, 1000) bytes
7 qdisc Queuing discipline pfifo, bfifo, pfifo_fast
8 ipfrag_low_thresh IP fragmentation lower threshold (0.25 0.5 0.75 1)
9 ipfrag_high_thresh IP fragmentation higher threshold (16384, 65536, 262144, 1048576, 4194304) bytes
10 udp_rmem_min Minimum UDP receive memory (0.019231 0.1 0.5)
11 rmem_default Default receive memory (0 0.25 0.5 0.75 1)
12 rmem_max Maximum receive memory (2304, 10418, 47105, 212992) bytes
13 udp_wmem_min Minimum UDP write memory (0.019231 0.1 0.5)
PHY IP UDP APP
Factors and Ranges
14
# Factor Description Range
14 wmem_default Default write memory (0 0.25 0.5 0.75 1)
15 wmem_max Maximum write memory (4608, 16537, 59349, 212992) bytes
16 udp_mem_min Minimum UDP memory (0.25 0.5 0.75)
17 udp_mem_pressure Pressure UDP memory (0 0.33338 0.5 0.75 1)
18 udp_mem_max Maximum UDP memory (95, 949, 9490, 94896) bytes
19 ROHC RObust Header Compression OFF
20 codec Audio codec type Opus, speex
21 codecBitrate Audio codec bitrate (8 16 24 32) kbps
22 FrameLengthAggregation Audio frame length aggregation (20 40 60) msec
23 interferenceCOR Interference Channel Occupancy Ratio 10%, 50%, 100%
24 sensing Background sensing activitity ON, OFF
PHY IP UDP APP
Factors and Ranges
15
Total number of experiments (with 24 factors)l = 6*5*5*5*5*3*4*5*3*4*4*3*5*4*3*5*4*2*4*3*3*2l = 5.59872×10¹² experimentsl = 20 M years (if 2 min. / experiment)l
Locating Array experimentsl = 109 experimentsl = less then 4 hours
Design space Range
16
Transmission Exposure Measures the electromagnetic exposure as felt by a receiver in SAR
(Specific Absorption Rate) units
Mean Opinion Score (MOS) Measures the audio quality of the speaker after the encoding and
transmission path
Experiment Outputs
17
Experimentation Challenges
• Interference generation using USRP does not work consistently
• Clients loose connection to Access point (speaker) when TxPower is reduced
• There used to be a bug inside the Speex decoder• Continuously bringing the Wi-Fi interface ON and OFF
sometime makes a node non-responding• Applying the LA factors on the system sometimes fail• Since the experiment takes very long time, watching
for a failed experiment is time taking• Singe output criteria required reports from multiple
sources (some devices might fail, other might not)• Variable outcomes (time dependent behavior)
THANK YOU FOR YOUR TIME
?Questions