Network Computing Laboratory Experiment Tutorial Network Computing Lab. 2005. 5. 19.

Network Computing Laboratory

Experiment Tutorial

Network Computing Lab.

2005. 5. 19

Network Computing Laboratory | 2

Korea Advanced Institute of Science and Technology

Table of contents

What is experiment? What is experiment design? Case study - CIGMA Three keys for experiment Step by step for experiment



What is experiment?

Proof of your proposal Convince people that your goal is well achieved Make your contribution to be clear and true Support that your technical problems are well solved



What is experiment design?

Experiment design 실험 계획표 작성 (Pre-report)

실험 목적1. Decide key aspects you want to show2. Choose the proper verification method

Demo, implementation and measurement, simulation, and analysis3. Identify metrics (dependent variables)4. Classify parameters (independent variables)

실험 환경 설정1. Decide key system components to be implemented

Plan to test your implemented component2. Prepare workload (benchmark, trace, synthetic workload)3. Plan to prepare test-bed (machine, OS, software, network link, simulation tools)

실험 예상1. Draw the expected graphs which you want to show (imaginary experiment w/ paper and

pencil)

Assignment Presentation slides of experiment design for your own experiment



Verification methods

1. Analysis 예 ) Quick sort = O(N log N)) Deterministic and Statistical Analysis

2. Simulation 예 ) Network protocol simulation (topology, bandwidth, link delay) 실험환경과 시스템을 모두 모델링 해서 검증하는 방법 Simulation 은 언제 하는가 ??

변수가 많고 시나리오가 복잡해서 분석이 불가능 할 때 구현 및 측정하기가 어렵거나 불가능 할 때 구현 및 측정하기 앞서서 미리 메커니즘 등을 검증하고 싶을 때

장점 : Easy to control, 단점 : Unrealistic

3. Implementation and measurement 예 ) 웹 서버 구현 후 , 초당 처리 수 측정



Case study – CIGMA: Motivation

CIGMA: aCtive Inventory service in Global e-MArket based on efficient catalog management

trave

l infor

mation

Internet

Travel ConsolidatorService

client A

cruise sites c1,c2,...

car-rental sitesr1,r2,...

travel information

hotel sitesh1,h2,...

web sitesd1,d2,..

vacation sitesv1,v2,...

airline sitesa1,a2,...

H

C

D

V

R

client

Highly dynamic e-commerce environment Constantly changing sales

item information Flooding information: a lot of

shopping sites “I want to know how I can

purchase goods with the best condition!!”



Case study – CIGMA: Goal

CIGMA: aCtive Inventory service in Global e-MArket based on efficient catalog management

trave

l infor

mation

Internet

Travel ConsolidatorService

client A

cruise sites c1,c2,...

car-rental sitesr1,r2,...

travel information

hotel sitesh1,h2,...

web sitesd1,d2,..

vacation sitesv1,v2,...

airline sitesa1,a2,...

H

C

D

V

R

client

CIGMA service and systemProvide aggregated catalog information from a lot of sitesAssure up-to-date informationServe large number of customers with fast response



Case study – CIGMA: Design and Implementation

Communication Module

update event+ modified data

ContentConverter

EventListener

invalidation msg+ modified data

Java Virtual Machine

ContentConversion

Table

updaterequest

(1)

(3) (4) (5)

(6)

ContentProvider'sDatabase

Triggers

EventReporter

Catalogtable (2) modified

data

CPMContent ProviderHandler

Content Provider Interface

invalidation msg+ modified data

(8)

(9)

Logger&

Monitor

Logging messageto SDBM

CacheUpdaterupdate

request

(7)

MCM

CPW

WSSLogic

Web ServerWeb Application Server

customer requests

contentretrieval

swappingmanager

Disk Storage

Memory Cache(Main Memory)



Case study – CIGMA: Experiment

Questions!!

What kinds of experiments do you want to do?

Why do you want to do those experiments?




What and Why - “Remind your goal and contribution” CIGMA should deal with a lot of shopping sites. So it should process

a large number of update requests from multiple sites Catalog update throughput

CIGMA should assure up-to-date information. So update delay should be small Catalog update delay from shopping site to CIGMA

CIGMA should handle a lot of customers requests. So it should process a number of customer requests at the same time Catalog browse throughput

CIGMA receives update requests and customer requests concurrently. So it should process both of requests Mixed throughput




One more Question!!

How do you do those experiments?




What to do after you decide metrics such as Catalog update throughput Catalog update delay Catalog browse throughput Mixed throughput

What do you need in order to measure those metrics?? Do you have machines and software needed for implementation? Do you have your implementation which is working well? Do you have something to generate the workload?

What kinds of parameters affect the experimental results?? What is your expectation about your experimental results??




What do you need in order to measure those metrics??

Do you have machines and software needed for implementation?

Prepare your machines and software.

Pentium III 1GHz, 2GB memory machine, 100M LAN Link, Linux OS, Tomcat server, FastDB Main Memory DB …

Do you have your implementation which is working well?

Implement your system’s core functionality to be verified and test your system.




What do you need in order to measure those metrics??

Do you have something to generate the workload?

Use existing benchmark tools. If not available, implement your own workload generator.

Browse throughput: Httperf benchmark used (c.f. TPC-W, SPEC-Web)

Update throughput: Update request generator implemented Update delay: WAN emulation tool used




What kinds of parameters affect the experimental results?? Find possible parameters Plan to choose and vary just one parameter at a time

What is your expectation about your experimental results?? Draw your expected graphs which show not only a

pattern of result (linearly increasing, logarithmically increasing, decreasing, or keeping constant)but also specific numbers

Help you to analyze your measured results Calculate your expected results mathematically




And then??

Experiment!! Experiment!! Experiment!! Analyze your results Verify your expectations Modify your parameters, implementations, even design




Example result graphs



Three keys for experiment

What Why How




What What do you want to show to people?

Strongly connected to what you propose as your system goals and your contributions

Quantitative

Q. What kinds of metrics can you use? E.g. Throughput, latency, execution time, storage usage, # of Disk I/O,

# of lookups, % of failed lookups, BW consumption, etc. vs. Qualitative

Q. What are core functionalities to show? E.g. Demo, screenshots




Why Why do you want to show it to people?

Also strongly connected to your system goals and your contributions

Q. Does it really prove your contribution?

Q. Does it show your achievement more effectively than others?




How How can you show it to people?

Experiments using real implementation and benchmark tool Simulations Demos



Step by step for experiment

Determining What and Why Setting experimental environment

Implementation Benchmark tools

While (hypothesis verified to be true || your approach turn to be wrong || give-up) {

Making hypothesis Expect results and draw your expected graphs

Verifying hypothesis Do experiment and get results Analyze results

Giving feedback from your analysis To experimental parameters, system tuning To your implementation To your system design

} Explaining your results and lessons if you succeed

Date post:	02-Jan-2016
Category:	Documents
Upload:	mary-bryant
View:	213 times
Download:	0 times

Network Computing Laboratory Experiment Tutorial Network Computing Lab. 2005. 5. 19.

Documents