as Testing of End-User Quality ofNetwork-Based...

North Carolina State University_____________________________________________CSC

NCSU__________________________________________________Department of Computer Science - 1

Dependability Benchmarking ofNetwork-Based Applications Viewed

as Testing of End-User Quality ofService

Mladen A. Vouk

http://renoir.csc.ncsu.edu/Faculty/Vouk



Mladen A. VoukProfessor

Tel: 919-515-7886 , FAX: [email protected]

Software Engineering LaboratoryComputer-Based Education

LaboratoryMultimedia Laboratory

Center for Advanced Computingand Communications

Department of Computer Science,Box 8206

North Carolina State UniversityRaleigh, NC 27695, U.S.A.



Support

•NSF•NCSU CACC, and other NCSU

sources•IBM•Alcatel•Bell South



End-User: Can I “Depend” onthis application, system, net…to support in an appliance-like

fashion what I do?

...



Key Trick

•Eliminate or standardize “non-essential” elements of theenvironment in which the softwareoperates

•Build a “standard” benchmarkingenvironment.

•Specific example - network-basedsoftware applications.



Questions

•What is an End-User view?•What is a workflow-based view

(scenarios)?•What is an appliance-based view?•What is QoS?•What is End-User QoS?•What is an SLA and SLS?

•Some answers and examples ...



End-User Quality Hyper-Surface

Availabilityof System

Response

Throughput “Ease of Use”Appliance-Like

Security

Support

Availabilityof Content

User needs andFault-Tolerance?

Services, Functions, ...



Framework:Benchmark + User Elements

SLA/SLS

Application

Middleware

OS

Com Protocols

End-User WorkflowsE.g., Professor,University, ISP ...

“Wire” Infrastructure

E.g., IP

E.g., SONET, ATM, “Light” ..

E.g., problem solvingenvironment, DB

E.g., CORBA-based



“Standard” Elements +User’s Elements

•User benchmarks (profile, whatconstitues a failure for this user,etc.)

•SLA/SLS benchmarks•Application benchmarks•Middleware benchmarks•OS benchmarks•Network and Infrastructure

benchmarks



Questions






Network-Based Education

•Modern learning environments areintegrated collections ofcooperating workflows, programs,tools, clients, and intelligentagents.

•They are distributed.•User satisfaction

is critical.



(Education) Workflows

•A series of (user-level) structuredactivities and computations that arisein education, training and learning,in general.



Cooperating Workflows•Education workflows are expected

to coexist, cooperate and evenmeld with other user workflows(e.g., business workflows, scientificworkflows, legislative workflows).

•As such they must supportcompatible interfaces andexpectations aboutquality of the delivered services.



“Horizontal” Workflows

Workflow Integration

Legal

Research

AssessmentBusiness

Education

Many Other Issues

Quality of ServiceNeeds

USERS



Operational Profile

Skill: S/W Expert 10

Application Package Developer (Author) 103Skill: Pedagogical and Content Expert

Package Selector (Instructor)Needed Skill: Knowledge of User Needs

Naive User

Needed Skill:Can Find Terminal

107

105

Support andGeneration ofInformation

Feedback and ErrorReports

System Software Developer

Advanced User

Needed Skill:Uses SuperComputers



Flow ModesSynchronousDelivery

AsynchronousDelivery

SynchronousCapture

Live1 or 2-wayinteractions

Live recording,On-demandplayback

AsynchronousCapture

“Tapes,”“Live-like”broadcast

“Tapes,”On-demandplayback



Major Support Modes•System Development•Courseware Development and

Management - Author Mode•Curriculum Development and

Management - Instructor Mode•Auto-Adaptive Teaching•Centralized Administration and Control•Local and Remote Storage and Processing•End-User Quality of Service Engines•Networking (the “Glue”)



Interaction/DeliveryModels

•TV model– Full motion video (e.g., MPEG2)– Possible two-way interaction– Usually high overhead

•Data model– Text, graphics, voice, possibly

compressed video– Interaction and collaboration– Low to medium overhead



When Audio Only?

•In many situations about 80-95%of relevant information is capturedthrough text, voice, and low-bandwidth interactions andanimations.

•Full motion video approach may beneeded only infrequently, butwhen it is needed it is essential.



When Video?•Interactions dependent on

interpretation of facial expressions,body language, color, ...

•Very complex moves, demos, ...•Exploratory data analysis and model

building•Lack of knowledge of exact learning

model•A r t•“Social” interactions• . . .



Constraints

•User Needs•Human capabilities•Technology•Financial limitations•Timely delivery•Social•Other ...



Matching•Student and system

capabilities must be explicitlymatched.

•In the future educationworkflows will need to at leastmatch that of other workflowswith which they interact.

• We use existing information aboutother workflows to estimate user-acceptable bounds for ET-Workflowsupport systems (e.g., end-user QoSmodels) .



Human Capacity•User interaction and ability to absorb

knowledge is a strong function ofcomputer-human interface (CHI), e.g. usercannot extract new information (learn and remember) fasterthan about 20 bits/second (i.e., differentiate among ~1,000,000"symbols" each second).

•Effective information transfer rates mayrequire different sets of "symbols" andpresentation rates (from several thousand bitsper second to many megabits per second).



Questions






Appliance-Like

•Should not impact existingworkflows in terms of attentionand additional operationalworkload - aid not distract (e.g., aseasy to use a whiteboard or ablackboard)

•Seamless integration of user-worklfows and technology -business model, effort-model, end-user fault-tolerance



Questions






Quality of Service (QoS)•“Classical” (network-based)

– Throughput, delay, loss, jitter

•Extended (user-based)– Reliability & Availability– End-User Response Time– Scalability– Usability– Functionality, Interoperability– SLA support, Security...– Other ...



End-User QoS Hyper-Surface

Reliability+ Downtime

Response

ThroughputJitter



Networks

Platforms &User Facilities

User Needs

Secu

rity

Qua

lity

of S

ervi

ceO

ther

?

Ver

tica

l Int

egra

tion

EducationWorkflows

Vertical Flows



Reliability andAvailability

•Once a user starts a ONEHOUR session, theprobability gettingthrough that sessionwithout any problemsmust be above 0.95

•Criterion is based on theNovaNET experiences.



UnAvailability

1 0 08 06 04 02 00.0001

.001

.01

.1

1

NovaNET (1988 - 1995)

Sys

tem

UnA

vaila

bilit

y

System Usage (Months)

Average

Instantaneous Model



Response Times•End-to-end response delay can

be a big problem in an educationenvironment.

•Synchronous end-to-endinteraction (round-trip) delaysthat consistently exceed about250 ms are often unacceptablewhen the interaction isconducted in the key-strokemode.



Network Delays (Summer ‘95)

Probability that Response Time is Network Traffic

LoadGood Acceptable Poor*

NCSU Low 0 . 9 9 6 3 0 . 0 0 2 0 0 . 0 0 1 7 Campus Medium 0 . 9 8 8 9 0 . 0 0 5 4 0 . 0 0 5 7

High 0 . 9 5 6 6 0 . 0 3 5 6 0 . 0 0 7 8Low 0 . 9 6 8 2 0 . 0 1 7 6 0 . 0 1 4 2

Internet Medium 0 . 9 5 0 2 0 . 0 1 3 0 0 . 0 3 6 8High 0 . 7 1 8 7 0 . 0 4 5 8 0 . 2 3 5 5

(*) Includes lost packets.



Throughput

•Principal driver is the problemsolving workflow.

•Usually takes one of the two forms•TV Model•Data Model



Questions






Service Level Agreement/Specs(How do we fulfill the following scenario?)

Actualcustomerinput

Networkreaction

ContracedSLA

ContractedClass

600 kbps800 kbps fromsilver

600 kbps

600 kbps fromeconomy

2 Mbps Super (EF)

3.8 Mbps 3 Mbps 3 Mbps Silver (AF)8 Mbps 1 Mbps 1 Mbps Economy (BE)



Service Level Agreement (2)

Actualcustomerinput

Networkreaction

Contracted SLA

ContractedClass

600 kbps600 kbps1.4 Mbpsfrom silver

2 Mbps Super (EF)

2.4 Mpbsfrom silver

3.8 Mbps

600 kbpsfromeconomy

3 Mpbs Silver (AF)

8 Mbps 1 Mpbs 1 Mbps Economy (BE)



Service Level Specs - How to implement?Do I need to/Can I Bid/Auction for Resources?

1

2

3

4

Example: WLS+MPEG2+HDTV (needs many “thin guaranteed streamswith dynamically allocated 3 to 10 Mbps streams”) + RSVP?+ COPS-based policy services ++DiffServ edges (Replace the WRR scheduler at node 2 with HOL scheduler & study its effect?).+ MPLS “cloud” (what about change traffic?)

MPLS-capable?



Issues

1

2

3

4

MPLS-capable

A B

GPDP

PEP

PDP

PDPPDP



Another Scenario• An ISP h as Poi nts o f Presen ce (POPs) at 4 0 cities i n t he c ou nt r y an d

a backb one p resence at 5 cities. Due t o t he h igh g r ow t h rate, it h a sbeen d ecid ed t o h ave a 4 n od e LAN backbon e at each POP th roughwh ich th ere is a lso access to th e ISP backbon e. Each POP h ostsserver s, p rovid es d ia l-in access to u ser s, an d is con n ected tor outes/access n odes fr om several t owns. T he ISP w o ul d like to h avea n RSVP/COPS /DiffServ/ MPLS solu tion at th e access n od es an d th ec ore n od e s. Som e of the r e qu i r ements are t hat n o e dge n ode or l inksh ou ld be load ed to m ore th an 75% of its cap acity, fa ilu re of asin gle n od e in th e n etwork sh ou ld n ot d isru p t an y u ser . Also, top r ovide a h ighly r e liable backbo ne, t wo switches are to be installe dat each backbon e site. Th e ISP p lan s to lease tw o OC3 lin ks from aPOP t o the associat e d b ackbo ne site a nd o ne OC12 l ink be twe e n t h eba ckbo n e sites.

• What is the recommended topology to maximizenetwork capacity and revenue?



(Edge nodes)

ISP Backbone

POP Backbone

POP-2 POP-40

POP BackbonePOP Backbone

city-1city-2

city-3

city-4city-5

POP-1



More on the Scenario

• Th e ISP est im ates th a t 5% of th e u ser t r affictr aver ses ou tsid e th e loca l POP. To p rovid ecom p etit ive Qu ality of Service, th e ISP p rojectsth at each ISP an d POP backbon e n od e sh ou ldsu p port 20% of change t r affic.

•How does it influence the projectedcapacity? Do I need to bid for just-in-time resources? Can I auctionsome of mine?



Hierarchical Model

Core

Edge nodes

Level-Nnodes



NC State Test-Bedfor End-User Oriented

Benchmarking of QoS forNetwork-Based I2

Applications



NCState.Net• World-class campus production network

– Fault-Tolerant (redundant, managed)– 4 Gbps backbone (plans to go to 10 Gbps)– 2 Gbps routing drops (fiber)– 1 Gbps building drops (fiber)– 100 Mbps floor drops (fiber)– 10/100 desktop (copper)– Nomadic computing + wireless– Being hardened to provide 6 to 7 9’s

availability in order to support critical end-users application (e.g., VoIP)



SiSiSiSi SiSi SiSi

4 X Gigabit Enet

InternetInternet2

NCNINCREN

RemoteAccess

Off CampusFacilities

OC48/OC3

T1/ISDN

POEEast Campus

32Interfaces

HLBNorth Campus

24Interfaces

SMDFSouth Campus

38Interfaces

CMDFCent. Campus

14Interfaces

1 Gig 1 GigFault-TolerantNomadicNet

Current NC State Production Backbone



Includes DiffServ andPolicy-Based Routing

DS1

DS2 DS3

GT

NCSU

I2

ProtectedFlow



NC State - Alcatel LinkAlcatel

NC State

I2

VideoVoiceData

VideoVoiceData







E-Lab

• Current: muti-location distributed facility

• Future: Large centralized + distributed labs

MultidisciplinaryShow-CaseFacility

NCState.NetR&D

DemoRoom

CollaboratoryEducation, Research, Service Application

R&D Labs

CNLTesting andAnalysisServices

NetDegreeProjectLab

NetDegreeTeachingLab

NetDegreeHi-TechClassroom Industrial Partners,

Other NC State Colleges,UNC System,Community Colleges,Schools, etc.



Australia



Televator









NC State WLS-AssistedVideo-Access-Node Classroom

WebLectureSystem

WebLectureSystem

WebLectureSystem

WebLectureSystem

WebLectureSystem

WebLectureSystem

WebLectureSystem

WebLectureSystem

WebLectureSystem

WebLectureSystem

WebLectureSystem

WebLectureSystem

ATM (Internet II)

Internet Wireless

Instructor

Phone

Modem

MPEG-2+ Web+QoS

MPEG-1 orRealVideoRealAudioWeb

RealAudio (Video), Web



End-User QoS Probes

•Standardized EU-QoS probes in allkey elements of NCState network.

•Constant/dynamic QoS reports(wrt. Benchmark metrics)transmitted to Network-OperationsCenter

•Matching of EU-QoS probe resultswith end-user SLAs and SLSs

•Corrective actions



SiSiSiSi SiSi SiSi

4 X Gigabit Enet

InternetInternet2

NCNINCREN

RemoteAccess

Off CampusFacilities

OC48/OC3

T1/ISDN

POEEast Campus

32Interfaces

HLBNorth Campus

24Interfaces

SMDFSouth Campus

38Interfaces

CMDFCent. Campus

14Interfaces

1 Gig 1 GigFault-TolerantNomadicNet

Current NC State Production Backbone

SiSiSiSi

SiSi SiSi



Benchmark Probes

SLA/SLS

Application

Middleware

OS

Com Protocols

End-User WorkflowsE.g., Professor,University, ISP ...

“Wire” Infrastructure

IP, TCP, UDP

Ethernet, ATM, “light”,DiffServ

Linux, W2000, CiscoR OS

Currently Open

Being constructed

Date post:	03-Oct-2020
Category:	Documents
Upload:	others
View:	0 times
Download:	0 times

as Testing of End-User Quality ofNetwork-Based...

Documents