SITAR: A Scalable Intrusion Tolerant Architecture for Distributed Services

Dr. Feiyi WangAdvanced Networking Research

MCNC Research Triangle Park, NC

Dr. Bahrat MadanElectrical Engineer Department

Duke University, NC

8/19/2002 OASIS 2002 Summer PI Meeting 2

SITAR Community

Service Oriented Solution

ClientsClients

ServersServers

ServersServers

8/19/2002 OASIS 2002 Summer PI Meeting 3

JavaSpaceTM Based Organization

Incoming Requests

ProxyServer

ProxyServer

Acceptance Monitor

Acceptance Monitor

AdaptiveReconfiguration

AdaptiveReconfiguration

BallotMonitor

BallotMonitor

ServerWrapper

ServerWrapper

COTS Servers

AuditMonitor

AuditMonitor

8/19/2002 OASIS 2002 Summer PI Meeting 4

Progress Status

Performance Study (JavaSpace & SITAR implementation)

Adaptive Reconfiguration Simulator

Performance & Security Modeling

Dynamic Content Checking

8/19/2002 OASIS 2002 Summer PI Meeting 5

Performance Impact of Entry Size

1 2 3 4 50

1

2

3

4

5

6

7

8

9

write

read

take

entry size (K)

0 10 20 30 40 50 60 70 80 90 1000

10

20

30

40

50

60

70

80

90

readtakewrite

entry size (K)

time

(m

s)tim

e (

ms)

8/19/2002 OASIS 2002 Summer PI Meeting 6

Impact of Number of Entries

4901

4903

4905

4907

4909

4911

4913

4915

4917

4919

4921

4923

4925

4927

4929

4931

4933

4935

4937

4939

4941

4943

4945

4947

4949

4951

4953

4955

4957

4959

49

61

4963

4965

4967

4969

4971

4973

4975

4977

4979

4981

4983

4985

4987

4989

4991

4993

4995

4997

4999

05

10

1520

2530354045505560

65

write

readtake

number of entries

time

(m

s)

8/19/2002 OASIS 2002 Summer PI Meeting 7

Impact to SITAR System

1.AcptWOR wrt

2.AcptWOR taken

3.ClientReq read

4.AcptReq wrt

5.ClientRes wrt

6.ClientRes read

7.ValRep wrt

8.ValRep read

9.ChosenRes wrt

0

25

50

75

100

125

150

175

200

225

181.99

126.13

207.12

23.64

66.3352.66

86.11

58.99

140.57

SITAR Operational Steps: A Break Down

time

(ms)

8/19/2002 OASIS 2002 Summer PI Meeting 8

Impact to SITAR System (Persistent Worker)

1.AcptWOR wrt

2.AcptWOR taken

3-1. ClReq (before)

3-2. ClReq (after)

4.AcptReq wrt

5.ClientRes wrt

6.ClientRes read

7.ValRep wrt

8.ValRep read

9.ChosenRes wrt

0

20

40

60

80

100

120

140

160

180

200

181.44

114.39

35.46

69.33

30.93

94.68

6068.01 68.03

151.77

time

(m

s)

SITAR Operational Steps: A Break Down

Persistent WorkerImproves performance

Adaptive Reconfiguration Simulation

8/19/2002 OASIS 2002 Summer PI Meeting 10

ARM Simulation

ARM Goals: maximum performance (low threat); maximum availability (high threat)

Simulation Goal: examine ARM model and algorithm in a controlled environment

Client WMPM BMAM

Server

Request

Response

8/19/2002 OASIS 2002 Summer PI Meeting 11

ARM Model

AssessmentAssessment

Allocation Decision

Allocation Decision

EvaluationEvaluation

Run time updates

Anomaly Events

Reconfiguration Directives

Application Models

Evaluation Results

New ResourceAllocations

8/19/2002 OASIS 2002 Summer PI Meeting 12

Simulation Framework

Key definition Resource Container (RC) Resources (RES)

Key events Threat level: manual injection or dynamically

changes by trigger events Trigger processing (performance, security) Status Change

8/19/2002 OASIS 2002 Summer PI Meeting 13

Configurations

Configuration 0 : 1 PS, 1 WM, 1 AM, 1 BM 1 : 1 PS, 3 WM, 1 AM, 1 BM (not for real system) 2 : 1 PS, 3 WM, 2 AM, 3 BM (not for real system) 3 : 1 PS, 2 WM, 2 AM, 1 BM (alternative to 1 & 2) 4 : 1 PS, 3 WM, 3 AM, 3 BM

Measurement: Connection Processing

0 1 3 5 6 2 7 4 9 10 8 11 13 12 15 14 18 16 17 19 20 22 21 24 27 25 26 23 28 29 31 30 34 32 33 35 36 37 38 39 40 41 43 42 44 47 46 45 49

0.00

2.50

5.00

7.50

10.00

12.50

15.00

17.50

20.00

22.50

25.00

27.50

First 50 Connections (TL0, 100 @ 5)

Processing Time

Elapsed Time

Connection ID Number

Sim

ula

tio

n T

ime

Un

its

1 0 4 3 6 2 8 5 9 12 7 11 14 10 13 17 20 15 18 16 19 22 25 21 24 26 29 23 30 28 27 33 31 36 35 32 34 37 38 42 39 41 43 44 40 46 47 45 48

0

5

10

15

20

25

30

35

40

45

First 50 Connections (TL4, 100 @ 5)

Processing Time

Elapsed Time

Connection ID Number

Sim

ula

tio

n T

ime

Un

its

Median (Processing Time: 6.62 Median (Elapsed Time): 12.723.87 3.87Standard Dev (Processing Time): Standard Dev (Elapsed Time):

Median (Processing Time): 21.72 Median (Elapsed Time): 27.726.86 6.86Standard Dev (Processing Time): Standard Dev (Elapsed Time):

Measurement: Threat Level Change

Reconfiguration Period: 8 time units (52 - 60)

Reconfiguration Period: 12 time units (52 - 64)

0 1 3 5 4 2 7 9 6 10 8 11 17 16 12 18 14 21 22 19 20 24 25 15 23 26 28 27 29 30 31 33 36 32 39 35 34 37 40 38 43 46 42 41 44 47 49 48 45

0

5

10

15

20

25

30

35

40

45

50

Manual Threat Level Change 0 to 2 (50 @ 5 on 10)

Processing Time

Elapsed Time

Connection ID Number

Sim

ula

tio

n T

ime

Un

its

0 2 1 4 3 6 10 15 17 16 18 12 19 5 13 20 21 14 24 22 23 26 25 29 30 28 31 27 32 34 36 33 35 38 37 40 39 41 43 42 47 44 45 46 49 48

0

5

10

15

20

25

30

35

40

45

50

55

60

Manual Threat Level Change 2 to 0 (50 @ 5 on 10)

Processing Time

Elapsed Time

Connection ID Number

Sim

ula

tio

n T

ime

Un

its

Measurement: Initial Conditions

4 2 5 6 3 0 9 10 8 11 12 13 14 15 16 18 17 19 20 22 21 24 23 27 26 29 31 30 33 35 34 36 38 39 37 41 42 43 44 47 48 45 49

0

5

10

15

20

25

30

35

40

Initial Conditions - Module Resource Pool 0 (TL0)

Process Time

Elapsed Time

Connection ID Number

Sim

ula

tio

n T

ime

Un

its

0 2 3 4 1 5 6 9 7 8 10 11 13 12 15 14 16 17 18 19 21 22 20 24 23 26 25 28 29 27 31 30 33 34 32 35 37 36 40 38 39 41 42 43 44 45 46 47 48 49

0

2.5

5

7.5

10

12.5

15

17.5

20

22.5

25

27.5

Initial Conditions - Module Resource Pool 5 (TL0)

Process Time

Elapsed Time

Connection ID Number

Sim

ula

tio

n T

ime

Un

its

Median (Processing Time): 7.58 Median (Elapsed Time): 13.583.57 5.03Standard Dev (Processing Time): Standard Dev (Processing Time):

Median (Processing Time) : 8.34 Median (Elapsed Time): 14.343.64 3.64Standard Dev (Processing Time): Standard Dev (Processing Time):

8/19/2002 OASIS 2002 Summer PI Meeting 17

Lesson Learned

Ripple effect of reconfiguration

Sensitivity of reconfiguration parameter

Minimize reconfiguration period

Care must be taken to decide when reconfiguration happens

Performance Modeling

8/19/2002 OASIS 2002 Summer PI Meeting 19

Performance + Security Modeling

Pure Performance

Performance in the presence of security threats and security failures

Use of analytical models Stochastic Reward Net (SRN)

Parameterization of these models Transition rates, branching probabilities,

distributions etc

20

Event timeline for single request

CLIENT

PM

ARM

AM

WM

BM

WorkerInventoryRequest

ClientRequest

WorkOrderRequests

AcceptableRequestEntry

WorkInventoryResponse

ClientRequestClientResponse

ValidationReport

ChosenResponse

ClientResponse

8/19/2002 OASIS 2002 Summer PI Meeting 21

Performance Variables

Module delays PM_NewConn_WIReq ARM_WIReq_WIRsp PM_WIRsp_WOReq AM_WOReq_ARE WM_ARE_CRsp AM_CRsp_VRep BM_VRep_ChRsp PM_ChRsp_Clnt

JavaSpace delays JSD_PM_ARM_WIReq JSD_ARM_PM_WIRsp JSD_PM_AM_WOReq JSD_AM_WM_ARE JSD_WM_AM_CRsp JSD_AM_BM_VRep JSD_BM_PM_ChRsp

8/19/2002 OASIS 2002 Summer PI Meeting 22

Pure-performance SRN

jsw

td arm

spawn

jsw

td

w ait

woreq

req

dly

jsw

jsw

td am jsr jsw

td

jsw

w oa

w ait

wm jsr w ait

wm

jsw

td

w ait

val

jsw

td

vote

jsw

td

w ait

CReq

pw

pwm

bm jsr w ait

resp

val

T_assign

T_w

ireq

T_t

d_w

ireq

T_w

rite

T_w rite

T_w

rite

T_w rite

T_w rite

T_w rite

T_w rite

td

td

T_delay

T_reqinit

jsw

T_w

irsp

T_w rite

T_t

d_w

irsp

T_w

oreq

s

T_t

d_w

oreq

T_t

d_w

oreq

T_s

paw

n

T_t

d_w

oreq

T_i

nit

T_i

nit

T_i

nit

T_r

ead

T_r

ead

T_r

ead

T_req_val

T_td_are

T_cots_rsp

T_w

rite

T_td_crsp

T_rsp_val

T_t

d_vr

ep

T_v

ote

T_t

d_ch

rsp

T_w rite

T_w oas

8/19/2002 OASIS 2002 Summer PI Meeting 23

Combining performance and security model

Current Assumptions One COTS server and one AM

Security states in SITAR system UP GD (Graceful Degradation) F (Fail) FS (Fail Safe) UC (Unmasked Compromise) MC (Masked Compromise)

P_pxy_Q P_pxy T_pxy

P_pxy_test P_pxy_test_Q

P_am_Q P_am T_am

P_am_Det

P_fire

P_sw

T_sw

P_am_test_Q

P_timeout

P_fire

P_am_test

T_am_testP_am_test_det

P_fire

P_bad_res

P_fireGD

F

FS

MC

UC

FP_fire

T3

T4

T5

T6

T7

T8

P_up

Performance SRN

1. Security model SRN

2. Places denote security states

3. Transitions model changes in security States.

4. Two models are coupled by the enabling functions.

Single COTS: Performance + security

8/19/2002 OASIS 2002 Summer PI Meeting 25

Multiple COTS

Conventional SRN: problem of token matching.

Colored Petri net: Different requests are assigned different color.

Modify SRN to include token color.

‘Place’ incorporates a FCFS queue that makes possible to combine tokens from the same request.

Dynamic Content Verification

8/19/2002 OASIS 2002 Summer PI Meeting 27

Dynamic Content Verification: What About HTML?

A mixture of tags for (1) encoding of format; (2) encoding of layout; (3) encoding of types of information content

While HTML can be described using a DTD, the vast majority of HTML on the Web is invalid

Fundamental limitation: lack of reliable, semantic encoding

8/19/2002 OASIS 2002 Summer PI Meeting 28

Dynamic Content Verification: Alternative Choice

Options Comparison to HTML

DHTML Dynamic HTML: A combination of HTML 4.0, Style Sheets and JavaScript.

XHTML 1) XHTML elements must be properly nested

2) XHTML document must be well formatted

3) Tag name must be in lower case

4) All XHTML elements must be closed

Zope Page Template

1) Provide a clean separation of data, logic and presentation

2) Allow you to treat XML contents as objects

XML/XLST 1) XML is not an HTML replacement; HTML was designed to display data and to focus on how data looks

2) XML was designed to describe data and to focus on what data is by using Document Type Definition (DTD) or XML Schema

8/19/2002 OASIS 2002 Summer PI Meeting 29

Dynamic Content Verification: Challenges

1. Confidentiality -- No one else can access or copy the data

2. Integrity -- The data isn't altered as it goes from the sender to the receiver

3. Authentication -- The document actually came from the purported sender

4. Nonrepudiability -- The sender of the data cannot deny that they sent it, and they cannot deny the contents of the data

5. Availability/Tolerance Capability – Continued service even when servers partially failed

SSL

XML

Digital

signature

SITAR

8/19/2002 OASIS 2002 Summer PI Meeting 30

Special Considerations

Differences that are not semantically significant Order of elements The amount of white space between attributes Whether or not that default values of attribute

are included in the source document

Solution: using Canonicalizer for preprocessing before digitally signed

8/19/2002 OASIS 2002 Summer PI Meeting 31

Modularization of SITAR

Capability Profile

Capability Profile

Running Profile

Running Profile

SITAR Modules

SITAR Modules

Module Configuration

Module Configuration

Local Policy(GUI Control)

Local Policy(GUI Control)

8/19/2002 OASIS 2002 Summer PI Meeting 32

Current Status Summary

Delivered final architecture reportDeveloped share-space based framework where different types of components, multiple instances of components can collaborate and provide SITAR services and demonstrated basic functionalities Presented 4 papers/fast abstracts to DSN’02 and its companion intrusion tolerance workshop

8/19/2002 OASIS 2002 Summer PI Meeting 33

Future Work

Near term Refinement of architecture Intelligent adaptive reconfiguration in action Performance evaluation and improvement

Next Phase: More intelligent ARM algorithm Strengthen space-based security Demonstrate support for other type of services

Thank You!