Validation Methodologies for Netcentric Warfare Simulations Wesley N. Colley, Ph.D. Center for...

Validation Methodologies for Netcentric Warfare Simulations

Wesley N. Colley, Ph.D.Center for Modeling, Simulation and Analysis

University of Alabama in Huntsville

CMSA SPARTA

Jared Biggs Chris Noller

Randy Harrell Jeff Roberts

Jim Walsh

Sponsor: Naval Air Warfare Center, Weapons Division

Validation for Netcentric Warfare Simulations

Presentation outline

• Motivation

• Validation Context

• Quantitative Validation

• InterTEC Joint Fires Exercise


Problem Statement / Project Focus

• Netcentric Warfare is new– Classic “face” validation by battle-experienced

SMEs not generally possible

• Networks are classically difficult to characterize– Nonlinear/brittle behaviors mean much more

careful quantitative analysis is necessary

• Focus: Develop quantitative validation techniques for NCW simulations.


TestComponent

PostProcessing

MetricsGathering

Real WorldTest Data

DesignMOP/MOE

MetricsAnalysis

V&VAnalysis

met

rics

responses

outputs

sta

tistic

s

experiment design

MathematicalV&V

stim

uli

validmetricvalues

Project Overview


Scope of Validation• Our Goal: Validate the NCW-specific aspects of systems (in

constructive simulation)

• Two possibilities:– Inherent NCW system (Link-16)

– NCW components of non-NCW system (Link-16 on F35)

NetcentricC2 Environment NCW System (e.g., Link-16)

Weapons System (e.g., F35)

Weapons System NCWInterface (F35 Link-16 Hardware)


NCW Effectiveness and Networks • NCW Goal: Full-spectrum information

dominancetimeliness

accuracyrelevance

InformationSuperiorityProduct

Easiest axis: timeliness“Simply” improve network

(Albert’s NCW, 2000)Network metric is thusTimeliness of messages


ServiceRegistry

Identity

InfrastructureInfrastructure

B2B Enterprise C2 Logicw/Distributed Services

L&

A

Pub/Sub

•Logon & Authenticate•Publish

•Sensor (SPY)•Distributed Sensor CPU•Fires (SM)•Alerts•PLI (Position)

•Subscribe•Threat•Environment•Blue Force•Combat ID (IFF)•ROE•Target ID•Alerts

Logon & Authenticate (L&A)

Subsc

ribe

Publis

h


•PLI (Position)•Subscribe

•Threat•Environment•Blue Force ID•Combat ID (IFF)•ROE•Alerts


•Sensor (MP-RTIP)•Forward Pass (SM)•PLI (Position)

•Subscribe•Hand-Off (SM)•Target ID (HRR)

Establishing ServicesEstablishing Services

What Does the Network Have to Do?

L&A

Pub/Sub

TrackService

AlertService

Publish Track

RC

V &

Pro

cess

Tra

ck/T

arge

t

Gener

ate A

lert/T

arge

t ID

RC

V A

lert/Target ID

RCV Alert/Target ID

Weapon Data Link

Weapon DeliveryWeapon Data Link

Weapon Provider

(Sensor to Shooter) (Sensor to Weapon)

Forward Pass R

cv

Enga

gem

ent w

/For

war

d Pa

ss

Delivering ServicesDelivering Services

DETECT

.


Network Composeability Table

Communications linkages

Dial-a-Comm Link Select the software programmable or networked radios connection and waveform

JTRS, VRC-99, EHF MDR, TC, Teleport, SHF/CA FCs, TCDL,MUOS

Compose the kind of connectivity and raw bandwidth you need to support your mission

Operational networkingDial-a-Network

Operational network formation to meet the force and mission needs (e.g. GIG-BE, TCS, JTF Warnet, EHF MDR)

GIG-BE, TCS, JTF Warnet, EHF MDR,

MUOS, TSAT

Compose the community of interests you need to support your mission

Information flow management

Dial-a-Precedence

Establish bandwidth allocation and priority for applications and key individuals

ADNS, BMAC QOS, TCS

Compose lanes with different speed limits and priority for your mission related data flows

Distributed information processing and storage

Dial-a-Computer

Establish roles of the computers that will support the mission, their interfaces to your FORCEnet, and the information managers who run them

NCES, XTCF, DJC2, OA, RAPIDS, CORBA, GRS, IT-21, NMCI, CAS, COWAN/CENTRIX

Compose your information management environment

Full spectrum military and civil affairs program applications

Dial-an-Application

Establish the operational cells and specific applications that will pubish information into the FORCEnet

GCCS-M, GCSS-M, OA, TMIP, IBV, RAMIS, NTCSS, TBMCS, ADOCS, JBMCS

Compose the sensing, planning, decision support, and weapons applications that will publish into the FORCEnet to support support your mission

Virtual collaborationDial-a-Meeting

Establish the collaborative environment that the participants in the mission use to coordinate actions and activities

GeoViz, IWS, DCTS, NetMeeting, VoIP, IP VTC

Compose the virtual rooms, participants, schedule, and battle rhythm

Functional, temporal, and geospacial visualization

Dial-a-GUI

Establish the standards for the form of presentation and FORCEnet subscription rules to be used to support your mission

GeoViz, WebCOP, K-WEB,

Compose the Alerting, Status Board, and COP views to be shared by the forceScads of systems with scads of uses


Recommendation• Use Missions × Means breakdown for validation

– Provides hierarchical decomposition of• Missions: the desired tasks or capabilities

• Means: the hardware and/or software assets available

– Successfully used by Petty, Colley for validation of JTLS in Terminal Fury ’05 (published at SIW Fall 2005)

• Select from broad NCW categories– Use M×M to drill down for specific systems

– Methodologies should be similar within broad categories


Missions × Means Example Ground Force Comm.

SecureConnection

Single Soldier

Receive

Voice Data Voice

GIG

JTR

S JTR

S L

AN

JTR

S r

adio

voic

eda

ta

Send

outg

oing

inco

min

g

voic

eda

talo

gon

band

wid

thal

loca

tion

Data

auth

en-

tica

te

= Evidence Item


Evidence Item• Element in M×M matrix where means

supports mission

• Each will have quantitative measures of performance (MOP)– Latency (e.g., time to authenticate connection)– Bandwidth consumed– Total packets sent/received– Compliance with protocol


Validation• Compare MOPs to known standards

– TCP/IP– JTRS specification– FORCEnet documentation, etc.

FORCEnet Architecture & Standards Volume I (Cont)

5.0 FORCEnet Functional Architecture5.1 DoD Architecture Framework Document

5.2 Mission Capability Packages

5.3 Communications and Networks

5.4 Intelligence, Surveillance, and Reconnaissance (ISR)

5.5 Distributed Services, Common Operational and Tactical

6.0 Implementation Reference Mission6.1 SEA STRIKE6.2 SEA SHIELD6.3 SEA BASE6.4 FORCEnet

AppendicesAppendix A AV-1

Appendix B ICD – FORCEnet Capabilities

Appendix C FORCEnet Service Category Definitions

Appendix D Allied and Coalition FORCEnet Architecture

Appendix E Human Systems Integration

Appendix F FORCEnet System Descriptions

Appendix G SP-21 Pillar POR Standards Compliance

Appendix H List of Acronyms

Appendix I References

Statement A: Approved for public release; distribution is unlimited (4/20/04)

F O R C E n e t A rc h ite c tu re & S ta n d a rd s V o lu m e I

T a b le o f C o n te n ts :1 .0 F O R C E n e t D e fin itio n

1 .1 F O R C E n e t O b je c tiv e -T ra ns form a tio n1 .2 A rc h ite c tu re O v e rv ie w

2 .0 B a c k g ro u n d2 .1 S e a P o w e r 21 P illa rs2 .2 F O R C E n e t P re c e p ts2 .3 F O R C E n e t A rc h ite c tu ra l S c op e2 .4 F O R C E n e t D e v e lo pm e n t P ro c e ss2 .5 F O R C E n e t A rc h ite c tu re Ap p ro a c h

3 .0 F O R C E n e t R e q u ire m e n ts3 .1 O p e ra tio n a l V ie w (O V -1 )3 .2 In itia l C a pa b ilit ie s D o c um e n t (IC D )3 .3 F le e t C a p a b ility N e e d s

4 .0 F O R C E n e t D riv e rs4 .1 D o D D riv e rs4 .2 N a v a l4 .3 C o m m erc ia l T ec h n o lo gy U p d a te4 .4 A llie d /C o a lit io n In te ro p e ra b ility

A r c h it e c t u r e & S ta n d a r d s

V o lu m e I

O p e r a t io n a l & S y s t e m s V ie w

O f f ic e o f t h e C h ie f E n g in e e r

S P A W A R 0 5

D is t r ib u t io n D : D is t r ib u t io n a u th o r i z e d to t h e D e p a rt m e n t o f D e f e n s e a n d U .S . D o D c o n t r a c t o rs o n l y ( O p e r a tio n a l U s e ) 0 3 N o v e m b e r 2 0 0 3 . O th e r re q u e s t s s h a l l b e r e fe r r e d t o t h e S p a c e a n d N a v a l W a r f a r e S y s t e m s C o m m a n d O f f i c e ( A r c h i t e c t u r e a n d S t a n d a rd s S P A W A R C o d e 0 5 2 ) o r P u b l i c A f f a i r s a n d C o r p o r a t e C o m m u n ic a t io n s O f f i c e ( S P A W A R 0 0 P ) .

“ D e s t ru c t io n N o t ic e ” - F o r u n c la s s i fi e d , l im i te d d is t r ib u tio n d o c u m e n t s , d e s t r o y b y a n y m e t h o d t h a t w i l l p r e v e n t d is c lo s u r e o f c o n t e n t s o r r e c o n s t r u c t io n o f th e d o c u m e n t . ”

F O R C E n e t

V e rs io n 1 .4 3 0 A p r il 2 0 0 4

S ta tem e nt A : A p p ro ve d fo r p u b lic re le a se ; d is tr ib u tio n is u n lim ite d (4 /2 0 /0 4 )


Quantitative Analysis• Experimental Design

– Flex system in statistically interesting ways

– Push simulations hard to identify problem areas

• Non-linear behaviors

– Networks are inherently susceptible to non-linear response

– Must create mathematical means of handling non-linearity

Network in non-linear regime

Mes

sage

Lat

ency

Runtime


Validation Testbed—NETE• Netcentric End-to-End Simulation• Based in Extend discrete event simulation

environment• Modeled NCW Elements:

– GIG BE, TSATs, Link-16– TADIL-J messages– AEGIS cruisers, FBX-T (Sea of Japan)

• Thread– DPRK launches missiles– Tracks formed, passed by GIG to STRATCOM


NETE Features• Realistic message processing• Link-16 message slotting, jitter• Latency computation• Threat tracks based on Lincoln Lab models



• < NETE Demo >

• Metric under test = TADIL-J latency from Link-16 hub in Sea of Japan back to STRATCOM


• 10 Gbits/sec– Very stable at low latency

throughout scenario

• 1.0 Gbits/sec– Latency grows as number

of tracks increases– Highly unstable run to

run

10.0 Gbits/sec

1.0 Gbits/sec

Bandwidth Sensitivity

Wide variationrun to run


0.8 Gbits/sec

0.1 Gbits/sec

• 0.8 Gbits/sec– Latency now grows

throughout run, most of the time

• 0.1 Gbits/sec– Nearly zero througput– Latency grows linearly as

run proceeds

Bandwidth Sensitivity


Quantitative Analysis• Onset of latency …

– rapid, non-linear, brittle, etc.

• Repeatability in this regime is limited– Statistics difficult to quantify

– Validity assessment very difficult with only MOPs, MOEs or a few live exercises as a guide

• Tack:– Quantify statistical behavior of simulations

– Validate where possible

– Assess likely validity breakdowns


Validity Roll-Up—Indexing• Recent NASA V&V work provides a roll-up framework• Index validity on 0 – 5 scale• Assign target validity index to each component• Measured validity vs. target validity generates validity gap

– How to change statistical measures into 0 – 5 scale– How to assign target validity?

012345

Measured

Target

Gap of 2

012345

Gap of 1

012345

Gap of 3

012345

No Gap

Component 1 Component 2 Component 3 Component 4


Validity Roll-Up• Assign risk associated with each component• Roll-up validity as sum of risk-weighted gaps

– May use max rather than sum (weakest link premise)

components

gap)validity ((risk) ii

Component:Validity scoreTarget ValidityRisk Factor




Module Validity Gap:

ii gap)validity ((risk)

components

gap)validity ((risk) ii

maxcomponents

or


InterTEC Joint Fires Exercise

REDRED

BLUEBLUE

RED = small enemy nationBLUE = small friendly nation

RED forces invade Boron-rich area in BLUE lands


Primary Battlespace Objects• Threat

– Integrated Air Defenses with CAP– Concentrations of Ground Forces Stationary and Mobile– Truck Convoy (HVT)

• Joint Force– C2

• CVN• AOC• GCCS-A• AEGIS• E-2C• E-3 AWACS

– Sensor Platforms• EC-135• JSTARS

– Strike Elements• F-16, F-22, F-35• F-18• MLRS• AEGIS

– ECM• EA6-B Prowler


Mission Thread Overview

• Carrier Air Wing and Joint Air Forces Conduct Coordinated Strike to Destroy HVT Truck Convoy

• Targeting Thread:– EW Provides Initial Indicators and Focused Search for JSTARS– JSTARS Provides Continuous Track for Targeting– E-3 and E-2 Coordinate Strike and Targeting– Also Control Strike Aircraft– EA-6 Provides SEAD Support– FA-18, F-16 Conduct Strike– AEGIS and MLRS also Included in Joint Strike– E-2 Provides SAM Location with EA-6/EC-135– Objective is Closely Spaced (in time) Weapon Arrival on Target

• Threat Uses IAMD to Disrupt Strike


Joint Fires Scenario in Extend (so far)

Link-16 models pulled from NETEGround models pulled from ESP (FCS simulation)

Enemyconvoy


Possible Network Topology

Surveillance Network Link-16

Surveillance Network Link-16 JRE

Strike Control Network Link-16

EPLRS


• < Joint Fires Scenario Demo >


• Currently underway at Point Mugu

• Awaiting post-exercise artifacts– Clearance recently granted

• Plan– Examine exercise in detail

• Model closely in Extend• Construct particular validation methodologies

– Compare to live artifacts• Assess validity• Identify weaknesses/gaps in methodologies

– Identify methodology weaknesses/gaps

Status of Exercise


“High-Level” Validation• Is the level of fidelity appropriate?

• Is the constructive setup appropriate?– If network is analyzed and computed ahead of time

• Is line-of-sight fidelity compromised?

• Do latencies change as assets move?

• Do connections change (a la cell tower hand-offs)?

• Red Force– Any networking capabilities modeled here?

– Jamming, other counter-measures on blue comm assets?

– Blue force anti-comm efforts against red?

• Other information axes– Accuracy, relevance (should realistic error rates be played?)


• Robust, quantitative validation needed for NCW simulations

• Missions × Means decomposition helps structure methodology

• Quantitative analysis just beginning– Still must address statistics of non-linear onset, etc.

• Validation roll-up methods provide an overall validity picture– “Recomposition” from M × M decomposition

• Support of Live Joint Fires Exercise just underway

Wrap-Up


Contact:Wesley N. Colley, Ph.D.

Senior Research Scientist, CMSA

[email protected]

Date post:	21-Jan-2016
Category:	Documents
Upload:	gerard-moody
View:	214 times
Download:	0 times

Validation Methodologies for Netcentric Warfare Simulations Wesley N. Colley, Ph.D. Center for...

Documents