Virtualization and

System of Systems Testing

Capt Donnie Mayop yMarine Corps Tactical Systems

Support Activity (MCTSSA)

Report Documentation Page Form ApprovedOMB No. 0704-0188

Public reporting burden for the collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering andmaintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information,including suggestions for reducing this burden, to Washington Headquarters Services, Directorate for Information Operations and Reports, 1215 Jefferson Davis Highway, Suite 1204, ArlingtonVA 22202-4302. Respondents should be aware that notwithstanding any other provision of law, no person shall be subject to a penalty for failing to comply with a collection of information if itdoes not display a currently valid OMB control number.

1. REPORT DATE APR 2010 2. REPORT TYPE

3. DATES COVERED 00-00-2010 to 00-00-2010

4. TITLE AND SUBTITLE Virtualization and System of Systems Testing

5a. CONTRACT NUMBER

5b. GRANT NUMBER

5c. PROGRAM ELEMENT NUMBER

6. AUTHOR(S) 5d. PROJECT NUMBER

5e. TASK NUMBER

5f. WORK UNIT NUMBER

7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) Marine Corps Tactical Systems Support Activity (MCTSSA),Box555171,Camp Pendleton,CA,92055-5171

8. PERFORMING ORGANIZATIONREPORT NUMBER

9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. SPONSOR/MONITOR’S ACRONYM(S)

11. SPONSOR/MONITOR’S REPORT NUMBER(S)

12. DISTRIBUTION/AVAILABILITY STATEMENT Approved for public release; distribution unlimited

13. SUPPLEMENTARY NOTES Presented at the 22nd Systems and Software Technology Conference (SSTC), 26-29 April 2010, Salt LakeCity, UT.

14. ABSTRACT

15. SUBJECT TERMS

16. SECURITY CLASSIFICATION OF: 17. LIMITATION OF ABSTRACT Same as

Report (SAR)

18. NUMBEROF PAGES

26

19a. NAME OFRESPONSIBLE PERSON

a. REPORT unclassified

b. ABSTRACT unclassified

c. THIS PAGE unclassified

Standard Form 298 (Rev. 8-98) Prescribed by ANSI Std Z39-18

AgendaAgenda

• Virtualization IntroG l B fit• General Benefits

• Virtualization in the Corps• Testing virtualization or testing with

virtualization• The Future

Virtualization Introductiontua at o t oduct o

• Virtualization is the abstraction of the real hardware of a real, physical, computer system, often a server, in order to allow for the sharing of that hardware among more than one virtual computer system.

Virtualization Overview

VirtualMachinesMachines(Clients and

Servers)

Real Real (Physical)Server and H t Host Software

From the Guest OperatingS stem Point of ieSystem Point of view

OperatingSystem

Device Drivers

System

Hardware

• A VM works just like a regular computer.• Virtual machine’s device drivers are replacedVirtual machine s device drivers are replaced

by device drivers that call virtual hardware instead of real hardware (but it does not knowinstead of real hardware (but it does not know that the hardware is virtual)

From the Host Operating S stem Point of VieSystem Point of View

Virtual Machines

VirtualizationSoftware

Real PhysicalServer

Scheduler

CPU

Firewall

Apache Tomcat

x86 Processor Virtualization

Memory Mgmt

Memory Disk NIC

Other Services

Virtualization Benefits

• More capability per piece of server h dhardware

• Disaster recovery/continuity of operationsDisaster recovery/continuity of operations• Greater training capability• More efficient testing

COC CapSet 2 Before Virtualization

IOW I lli O i W k i► IOW – Intelligence Operations Workstation► IOS – Intelligence Operations Server► CPOF – Command Post of the Future

COC CapSet 2 After Virt ali ationAfter Virtualization

0 Laptops 0 Switches

CPOF Workstation

1 Server- $5,000 1 Terminal $500 Total $5,500

CPOF Workstation

CPOF Workstation

10SV3

CPOF Workstation

0

CPOF Data Bridge

Server

* Still not at Server Capacity

Disaster Recoveryy• Most virtualization solutions provide aMost virtualization solutions provide a

variety of methods to increase the ability to recover from disasterto recover from disaster.

• vMotion, Live Migration• Fault Tolerance• High Availability• High Availability• Most assume VMs are running on some

sort of robust shared storage (NAS, SAN etc…))

Greater Training Capabilityg p y

CPOFCiient CPOFCiient

CPOF Ciient CPOFCilent

Network lnfonnatlon Subnet 192..168.10.0

Subnet Mask 255.255.255.128

Physical Machln&S .16-.38 CPOF Client Laptops

.42 vCenter Server .39 ESX Server 1 40 ESX Server 2 .41 ESX Server 3

VMs .51-.54 Student 1 Servers .55-.58 Student 2 Servers .59-.62 student 3 Servers .63-.66 Stude11t 4 Servers .67-.70 Stude11t 5 Servers .71-.74 Student 6 Servers .7$-. 78 Student 7 Servers .79-.82 student 8 Servers .83·.86 stude11\ 9 Servers

.87-.90 Studet1t 10 Servers

.91 -.94 Studet1t 11 Servers

.9$-.98 Student 12 Servers

CPOF Training Network Diagram 12 May 2009

Primary POC Mimi Chung 760-725-0657

CPOFCiient CPOI'"Ctlont CPOF Cli.ent CPOI= Cllont

CPOFCiient CPOF CIIent CPOF Client CPOFCIIent ESX Server 1 192.168.10.39

ESXSetver2 192.168.10.40

ESX Server3 192.168 10 41

Test Benefits

• Less people and physical gear to run testsLess people and physical gear to run tests– One person can run multiple virtual machines.

Hardware limitations are a huge problem as we– Hardware limitations are a huge problem as we get more missions and the same funding.

• Easily set up lab and test environment• Easily set up lab and test environment– More rapid setup of test network architecture &

virtual machinesvirtual machines– Test thread development and vetting

Redline test procedures and dry run testing– Redline test procedures and dry run testing– Simulation/Stimulation

Network data capture and analysis much the– Network data capture and analysis much the same

Real World ExamplepDRAFT Version- 3.5.2

For Official Use Only MC3T Virtualization Proof of Concept Test Environment Topology

For Official Use Only

DRAFT Version- 3.5.2

---._._ __ -·-·------ · .......... .__ ··-·-· .. ----·--·-..... __

A Notional (All Ph sical) Test(All Physical) Test

% Hardware Required Over Time

100.00%

70.00%

80.00%

90.00%

Test Complete

50.00%

60.00%

70.00%

30.00%

40.00%

0.00%

10.00%

20.00%

18-Dec 7-Jan 27-Jan 16-Feb 8-Mar 28-Mar 17-Apr 7-May

Virtualization Supported Testing (SAME SCALE)Testing (SAME SCALE)

% Hardware Required Over Time

100.00%

70 00%

80.00%

90.00%Test Complete - PHYSICAL

50.00%

60.00%

70.00%

With t Vi t li it

30.00%

40.00%Without VirtualizaitonWith Virtualization

10.00%

20.00% Test Complete - VIRTUAL

0.00%18-Dec 7-Jan 27-Jan 16-Feb 8-Mar 28-Mar 17-Apr 7-May

Marine Corps Use ofVirt ali ation

• Tactical Collaborative Work Suite (TCWS)

VirtualizationTactical Collaborative Work Suite (TCWS)– 20+ virtual servers hosted on 5 physical servers

• Data Delivery System-Replacement and Modular (DDS-R and DDS-M)– 2 physical servers with a storage area network (SAN)

• Combat Operations Center (COC)• Combat Operations Center (COC)– COC is virtualizing.

• Expeditionary Fighting Vehicle (EFV) Command and Control variant– Virtualizing servers for continuity of operations

• MCNOSC & MITSCsMCNOSC & MITSCs– Fully virtualized (as much as possible)

• Over 1,000 VMs in Iraq and Afghanistan right now.q g g

Relative Overall ProgressProgress

MCNOSCMCTSSA

MCIAMARFORRES

MARCENTMARCENTMARFORPAC

22 MEUHQMC / PP&O

MCTOGIII MEF CE1ST MAW2ND MAW

HQMC / ARIHQMC / M&RA

SSC-LANT / NECCSSC-LANT / COC

3RD MAW3 MAW

MARCENT- DesktoppMARFORRES- Desktop

TECOMMCEITS

MCTSSA/Marine CorpsInd str Cooperation

• Non standard use casesIndustry Cooperation

– Much of what MCTSSA virtualizes is client based– The Marine Corps is doing a lot of server based– The Marine Corps is doing a lot of server based

virtualization, but ruggedized and expeditionary in naturenature.

– Combat Operations Center and the Expeditionary Fighting Vehicle both use the TACLINK 3000 nonFighting Vehicle both use the TACLINK 3000, non standard interfaceM f th C4I t i th M i C– Many of the C4I assets in the Marine Corps inventory also use non-standard interfaces.V i i d d i i i h– Very inexperienced systems administrators with high turnover.

Where MCTSSA is

• Current use of virtualization – Virtualization of clients and servers in the C4I architecture in order to create a virtual Marine Expeditionary Force to develop, test and certify C4I assets and to allow more efficient testing with real assetsassets and to allow more efficient testing with real assets.

• Have virtualized:Intelligence Operations Workstation (IOW) V1/V2– Intelligence Operations Workstation (IOW) V1/V2,

– Intelligence Operations Server (IOS) V3, – Command and Control Personal Computer p– Joint Tactical Collaborative Workstation (JTCW) – InterTEC Tool Suite, – C3Driver Tool Set, – Combat Operations Center (COC) and Command Post of the Future

(CPOF)(CPOF)

– Researching Sun SPARC virtualization and more.

Where MCTSSA Is

• Support Centerpp– VMs used for training at the Support Center.– Support Center designated the Tier 1 Help Desk for pp g p

virtualization issues in the Marine Corps.

• Deployed Supportp y pp– Deployed support teams already receiving training on

virtualization IOT help the Operating Forces while p p gdeployed.

– Deployed Support now has their own sandbox lab

• Test Branch– Used virtual machines for validating and vetting test g g

threads and operator training in MC3T and JTCW tests.

Where MCTSSA Is Going

• Virtualization Center of Excellence– A place where all aspects of virtualization in garrison

and tactical areas can be explored, researched, developed tested and supported through all phasesdeveloped, tested and supported through all phases and aspects of the acquisition cycle.

• Tactical use and integration of virtualization for continuity of operations and lessen footprint/lighter loadsof operations and lessen footprint/lighter loads.

• Garrison development, use testing and support of virtualization solutions, including all aspects of software and hardware related to the many add on softwareand hardware related to the many add-on software packages developed for VMware virtualization solutions

• Research and development of non-VMware, non-standard virtualization technologies to meet the Marinestandard virtualization technologies to meet the Marine Corps current and future use cases.

• Development of best practices, deployed support teams t i d i i t li ti d 24/7/365 l b l h b k ttrained in virtualization and 24/7/365 global reach back to trained virtualization administrators with the capability of rapidly fixing almost any problem.

Take Aways

• Use of Virtual Machines is in support of and does not

y

ppreplace testing with and of real machines, except when used as simulation/stimulation in a test.

• Shortens time required for physical test by identifying and mitigating of incidents in virtual environment g gbefore physical environment is even built.

• Virtualization in support of physical testing results in aVirtualization in support of physical testing results in a much more efficient use of physical gear – Minimize time the gear is sitting in the physical environmentMinimize time the gear is sitting in the physical environment– Minimizes the need for physical gear in thread development and

vetting, operator training, dry run testing and troubleshooting

• More tests with the same amount of gear and less money

But Wait, There’s More,• Use of Virtual Machines is in support of and

d t l t ti ith d f ldoes not replace testing with and of real machines, except when used as simulation/stimulation in a test?????simulation/stimulation in a test?????

• DoD Policy is that if any simulation is used in an acquisition program it must be verified, q p g ,validated and accredited (VV&A)– What if the system was developed originally as a

VM or if it spent part of it’s life in development as aVM, or if it spent part of it s life in development as a VM and then was deployed as a physical box?

– What if the system is deployed as a VM or set of VMs? (NECC)

– The line in increasingly becoming blurryEven if the system is deployed as a VM certain– Even if the system is deployed as a VM, certain caveats must be set (RAM reservations, CPU reservations, hard disk size, etc…)

QUESTIONS ?

Word to the Wise• Don’t believe the hypeo t be e e t e ype

– There can be some serious digressions from reality when you speak with folks from differing camps or vendorsDo your own research don’t rely on vendors to tell you the truth– Do your own research, don t rely on vendors to tell you the truth

• Proof of Concept is cheap, implementation probably won’t bewon t be

• Invest in some sort of higher quality shared storage for business critical production use to ensure you canfor business critical production use to ensure you can take advantage of the disaster recovery and continuity of operations use cases

• Good high speed networking is essential for anything other than simple proof of concept use

• Open source virtualization has some interesting offerings

Virtualization Links• VMware

– www.vmware.com

• Citrix and Xensource– www.xensource.com

• Hyper V– http://www.microsoft.com/windowsserver2008/en/us/hyperv.aspx

• Sun (Solaris-based)– http://www.sun.com/solutions/virtualization/

• Oracle (mostly for DB)– http://www.oracle.com/virtualization/index.html

• There are many, many more….