The Evolution of R&E b f iCyber‐Infrastructure in West 

VirginiaVirginia 

A West Virginia TaleA West Virginia Tale…

• West Virginia has long been involved in R&E Networking (more than 40 years), butg ( y ),

• …West Virginia has struggled with ..LATA T iff th l k f lti l i– LATAs, Tariffs, the lack of multiple service providers and carriers, little competition among b d idth id ll tl b d idthbandwidth providers, unusually costly bandwidth

• … as Bob Dylan said…– “The times they are a changin”

Internet2 Spring 2012 Member Meeting

A West Virginia TaleA West Virginia Tale…• A confluence of events• A confluence of events…

– Over the past two years…WV BTOP A d• WV BTOP Award

• New R&E Infrastructure at WVNET• Cyber‐Infrastructure Projects at WVU and Marshall

• High bandwidth connectivity – Green Bank to WVU, I2 and beyond

• NOAA’s new high performance network in WV• …

Internet2 Spring 2012 Member Meeting

PanelistsPanelists

• Don McLaughlin, West Virginia University

• Michael Lambert, Three Rivers OpticalMichael Lambert, Three Rivers Optical Exchange (3ROX)

O’H l W Vi i i N k• Dan O’Hanlon, West Virginia Network (WVNET)

• Allen Daugherty, West Virginia Network (WVNET)(WVNET)

Internet2 Spring 2012 Member Meeting

PanelistsPanelists

• Arnold Miller, Marshall University

• Tim Williams, West Virginia UniversityTim Williams, West Virginia University

• David Halstead, National Radio Astronomy Ob (N A0)Observatory (NRA0)

• Jeff Flick, National Oceanic and Atmospheric , pAdministration (NOAA)

Internet2 Spring 2012 Member Meeting

PSCPITTSBURGH SUPERCOMPUTING CENTER

WV and Regional/National R&E Networks

Michael H LambertThree Rivers Optical Exchange

Pittsburgh Supercomputing Center

PSCPITTSBURGH SUPERCOMPUTING CENTER

Outline• Overview of 3ROX• Services• Architecture

2

PSCPITTSBURGH SUPERCOMPUTING CENTER

Overview of 3ROX• GigaPoP/RON serving western

Pennsylvania and West Virginia– The latter in conjunction with OARnet

• Based at PSC, part of Carnegie Mellon University

• Original NSFnet regional (PREPnet)• Have active research component

(Web10G)3

PSCPITTSBURGH SUPERCOMPUTING CENTER

Services• “Traditional” commodity Internet transit

– Multiple providers• Commodity peering (TR-CPS)• “Regional” peering

– SoX, OARnet, Drexel• Local peering

– Several local residential and business broadband providers

4

PSCPITTSBURGH SUPERCOMPUTING CENTER

More Services• Research and education networks

– Internet2– National LambdaRail– ESnet– XSEDE (wearing PSC hat)

• Limited colocation (DR, etc)• Data archiving (wearing PSC hat)

– Important given newer requirements for NSF proposals 5

PSCPITTSBURGH SUPERCOMPUTING CENTER

Architecture• Four PoPs in Pittsburgh metro area

– Connected by leased fiber– Cisco 15454s– 10 Gb/s now; 100 Gb/s capabilities

through NSF ARI-R2 (in progress)• Ethernet switches in two PoPs

– Cisco 6509s• Layer-2 core

6

PSCPITTSBURGH SUPERCOMPUTING CENTER

More Architecture• Providers, peers land on routers

– Mix of various Junipers, Ciscos– Think impedance-matching between

outside world and core– GbE and 10 GbE now; looking at 100 GbE

R&E router• Member connections land on switches

– Mix of GbE and 10 GbE– Possible upgrades in future

7

PSCPITTSBURGH SUPERCOMPUTING CENTER

Layer-2 Core• Routing for members

– Separate VLAN for each member (overkill but good for statistics)

– Route servers for routing• Quagga’s bgpd on FreeBSD• Third-party BGP• Fine-grained control of routing

8

PSCPITTSBURGH SUPERCOMPUTING CENTER

Advantages of Layer-2 Core• Layer-2 services easy to configure

– ION (or successors)• Nothing dynamic yet

– Sherpa– Dedicated member-member VLANs

• Devices easily added– Eg, for OpenFlow or other SDN

9

WVNET’s NEW RESEARCH & EDUCATION NETWORK FOR 

WEST VIRGINIAWEST VIRGINIA

SEN BYRD moved us to WESTSEN. BYRD moved us to WEST VIRGINIA – What do we do NOW?!

A BIT OF HISTORY & CONTEXTA BIT OF HISTORY & CONTEXT• For almost 40 years WVNET has provided network services to universities• For almost 40 years, WVNET has provided network services to universities 

and researchers in West Virginia.– According to I2 CEO David Lambert, who worked at WVU in the late 1960’s, 

WVNET was the first statewide research & education network in the nation.WVNET was the first statewide research & education network in the nation.

• In 1984, WVNET became a member of BITNET (Because It's Time NETwork).  At that time, BITNET provided communications with over 60 

b i i i d h k i l di ARPANETmember institutions and gateways to other networks including ARPANET, CSNET, CCNET, USENET, and EDUCOM's MAILNET.

• Later that year WVNET gained access to SURAnet a regional data• Later that year, WVNET gained access to SURAnet, a regional data communications network established under a grant by the National Science Foundation to the Southeastern University Research Association (SURA).( )– SURAnet connected users in twelve southeastern states and the District of 

Columbia, and also provided gateways to a world‐wide system of networks called the Internet.

A BIT OF HISTORY & CONTEXT (cont )A BIT OF HISTORY & CONTEXT (cont.)

I 1994 NCSA M i W ld Wid W b li t i t ll d th• In 1994, a NCSA Mosaic World Wide Web client was installed on the WVNET VMS cluster. The Mosaic client provided access to the Internet WWW using X Window System displays. WVNET worked with the US domain name authority at the Information Scienceswith the US domain name authority at the Information Sciences Institute of the University of Southern California to transfer control of cc.wv.us, gen.wv.us, lib.wv.us, state.wv.us, and tech.wv.us from OARnet to WVNET.OARnet to WVNET.

• In 1997, WVNET was accepted as an affiliate member of Internet2.  WVNET is the main provider of Internet access within the state ofWVNET is the main provider of Internet access within the state of West Virginia and has served the state for over 20 years.

• WVNET’s Internet connectivity has grown from a T1 to multipleWVNET s Internet connectivity has grown from a T1, to multiple T1s, a DS3, multiple DS3s, and then an OC3 and multiple OC3s.

A BIT OF HISTORY & CONTEXT (cont )A BIT OF HISTORY & CONTEXT (cont.)

WVNET tl t b t 5 Gb f I t t• WVNET currently aggregates about 5 Gbps of Internet traffic for K‐12, libraries, colleges and universities, and state agencies. It then carries that traffic to Sprint using OC12s.

• WVNET will be replacing that with a 10 Gbps Ring from GTT‐ZAYO using Cisco equipment from a $3 5 million NTIAGTT ZAYO using Cisco equipment from a $3.5 million NTIA BTOP grant.  The equipment will scale to 100 Gbps.

Thi h ll d t t ith l f d l & t t• This has allowed us to partner with several federal & state agencies to aggregate about 40 Gbps of new Internet & I2 traffic to Pittsburgh & Columbus with 3ROX & OARnet.

InternetInternet 2

10GB

Carrier CoLoInternet

Internet 2

10GB

WVNETWVU

Carrier CoLo

10GB

10GB

10GB

10GB

CapitolMarshall U

WVNETProposed 10gb

Network

Enhancing Inter‐ and Intra‐campusEnhancing Inter‐ and Intra‐campus Cyber‐Infrastructure at West Virginia’s Research 

InstitutionsInstitutionsEPSCoR RII C2: EPS‐1006576EPSCoR RII C2: EPS 1006576 

Marshall University

What is it?What is it?

S SC b ild C b• NSF EPSCoR grant to build out Cyber‐Infrastructure in WV.

• Sponsored and Inspired from HEPC Vice‐Chancellor for Research.

• Shared between WVU and MU• WVU to build out Campus Interconnection andWVU to build out Campus Interconnection and their core infrastructure

• MU to establish SEGP opportunity within theMU  to establish SEGP opportunity within the state encompassing the largest audience possible

StrategyStrategy

ld ’ b h• Build upon MU’s recent membership in Internet2 and other synergies:– MU’s need to broaden research resources, communication and interconnection.

– allow MU to bid on Internet2 access if the FCC rural health pilot grant decided to include it in htheir program….

– Strengthen resources for CI‐Train grant between k d (Arkansas and West Virginia (WVU, MU, 

WVStateU)…...

RationaleRationale

GoalGoal• Establish SEGP opportunity within the state encompassing the largest• Establish SEGP opportunity within the state encompassing the largest 

audience possible.   Meant to include:– Higher education ‐ 2 yr, 4yr, public and private– K‐12 public and private– K‐12  public and private– Public Libraries, museums and other educational/research cultural initiatives– State and local government and the Judiciary– The WV state government networkThe WV state government network– The BTOP grant core network– WVNET– Health Care (those not covered in any FCC Rural Health Care Pilot Program)( y g )– Non profit research and development organizations– Others

• Other organizations and entities within the state that may at various times be ll b ti ith th b d lt f h ll b ti i d dcollaborating with the above and as a result of such collaboration require advanced 

networking services to fulfill their collaborative role.

What we have doneWhat we have done

bli h d h S G i i h b d• Established the WV SEGP consortium with board oversight– MU is SEGP connector– OARNet is Internet2 Connector– Begun by reconnecting to WVNET and sharing the 1G link to internet2 through OARNet.

h f h• In the process of revising the SEGP Agreement to– include WVNET as an alternative SEGP Connector– Include 3ROX/Drexel as an alternative Internet2 connector through WVNET.

GovernanceGovernance• Interim governance of the Consortium will be provided by a Board of• Interim governance of the Consortium will be provided by a Board of 

Advisors (the “Board”) consisting of the following voting members:

A representative of the Sponsoring Institution Marshall University who will– A representative of the Sponsoring Institution, Marshall University, who will also act as chair and convener of the Board.

– A representative of the West Virginia Higher Education Policy Commission.– A representative of the West Virginia Council for Community and Technical ep ese tat e o t e est g a Cou c o Co u ty a d ec ca

College Education.  – A representative of the West Virginia Department of Education.– A representative of the Secretary of Education and the Arts.– A representative of WVNET.– A representative of The WV State Government network, i.e., the Chief 

Technology Officer or their designate.A representative of the BTOP grant core network– A representative of the BTOP grant core network.

– A representative of the WV TeleHealth Alliance.

R&ER&E CYBER-INFRASTRUCTURE AT WEST VIRGINIA UNIVERSITY

WEST VIRGINIA UNIVERSITYWEST VIRGINIA UNIVERSITY CORE NETWORK UPGRADE

Joint venture betweenJoint venture between • WVU Office of Research• WVU Office of Information Technology• WV Higher Education Policy Commission• WV Higher Education Policy Commission

WEST VIRGINIA UNIVERSITYWEST VIRGINIA UNIVERSITY CORE NETWORK UPGRADECORE NETWORK UPGRADE• Build a high-speed cyberinfrastructure for education and

researchresearch• Enhance intercampus and intracampus network

infrastructure bandwidth from a 1 Gigabit per second g p(Gbps) core architecture to a 10 Gbps core architecture with 1Gbps at each desktop. Id tif iti l b ildi t t i iti l• Identify critical campus buildings to support initial connectivity

WEST VIRGINIA UNIVERSITY

Layer 3 only CoreCORE NETWORK UPGRADE• Layer 3 only Core• Equal-cost Multi-path Routing through Core to ISP• Designed for future scalability to switching capacity ofDesigned for future scalability to switching capacity of

more than 17 terabits per second• Provides up to 132 ports of 10 GbE per system

Abilit t id 10 GbE ti t h b ildi• Ability to provide 10 GbE connections to each building (with the addition of 10 GbE equipment at the building POP)

• Support for emerging 40 Gbps and 100 Gbps Ethernet standards

• Lossless nondisruptive upgrades for zero service• Lossless, nondisruptive upgrades for zero service downtime

• Efficient power and cooling design

WVU CORE FY 2009WVU CORE FY 2009

WVU CORE WITH 10 GBBUILDING CONNECTIONS 2011

WEST VIRGINIA UNIVERSITYWEST VIRGINIA UNIVERSITY CORE NETWORK UPGRADECORE NETWORK UPGRADE• Total Budget $1.8 million• Revenue Sources – EPSCor grant, WVU

OIT Provost Office in-kind donation fromOIT, Provost Office, in-kind donation from vendor

• Project delivered on time and on budget

WEST VIRGINIA UNIVERSITY’SWEST VIRGINIA UNIVERSITY’S INTERNET2 LINKINTERNET2 LINK• WVU’s research network connection to

3ROX, Internet2, and others…• Upgraded in January• Upgraded in January…

– …from 155 Mbps to 10 Gbps– 64+fold increase in research bandwidth

Supports WVU NETL and NRAO Green– Supports WVU, NETL and NRAO-Green Bank

NRAO Internet2 Panel PresentationA il 24th 2012 A li VAApril 24th 2012, Arlington, VA

Community Input on Radio Data Deluge (from quiet places)

Atacama Large Millimeter/submillimeter ArrayExpanded Very Large Array

David M. Halstead, CIO

Expanded Very Large ArrayRobert C. Byrd Green Bank Telescope

Very Long Baseline Array

21st Century NRAOComplementary research facilities under Open SkiesComplementary research facilities under Open Skies policy

Jansky GBTyVLA

VLBA ALMA

2

Radio Imaging: A RevelationCentaurus A; Not so black

Planets and ring of dust

W50 SNR: SS 433 accreting neutron star

gorbiting star Fomalhaut

3

Green Bank single disk T:2 A 1 Pi l 0 DWDM ( i l !)2 Acres,1 Pixel, 0 DWDM (or wireless!)

QUIET PLEASE!3 TeraBytes/night45Mbps….

4

ALMA: High, Dry & Data Rich (LlamaNet?)

AUI Board Meeting – October 20-21, 2011 5

Extreme conditions, no economy of scale

Internet 2 April 2012 6

HPC: ALMA Correlator @ 16,200 feet

Tunable Filter Bank Card Correlator CardCorrelator Quadrant

•Receives signals from 50x12m antennae•2551 printed circuit boards total in system•8192 Altera Stratix II FPGAs on TFB cards•32768 custom correlator chips with 4096 processors for multiply-and-add calculations•Cross-correlation rate 17 Peta ops/sec•Output specified at 6-60MBytes/sec

7

NRAO Archive Growth

12000

NRAO Aggregate Archive Capacity

h k ( )

8000

10000EVLA Archive Backup (TB)

EVLA Archive (TB)

ALMA Archive (TB)

GBT Archive (TB)

4000

6000

Storage (TB)

2000

02011 2012 2013 2014 2015

FY

8Internet 2 April 2012

NRAO Archive Strategy

Ope

ratio

ns

VLBA

Main ALMA Archive (SCO)

Telescop

e O

NGAS Arch.

Socorro Data Center

CharlottesvilleData Center

External Facility Data 

Center /ODMG

Center NGAS Architecture

OSO

/O

Proprietary/Public Data  Public Data

OSO

/VAO

Virtual Astronomy Observatory (VAO) Standards & Infrastructure

Science Portal, CASA, Archive Access Tool, Helpdesk, Forums etc.

Internet 2 April 2012 9

NRAO Network Architecture UpgradesVLA, EVLA, VLBA

GBT

GBT Pulsar

GBT

ALMA/EVLA

10

EVLA

Communication:- How hard can it be

• Gigabit to Chile: • 10Gigs to Green Bank– NOAO/AURA– ESO– REUNA

– WV State, BTOP, – Frontier Communications– WVUREUNA

– Red CLARA– RNP

AMPATH

WVU– PSC– 3ROX

I 2– AMPATH– Internet2

– Internet2

Internet 2 April 2012 11

N A T I O N A L O C E A N I C A N D A T M O S P H E R I C A D M I N I S T R A T I O N

NOAA Presence in West VirginiaNOAA Presence in West Virginia

Jeff FlickNOAA

April 24, 2012

1

N A T I O N A L O C E A N I C A N D A T M O S P H E R I C A D M I N I S T R A T I O N

NOAA in West VirginiaNOAA in West Virginia

Charleston, West Virginia• National Weather Service

Forecast Office

Fairmont, West Virginia• Development High Performance Computing

• Operational High Performance Computing Backup

• Geostationary Operational Environmental Satellites R SeriesEnvironmental Satellites R-Series Remote Backup Facility

• Security Operations Center• Advanced Weather Interactive Processing System Network Control Facility Backup

Statewide• Observational platforms and Weather Radio transmitters supporting forecast,

2

Observational platforms and Weather Radio transmitters supporting forecast, warning and other public service programs of the National Weather Service

• Monitoring stations supporting air quality and climate research

N A T I O N A L O C E A N I C A N D A T M O S P H E R I C A D M I N I S T R A T I O N

NOAA in Fairmont West VirginiaNOAA in Fairmont, West Virginia

Current and Upcoming Capabilities at The Vertex CenterCurrent and Upcoming Capabilities at The Vertex Center•Environmental Security Computing Center / Development High Performance Computing (HPC) System

•Geostationary Operational Environmental Satellites R-Series (GOES-R) Remote Geostat o a y Ope at o a o e ta Sate tes Se es (GO S ) e oteBackup Facility

Existing Capabilities at NASA IV&V Facility•Operational High Performance Computing Backup Systemp g p g p y•Security Operations Center•Advanced Weather Interactive Processing System (AWIPS) Network Control Facility Backup & Backup Master Ground Stationy p p

3

Observations Computing Forecast

Confidentiality, Integrity, and Availability of Systems

N A T I O N A L O C E A N I C A N D A T M O S P H E R I C A D M I N I S T R A T I O N

GOES‐R Remote Back‐upp(RBU) Facility

NOAA’ ti l t i• NOAA’s operational posture requires a remote back-up facility from which NOAA can command and control the GOES-R satellites in the event the primary operating p y p glocations (NSOF & WCDAS) become disabled.

• After a comprehensive nation-wide review of various locations, the GOES-R Program selected the I-79 Technology Park R h C t i F i t WV th

RBU Facility at the I-79 Technology Park Research Center in Fairmont, WV

Research Center in Fairmont, WV as the site for its Remote Back-Up (RBU) facility.

• In cooperation with the General Services Administration (GSA), a 10- year lease was awarded in December 2009 by GSA with an option for an additional five years

4

with an option for an additional five years.

N A T I O N A L O C E A N I C A N D A T M O S P H E R I C A D M I N I S T R A T I O N

High Performance Computing

Development HPC

Research & Development

Development HPC•Location

– Fairmont, WV (The Vertex Center)– Fit-out of GSA leased space underway– Fit-out of GSA leased space underway

•Configuration– Computing delivered end of FY2011– >382 trillion calculations/sec– >382 trillion calculations/sec– 50PB Archive

•Significance– Support development of weather and seasonal to inter-annual– Support development of weather and seasonal to inter-annual

climate model predictions bound for operational implementation

5

N A T I O N A L O C E A N I C A N D A T M O S P H E R I C A D M I N I S T R A T I O N

AWIPS Network Control Facility yBackup Site

Location

• NASA IV&V (Fairmont, WV)

Configuration

• Primary backup facility for the AWIPS network, supportingPrimary backup facility for the AWIPS network, supporting both the network communications and AWIPS centralized services

• Houses fully operational AWIPS system, critical infrastructure servers, and alternate wide area and satellite broadcast network communicationsnetwork communications

• Nearly identical to the Network Control Facility (NCF) located in Silver Spring MDin Silver Spring, MD

6

N A T I O N A L O C E A N I C A N D A T M O S P H E R I C A D M I N I S T R A T I O N

NOAA in Charleston WVNOAA in Charleston, WV

• National Weather Service• National Weather ServiceWeather Forecast Office (WFO)– Operates 24/7 365 day/year

– Provides weather, water, and climate forecasts and warnings

– Serves public, media, emergency managers and p , , g y glaw enforcement officials, aviation community, agricultural interests, business

– Disseminated through multiple systems, g p yincluding satellite, internet and NOAA Weather Radio All Hazards

– Forecasters provide on-site, detailed weather support for emergencies h ildfi fl d h i l ill d f j ff tsuch as wildfires, floods, chemical spills, and for major recovery efforts

– Office conducts local outreach and education programs to schools, officials, local government, media – essential to ensure appropriate response to severe weather or other hazardsresponse to severe weather or other hazards

N A T I O N A L O C E A N I C A N D A T M O S P H E R I C A D M I N I S T R A T I O N

Security Operations Center (SOC)

• Located at the NASA IV&V building in Fairmont

Security Operations Center (SOC)

Located at the NASA IV&V building in Fairmont WV

• Provides enterprise view of situationalProvides enterprise view of situational awareness for threats and risks across NOAA

• Monitors critical infrastructure intelligently and inMonitors critical infrastructure intelligently and in real-time

• Proactively supports IT Security Incident• Proactively supports IT Security Incident Response by shortening the time between event detection and response to incidentp

• Provides Risk and Compliance measurements

• Provides Trusted Internet Connection services• Provides Trusted Internet Connection services

8

Proactive IT Security Posture Positively Impacts Mission Operations

N A T I O N A L O C E A N I C A N D A T M O S P H E R I C A D M I N I S T R A T I O N

N‐WaveN Wave

9

Th t’ tThat’s our story…

d ’ ti ki t it…and we’re sticking to it.

Questions?