1 America Connects to Europe (ACE) (SCI - 0962973) Year 5 Quarter 3 Quarterly Report 1-September-2014 through 30-November-2014 Jennifer Schopf – Principal Investigator (Prepared by, Jennifer M. Schopf, Alice Jackson) Summary During the quarter of September 1,2104 through November 30, 2014, the America Connects to Europe (ACE) project continued its collaborative and engineering activities to support international research. This report outlines collaborations, software and systems work, operational activities, and usage statistics for the project. Highlights include the decommissioning of the Paris-ManLan circuit, ongoing discussions with application scientists, and several presentations at fall meetings. 1. ACE Overview The America Connects to Europe (ACE) project supports a series of circuits and services between the US East coast and Europe. In the current set up, these circuits are: • Three 10G circuits between WIX (McLean, VA) and Frankfurt. These circuits are lagged together and load balanced for performance, and sometimes they are reported on as a single unit. Two of these links are funded by NSF, the third is an in-kind contribution by GEANT. • Three 10G circuits between MAN LAN (New York City) and Amsterdam. These circuits are lagged together and load balanced for performance, and sometimes they are reported on as a single unit. One of these links is funded by NSF, the other two are in-kind contributions by GEANT. • One 10G circuit between Paris and New York to connect Internet2’s ION service to GEANT’s AutoBAHN service and provide bandwidth on demand services for researchers. This circuit is fully funded by GEANT. This circuit was discontinued by GEANT on October 6, 2014. • One 10G circuit between StarLight in Chicago and Amsterdam for use in SDN experiments. This circuit is fully funded by NSF. These circuits are used in production to support a wide variety of science applications, including but not limited to physics (LHC), astronomy (e-VLBI), and biomedical research (GENIUS). In addition, through a supplement to the award, the ACE project is now directly supporting application use of 100G testbeds and tool development for 100G links. Overall, this award supports tool development, software defined networking (SDN) experimental work, and measurement and security activities. Please note that some of the activities (outreach to Africa, perfSONAR training, etc.) are also included in the TransPAC quarterly report project, as appropriate, as joint funding supports these efforts.
Transcript
1
America Connects to Europe (ACE) (SCI - 0962973)
Year 5 Quarter 3 Quarterly Report 1-September-2014 through 30-November-2014
Jennifer Schopf – Principal Investigator (Prepared by, Jennifer M. Schopf, Alice Jackson)
Summary
During the quarter of September 1,2104 through November 30, 2014, the America Connects to Europe (ACE) project continued its collaborative and engineering activities to support international research. This report outlines collaborations, software and systems work, operational activities, and usage statistics for the project. Highlights include the decommissioning of the Paris-ManLan circuit, ongoing discussions with application scientists, and several presentations at fall meetings.
1. ACE Overview The America Connects to Europe (ACE) project supports a series of circuits and services between the US East coast and Europe. In the current set up, these circuits are:
• Three 10G circuits between WIX (McLean, VA) and Frankfurt. These circuits are lagged together and load balanced for performance, and sometimes they are reported on as a single unit. Two of these links are funded by NSF, the third is an in-kind contribution by GEANT.
• Three 10G circuits between MAN LAN (New York City) and Amsterdam. These circuits are lagged together and load balanced for performance, and sometimes they are reported on as a single unit. One of these links is funded by NSF, the other two are in-kind contributions by GEANT.
• One 10G circuit between Paris and New York to connect Internet2’s ION service to GEANT’s AutoBAHN service and provide bandwidth on demand services for researchers. This circuit is fully funded by GEANT. This circuit was discontinued by GEANT on October 6, 2014.
• One 10G circuit between StarLight in Chicago and Amsterdam for use in SDN experiments. This circuit is fully funded by NSF.
These circuits are used in production to support a wide variety of science applications, including but not limited to physics (LHC), astronomy (e-VLBI), and biomedical research (GENIUS). In addition, through a supplement to the award, the ACE project is now directly supporting application use of 100G testbeds and tool development for 100G links. Overall, this award supports tool development, software defined networking (SDN) experimental work, and measurement and security activities. Please note that some of the activities (outreach to Africa, perfSONAR training, etc.) are also included in the TransPAC quarterly report project, as appropriate, as joint funding supports these efforts.
2
2. Financials This quarter, Schopf was finally able to get a fully updated accounting of the ACE project financials from the IU Finance office. This included not only being able to correct past errors but to get a ground truth of the state of the finances for the project as a whole. Assuming Year 5 expenses proceed on course, this project is estimated to be approximately $1,150,000 underspent at the projects current end date of May 31, 2015. Further investigation found a severe underspend in the first 2 years of the project, which the current PI was unaware of. In order to address this issue, Schopf met with NSF PO Thompson September 30th to discuss future plans and alternatives. The project will request for a one-year No Cost Extension to carry out the plans, and a short version of the agreed upon projects include:
• Continue all staffing through May 2016 • Continue production circuits from WIX and ManLan through December 2016 • Continue funding for GlobalNOC support through December 2016 • Fund production IU systems support through December 2016 (the accounting
review this quarter discovered this had not been being properly billed and paid, although services have been rendered since the project start)
• Discontinue Chicago circuit in May 2015 • Upgrade WIX and MAN LAN PoPs to collect active and passive monitoring data • Expand monitoring work to include an extension project titled “perfSONAR
Everywhere” with the Network Startup Resource Center (NSRC), with the goal of deploying low-cost perfSONAR nodes in 2 emerging regions
• Additional application support and workshops With these plans in place, the project should spend out its balance by the end of the sixth year.
3. Staffing Prior to this quarter, project staff consisted of:
• Jennifer Schopf, Director • Andrew Lee, primary ACE senior network engineer (100%) • Arvind Gopu, application support staff (50%) • Abinahav Thota, 100G Lustre support (50%) • Robert Henschel, 100G consultant in IU Research Technologies (10%) • Dan Doyle, International and 100G perfSONAR support (25%) • Michael Johnson, International and 100G perfSONAR support (25%) • Scott Chevalier, primary contact for GlobalNOC support desk (25%) • Alice Jackson, administration (5%) • Dale Smith, network consultant (5%)
No staffing changes took place this quarter.
4. Collaborations, Travel, and Training ACE staff continued to grow collaborations in Europe over the quarter with the goal of better understanding collaborative science use of the ACE links and supporting use of emerging network technologies.
3
Lee attended the LHC Open Network Environment/Optical Private Network meeting (http://indico.cern.ch/event/318811/) in Ann Arbor, MI, September 15-16. While there, Lee discussed with LHCONE and LHCOPN members support models for smaller facilities with performance issues. Lee also discussed usage of the trans-Atlantic circuits by the LHC community, including the shifting of traffic onto the Department of Energy’s Energy Science Network (ESNet). Lee received Belle-2 update and discussed the impact on trans-Pacific data transfer capabilities. September 29th- October 1st, Lee attended the14th Annual Global LambdaGrid Workshop (Glif), (http://www.glif.is/meetings/2014/) in Queenstown, New Zealand, where he observed early trans-Pacific 100g demos. While in New Zealand, Lee also discussed with engineers from SANREN (South Africa) the impact of SKA-MID and other science activities and their potential network demands.
October 5th – 9th, Gopu got a chance to interact with scientists from LSST and SKA at the Astronomical Data Analysis Software and Systems (ADASS2014), in Calgary, Canada.
PRAGMA 27 was hosted in Bloomington, Indiana this year on October 15ht-17th. Schopf, a member of the organizing committee, ran a workshop on Experimental Networking. Lappa attended as a member of the Experimental Networking Panel and delivered his presentation on CERNET. The project also supported Brent Sweeny returning to IU to participate in this meeting and speak at the networking workshop.
October 26th - 30th at the Internet2 Technology Exchange meeting, Schopf, Lappa, Lee, Doyle, Johnson, and Gopu attended. Staff members discussed SDN and measurement technologies going forward with Jin Tanaka and Kaichiro Ikeda of KDDI. Several team members attended Internet2 TechEx conference to meet with perfSONAR developer team from around the world to suggest and discuss 3.5 milestones. Gopu met representatives from the ICT-BioMed consortium - these conversations led to a subsequent conference call to discuss three potential use-cases. Following that, however, the use-cases were not feasible because of lack of resources on the ICT-BioMed side.
Schopf, Lee, and Lappa attended Supercomputing in New Orleans, November 16-21. Schopf co- organized “Innovating the Network for Data-Intensive Science” workshop. Lappa and Lee met with representatives of the R&E networks of Philippines, Malaysia, and Singapore to discus collaborations and with JGN-X to discuss SDN roles for TransPAC.
5. Software and Systems Work
A. Tool Development The development of GlobalNOC tools, funded in small part by ACE, continued this quarter with patches and upgrade to existing tools. No new tools were developed.
4
As part of the larger perfSONAR development team, Doyle and Johnson worked on testing and documenting new releases of the toolkit. The IU team also has shifts for the "Person of the Week”, whose function is to be responsible for monitoring the various user support lists and ensuring that a response is given in a timely manner, whether escalating to others developers or handling the question personally. Some development work provided fixes to bugs and enhancements to visualizations in the UI to support 3.4.1 release of perfSONAR. The team also worked to help create the 3.5 milestones and teams, including attending a full day eeting after the I2 TechExchange meeting. For the Lustre over 100G project, we continued testing with the hosts in Aachen and Dresden, but on their regular research network, as well as on the IU internal network. We began running the test application with reads and writes from 1K to 100MB chunks using the application we developed, and then examined the Lustre activity on the client and the server to understand how Lustre responds to I/O requests, both big and small. This helped us to identify that Lustre is using read aheads at all times, which means we should be targeting applications that do sequential reads for our work here.
B. Software Defined Networking (SDN) Activities Currently, the ACE network supports SDN and OpenFlow as part of our partnership with Internet2 and GÉANT. There is currently an Inter Domain Controller (IDC) at MAN LAN, which makes it possible to create dynamic circuits through MAN LAN using OSCARS. This will enable us to provide Layer 2 Virtual Circuits dynamically instead of relying on static services. US researchers using the Internet2 ION facility are able to create dynamic circuit to Europe using the ACE networks.
C. Measurement Activities
Conversations have stalled with the Europeans to get a maddash in place over the ACE links, with the hope they will start up again in the new year.
Conversations also continue with Internet2 and DANTE/GEANT to collect flow data on the ACE links. With the identified budget surplus, we began discussions with Internet2 to collect our own active and passive measurement data, but met with significant resistance to this approach. We hope this will be resolved in the next few months and data collection will finally begin. On the EU side, they are still putting in place the proper collectors but have agreed to share some data as soon as their systems are fully in place.
Conversations began with the Network Startup Resource Center for the perfSONAR Everywhere project, which is being enabled by the current budget surplus. The two emerging region sites were identified: Kenya and the Philippines. A job description for the main IU staff member to oversee this work was drafted and is in IU HR process. Once the staffing is in place, we will identify candidate equipment to purchase.
D. Application Support over 100G Support for application use of the 100G ANA link continues to have issues that have nothing to do with staff outreach or the ANA 100G link itself, but everything to do with the lack of 10G reliable links between data sources and sinks. Gopu continues to reach out to groups moving large data sets at various meetings. These discussions included:
5
• EBI: We facilitated experiments between the University of Chicago and EBI, and
this was presented at a workshop at SC. • PANSTARS: We continue to try to follow up on leads from Q2.
• LSST: We continue to try to follow up on leads from Q2.
• GeoScience: Several geoscientists seemed interested in our work at various meetings, but initial uptake has not been successful due to lack of participation on the scientist side.
• Belle2/DANTE: We continued to meet periodically with our DANTE colleagues.
• ICT-BioMed: Exploring a possible use-case with this group.
• LHC-One: Discussions are ongoing, but the group is basically independent of our work.
• Open Science Grid and Europe continue, mainly via conversations with Rob Quick about a couple of groups he works with out of Europe. The project is investigating upgrading their links in the EU to take advantage of the increased trans-Atlantic bandwidth.
• LIGO: We continue to follow leads provided by Scott Korrander to try to get uptake with the project. Latest contact, Staurt Anderson, is not responding to emails.
• Mike Sullivan introduced Gopu to several possible leads at the tech exchange, including Anil Srivastava @ OSHL, Cezary Mazurek @ PSNL, Ashok Kolaskar, and Sandra Gessing @ Notre Dame.
Discussions began with ESnet to support a follow-on activity to the Climate Science Cross Connect, using in part surplus funding. The International Center for Advanced Internet Research (iCAIR) at Northwestern University and its partner organizations have utilized the ACE Link between the StarLight International/National Communications Exchange Facility in Chicago, Illinois and the NetherLight facility in Amsterdam, Netherlands. This layer 2 10 Gbps path supported multiple projects during the quarter – details will be included in the final report.
6. Operational Activities There were two main outages during Quarter 3, both undersea fiber cuts. The first, Sept 3-10 (ticket 437:127), affected one of the MAN LAN-Amsterdam links. Our relationship with the vendor kept updates flowing and the community aware of ongoing repair efforts. An outage of this duration is expected with undersea fiber. The second was occurred November 19-30, (ticket 481:127), and affected one of the WIX-Frankfurt links and one of the MAN LAN-Amsterdam links. Again the relationship with the vendor made this easy to update throughout the repair process with the community. The GlobalNOC Phone system was upgraded from version 3.0 to 4.0 for better reporting and reliability. This upgrade enables reporting on specific calling queue numbers and logging of calls for historical reference.
6
A. Circuit Procurement On October 6, 2015, the Paris-MAN LAN circuit, part of the GEANT in kind contribution to the ACE project, was discontinued due to expense and lack of use. GEANT is in the planning stages for a 100G link that may replace this path.
B. Traffic Graphs Figures 1 and 2 shows the aggregated traffic for the three 10G circuits from WIX to Frankfurt. The circuits are lagged together to create a total capacity of 30Gbps. Figures 3 and 4 show the aggregated traffic for the three 10G circuits from MAN LAN to Amsterdam. The circuits are lagged together to create a total capacity of 30Gbps. In addition to these 4 circuits, ACE supports two other circuits. The New York to Paris link is funded by the GEANT project to support Internet2’s ION service to GEANT’s AutoBAHN service, and Figures 5 and 6 shows that traffic. The second is a link from Chicago to Amsterdam, which is used as a test-bed for SDN. The usage of the Chicago to Amsterdam link consisted of lightpaths for many diverse projects involving US-funded and Europe-funded projects. The VLAN assignments on this link are presented in Appendix A. Traffic information for this link is not available.
Figure 1. Aggregated traffic using maximum daily values on the 30G Lag between WIX and Frankfurt for September 1, 2014 through November 30, 2014.
7
Figure 2. Aggregated traffic using smoothed average daily values on the 30G Lag between WIX and Frankfurt for September 1, 2014 through November 30, 2014.
Figure 3. Aggregated traffic using maximum daily values on the 30G Lag between MAN LAN and Amsterdam for September 1, 2014 through November 30, 2014.
8
Figure 4. Aggregated traffic using smoothed average daily values on the 30G Lag between MAN LAN and Amsterdam for September 1, 2014 through November 30, 2014.
Figure 5. Aggregated traffic using maximum daily values on the 10G circuit between New York and Paris for September 1, 2014 through November 30, 2014. Please note the units on the Y-axis.
9
Figure 6. Aggregated traffic using smoothed average daily values on the 10G circuit between New York and Paris for September 1, 2014 through November 30, 2014. Please note the units on the Y-axis.
10
C. Trouble Tickets During this quarter, there were 21 tickets for unscheduled maintenance, detailed in Table 1, and 14 tickets for scheduled maintenance, detailed in Table 2. Additional trouble ticket information is available at https://tick.globalnoc.iu.edu/fp_tools/public_ticket_viewer/index.cgi .
Table 2. Tickets for scheduled maintenance on ACE circuits, September 1, 2014 – November 30, 2014
13
D. Downtime and Availability
Table 3 below displays downtime for core nodes on the project. Table 4 lists downtime for the project’s circuits. ACE Core Nodes Down Time
Reporting Period
Availability 52 Week Availability
0 hr 0 min 100.00% 100.00% Aggregate ACE Core Nodes 0 hr 0 min 100.00% 100.00%
Table 3. Downtime and availability for ACE core nodes.
ACE Backbone Circuits Down Time Reporting Period
Availability 52 Week Availability
GLOBAL-ACE-MANLAN-O192-01500 281 hr 24 min 87.12% 96.20%
GLOBAL-ACE-MANLAN-O192-01501 14 hr 7 min 99.35% 99.13%
GLOBAL-ACE-WIX-O192-01502 205 hr 55 min 90.57% 94.98%
GLOBAL-ACE-WIX-O192-01504 89 hr 15 min 95.91% 98.34%
GLOBAL-ACE-MANLAN-O192-01505 14 hr 45 min 99.32% 99.02%
GLOBAL-FRK-WASH2-10GE-01507 280 hr 4 min 87.18% 96.087%
MAN-NEWY32AOA-PARIS-10GE-01618 13 hr 36 min 99.38% 97.59% Aggregate All ACE Backbone Circuits 899 hr 6 min 94.12% 97.34%
Table 4. Downtime and availability for ACE circuits.
7. Security Events and Activities Basic security measures were maintained during this quarter and no security incidences were reported. Basic security work is included as part of the ongoing funded support ACE receives from the GlobalNOC.
8. Reporting against Objectives June-2014 thru August-2014 1. Overall
a. Adjust future plans according to updated budget information 2. Collaboration and Demonstrations
a. Attend variety of large scale international meetings b. Seek out additional end users to support in use of 100G testbeds
3. Systems and Software Work a. Continue support for perfSONAR over great than 10G circuits b. Began new project perfSONAR everywhere to explore use of inexpensive
perfSONAR nodes in emerging regions c. Investigate passive monitoring to 60G LAG circuits d. Additional Lustre work in support of supplement
4. Operational Activities a. Discontinue Paris-MAN LAN circuit b. Continue full support of 7 circuits
14
9. Plans for December-2014 thru January-2014 1. Overall
a. New director to review activities and adjust as needed b. Continued addit ional f inancial oversight of project c. Put in place additional project tracking d. Hire new staff member to work on perfSONAR Everywhere project
2. Collaboration and Demonstrations a. Attend variety of large scale international meetings b. Plan and implement SDN experiments, and work in multi-domain c. Seek out additional end users to support in use of 100G testbeds d. Use additional monitoring information to increase outreach and
support of circuit end users, verifying performance 3. Systems and Software Work
a. Add active monitoring to 60G LAG circuits b. Add passive monitoring to 60G LAG circuits c. Additional Lustre work in support of supplement
4. Operational Activities a. Continue full support of 7 circuits
15
APPENDIX A. VLAN assignment on Chicago-Amsterdam Link The table below shows the VLAN assignment on the Chicago to Amsterdam circuit on the last date of this reporting period.
NetherLight Service ID VLANs Description Global ID 5048VL-EVL-Cinegrid-UvA-IRNC
21 -
5040VL_CNPEK-NLAMS_CSTnet(private –peer)
137 Starlight side vlan 137-Netherlight side vlan 136
