Date post: | 03-Jan-2016 |
Category: |
Documents |
Upload: | austin-lewis |
View: | 213 times |
Download: | 0 times |
Sponsored by the National Science Foundation
GENI I&M and MonitoringGENI Engineering Conference 14
Boston, MA
Sarah Edwards
Chaos Golubitsky
Jeanne Ohren
July 9, 2012www.geni.net
Sponsored by the National Science Foundation 2July 9, 2012
Introduction
• Useful data lives everywhere in GENI– Relationships: slices, slivers, users, resources– Counters: interface traffic, OpenFlow flowspace rules– Measurements: CPU load, memory, bandwidth, latency– Health status: reachability, API functionality
• We can use this information to…– Troubleshoot issues– Optimize configurations– Help experimenters understand their slice resources– Help experimenters analyze their experiments
• How do we help each other bring it all together?
Sponsored by the National Science Foundation 3July 9, 2012
Agenda
• Introduction– Sarah Edwards, GPO
• Guest Speakers:– Kevin Bohan, GMOC
• GMOC Monitoring Demonstration– Anirban Mandal, RENCI
• Client Authentication & Authorization for GENI XMPP Messaging Service
– Martin Swany, Indiana University• GEMINI: Active Network Measurement
– Prasad Calyam, OSC• Measurements on Layer 2 and OpenFlow Paths
• Bringing It All Together– Jeanne Ohren, GPO
• Discussion
Sponsored by the National Science Foundation 4July 9, 2012
• GMOC Monitoring Demonstration– Kevin Bohan, GRNOC
Sponsored by the National Science Foundation 5July 9, 2012
• Client Authentication & Authorization for GENI XMPP Messaging Service– Anirban Mandal, RENCI
Sponsored by the National Science Foundation 6July 9, 2012
• GEMINI: Active Network Measurement– Martin Swany, Indiana University
Sponsored by the National Science Foundation 7July 9, 2012
• Measurements on Layer 2 and OpenFlow Paths – Prasad Calyam, OSC
Sponsored by the National Science Foundation 8July 9, 2012
• Bringing It All Together– Jeanne Ohren, GPO
Sponsored by the National Science Foundation 9July 9, 2012
Bringing It All Together
• Useful data lives everywhere in GENI– Relationships: slices, slivers, users, resources– Counters: interface traffic, OpenFlow flowspace rules– Measurements: CPU load, memory, bandwidth, latency– Health status: reachability, API functionality
• We can use this information to…– Troubleshoot issues– Optimize configurations– Help experimenters understand their slice resources– Help experimenters analyze their experiments
• How do we help each other bring it all together?
Sponsored by the National Science Foundation 10July 9, 2012
Bringing It All Together
• Let’s discuss a couple of examples of issues to consider when working on projects– Data Naming– Data Transport
• Let’s walk through some of the types of data that are being collected or are planned to be collected soon
Sponsored by the National Science Foundation 11July 9, 2012
Data Naming ExampleScenario 1
• Scenario 1 – Consistent naming of resources and devices– Resources on two aggregates are sharing a network link, each
referencing an endpoint. – Each aggregate names their endpoint when submitting data about
the link.– The names must be consistent in order for the consumer to be
able to relate the data from both endpoints.
Aggregate A Aggregate B
Sponsored by the National Science Foundation 12July 9, 2012
Slice: urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+joslice 550e8400-e29b-41d4-a716-446655440000
Data Naming ExampleScenario 2
• Scenario 2 – Globally unique and consistent naming– Two aggregates are reporting data on their active slivers including
to which slice the sliver belongs.– Aggregate A reports a sliver on the slice by URN
(e.g. urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+joslice)– Aggregate B reports a sliver on the slice by UUID
(e.g. 550e8400-e29b-41d4-a716-446655440000 )– The experimenter who created the slice may report I&M data on
that slice by slice name (e.g. joslice).
Sliver A Sliver B
Sponsored by the National Science Foundation 13July 9, 2012
Data Naming ExampleScenario 2 (cont’d)
• Scenario 2 – Globally unique and consistent naming– The consumer of the data may need to determine if these two
slivers belong to the same slice.– Without consistent naming and namespaces, the consumer of the
data has to figure out if and how the two slivers and the experiment data relate.
– This is already being addressed by GENI AM API v3 by using the combination of URN and UUID. Monitoring and some I&M projects are adopting the same slice naming.
– URN + UUID provides uniqueness over time and space.– How does this affect other projects?– What are some other examples?
Sponsored by the National Science Foundation 14July 9, 2012
Data Transport ExampleScenario
• Scenario– As an aggregate, I collect data about the slivers that I
manage, the resources assigned to those slivers, the resources that I have available, etc and report that data to GMOC.
– As an experimenter, I am interested in what resources are available at each of the aggregates.
– As an operator, I am interested in statistics on the slivers that have been created/deleted over a period of time.
Sponsored by the National Science Foundation 15July 9, 2012
Data Transport ExampleHow data is accessed today
• How do each of these consumers access this data?– Aggregates (ExoGENI, InstaGENI, MyPLC)
• Push data to GMOC at regular intervals using the GMOC APIs• Currently access control is using non-GENI credentials
– GENI Clearinghouse (Future)• Will provide an API to pull data on slices, users, and projects.
– IMF and others• Provides a pub/sub interface to allow interested parties with the appropriate
credentials to subscribe to data events
– I&M (GEMINI, GIMI, INSTOOLS)• Provide the ability for the user to push data to an archive on iRODS with
metadata.• iRODS account holders can control and track who has access to archived data
Sponsored by the National Science Foundation 16July 9, 2012
Data Transport ExampleAccess Control and Reliability
• Access control – How do we ensure that the appropriate people are able to access
the data?– How do we ensure that the wrong people do not get to the data?– How do we keep the access control from getting too complicated
for the users?
• Reliability– How do we ensure the data makes it to the other end
uncorrupted?– How do we ensure that the data is getting recorded correctly?
• How can we all walk away from the table with access to good, reliable data?
Sponsored by the National Science Foundation 17July 9, 2012
Data Sources
• Relational data collected by GMOC• Time-series data collected by GMOC• Active network measurement data collected by I&M tools• Passive host measurement data collected by I&M tools• Measurement Data Object Descriptor• Other independent monitoring tools
Sponsored by the National Science Foundation 18July 9, 2012
Data Sources
• Relational data collected by GMOC– Physical location of aggregate resources– Points of Contact (POC) for each aggregate– Slice Authority Info
• type, version, operating organization, etc. – Aggregate Info
• name, version, type, etc. – Slivers for each aggregate – Sliver data
• who created them, when they were created, current state, etc. – Data about resources within each aggregate
• VM servers, routers, etc. – Mapping of resources to slivers – Data about interfaces on resources
• MAC/IPv4/IPv6 addresses, VLAN tags, netmask, etc.
• Schema: http://groups.geni.net/geni/attachment/wiki/GENIMetaOps/gmocv3.rng
Sponsored by the National Science Foundation 19July 9, 2012
Data Sources
• Time-series data collected by GMOC– CPU utilization– Disk Utilization
• per partition
– Number of active VMs• for hypervisors
– Interface traffic counters • TX/RX pps, TX/RX bps
– OpenFlow stats (per datapath and per sliver)• ports, RO/RW rules, TX/RX messages, breakdown of messages by type
– Health checks • AM is accessible via AM API
• Details: http://groups.geni.net/geni/wiki/GENIMetaOps/DraftMonitoringMetrics
Sponsored by the National Science Foundation 21July 9, 2012
Data Sources
• GEMINI– Provides tools to collect active network measurements
• Bandwidth, latency
– Provides tools to collect passive network and host measurements• CPU utilization, memory usage, network traffic count
– Data will be stored in measurement store service (coming soon)• Will provide pub/sub interface and support high-rate data transfers
– Experiment topology and service data stored in UNIS service• Queryable history of topology changes
– Data can be pushed to iRODS archive• Command line interface with access control• Web interface with access control• Searchable
Sponsored by the National Science Foundation 23July 9, 2012
Data Sources
• GIMI– Provides tools to collect data from experiment nodes
• bandwidth, delay jitter, datagram loss data • CPU load, memory usage, per-process state, system usage data
– Collected on OML server– Data can be pushed to iRODS archive
• Command line interface with access control• Web interface with access control• Searchable
Sponsored by the National Science Foundation 25July 9, 2012
Data Sources
• Measurement Data Object Descriptor (MDOD)– Measurement data objects have associated metadata that
provides information on the schema and provenance of the data– Would like to extend MDOD to cover all types of objects, i.e.,
software images– Would like to use MDOD schema to define Event Record schema– Plan to archive measurement data objects in an archive system
based on iRODS– Facilitates searching and correlating data– I&M group has completed v1 of MDOD schema
• Working towards a simpler v2
Sponsored by the National Science Foundation 26July 9, 2012
Data Sources
• Other Independent Monitoring Data Sources– PlanetLab Monitoring - CoMon
• http://comon.cs.princeton.edu• Provides monitoring statistics at both a node level and a slice level• Only covers regular PLC nodes
– ProtoGENI Monitoring• Node Control Center: https://www.emulab.net/nodecontrol_list.php3?showtype=pcs
• Shared Pool: https://www.emulab.net/showpool.php
• Testbed Node Availability Stats: https://www.emulab.net/node_usage/
• Experiment Information Listing: https://www.emulab.net/showexp_list.php3?showtype=all&sortby=name&thumb=1
• Encourage new independent tools that provide monitoring or I&M info– more accessible and usable across all of GENI if people
collaborate and use interfaces like those we are reviewing today
Sponsored by the National Science Foundation 27July 9, 2012
Discussion
– Data Naming• How have lack of globally unique and consistent naming
affected other projects?• What are some other data naming examples?
– Data Transport• What are you using that others might find useful?• How can we all walk away from the table with access to good,
reliable data?
– What other data sharing issues have you encountered?– Data Resources
• What other data resources should we all know about?