perfSONARMulti-LS
Jason Zurawski
Martin Swany
04/22/23Swany
OverviewIntroduction
Functionality
Dev Status
Future Plans
Demo
04/22/23Swany
IntroductionA single LS instance would have problems scaling to the increased volume of global deployment
Need multiple LS instancesThey need a way to discover one another
The key idea is to relate local LS instances through the exchange of information 'summaries'.
This transformed data is smaller, yet attempts to retain necessary lookup properties.
04/22/23Swany
IntroductionThrough federation it is possible to link many LS instances on the inter-domain level, as well as extending this notion between domains through the use of a dedicated representative (leader).
Queries will attempt to be resolved locally first, before either iterating through multiple layers of LS, or simply returning a pointer to the proper location to query.
04/22/23Swany
IntroductionLocal LS Schematic
04/22/23Swany
IntroductionGlobal LS Schematic
04/22/23Swany
Functionality
XSLT Transformation
New XMLDB organization, procedures
Join Algorithm
Token Passing Algorithm
Leader Election / Leader Mirroring
Query Interface
XML Formats
04/22/23Swany
Functionality - XSLT
XSLT is a part of the The Extensible Stylesheet Language Family (XSL)
XSLT is a language for transforming XML documents into other XML documents.
Idea:Define a set of definitions that match a specific pattern.
Feed source documents that fit this pattern, get back a new document defined by the stylesheet
04/22/23Swany
Functionality - XSLTExample (initial Document):
04/22/23Swany
Functionality - XSLTExample (stylesheet):
04/22/23Swany
Functionality - XSLTExample (final document):
04/22/23Swany
Functionality - XSLT
Use in the LS:Define 'Local' and 'Global' patterns summarize the contents of 'LSStore'.
LocalCompact both the service definition (MP, MA, etc.), and the actual metadata description (interface, endPoint, etc).
GlobalSame idea, but compact multiple services/metadata instances further (reduce domain scope, etc.)
04/22/23Swany
Functionality - XSLTUse in the LS:
All summarized data is stored in the 'LSCache' file.
Periodic event performs the summarization of LSStore information, stores results in LSCache.
Summaries sent by peer LS instances are stored in LSCache
LSCache is subject to the same cleanup rules as LSStore (should summary life be longer?)
04/22/23Swany
Functionality - XSLTOpen Issues
Format/Style of summaries depends on how much information an LS is willing to share
Formats must be similar (to the same generic pattern of the LSStore) or queries are useless.
If an LS drops out of the ring, how long before it's summary should be removed from other's caches?
Should be a function of the size and speed of the ring.
04/22/23Swany
Functionality - Joining
When a new LS comes online, it will try to join up with other LS instances.
For now depends on list of 'known' LS deployments (don't need to be active, but could be)
List will be updated over time
Once other running LS instances are made aware of a new member, the logical ring can grow.
04/22/23Swany
Functionality - Joining
Joining Example:LS1 is started, looks through its peer list:
04/22/23Swany
Functionality - Joining
Joining Example:LS2 Inserts LS1 into it's peer list, and will inform other members (LS3) as well.
04/22/23Swany
Functionality - Joining
Joining Example:LS1 is added to the ring.
04/22/23Swany
Functionality - Joining
Joining Example:The Peer List is kept 'updated' by an artificial mechanism meant to measure how well each LS knows each other:
04/22/23Swany
Functionality - JoiningPeer List:
The 'familar' parameter is incremented/decremented (max=3 min=0) to indicate when the last communication between a source and destination LS.
When starting, all values are set to '0'.
Each update from an LS increments the value
If an LS can't contact a particular LS, the value is decremented.
04/22/23Swany
Functionality - JoiningPeer List:
A Peer List with all 0 values means needing to rejoin the ring again.
Any time a single LS reaches 0 they are taken out of the ring.
04/22/23Swany
Functionality – Token Passing
The Join procedure touches on the nature of token passing.
Basic Idea:Pass around 'your' list of active peers.
Each LS will add 'new' peers to their own list
After waiting some time (token delay) a summary message (from the 'LSCache') will be sent to all active peers.
Token is passed again.
04/22/23Swany
Functionality – Token Passing
LS1 is currently holding the token. It is now time to pass it:
04/22/23Swany
Functionality – Token Passing
Summary information is retrieved from LSCache through an XMLDB query.
A summary Message is created, each Peer gets the message; update the peer list for success/failure
Create a Token Message from the Peer list, pass it to the 'next' LS.
04/22/23Swany
Functionality – Token Passing
Same Idea, new LS:
04/22/23Swany
Functionality – Token Passing
What about new additions to the Ring?When a new LS joins, only one LS instance in the ring knows at first
When the 'new' LS gets the token for the first time it will send its own summary to the group
New summaries are treated just like a Join.
04/22/23Swany
Functionality – Token PassingAdding a new member:
04/22/23Swany
Functionality – Token PassingAdding a new member:
04/22/23Swany
Functionality – Leader ElectionEach Local group must be aware of the 'Leader' that communicates to the outside world:
Sends/Receives Group Summaries from related Domains
Acts as a 'non-authoritative' answer to queries that may need to go outside of the domain
How to 'elect' this leader?
04/22/23Swany
Functionality – Leader ElectionElection:
Take your hostname as an IP, convert to a 'long' value:
127.0.0.1 = 127*256^2 + 0*256^2 + 0*256^1 +1*256^0 = 2130706432
Assign some 'rank' between 0 and 5(?) (this can be used to ensure a certain LS is always the leader)
CAT 1 2130706432 = 12130706432
04/22/23Swany
Functionality – Leader ElectionThese ID values are used when communicating the summaries and tokens around the ring
The 'smallest' ID is the leader of a group.
The next 1 or 2 smallest entries can be selected to be 'Secondary' and 'Tertiary' LS instances in the logical ring
Are given the Global summaries in case the leader fails.
04/22/23Swany
Functionality – Query Interface
A query, be it from a client or a service, needs to respond in a uniform manner:
Iterative:A query that results in information being NOT on the inital LS returns only the 'location' (i.e. LS information) of where the true data lives.
Recursive:A query will iterate through LS instances automatically returning the information, and caching this data as it progresses.
04/22/23Swany
Functionality – Query InterfaceIterative Queries:
Simple Xpath/XQuery statements can be created to search both the LSCache/LSStore and return either the requested data, or the LS that has the requested data.
This requires either expert knowledge of Xpath (a pretty tall assumption) or publishing 'cookbook' queries.
04/22/23Swany
Functionality – Query Interface
Iterative Queries:declare namespace nmwg="http://ggf.org/ns/nmwg/base/2.0/";
declare namespace nmwgt="http://ggf.org/ns/nmwg/topology/2.0/";
for $m in /nmwg:store[@type='LSStore' or @type='LSCache']/nmwg:metadata
for $d in /nmwg:store[@type='LSStore' or @type='LSCache']/nmwg:data
where $m/@id=$d/@metadataIdRef and $d//nmwgt:hostName/text()[fn:matches(.,'.*.abilene.ucaid.edu')]
return $m
This particular query is looking for anything on the Abilene network, it will return either an LS metadata, or the actual MP metadata (if it is local to this LS).
04/22/23Swany
Functionality – Query InterfaceRecursive Queries:
May be able to get away with using the same Xpath expression, but requires logic to send the query to the LS in question when we get a Hit.
Logic to Cache results is needed as well.Do we treat the returned results as 'registration' data, or store it in our LSCache?
04/22/23Swany
Functionality – XML FormatsIntroduced a new message:
Token
Created a new function for an old message:
LSRegistration, used for summaries
Created 2 new storage mediums:
Peer List
Cache
04/22/23Swany
Functionality – Token MSGType = LSToken
Use Parameters (at the msg level) to indicate if this is a 'Local' or 'Global' token.
Body consists of a peer list.
04/22/23Swany
Functionality – Token MSGFormat:
04/22/23Swany
Functionality – Reg MSGType=LSRegisterRequest
Use Parameters (at the msg level) to indicate if this is a 'Local' or 'Global' token, and that it is a summary message.
Body consists of a metadata tag describing the sending LS, and related data that is a summary of that LS's LSStore.
04/22/23Swany
Functionality – Reg MSGFormat:
04/22/23Swany
Functionality – Peers StoreType=LS(Local|Global)Peers
Same Idea as the LSStore/LSStore-control, meant to house peer entries for each level.
04/22/23Swany
Functionality – Cache StoreType=LSCache
Same Idea as the LSStore/LSStore-control, meant to house cache information that we created, or others give us.
04/22/23Swany
Dev Status
Completed:XSLT, XML DB organization, Peer Tracking, Join Alg., Token Passing Alg., Leader Election, Leader Mirroring, eXist changes (xquery+update instead of xupdate, v1.1)
In Progress:Testing, recursive LS queries, finalize XML formats, experiment on timing and efficiency, code cleanup, error handling
ToDo:Junit, using jar repository?, merge recent trunk changes into my branch, What becomes of my branch?
04/22/23Swany
Future PlansComplete testing/experimentation
Start Deployment to various locations
Merge into trunk in preparation for next release
04/22/23Swany
DemoDemo:
Two installations of MLS, we will be simulating 'local' interaction only
Different data will be registered to each LS, it will be summarized, and exchanged via the token passing protocol
This will be verified by examining the eXist DB contents.
Lookup Service
Lookup Service
Service DiscoveryFrom the client
System component inter-communication
Broad Design ConsiderationsUse existing tooling as much as possible
Flexible levels of information exchange
Scalability via hierarchical organization
Tolerant of component failures
Service Registration with an LS
Express a perfSONAR service as a SubjectReuse of existing tools and protocol elements inside the perfSONAR tool namespace
Each service creates a self-identification record consisting of contact point and administrative detailsEncode information to be advertised as Data
For an MA, encode Metadata of stored measurements inside Data
LS Registration with other LSes
Create registration infoset that summarizes local registration data
Including other known LS instances
Periodically register self-information and registration infoset with other LS services in the same scope
A service (including an LS) may register in multiple scopes
Multi-LS
Simple case in whichLS instances register
with one another
Multi-LS
Group nodes by someadministrative scope
(DNS, X.500)
Full mesh registrationhas scalability issues
Circulate control token
Multi-LS
Token rotation protocolalso allows for leader
election of “designatedLS”
Multi-LS
Hierarchyof domains
formed
LS Registration Protocol OverviewRegistration token includes current LS set
This allows the ring to heal in the case of failures
Initial leader can potentially be chosen based on a variety of metrics
Currently, static priority and IP address are used (allows an administrator to designate defaults)
Leader initiates token at tunable target interval, but all nodes notice absence of token to regenerate/re-elect
Propagation of Registration Information
Essentially, summarize registrations and propagateMust be easy to changeWe don’t know all the answers here and leaving as much room as possible for future customization is very desirable
Approach: use XML Stylesheet Language Transformations (XSLT)
This transformation language allows us to change one XML document into another formGeneric, standalone, portable, language-independent, extensible
“Fast” may not be among these
Status of Prototype
Basic LS exists in the upcoming distribution
Implementation by Maciej Glowiak at PSNC
Multi-LS features implemented in early form
Implementation by Jason Zurawski at U. Delaware
Status of Prototype
Summarization stylesheets
Token protocolHandles nodes joining and leaving
“Internal” and “External” summaries maintained in the XML store
Design Issues
UDDIDesigned with certain things in mind, thus fixed fields don’t map well to our purposes
Version 3 has extensibility via arbitrary XMLIf the majority information of the goes into this XML document, what do we gain?
Generic XML databaseFlexibility, but standards are evolving
Design Issues
Query and update mechanismsXPath certainlyXQuery very likelyXUpdate necessary?XSLT
Iterative vs RecursiveIterative queries are more simpleRecursive queries allow opportunities to speed responses by caching data
Design Issues
DiscoveryCurrently, use a “cached-zone” like DNS in that knowing one server will allow us to get a relatively current list
Better solutions are possiblemDNS
Multicast and Anycast