Date post: | 14-Jan-2016 |
Category: |
Documents |
Upload: | ashlynn-blackstock |
View: | 213 times |
Download: | 0 times |
Implementing Declarative Overlays
From two talks by:Boon Thau Loo1
Tyson Condie1, Joseph M. Hellerstein1,2, Petros Maniatis2, Timothy Roscoe2, Ion Stoica1
1University of California at Berkeley, 2Intel Research Berkeley
P2
Overlays Everywhere…
Overlay networks are widely used today: Routing and forwarding component of large-
scale distributed systems Provide new functionality over existing
infrastructure
Many examples, variety of requirements: Packet delivery: Multicast, RON Content delivery: CDNs, P2P file sharing, DHTs Enterprise systems: MS Exchange
Overlay networks are an integral part of many large-scale distributed systems.
Overlay networks are an integral part of many large-scale distributed systems.
Problem
Non-trivial to design, build and deploy an overlay correctly: Iterative design process:
Desired properties Distributed algorithms and protocols Simulation Implementation Deployment Repeat…
Each iteration takes significant time and utilizes a variety of expertise
The Goal of P2
Make overlay development more accessible: Focus on algorithms and protocol designs, not
the implementation
Tool for rapid prototyping of new overlays: Specify overlay network at a high level Automatically translate specification to protocol Provide execution engine for protocol
Aim for “good enough” performance Focus on accelerating the iterative design
process Can always hand-tune implementation later
Outline
Overview of P2Architecture By Example Data Model Dataflow framework Query Language
ChordAdditional Benefits Overlay Introspection Automatic Optimizations
Conclusion
6
All-Pairs Reachability
R2: reachable(S,D) link(S,Z), reachable(Z,D)
R1: reachable(S,D) link(S,D)
Input: link(source, destination)Output: reachable(source, destination)
“For all nodes S,D, If there is a link from S to D, then S can reach D”.
link(a,b) – “there is a link from node a to node b”
reachable(a,b) – “node a can reach node b”
7
All-Pairs Reachability
R2: reachable(S,D) link(S,Z), reachable(Z,D)
R1: reachable(S,D) link(S,D)
Input: link(source, destination)Output: reachable(source, destination)
“For all nodes S,D and Z, If there is a link from S to Z, AND Z can reach D, then S can reach D”.
8
All-Pairs Reachability
R2: reachable(S,D) link(S,Z), reachable(Z,D)
R1: reachable(S,D) link(S,D)
Input: link(source, destination)Output: reachable(source, destination)
“For all nodes S,D, If there is a link from S to D, then S can reach D”.
link(a,b) – “there is a link from node a to node b”
reachable(a,b) – “node a can reach node b”
9
All-Pairs Reachability
R2: reachable(S,D) link(S,Z), reachable(Z,D)
R1: reachable(S,D) link(S,D)
Input: link(source, destination)Output: reachable(source, destination)
“For all nodes S,D and Z, If there is a link from S to Z, AND Z can reach D, then S can reach D”.
10
Towards Network Datalog
Specify tuple placement Value-based partitioning of tables
Tuples to be combined are co-located Rule rewrite ensures body is always single-site
All communication is among neighbors No multihop routing during basic rule
execution Link-restricted rules: Enforced via simple
syntactic restrictions
11
All-Pairs Reachability
R1: reachable(@S,D) link(@S,D)
R2: reachable(@S,D) link(@S,Z), reachable(@Z,D)
Network Datalog
Query: reachable(@M,N)
@S
D
@a
b
@a
c
@a
d
reachableOutput
table:
Input table:
Query: reachable(@a,N)
@S
D
@c b
@c d
link@
SD
@b
c
@b
a
link@
SD
@a
b
link @
SD
@d
c
link
b dca
@S
D
@b
a
@b
c
@b
d
reachable @
SD
@c a
@c b
@c d
reachable @
SD
@d
a
@d
b
@d
c
reachable
Location Specifier “@S”
Query: reachable(@a,N)
12
All-Pairs Reachability
R2: reachable(S,D) link(S,Z), reachable(Z,D)
R1: reachable(S,D) link(S,D)
Input: link(source, destination)Output: reachable(source, destination)
“For all nodes S,D, If there is a link from S to D, then S can reach D”.
link(a,b) – “there is a link from node a to node b”
reachable(a,b) – “node a can reach node b”
13
All-Pairs Reachability
R2: reachable(S,D) link(S,Z), reachable(Z,D)
R1: reachable(S,D) link(S,D)
Input: link(source, destination)Output: reachable(source, destination)
“For all nodes S,D and Z, If there is a link from S to Z, AND Z can reach D, then S can reach D”.
14
R1: path(@S,D,P) link(@S,D), P=(S,D).
R2: path(@S,D,P) link(@S,Z), path(@Z,D,P2), P=SP2.
@S
D P @S
D P
@c d [c,d]
Query Execution
@S
D P @S
D P
Query: path(@a,d,P)
Neighbor table:
@S
D
@c b
@c d
link@S D
@b c
@b a
link@
SD
@a
b
link @S D
@d c
link
b dca
path path path
Forwarding table:
Chord Model
Relational data: relational tables and tuples Two kinds of tables: Stored, soft state:
E.g. neighbor(Src,Dst), forward(Src,Dst,NxtHop) Transient streams:
Network messages: message (Rcvr, Dst) Local timer-based events: periodic (NodeID,10)
...
Netw
ork In Dataflow
...
...
Netw
ork Out D
ataflow
node
Local Tables
route ...
Example: Ring Routing
3
28
15
1840
60
58 13
37
0
56
42
222433
Each node has an address and
an identifier
Each object has an
identifier.
Every node
knows its successor
Objects “served” by successor
3
28
15
1840
60
58 13
37
Ring State
node(IP40,40) succ(IP40,58,IP58)
node(IP58,58) succ(IP58,60,IP60)
Stored tables: node(NAddr, N) succ(NAddr, Succ, SAddr)
Example: Ring lookup
3
28
15
1840
60
58 13
37
0
56
42
222433
Find the responsible node for a given key k?
n.lookup(k)if k in (n, n.successor)
return n.successor.addr
elsereturn n.successor.
lookup(k)
lookup(IP40,IP37,59)
Ring Lookup Events
3
28
15
1840
60
58 13
37
node(IP40,40) succ(IP40,58,IP58)
Event streams: lookup(Addr, Req, K) response(Addr, K, Owner)
lookup(IP37,IP37,59)
response(IP37,59,IP60)
lookup(IP58,IP37,59)
node(IP58,58) succ(IP58,60,IP60)
n.lookup(k)
if k in (n, n.successor) return n.successor.addrelse return n.successor. lookup(k)
Query Language: Overlog
Datalog rule syntax: <head> <condition1>, <condition2>, … , <conditionN>.
Overlog rule syntax: <Action> <event>, <condition1>, … , <conditionN>.
Query Language: Overlog
Event: RECEIVE lookup(NAddr, Req, K)
Condition: lookup(NAddr, Req, K) & node(NAddr, N) & succ(NAddr, Succ, SAddr) & K in (N, Succ]
Action: SEND response(Req, K, SAddr) to Req
response@Req(Req, K, SAddr)
lookup@NAddr(Naddr, Req, K),node@NAddr(NAddr, N),
succ@NAddr(NAddr, Succ, SAddr), K in (N,Succ].
Overlog rule syntax: <Action> <event>, <condition1>, … , <conditionN>.
P2-Chord
Chord Routing, including: Multiple successors Stabilization Optimized finger
maintenance Failure recovery
47 OverLog rules13 table definitionsOther examples:
Narada, flooding, routing protocols
10 pt font
Introspection with Queries
Unifying framework for debugging and implementation Same query language, same platform
Execution tracing/logging Rule and dataflow level Log entries stored as tuples and queried
Correctness invariants, regression tests as queries: “Is the Chord ring well formed?” (3 rules) “What is the network diameter?” (5 rules) “Is Chord routing consistent?” (11 rules)
With Atul Singh (Rice) and Peter Druschel (MPI)
Automatic OptimizationsApplication of traditional Datalog optimizations to network routing protocols (SIGCOMM 2005)Multi-query sharing:
Common “subexpression” elimination Caching and reuse of previously computed results Opportunistically share message propagation across
rules Join
lookup.Addr =
node.Addr
Joinlookup.Ad
dr = succ.Addr
lookup SelectK not in (N,Succ)
Projectlookup(SAddr
,Req,K)
lookup
ProjectResponse(Req,K,SAddr)
SelectK in
(N,Succ)
responseJoinlookup.Add
r = node.Addr
lookupJoin
lookup.Addr =
succ.Addr
Automatic Optimizations
Cost-based optimizations Join ordering affects performance
Joinlookup.Add
r = node.Addr
Joinlookup.Add
r = succ.Addr
SelectK not in (N,Succ)
lookup Projectlookup(SAddr
,Req,K)
lookup
ProjectResponse(Req
,K,SAddr)
SelectK in
(N,Succ)
response
Joinlookup.Add
r = node.Addr
Joinlookup.Add
r = succ.Addr
Future Work
“Right” languageFormal data and query semantics Static analysis Optimizations Termination Correctness
Conclusion
P2: Declarative Overlays Tool for rapid prototyping new overlay
networks
Declarative Networks Research agenda: Specify and
construct networks declaratively Declarative Routing : Extensible
Routing with Declarative Queries (SIGCOMM 2005)