Kalman Graffi - Efficiency and Information Management in Peer-to-Peer Systems

2008_Kalman.Graffi_QuaP2P_Kolloquium_Efficiency.Management.ppt

KOM - Multimedia Communications LabProf. Dr.-Ing. Ralf Steinmetz (director)

Dept. of Electrical Engineering and Information TechnologyDept. of Computer Science (adjunct professor)

TUD – Technische Universität Darmstadt Merckstr. 25, D-64283 Darmstadt, Germany

Tel.+49 6151 164959, Fax. +49 6151 166152 www.KOM.tu-darmstadt.de

© author(s) of these slides 2008 including research results of the research network KOM and TU Darmstadt otherwise as specified at the respective slide8. April 2023

Dipl.-Math. Dipl.-Inform. Kalman Graffi

[email protected]

Efficiency and Information Management in Peer-to-Peer Systems

QuaP2P Kolloquium, 9. April 2008

7.31.10.25

peer-to-peer.info

12.5.7.31

95.7.6.10

86.8.10.18

planet-lab.orgberkeley.edu 89.11.20.15

Efficiency and Information Management

KOM – Multimedia Communications Lab 2

Overview

1 The Peer-to-Peer Paradigm

1.1 Trends in Peer-to-Peer Research

1.2 Efficiency in Peer-to-Peer Systems

2 Towards QoS & Emergency Call Handling

2.1 Serious Application: Emergency Call Handling

2.2 Towards QoS for Overlay Flows

2.3 Overlay Bandwidth Management

3 Lessons Learned for QoS in P2P Systems

4 Towards a Kind of „Efficiency Management”

4.1 Current State of Efficiency Management

4.2 Our Vision of an Efficiency Management Lifecycle

4.3 Efficiency Management Module

4.4 Current Research Focus: Improved Architecture for Efficiency Management

5 Lessons Learned for Efficiency Management in P2P Systems


The Peer-to-Peer Paradigm

Peer-to-Peer Systems: Users of a system provide the infrastructure of the system Service is provided from users/peers to users/peers Peer-to-Peer overlays:

virtual networks, providing new functionality E.g. Distributed Hash Tables, Keyword-based Search

Evolution of applications File sharing:

No Quality of Service (QoS) requirements Voice over IP

Real-time requirements Video-on-demand

Real-time and bandwidth requirements

H(„my data“)

= 31072207

7.31.10.25

peer-to-peer.info12.5.7.31

95.7.6.10

86.8.10.18

planet-lab.orgberkeley.edu

29063485

201116221008709

611

61.51.166.150

?

1


Trends in Peer-to-Peer Research

Quality aspects gain importance Reliability: expected professionalism Ease of Use:

Multimedia and interactivity

Critical success factor for complex P2P applications modular P2P applications

Quality aspects: Adaptability – to scenario, system scale Validity – of stored data Trust – of users and mechanisms Efficiency – ratio between performance and costs

Costs Security

Quality of P2P Systems

Retrievability

Coherence

Consistency

Correctness

PerformanceScalability

Flexibility

Stability

Dependability

Service Provisioning

Overlay Operations

Individual Node

Complete System

IP Infrastructure

Availability

Reliability

Robustness/ Fault tolerance

Integrity

Confidentiality

Authentication

Non- repudiation

TrustValidityEfficiencyAdaptability

Costs Security

Quality of P2P Systems

Retrievability

Coherence

Consistency

Correctness

PerformanceScalability

Flexibility

Stability

Dependability

Service Provisioning

Overlay Operations

Individual Node

IP Infrastructure

Availability

Reliability

Robustness/ Fault tolerance

Integrity

Non-

TrustValidityEfficiencyAdaptability

1.1


Efficiency in Peer-to-Peer Systems

Future Peer-to-Peer based applications Modular, component based composition A module has to

be highly efficient provide Quality of Service

Efficiency can be viewedin three aspects: Resource usage

in the peer in the overlay in the network

Question in peer-centric view: local knowledge vs. system wide solution Devices

IP Infrastructure

3

$

$

P2P Overlay

2

^Peer

IN OUT

10100110110001000110

1

1.2


Efficiency Management in Peers

Taxonomy on Sched. and AQMMechanisms in P2P ScenariosK. Graffi et al.Technical Report KOM-2007-1/2, 2007

Taxonomy of Scheduling and Active Queue Mechanisms on Overlay Flows

Welche Strategien aus der Netzwerk-schicht sind auch im Overlay anwendbar?

01/2007

1. Message SchedulingBefore:

After:

2. Queue ManagementBefore:After:

Queue Limit

Overlay Bandwidth Management – Scheduling and Active Queue Management of Overlay Flows

Overlay Bandwidth Mana-gement: Sched. and AQM of Overlay Flows, K.Graffi et al., IEEE LCN ‘07

Wie kann man QoS für Overlay Flows erbring-en? Welche Verfahren sind geeignet? Sind lokale Lösungen aus-reichend?

10/2007

Wie kann man QoS (geringe Latenz, Loss) für P2P-basierte Notrufdienste erbringen? Anwendung der Erkenntnisse.

ECHoP2P: Emergency Call Handling over P2P Overlays

ECHoP2P: Emergency CallHandling over P2P OverlaysK.Graffi, A.Kovacevic, et al. P2P-NVE ‘07

11/2007

Connect me to an emergency station!

Emergency Call Handling

QoS Provisioning

05/2007

P2P Forschung - Übersicht und Herausforderungen, K.Graffi, Quap2P, it-Informa-tion Technology, 2007

Peer-to-Peer Research – Overview and Challenges

Was ist Peer-to-Peer? Gemeinschaftsarbeit von QuaP2P mit einem Überblick zu Peer-to-Peer und seinen Herausforder-ungen.

Network Wrapper

Overlay Layer

User

UDP TCP

Online-timeModel

Behavior

PackageLoss

DelayModel

Bandwidth

Kademlia

Sim

ulatio

n E

ng

ine

Application LayerDistribution

Strategy

Chord

ReplicationStrategy


Efficiency Management in P2P Networks

05/2008

Load Balancing for Multimedia Streaming in Heterogeneous Peer-to-Peer SystemsK.Graffi,…,A.Kovacevic, et al. ACM NOSSDAV ‘08

Load Balancing for Multimedia Streaming in Heterogeneous Peer-to-Peer Systems

Wie kann die Ressourc- en und Aufgabenver-teilung in P2P Netzen fair gestaltet werden?

Wie kann man Lasten-verteilung und Hetero-genität unterstützen?

Eine allgemeines Over-Overlay für Attribute-based Peer Search. Wie gestaltet man skalierende Informationsarchitekturen?

Information and Efficiency Management: Attribute-based Peer Search for Structure P2P Overlays

Towards and Information and Efficiency Management Arch… K. Graffi et al. Technical Report KOM-2008-2, 2008

4/2008

7.31.10.25


95.7.6.10

86.8.10.18

graffi.org-lifesocial.org 130.83.139.139

KBR Layer

UnifiedID Space

with Overlay specific IDs

KBR using Unified ID Space

CoordinatorSupport Peer

Peer

From Cells to Organisms: Long-Term Guarantees on Service Provisioning in P2P NetworksK. Graffi, A.Kovacevic, et al. ACM SIGAPP NOTERE ’08

From Cells to Orga-nisms: Long-term Guarantees on Service Provisioning in Peer-to-Peer Networks

Wie kann man die Optimierung der Res- sourcenverteilung im Overlay als Aufgabe des Overlays gestalten?

06/2008

Information and Efficiency Management in the case of FreePastry

Information and Efficiency Management in the Case of FreePastry, K.Graffi et al., to be published ‘08

Ein Statistik-Modul (für Overlay Service Provider) und Attribute-based Peer Search.

3/2008

7.31.10.25


95.7.6.10

86.8.10.18



Towards QoS & Emergency Call Handling

Connect me to an emergency

station!

Emergency Call Handling

QoS Provisioning

2


Serious Future Peer-to-Peer Applications

Application Areas To exploit self-organization abilities of P2P

Catastrophe scenarios require robust mechanisms E.g. coping with churn

Example: Emergency Call Handling Hard QoS requirements Peer-to-peer mechanisms provide

failure-tolerance and Quality of Service

See how the efficiency can be improved in this example Learn from it


Serious Application: Emergency Call Handling

Emergency Call Handling is not supported in VoIP (Skype) 2009: mandatory for VoIP providers P2P fits: all-IP, scalable,

but Quality of Service?

Requirements1. Location critical service:

Find closest/responsible Emergency Station

2. Quality of Service for P2P flows needed QoS policy: low delay, low loss

contact Emergency Station as soon as possible without message loss

Goal: How to solve problem locally ? OR

do we need system wide management? Alabama Emergency Zones

Snapshot of the simulated scenarios

Source: US census

Source: NENA

Paper at: K. Graffi et al., “ECHoP2P…”, Int. Workshop on P2P-NVE, Nov. 2007

Population density in Alabama

2.1


Our Approach for P2P-based Emergency Call Handling

Challenge 1: Location-based search requirements

Approach: Globase.KOM - Geographical LOcation BAsed SEarch Engineered for requirements of location based services A logical neighbor is a geographical neighbor (like in CAN) Tree structure enables search/lookup in O(log N)

Extended with following search mechanisms: Closest peer (Emergency Station) Peer fulfilling a specific criteria (responsibility)

A

B

C

D

E

G

H

I

JK

F

A

B C D E

F G H I JK Search Query

Paper at: A. Kovacevic et al., “Location Awareness…”, Special Issue of the Proc. of the IEEE on Adv. In Distr. Multim. Comm., Jan. 2008


Towards QoS for Overlay Flows

Types of flows in P2P systems

Layer 4 traffic flows: Underlay perspective Out of scope

Direct P2P communication: File transfer, application traffic, … Few, but large data streams (elephants) (Often) with low priority for the system

Overlay flows Multi-hop: maintenance, user queries… Many, small messages (mice) Varying relevance for system

Provide QoS to overlay flow according to its relevance

2.2

P2P Overlay

2

^Peer

IN OUT

10100110110001000110

1


Scheduling and Active Queue Management

Learn from network layer Scheduling

Reorder tasks/messages in queue Reference: First-in-First-Out

Active Queue Management Drop msg. upon congestion of queue Reference: Drop Tail

Research challenge: how to apply on overlay layer?

1. Message SchedulingBefore:

After:

2. Queue ManagementBefore:

After:Queue Limit

MessageSched.+AQM

See: K. Graffi et al., “Taxonomy on Scheduling/AQM Strategies…” Tech. Report, KOM-TR-2007-1,2, TU Darmstadt


Analyzing Overlay Flows in Kademlia

Simulation of Kademlia

Setup: PeerfactSim.KOM 10,000 peers, all join,

random store/lookup

Metrics: # of contacts per peer # of msg per contact

Observation Huge number of contacts with few messages each Overlay “flows” and traditional flows differ significantly

Conclusion Stateless scheduler needed Quality of service information are stored in messages

Paper at: K. Graffi et al., “Overlay Bandwidth Management …” in Proc. of IEEE Loc. Comp. Networks, Oct. 2007


Overlay Bandwidth Management

Novel substrate “Network Wrapper” Between overlay and transport layer:

Queues messages Applies Scheduling and AQM solution: HiPNOS.KOM

HiPNOS.KOM: Highest Priority First, No Starvation Introduce message priorities for Loss and Delay AQM: at congestion, drop message with lowest loss-prio. Scheduling: at free bandwidth, send message with highest delay-prio. Avoid starvation: Periodically increase delay-prio. of queued messages

Properties of HiPNOS.KOM Focus on QoS for overlay flows Easy to apply on existing overlays


2.3


Overlay Bandwidth Management Results

Observation: Proportional relations:

Delay to delay-priority Loss to loss-priority

Results: HiPNOS.KOM provides QoS

Regarding delay and loss According to chosen priorities



Lessons Learned for QoS in P2P Systems

Results for Scheduling and AQM Delay and delay-priority, loss and loss-priority are proportional Emergency Calls have always highest priority All other messages have lower priority Quality of service can be provided

Lessons learned:IF … known:

Optimization criteria Set of all alternatives

THEN mechanisms for Quality of Service are easy to adopt

Question from beginning:How to solve problem locally ? OR do we need system wide management?

Solve it system wide requires information Necessary for efficient decisions in distributed systems Often missing in Peer-to-Peer systems

3


Towards a Kind of „Efficiency Management”

Peers

αβλ

μ

Parameters

f(α, β)=…=xg(λ, μ)=…=yh(α, λ)=…=z

ModelsInterpreted state

Architecture

Choose priorities

Efficiency Management Architecture

Analysis, Modeling and Interpretation

Using Info. to Gain

Efficiency

4


Current State of Efficiency Management

Each functional layerhas its own information/analysis architecture

To gather, analyze layer specific information Examples

BitTorrent: for tit-for-tat peer selection Replication: which data, on which peers

Give me 10 peers with specific availability

Skype: for Superpeer selection Find one peer with specific bandwidth and online time

Network wrapper: underlay awareness Identify most sent message type

Security: Find 5 trustworthy peers

PASTPAST

See: K. Graffi et al., “Towards an Information and Efficiency Management Architecture…” Technical Report, KOM-TR-2008-2, TU Darmstadt

4.1


Desired State of Efficiency Management

Each functional layerhas its own information/analysis architecture

To gather, analyze layer specific information

Common basic functionality can be “abstracted”, i.e. “extracted” To gather layer specific information

Attribute-based peer search To analyze information, (derive optimization goals) To apply results for better decisions

Separate Information/Efficiency Management Layer for this task



Our Vision of an Efficiency Management Lifecycle

Efficiency Management System:

To engineer & to build architecture To gather information from peers To retrieve system parameters

To analyze component To use system model To prepare statistics To interpret system state

With application component To Provide QoS

Based on above issues

Peersα

βλμ

Parameters

f(α, β)=…=xg(λ, μ)=…=yh(α, λ)=…=z

ModelsInterpreted state

Architecture

Choose priorities


Analysis, Modeling and Interpretation

Using Info. to Gain

Efficiency

Paper at: K. Graffi et al., “From Cells to Organisms …” in Proc. of ACM SIGMM NOTERE, June 2008

4.2


Efficiency Management Module

Offers:

Statistics / Monitoring Number of peers, avg. online times Aggregable information

Attribute-based peer search Give me:

3 peers with Storage space > 20Mb Bandwidth > 100kb/s

Uses: Can be used on all structured P2P overlay Requires Key-based Routing (KBR)


4.3

EfficiencyManagement

Module

KBR


First Version of the Efficiency Mgmt. Module

First step: specific solution for FreePastry For Monitoring / Statistics Attribute-based peer search

Built a module for FreePastry Using Scribe to spread data Pub/sub principle

Module can be used!

Valuable component forPIPE Architecture

7.31.10.25

peer-to-peer.info

12.5.7.31

95.7.6.10

86.8.10.18


Underlay:The Internet

StructuredOverlay: DHT


System

Monitoring, Statistics


Current Research Focus: Improved Architecture for Efficiency Management

For all structured P2P overlays Covered by common API KBR Usable by all functional layers in a P2P system

Function: Overlay-independence Monitoring capabilities Attribute-based peer search

Features: Scalability (# of info and peers) Robustness (double-churn) Load-balancing Considering peer heterogeneity Adapting to usage patterns Low overhead


7.31.10.25

peer-to-peer.info

12.5.7.31

95.7.6.10

86.8.10.18


StructuredOverlay: DHT

Underlay:The Internet

KBR Layer

UnifiedID Space

with Overlay specific IDs



System


Peer

4.4



Efficiency Management Architecture Built on underlying structured overlay Communicates via common API

Route to PeerID Just an add-on, easy to deploy

Principle Each node publishes information

updates in the architecture (bottom up) Update-tree is established Each node knows where to send updates to Queries are processed bottom up


UnifiedID Space



Peer


Efficiency Management Architecture Details

Over-overlay: ID space separated in intervals (domains) Peer responsible for a specific ID (e.g. middle) is responsible for ID domain Peers in the domain send updates to this Coordinator Updates propagated upwards the tree

Supporting Peers for Load Balancing Coordinator may chose Supporting Peers Good peers chosen by 50/50 ratio

Pick e.g. 20 best peers in the domain Best 10 peers in domain advertised one level up Second best 10 peers can be used as support

Workload can be delegated to supporting peers Tree depth / peer load adjustable

IDSpace

Peers

Coordinator Supporting Peer



Queries in the Efficiency Management System

Query Type: Give me M peers Fulfilling specific requirements on

Bandwidth, storage space, computational capabilities, Online time, peer load, reputation … (wide set of requirements definable)

Query processing First sent to coordinator of lowest domain Query traverses bottom-up, until M matching peers found Result is sent then to requesting peer Tradeoff:

Upper peers in tree know more Load should be kept on lower levels of the tree



Structure of the Efficiency Management Arch.

Query Performance: O(log N) hopsScalability:

Tree-structure of coordinators form information architecture Supporting peers: Strong peers can take the load

Robustness: No additional maintenance needed (done by structured overlay) Any peer can fail, no unwanted effects



Example Application: Replication Layer

Content storage in P2P systems Churn is a problem

Data may get lost

Replication is a solution

Challenges Which files to replicate?

Most requested, rarest? At which peers?

Most reliable? Highest bandwidth? How many replicas?

Depends on requirements on availability. By which peers?

Efficiency Management System allows for answers


Lessons Learned for Efficiency Management in P2P Systems

Information Management is just ONE part of the Efficiency Management Lifecycle

Next steps: Publish Technical Report on distributed Efficiency Management Architecture on

IEEE P2P 2008 Fine tune on some aspects / limitations

Long-term vision: P2P network regulates itself

According to QoS constraints towards efficiency From self-organization of the peers to self-consciousness of the system* Requires P2P modeling, data mining, distributed decision making, …

Upcoming Applications: P2P-based Grid: Share resources, negotiate service in return with the system* Modularized, layer-interactive, complex applications

* Paper at: K. Graffi et al., “From Cells to Organisms …” in Proc. of ACM SIGMM NOTERE, June 2008

5


Questions?

Kalman Graffi

Peer-to-Peer Research Group

Dept. of Electrical Engineering and Information TechnologyMultimedia Communications Lab · KOM

Merckstr. 25 · 64283 Darmstadt · GermanyPhone(+49) 6151 – 16 49 59Fax (+49) 6151 – 16 61 [email protected]

Further information: http://www.KOM.tu-darmstadt.de/

Publications: http://www.KOM.tu-darmstadt.de/Research/Publications/publications.html

Date post:	17-Nov-2014
Category:	Documents
Upload:	kalman-graffi
View:	1,076 times
Download:	2 times

Kalman Graffi - Efficiency and Information Management in Peer-to-Peer Systems

Documents