Design and Design and Implementation of Implementation of

HTTP-Gnutella GatewayHTTP-Gnutella Gateway

Baoning Wu (baw4)Baoning Wu (baw4)Wei Zhang (wez5)Wei Zhang (wez5)

CSE DepartmentCSE DepartmentLehigh UniversityLehigh University

MotivationMotivation Peer-to-peer networking is a hot topic.Peer-to-peer networking is a hot topic. Can P2P nodes search and get files Can P2P nodes search and get files

from Web sites?from Web sites? Can one P2P network search and get Can one P2P network search and get

files from other P2P networks?files from other P2P networks?

In our project, we have built a special In our project, we have built a special gateway between Gnutella and Web gateway between Gnutella and Web sites.sites.

Related WorkRelated Work David McNab has launched Freenet search David McNab has launched Freenet search

engine.engine. Asiayeah is a Gnutella search engine.Asiayeah is a Gnutella search engine. Filedonkey.com is an Edonkey search Filedonkey.com is an Edonkey search

engine.engine. Kalepa Networks , Inc is doing work about Kalepa Networks , Inc is doing work about

connecting different P2P systems.connecting different P2P systems.

Our work is kind of reverse to all above Our work is kind of reverse to all above works.works.

Mechanism of Gnutella Mechanism of Gnutella SearchingSearching

Node A sends a query to its neighbor B; Node A sends a query to its neighbor B; Node B boardcasts the query to its neighors C, D; Node B boardcasts the query to its neighors C, D; Node C has the objects node A needs and then returns a Node C has the objects node A needs and then returns a

query hit message to node B; query hit message to node B; Node B forwards the query hit message by consulting Node B forwards the query hit message by consulting

the local states.the local states.

Architecture of HTTP-Architecture of HTTP-Gnutella GatewayGnutella Gateway

Mechanism of the Mechanism of the gatewaygateway

1.1. Node A broadcasts a query message directly or Node A broadcasts a query message directly or indirectly to the HTTP-Gnutella gateway; indirectly to the HTTP-Gnutella gateway;

2.2. The HTTP-Gnutella gateway forwards the translated The HTTP-Gnutella gateway forwards the translated query message to search engine; query message to search engine;

3.3. The search engine returns a bunch of query results The search engine returns a bunch of query results to the gateway; to the gateway;

4.4. The gateway translates the results into Gnutella The gateway translates the results into Gnutella formats and then forwards them to node A;formats and then forwards them to node A;

5.5. If node A initializes a download requests to the If node A initializes a download requests to the gateway, the gateway will translate the Gnutella gateway, the gateway will translate the Gnutella request into a well-formatted HTTP request to the request into a well-formatted HTTP request to the Web server; Web server;

6.6. The gateway fetches the data from the Web server; The gateway fetches the data from the Web server; 7.7. The gateway forwards the data from the Web server The gateway forwards the data from the Web server

to node A.to node A.

Handle Query MessagesHandle Query Messages

We still use the original Gnutella We still use the original Gnutella mechanism to judge whether to mechanism to judge whether to forward the message or not.forward the message or not.

The gateway captures all of queries The gateway captures all of queries with hops# < 5 and sends them to with hops# < 5 and sends them to search engine.search engine.

Search Engine APISearch Engine API

Google search engine API has a limit Google search engine API has a limit of up to 1,000 requests per day.of up to 1,000 requests per day.

Search engine API consists of three Search engine API consists of three main functions:main functions: Query conversionQuery conversion Extraction of URLsExtraction of URLs Measurement of content sizeMeasurement of content size

Generate Query Hit Generate Query Hit MessagesMessages

Two considerations:Two considerations: Let Gnutella nodes contact Web servers Let Gnutella nodes contact Web servers

directlydirectly Let the gateway work as a proxyLet the gateway work as a proxy

The gateway fills its own IP address The gateway fills its own IP address and a specific port number (currently and a specific port number (currently 9999) in the query hit messages.9999) in the query hit messages.

File names are URLs of Web objects.File names are URLs of Web objects.

Downloading ServiceDownloading Service Translate Gnutella download request Translate Gnutella download request

into a well-formatted HTTP request. into a well-formatted HTTP request. e.g.e.g.

GET /get/1234/http://www.foo.com/foo.mp3 HTTP/1.1GET /get/1234/http://www.foo.com/foo.mp3 HTTP/1.1

User-Agent: GnutellaUser-Agent: Gnutella

Host: 123.123.123.123:6346Host: 123.123.123.123:6346=>=>

GET http://www.foo.com/foo.mp3 HTTP/1.1GET http://www.foo.com/foo.mp3 HTTP/1.1

User-Agent: GnutellaUser-Agent: Gnutella

Host: www.foo.comHost: www.foo.com

It should handle Gnutella handshakes It should handle Gnutella handshakes properly.properly.

It also records the bytes transferred.It also records the bytes transferred.

Problems & SolutionsProblems & Solutions

Irregular handshakesIrregular handshakes We handle all possibilitesWe handle all possibilites

File sizeFile size We use HTTP HEAD request to get file We use HTTP HEAD request to get file

sizesize Broken Pipe signalBroken Pipe signal

We use forked processWe use forked process

Experiment ResultsExperiment Results

OutlineOutline Basic verification and validationBasic verification and validation Log file formatLog file format Results #1 to #4Results #1 to #4

Basic Verification & Basic Verification & ValidationValidation

Run our special gateway on machine 1 Run our special gateway on machine 1 and run a normal gtk-gnutalla client on and run a normal gtk-gnutalla client on machine 2. After machine 2 connects to machine 2. After machine 2 connects to machine 1, we use machine 2 to send machine 1, we use machine 2 to send query messages and downloading query messages and downloading request to machine 1.request to machine 1.

For downloaded files from machine 1, For downloaded files from machine 1, we use wget to get the same file from we use wget to get the same file from web server directly and use diff to test web server directly and use diff to test if they are identical. if they are identical.

Log File FormatLog File Format

Log 1Log 1 Time stamp, MUID, IP address, Type, Time stamp, MUID, IP address, Type,

QueryQuery

Log 2Log 2 Time stamp, IP address, URL, Size, Time stamp, IP address, URL, Size,

Code, SuccessCode, Success

Results #1Results #1

No. of Query messages: 319,245No. of Query messages: 319,245 No. of Query Hit messages: 930,860No. of Query Hit messages: 930,860 No. of served requests: 113,391No. of served requests: 113,391 Average Response Time: 16.33 Average Response Time: 16.33

secondsseconds

Result #2Result #2

1 2 3 4 5 6 7 8 90

10000

20000

30000

40000

50000

60000

70000

80000

90000

100000

number of responses

nu

mb

er

of r

eq

ue

sts

Result #3Result #3 No. of Downloading requests: 952No. of Downloading requests: 952 No. of Different IP addresses: 67No. of Different IP addresses: 67 No. of served Requests: 945No. of served Requests: 945 No. of sucessfully served requests: 740No. of sucessfully served requests: 740 Total size transfered: 244,227,881 bytesTotal size transfered: 244,227,881 bytes Average response time: 3.15 secondsAverage response time: 3.15 seconds Average total download time: 15.92 Average total download time: 15.92

seconds seconds

Result #4Result #4

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 200

10

20

30

40

50

60

70

80

90

100

110

120

different sites

num

ber

of d

ownl

oade

d fil

es

Future WorkFuture Work

Support a variety of file types and Support a variety of file types and measure their popularitymeasure their popularity

Build a gateway to connect different Build a gateway to connect different P2P systemsP2P systems

Deployment of such gatewaysDeployment of such gateways

ConclusionConclusion

An HTTP-Gnutella gateway was built An HTTP-Gnutella gateway was built and worked for the Gnutella users.and worked for the Gnutella users.

Only 5 days, the gateway transferred Only 5 days, the gateway transferred about 244MB data from the Web sites about 244MB data from the Web sites to the Gnutella nodes.to the Gnutella nodes.

The systems achieved all goals of our The systems achieved all goals of our design.design.

Question?Question?