Covert Communication in a DarkNetwork

A major new version of freenet

Ian Clarke and Oskar Sandberg

The Freenet Project

Ian Clarke - 2005 – p. 1

Introduction

• We have long been interested in decentralised“Peer to Peer” networks. Especially Freenet.

Ian Clarke - 2005 – p. 2

Introduction

• We have long been interested in decentralised“Peer to Peer” networks. Especially Freenet.

• But when individual users come under attack,decentralisation is not enough.

Ian Clarke - 2005 – p. 2

Introduction

• We have long been interested in decentralised“Peer to Peer” networks. Especially Freenet.

• But when individual users come under attack,decentralisation is not enough.

• Future networks may need to limit connections totrusted friends.

Ian Clarke - 2005 – p. 2

Introduction

• We have long been interested in decentralised“Peer to Peer” networks. Especially Freenet.

• But when individual users come under attack,decentralisation is not enough.

• Future networks may need to limit connections totrusted friends.

• The next version of Freenet will be based on thisphilosophy, a so called Dark Network.

Ian Clarke - 2005 – p. 2

Overview of “Peer to Peer” net-works

• Information is spread across many inter-connected computers

Ian Clarke - 2005 – p. 3

Overview of “Peer to Peer” net-works

• Information is spread across many inter-connected computers

• Users want to find information

Ian Clarke - 2005 – p. 3

Overview of “Peer to Peer” net-works

• Information is spread across many inter-connected computers

• Users want to find information• Some are centralised (eg. Napster), some are

semi- centralised (eg. Kazaa), others aredistributed (eg. Freenet)

Ian Clarke - 2005 – p. 3

Light P2P Networks

• Examples: Gnutella, Freenet, Distributed HashTables

Ian Clarke - 2005 – p. 4

Light P2P Networks

• Examples: Gnutella, Freenet, Distributed HashTables

• Advantage: Globally scalable with the rightrouting algorithm

Ian Clarke - 2005 – p. 4

Light P2P Networks

• Examples: Gnutella, Freenet, Distributed HashTables

• Advantage: Globally scalable with the rightrouting algorithm

• Disadvantage: Vulnerable to “harvesting”, ie.people you don’t know can easily discoverwhether you are part of the network

Ian Clarke - 2005 – p. 4

Dark or “Friend to Friend” P2PNetworks

• Peers only communicate directly with “trusted”peers

Ian Clarke - 2005 – p. 5

Dark or “Friend to Friend” P2PNetworks

• Peers only communicate directly with “trusted”peers

• Examples: Waste

Ian Clarke - 2005 – p. 5

Dark or “Friend to Friend” P2PNetworks

• Peers only communicate directly with “trusted”peers

• Examples: Waste• Advantage: Only your trusted friends know you

are part of the network

Ian Clarke - 2005 – p. 5

Dark or “Friend to Friend” P2PNetworks

• Peers only communicate directly with “trusted”peers

• Examples: Waste• Advantage: Only your trusted friends know you

are part of the network• Disadvantage: Networks are disconnected and

small, they typically don’t scale well

Ian Clarke - 2005 – p. 5

The Small-World Phenomenon

• In "small-world" networks short paths existbetween any two peers

Ian Clarke - 2005 – p. 6

The Small-World Phenomenon

• In "small-world" networks short paths existbetween any two peers

• People tend to form this type of network (asshown by Milgram experiment)

Ian Clarke - 2005 – p. 6

The Small-World Phenomenon

• In "small-world" networks short paths existbetween any two peers

• People tend to form this type of network (asshown by Milgram experiment)

• Short paths may exist but they may not be easy tofind Ian Clarke - 2005 – p. 6

Navigable Small-World Net-works

• Concept of similarity or “closeness” betweenpeers

Ian Clarke - 2005 – p. 7

Navigable Small-World Net-works

• Concept of similarity or “closeness” betweenpeers

• Similar peers are more likely to be connectedthan dissimilar peers

Ian Clarke - 2005 – p. 7

Navigable Small-World Net-works

• Concept of similarity or “closeness” betweenpeers

• Similar peers are more likely to be connectedthan dissimilar peers

• You can get from any one peer to any othersimply by routing to the closest peer at each step

Ian Clarke - 2005 – p. 7

Navigable Small-World Net-works

• Concept of similarity or “closeness” betweenpeers

• Similar peers are more likely to be connectedthan dissimilar peers

• You can get from any one peer to any othersimply by routing to the closest peer at each step

• This is called “Greedy Routing”

Ian Clarke - 2005 – p. 7

Navigable Small-World Net-works

• Concept of similarity or “closeness” betweenpeers

• Similar peers are more likely to be connectedthan dissimilar peers

• You can get from any one peer to any othersimply by routing to the closest peer at each step

• This is called “Greedy Routing”• Freenet and “Distributed Hash Tables” rely on

this principal to find data in a scalabledecentralised manner

Ian Clarke - 2005 – p. 7

Data Networks

• Data Networks (also DHTs) work by assigningeach document with a numerical address or key.

Ian Clarke - 2005 – p. 8

Data Networks

• Data Networks (also DHTs) work by assigningeach document with a numerical address or key.

• Each node is then assigned some section of the“keyspace” in which to specialize.

Ian Clarke - 2005 – p. 8

Data Networks

• Data Networks (also DHTs) work by assigningeach document with a numerical address or key.

• Each node is then assigned some section of the“keyspace” in which to specialize.

• When data is inserted, it is routed towards nodesthat specialize in its part of the keyspace.

Ian Clarke - 2005 – p. 8

Data Networks

• Data Networks (also DHTs) work by assigningeach document with a numerical address or key.

• Each node is then assigned some section of the“keyspace” in which to specialize.

• When data is inserted, it is routed towards nodesthat specialize in its part of the keyspace.

• When data is requested, the query routedlikewise.

Ian Clarke - 2005 – p. 8

Application

How can we apply small-world theory to routing in aDark peer to peer network?

Ian Clarke - 2005 – p. 9

Application

How can we apply small-world theory to routing in aDark peer to peer network?

• A Darknet is, essentially, a social network ofpeoples trusted relationships.

Ian Clarke - 2005 – p. 9

Application

How can we apply small-world theory to routing in aDark peer to peer network?

• A Darknet is, essentially, a social network ofpeoples trusted relationships.

• If people can route in a social network, then itshould be possible for computers.

Ian Clarke - 2005 – p. 9

Application

How can we apply small-world theory to routing in aDark peer to peer network?

• A Darknet is, essentially, a social network ofpeoples trusted relationships.

• If people can route in a social network, then itshould be possible for computers.

• Jon Kleinberg explained in 2000 howsmall-world networks can be navigable.

Ian Clarke - 2005 – p. 9

Kleinberg’s Result

• The possibility of routing efficiently depends onthe proportion of connections that have differentlengths with respect to the “position” of thenodes.

Ian Clarke - 2005 – p. 10

Kleinberg’s Result

• The possibility of routing efficiently depends onthe proportion of connections that have differentlengths with respect to the “position” of thenodes.

• If the positions are in a ring,the proportion of connectionswith a certain length should beinverse to the length:

Ian Clarke - 2005 – p. 10

Kleinberg’s Result

• The possibility of routing efficiently depends onthe proportion of connections that have differentlengths with respect to the “position” of thenodes.

• If the positions are in a ring,the proportion of connectionswith a certain length should beinverse to the length:

• In this case a simplegreedy routing algorithmperforms inO(log2

n) steps.Ian Clarke - 2005 – p. 10

Kleinbergs Result, cont.

Ian Clarke - 2005 – p. 11

Kleinbergs Result, cont.

Ian Clarke - 2005 – p. 11

Kleinbergs Result, cont.

But in a social network, how do we see if one personis closer to the destination than another?

Ian Clarke - 2005 – p. 11

Application, cont.

Is Alice closer to Harry than Bob?

Ian Clarke - 2005 – p. 12

Application, cont.

Is Alice closer to Harry than Bob?

• In real life, people presumably use a large numberof factors to decide this. Where do they live?What are their jobs? What are their interests?

Ian Clarke - 2005 – p. 12

Application, cont.

Is Alice closer to Harry than Bob?

• In real life, people presumably use a large numberof factors to decide this. Where do they live?What are their jobs? What are their interests?

• One cannot, in practice, expect a computer toroute based on such things.

Ian Clarke - 2005 – p. 12

Application, cont.

Is Alice closer to Harry than Bob?

• In real life, people presumably use a large numberof factors to decide this. Where do they live?What are their jobs? What are their interests?

• One cannot, in practice, expect a computer toroute based on such things.

• Instead, we let the network tell us!

Ian Clarke - 2005 – p. 12

Application, cont.

• Kleinberg’s model suggests: there should be fewlong connections, and many short ones.

Ian Clarke - 2005 – p. 13

Application, cont.

• Kleinberg’s model suggests: there should be fewlong connections, and many short ones.

• We can assign numerical identities placing nodesin a circle, and do it in such a way that this isfulfilled.

Ian Clarke - 2005 – p. 13

Application, cont.

• Kleinberg’s model suggests: there should be fewlong connections, and many short ones.

• We can assign numerical identities placing nodesin a circle, and do it in such a way that this isfulfilled.

• In other words, we “reverse engineer” the nodespositions based on the connections in thenetwork.

Ian Clarke - 2005 – p. 13

Application, cont.

• Kleinberg’s model suggests: there should be fewlong connections, and many short ones.

• We can assign numerical identities placing nodesin a circle, and do it in such a way that this isfulfilled.

• In other words, we “reverse engineer” the nodespositions based on the connections in thenetwork.

• Then greedy route with respect to thesenumerical identities.

Ian Clarke - 2005 – p. 13

The Method

• When nodes join the network, they choose aposition on the circle randomly.

Ian Clarke - 2005 – p. 14

The Method

• When nodes join the network, they choose aposition on the circle randomly.

• They then switch positions with other nodes, soas to minimize the product of the edge distances.

Ian Clarke - 2005 – p. 14

The Method, cont.

An advantageous switch of position:

Ian Clarke - 2005 – p. 15

The Method, cont.

An advantageous switch of position:

Ian Clarke - 2005 – p. 15

The Method, cont.

Some notes:

Ian Clarke - 2005 – p. 16

The Method, cont.

Some notes:

• Switching is essential!

Ian Clarke - 2005 – p. 16

The Method, cont.

Some notes:

• Switching is essential!• Because this is an ongoing process as the network

grows (and shrinks) it will be difficult to keeppermanent positions.

Ian Clarke - 2005 – p. 16

Simulations

We have simulated networks in three different modes:

Ian Clarke - 2005 – p. 17

Simulations

We have simulated networks in three different modes:

• Random walk search: “random”.

Ian Clarke - 2005 – p. 17

Simulations

We have simulated networks in three different modes:

• Random walk search: “random”.• Greedy routing in Kleinberg’s model with

identities as when it was constructed: “good”.

Ian Clarke - 2005 – p. 17

Simulations

We have simulated networks in three different modes:

• Random walk search: “random”.• Greedy routing in Kleinberg’s model with

identities as when it was constructed: “good”.• Greedy routing in Kleinberg’s model with

identities assigned according to our algorithm(2000 iterations per node): “restored”.

Ian Clarke - 2005 – p. 17

Simulations, cont.

The proportion of queries that succeeded within(log2 n)2 steps, wheren is the network size:

Ian Clarke - 2005 – p. 18

Simulations, cont.

The proportion of queries that succeeded within(log2 n)2 steps, wheren is the network size:

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

1000 10000 100000

Suc

c

Network Size

randomgood

restored

Ian Clarke - 2005 – p. 18

Simulations, cont.

The average length of the successful routes:

Ian Clarke - 2005 – p. 19

Simulations, cont.

The average length of the successful routes:

0

20

40

60

80

100

120

140

160

180

1000 10000 100000

Ste

ps

Network Size

randomgood

restored

Ian Clarke - 2005 – p. 19

Results

• Simulated networks are only so interesting, whatabout the real world?

Ian Clarke - 2005 – p. 20

Results

• Simulated networks are only so interesting, whatabout the real world?

• We borrowed some data from orkut.com. 2196people were spidered, starting with Ian.

Ian Clarke - 2005 – p. 20

Results

• Simulated networks are only so interesting, whatabout the real world?

• We borrowed some data from orkut.com. 2196people were spidered, starting with Ian.

We have also tried it on other datasets (e.g. “the PGPweb of trust”.)

Ian Clarke - 2005 – p. 20

Results, cont.

• The set was spidered so as to be comparativelydense (average 36.7 connections per person).

Ian Clarke - 2005 – p. 21

Results, cont.

• The set was spidered so as to be comparativelydense (average 36.7 connections per person).

• It contains mostly American techies andprogrammers. Some are probably in this room.(No Brazilians...)

Ian Clarke - 2005 – p. 21

Results, cont.

• The set was spidered so as to be comparativelydense (average 36.7 connections per person).

• It contains mostly American techies andprogrammers. Some are probably in this room.(No Brazilians...)

• The degree distri-bution is approxi-mately Power-Law:

0

200

400

600

800

1000

1200

0 50 100 150 200 250 300

Degree

Frequency

Ian Clarke - 2005 – p. 21

Results, cont.

Searching the Orkut dataset, for a maximum oflog2(n)2 steps.

Success RateMean StepsRandom SearchOur Algorithm

Ian Clarke - 2005 – p. 22

Results, cont.

Searching the Orkut dataset, for a maximum oflog2(n)2 steps.

Success RateMean StepsRandom Search 0.72 43.85Our Algorithm

Ian Clarke - 2005 – p. 22

Results, cont.

Searching the Orkut dataset, for a maximum oflog2(n)2 steps.

Success RateMean StepsRandom Search 0.72 43.85Our Algorithm 0.97 7.714

Ian Clarke - 2005 – p. 22

Results

Clipping degree at 40 connections. (24.2 connectionsper person.)

Success RateMean StepsRandom SearchOur Algorithm

Ian Clarke - 2005 – p. 23

Results

Clipping degree at 40 connections. (24.2 connectionsper person.)

Success RateMean StepsRandom Search 0.51 50.93Our Algorithm

Ian Clarke - 2005 – p. 23

Results

Clipping degree at 40 connections. (24.2 connectionsper person.)

Success RateMean StepsRandom Search 0.51 50.93Our Algorithm 0.98 10.90

Ian Clarke - 2005 – p. 23

Results

Clipping degree at 40 connections. (24.2 connectionsper person.)

Success RateMean StepsRandom Search 0.51 50.93Our Algorithm 0.98 10.90

Our algorithm takes advantage of there being peoplewho have many connections, but it does not dependon them.

Ian Clarke - 2005 – p. 23

How will Freenet use it?

• We wish to make this work in the wild, withthousands of users

Ian Clarke - 2005 – p. 24

How will Freenet use it?

• We wish to make this work in the wild, withthousands of users

• Key concerns:

Ian Clarke - 2005 – p. 24

How will Freenet use it?

• We wish to make this work in the wild, withthousands of users

• Key concerns:• Preventing malicious behaviour

Ian Clarke - 2005 – p. 24

How will Freenet use it?

• We wish to make this work in the wild, withthousands of users

• Key concerns:• Preventing malicious behaviour• Ensuring ease of use

Ian Clarke - 2005 – p. 24

How will Freenet use it?

• We wish to make this work in the wild, withthousands of users

• Key concerns:• Preventing malicious behaviour• Ensuring ease of use• Storing data (LRU currently implemented)

Ian Clarke - 2005 – p. 24

Preventing Malicious Behaviour

Threats:

• Selection of identity to attract certain data

Ian Clarke - 2005 – p. 25

Preventing Malicious Behaviour

Threats:

• Selection of identity to attract certain data• Manipulation of other node’s identities

Ian Clarke - 2005 – p. 25

Ensuring ease of use

• Peers will need to be “always on”

Ian Clarke - 2005 – p. 26

Ensuring ease of use

• Peers will need to be “always on”• Peer introduction

Ian Clarke - 2005 – p. 26

Ensuring ease of use

• Peers will need to be “always on”• Peer introduction

• Email

Ian Clarke - 2005 – p. 26

Ensuring ease of use

• Peers will need to be “always on”• Peer introduction

• Email• Phone

Ian Clarke - 2005 – p. 26

Ensuring ease of use

• Peers will need to be “always on”• Peer introduction

• Email• Phone• Trusted third party

Ian Clarke - 2005 – p. 26

Ensuring ease of use

• Peers will need to be “always on”• Peer introduction

• Email• Phone• Trusted third party

• What about NATs and firewalls

Ian Clarke - 2005 – p. 26

Ensuring ease of use

• Peers will need to be “always on”• Peer introduction

• Email• Phone• Trusted third party

• What about NATs and firewalls• Could use UDP hole- punching (as used by

Dijjer, Skype)

Ian Clarke - 2005 – p. 26

Ensuring ease of use

• Peers will need to be “always on”• Peer introduction

• Email• Phone• Trusted third party

• What about NATs and firewalls• Could use UDP hole- punching (as used by

Dijjer, Skype)• Would require third- party for negotiation

Ian Clarke - 2005 – p. 26

Freenet

• Much of the the next Freenet version has beenimplemented

Ian Clarke - 2005 – p. 27

Freenet

• Much of the the next Freenet version has beenimplemented

• Routing, as described above, and withTCP-inspired load balancing

Ian Clarke - 2005 – p. 27

Freenet

• Much of the the next Freenet version has beenimplemented

• Routing, as described above, and withTCP-inspired load balancing

• Large scale testing will, of course, be thetrial-by-fire.

Ian Clarke - 2005 – p. 27

Freenet

• Much of the the next Freenet version has beenimplemented

• Routing, as described above, and withTCP-inspired load balancing

• Large scale testing will, of course, be thetrial-by-fire.

• More will be known by the time of theconference!

Ian Clarke - 2005 – p. 27

Conclusion

We believe very strongly that building a navigable,scalable Dark network is possible.And it is beingdone!

Ian Clarke - 2005 – p. 28

Conclusion

We believe very strongly that building a navigable,scalable Dark network is possible.And it is beingdone!

• There is still much work to do on the theory.

Ian Clarke - 2005 – p. 28

Conclusion

We believe very strongly that building a navigable,scalable Dark network is possible.And it is beingdone!

• There is still much work to do on the theory.• Can other models work better?

Ian Clarke - 2005 – p. 28

Conclusion

We believe very strongly that building a navigable,scalable Dark network is possible.And it is beingdone!

• There is still much work to do on the theory.• Can other models work better?• Can we find better selection functions for

switching?

Ian Clarke - 2005 – p. 28

Conclusion

We believe very strongly that building a navigable,scalable Dark network is possible.And it is beingdone!

• There is still much work to do on the theory.• Can other models work better?• Can we find better selection functions for

switching?• It needs to be tested on more data.

Ian Clarke - 2005 – p. 28

Conclusion, cont.

• We have learned the hard way that practice ismore difficult than theory.

Ian Clarke - 2005 – p. 29

Conclusion, cont.

• We have learned the hard way that practice ismore difficult than theory.• Security issues are very important.

Ian Clarke - 2005 – p. 29

Conclusion, cont.

• We have learned the hard way that practice ismore difficult than theory.• Security issues are very important.• How the network is deployed will affect how

well it works.

Ian Clarke - 2005 – p. 29

Conclusion, cont.

• We have learned the hard way that practice ismore difficult than theory.• Security issues are very important.• How the network is deployed will affect how

well it works.

People who are interested can join the discussion athttp://freenetproject.org/.

Ian Clarke - 2005 – p. 29