Covert Communication in a DarkNetwork
A major new version of freenet
Ian Clarke and Oskar Sandberg
The Freenet Project
Ian Clarke - 2005 – p. 1
Introduction
• We have long been interested in decentralised“Peer to Peer” networks. Especially Freenet.
Ian Clarke - 2005 – p. 2
Introduction
• We have long been interested in decentralised“Peer to Peer” networks. Especially Freenet.
• But when individual users come under attack,decentralisation is not enough.
Ian Clarke - 2005 – p. 2
Introduction
• We have long been interested in decentralised“Peer to Peer” networks. Especially Freenet.
• But when individual users come under attack,decentralisation is not enough.
• Future networks may need to limit connections totrusted friends.
Ian Clarke - 2005 – p. 2
Introduction
• We have long been interested in decentralised“Peer to Peer” networks. Especially Freenet.
• But when individual users come under attack,decentralisation is not enough.
• Future networks may need to limit connections totrusted friends.
• The next version of Freenet will be based on thisphilosophy, a so called Dark Network.
Ian Clarke - 2005 – p. 2
Overview of “Peer to Peer” net-works
• Information is spread across many inter-connected computers
Ian Clarke - 2005 – p. 3
Overview of “Peer to Peer” net-works
• Information is spread across many inter-connected computers
• Users want to find information
Ian Clarke - 2005 – p. 3
Overview of “Peer to Peer” net-works
• Information is spread across many inter-connected computers
• Users want to find information• Some are centralised (eg. Napster), some are
semi- centralised (eg. Kazaa), others aredistributed (eg. Freenet)
Ian Clarke - 2005 – p. 3
Light P2P Networks
• Examples: Gnutella, Freenet, Distributed HashTables
Ian Clarke - 2005 – p. 4
Light P2P Networks
• Examples: Gnutella, Freenet, Distributed HashTables
• Advantage: Globally scalable with the rightrouting algorithm
Ian Clarke - 2005 – p. 4
Light P2P Networks
• Examples: Gnutella, Freenet, Distributed HashTables
• Advantage: Globally scalable with the rightrouting algorithm
• Disadvantage: Vulnerable to “harvesting”, ie.people you don’t know can easily discoverwhether you are part of the network
Ian Clarke - 2005 – p. 4
Dark or “Friend to Friend” P2PNetworks
• Peers only communicate directly with “trusted”peers
Ian Clarke - 2005 – p. 5
Dark or “Friend to Friend” P2PNetworks
• Peers only communicate directly with “trusted”peers
• Examples: Waste
Ian Clarke - 2005 – p. 5
Dark or “Friend to Friend” P2PNetworks
• Peers only communicate directly with “trusted”peers
• Examples: Waste• Advantage: Only your trusted friends know you
are part of the network
Ian Clarke - 2005 – p. 5
Dark or “Friend to Friend” P2PNetworks
• Peers only communicate directly with “trusted”peers
• Examples: Waste• Advantage: Only your trusted friends know you
are part of the network• Disadvantage: Networks are disconnected and
small, they typically don’t scale well
Ian Clarke - 2005 – p. 5
The Small-World Phenomenon
• In "small-world" networks short paths existbetween any two peers
Ian Clarke - 2005 – p. 6
The Small-World Phenomenon
• In "small-world" networks short paths existbetween any two peers
• People tend to form this type of network (asshown by Milgram experiment)
Ian Clarke - 2005 – p. 6
The Small-World Phenomenon
• In "small-world" networks short paths existbetween any two peers
• People tend to form this type of network (asshown by Milgram experiment)
• Short paths may exist but they may not be easy tofind Ian Clarke - 2005 – p. 6
Navigable Small-World Net-works
• Concept of similarity or “closeness” betweenpeers
Ian Clarke - 2005 – p. 7
Navigable Small-World Net-works
• Concept of similarity or “closeness” betweenpeers
• Similar peers are more likely to be connectedthan dissimilar peers
Ian Clarke - 2005 – p. 7
Navigable Small-World Net-works
• Concept of similarity or “closeness” betweenpeers
• Similar peers are more likely to be connectedthan dissimilar peers
• You can get from any one peer to any othersimply by routing to the closest peer at each step
Ian Clarke - 2005 – p. 7
Navigable Small-World Net-works
• Concept of similarity or “closeness” betweenpeers
• Similar peers are more likely to be connectedthan dissimilar peers
• You can get from any one peer to any othersimply by routing to the closest peer at each step
• This is called “Greedy Routing”
Ian Clarke - 2005 – p. 7
Navigable Small-World Net-works
• Concept of similarity or “closeness” betweenpeers
• Similar peers are more likely to be connectedthan dissimilar peers
• You can get from any one peer to any othersimply by routing to the closest peer at each step
• This is called “Greedy Routing”• Freenet and “Distributed Hash Tables” rely on
this principal to find data in a scalabledecentralised manner
Ian Clarke - 2005 – p. 7
Data Networks
• Data Networks (also DHTs) work by assigningeach document with a numerical address or key.
Ian Clarke - 2005 – p. 8
Data Networks
• Data Networks (also DHTs) work by assigningeach document with a numerical address or key.
• Each node is then assigned some section of the“keyspace” in which to specialize.
Ian Clarke - 2005 – p. 8
Data Networks
• Data Networks (also DHTs) work by assigningeach document with a numerical address or key.
• Each node is then assigned some section of the“keyspace” in which to specialize.
• When data is inserted, it is routed towards nodesthat specialize in its part of the keyspace.
Ian Clarke - 2005 – p. 8
Data Networks
• Data Networks (also DHTs) work by assigningeach document with a numerical address or key.
• Each node is then assigned some section of the“keyspace” in which to specialize.
• When data is inserted, it is routed towards nodesthat specialize in its part of the keyspace.
• When data is requested, the query routedlikewise.
Ian Clarke - 2005 – p. 8
Application
How can we apply small-world theory to routing in aDark peer to peer network?
Ian Clarke - 2005 – p. 9
Application
How can we apply small-world theory to routing in aDark peer to peer network?
• A Darknet is, essentially, a social network ofpeoples trusted relationships.
Ian Clarke - 2005 – p. 9
Application
How can we apply small-world theory to routing in aDark peer to peer network?
• A Darknet is, essentially, a social network ofpeoples trusted relationships.
• If people can route in a social network, then itshould be possible for computers.
Ian Clarke - 2005 – p. 9
Application
How can we apply small-world theory to routing in aDark peer to peer network?
• A Darknet is, essentially, a social network ofpeoples trusted relationships.
• If people can route in a social network, then itshould be possible for computers.
• Jon Kleinberg explained in 2000 howsmall-world networks can be navigable.
Ian Clarke - 2005 – p. 9
Kleinberg’s Result
• The possibility of routing efficiently depends onthe proportion of connections that have differentlengths with respect to the “position” of thenodes.
Ian Clarke - 2005 – p. 10
Kleinberg’s Result
• The possibility of routing efficiently depends onthe proportion of connections that have differentlengths with respect to the “position” of thenodes.
• If the positions are in a ring,the proportion of connectionswith a certain length should beinverse to the length:
Ian Clarke - 2005 – p. 10
Kleinberg’s Result
• The possibility of routing efficiently depends onthe proportion of connections that have differentlengths with respect to the “position” of thenodes.
• If the positions are in a ring,the proportion of connectionswith a certain length should beinverse to the length:
• In this case a simplegreedy routing algorithmperforms inO(log2
n) steps.Ian Clarke - 2005 – p. 10
Kleinbergs Result, cont.
Ian Clarke - 2005 – p. 11
Kleinbergs Result, cont.
Ian Clarke - 2005 – p. 11
Kleinbergs Result, cont.
But in a social network, how do we see if one personis closer to the destination than another?
Ian Clarke - 2005 – p. 11
Application, cont.
Is Alice closer to Harry than Bob?
Ian Clarke - 2005 – p. 12
Application, cont.
Is Alice closer to Harry than Bob?
• In real life, people presumably use a large numberof factors to decide this. Where do they live?What are their jobs? What are their interests?
Ian Clarke - 2005 – p. 12
Application, cont.
Is Alice closer to Harry than Bob?
• In real life, people presumably use a large numberof factors to decide this. Where do they live?What are their jobs? What are their interests?
• One cannot, in practice, expect a computer toroute based on such things.
Ian Clarke - 2005 – p. 12
Application, cont.
Is Alice closer to Harry than Bob?
• In real life, people presumably use a large numberof factors to decide this. Where do they live?What are their jobs? What are their interests?
• One cannot, in practice, expect a computer toroute based on such things.
• Instead, we let the network tell us!
Ian Clarke - 2005 – p. 12
Application, cont.
• Kleinberg’s model suggests: there should be fewlong connections, and many short ones.
Ian Clarke - 2005 – p. 13
Application, cont.
• Kleinberg’s model suggests: there should be fewlong connections, and many short ones.
• We can assign numerical identities placing nodesin a circle, and do it in such a way that this isfulfilled.
Ian Clarke - 2005 – p. 13
Application, cont.
• Kleinberg’s model suggests: there should be fewlong connections, and many short ones.
• We can assign numerical identities placing nodesin a circle, and do it in such a way that this isfulfilled.
• In other words, we “reverse engineer” the nodespositions based on the connections in thenetwork.
Ian Clarke - 2005 – p. 13
Application, cont.
• Kleinberg’s model suggests: there should be fewlong connections, and many short ones.
• We can assign numerical identities placing nodesin a circle, and do it in such a way that this isfulfilled.
• In other words, we “reverse engineer” the nodespositions based on the connections in thenetwork.
• Then greedy route with respect to thesenumerical identities.
Ian Clarke - 2005 – p. 13
The Method
• When nodes join the network, they choose aposition on the circle randomly.
Ian Clarke - 2005 – p. 14
The Method
• When nodes join the network, they choose aposition on the circle randomly.
• They then switch positions with other nodes, soas to minimize the product of the edge distances.
Ian Clarke - 2005 – p. 14
The Method, cont.
An advantageous switch of position:
Ian Clarke - 2005 – p. 15
The Method, cont.
An advantageous switch of position:
Ian Clarke - 2005 – p. 15
The Method, cont.
Some notes:
Ian Clarke - 2005 – p. 16
The Method, cont.
Some notes:
• Switching is essential!
Ian Clarke - 2005 – p. 16
The Method, cont.
Some notes:
• Switching is essential!• Because this is an ongoing process as the network
grows (and shrinks) it will be difficult to keeppermanent positions.
Ian Clarke - 2005 – p. 16
Simulations
We have simulated networks in three different modes:
Ian Clarke - 2005 – p. 17
Simulations
We have simulated networks in three different modes:
• Random walk search: “random”.
Ian Clarke - 2005 – p. 17
Simulations
We have simulated networks in three different modes:
• Random walk search: “random”.• Greedy routing in Kleinberg’s model with
identities as when it was constructed: “good”.
Ian Clarke - 2005 – p. 17
Simulations
We have simulated networks in three different modes:
• Random walk search: “random”.• Greedy routing in Kleinberg’s model with
identities as when it was constructed: “good”.• Greedy routing in Kleinberg’s model with
identities assigned according to our algorithm(2000 iterations per node): “restored”.
Ian Clarke - 2005 – p. 17
Simulations, cont.
The proportion of queries that succeeded within(log2 n)2 steps, wheren is the network size:
Ian Clarke - 2005 – p. 18
Simulations, cont.
The proportion of queries that succeeded within(log2 n)2 steps, wheren is the network size:
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
1000 10000 100000
Suc
c
Network Size
randomgood
restored
Ian Clarke - 2005 – p. 18
Simulations, cont.
The average length of the successful routes:
Ian Clarke - 2005 – p. 19
Simulations, cont.
The average length of the successful routes:
0
20
40
60
80
100
120
140
160
180
1000 10000 100000
Ste
ps
Network Size
randomgood
restored
Ian Clarke - 2005 – p. 19
Results
• Simulated networks are only so interesting, whatabout the real world?
Ian Clarke - 2005 – p. 20
Results
• Simulated networks are only so interesting, whatabout the real world?
• We borrowed some data from orkut.com. 2196people were spidered, starting with Ian.
Ian Clarke - 2005 – p. 20
Results
• Simulated networks are only so interesting, whatabout the real world?
• We borrowed some data from orkut.com. 2196people were spidered, starting with Ian.
We have also tried it on other datasets (e.g. “the PGPweb of trust”.)
Ian Clarke - 2005 – p. 20
Results, cont.
• The set was spidered so as to be comparativelydense (average 36.7 connections per person).
Ian Clarke - 2005 – p. 21
Results, cont.
• The set was spidered so as to be comparativelydense (average 36.7 connections per person).
• It contains mostly American techies andprogrammers. Some are probably in this room.(No Brazilians...)
Ian Clarke - 2005 – p. 21
Results, cont.
• The set was spidered so as to be comparativelydense (average 36.7 connections per person).
• It contains mostly American techies andprogrammers. Some are probably in this room.(No Brazilians...)
• The degree distri-bution is approxi-mately Power-Law:
0
200
400
600
800
1000
1200
0 50 100 150 200 250 300
Degree
Frequency
Ian Clarke - 2005 – p. 21
Results, cont.
Searching the Orkut dataset, for a maximum oflog2(n)2 steps.
Success RateMean StepsRandom SearchOur Algorithm
Ian Clarke - 2005 – p. 22
Results, cont.
Searching the Orkut dataset, for a maximum oflog2(n)2 steps.
Success RateMean StepsRandom Search 0.72 43.85Our Algorithm
Ian Clarke - 2005 – p. 22
Results, cont.
Searching the Orkut dataset, for a maximum oflog2(n)2 steps.
Success RateMean StepsRandom Search 0.72 43.85Our Algorithm 0.97 7.714
Ian Clarke - 2005 – p. 22
Results
Clipping degree at 40 connections. (24.2 connectionsper person.)
Success RateMean StepsRandom SearchOur Algorithm
Ian Clarke - 2005 – p. 23
Results
Clipping degree at 40 connections. (24.2 connectionsper person.)
Success RateMean StepsRandom Search 0.51 50.93Our Algorithm
Ian Clarke - 2005 – p. 23
Results
Clipping degree at 40 connections. (24.2 connectionsper person.)
Success RateMean StepsRandom Search 0.51 50.93Our Algorithm 0.98 10.90
Ian Clarke - 2005 – p. 23
Results
Clipping degree at 40 connections. (24.2 connectionsper person.)
Success RateMean StepsRandom Search 0.51 50.93Our Algorithm 0.98 10.90
Our algorithm takes advantage of there being peoplewho have many connections, but it does not dependon them.
Ian Clarke - 2005 – p. 23
How will Freenet use it?
• We wish to make this work in the wild, withthousands of users
Ian Clarke - 2005 – p. 24
How will Freenet use it?
• We wish to make this work in the wild, withthousands of users
• Key concerns:
Ian Clarke - 2005 – p. 24
How will Freenet use it?
• We wish to make this work in the wild, withthousands of users
• Key concerns:• Preventing malicious behaviour
Ian Clarke - 2005 – p. 24
How will Freenet use it?
• We wish to make this work in the wild, withthousands of users
• Key concerns:• Preventing malicious behaviour• Ensuring ease of use
Ian Clarke - 2005 – p. 24
How will Freenet use it?
• We wish to make this work in the wild, withthousands of users
• Key concerns:• Preventing malicious behaviour• Ensuring ease of use• Storing data (LRU currently implemented)
Ian Clarke - 2005 – p. 24
Preventing Malicious Behaviour
Threats:
• Selection of identity to attract certain data
Ian Clarke - 2005 – p. 25
Preventing Malicious Behaviour
Threats:
• Selection of identity to attract certain data• Manipulation of other node’s identities
Ian Clarke - 2005 – p. 25
Ensuring ease of use
• Peers will need to be “always on”
Ian Clarke - 2005 – p. 26
Ensuring ease of use
• Peers will need to be “always on”• Peer introduction
Ian Clarke - 2005 – p. 26
Ensuring ease of use
• Peers will need to be “always on”• Peer introduction
Ian Clarke - 2005 – p. 26
Ensuring ease of use
• Peers will need to be “always on”• Peer introduction
• Email• Phone
Ian Clarke - 2005 – p. 26
Ensuring ease of use
• Peers will need to be “always on”• Peer introduction
• Email• Phone• Trusted third party
Ian Clarke - 2005 – p. 26
Ensuring ease of use
• Peers will need to be “always on”• Peer introduction
• Email• Phone• Trusted third party
• What about NATs and firewalls
Ian Clarke - 2005 – p. 26
Ensuring ease of use
• Peers will need to be “always on”• Peer introduction
• Email• Phone• Trusted third party
• What about NATs and firewalls• Could use UDP hole- punching (as used by
Dijjer, Skype)
Ian Clarke - 2005 – p. 26
Ensuring ease of use
• Peers will need to be “always on”• Peer introduction
• Email• Phone• Trusted third party
• What about NATs and firewalls• Could use UDP hole- punching (as used by
Dijjer, Skype)• Would require third- party for negotiation
Ian Clarke - 2005 – p. 26
Freenet
• Much of the the next Freenet version has beenimplemented
Ian Clarke - 2005 – p. 27
Freenet
• Much of the the next Freenet version has beenimplemented
• Routing, as described above, and withTCP-inspired load balancing
Ian Clarke - 2005 – p. 27
Freenet
• Much of the the next Freenet version has beenimplemented
• Routing, as described above, and withTCP-inspired load balancing
• Large scale testing will, of course, be thetrial-by-fire.
Ian Clarke - 2005 – p. 27
Freenet
• Much of the the next Freenet version has beenimplemented
• Routing, as described above, and withTCP-inspired load balancing
• Large scale testing will, of course, be thetrial-by-fire.
• More will be known by the time of theconference!
Ian Clarke - 2005 – p. 27
Conclusion
We believe very strongly that building a navigable,scalable Dark network is possible.And it is beingdone!
Ian Clarke - 2005 – p. 28
Conclusion
We believe very strongly that building a navigable,scalable Dark network is possible.And it is beingdone!
• There is still much work to do on the theory.
Ian Clarke - 2005 – p. 28
Conclusion
We believe very strongly that building a navigable,scalable Dark network is possible.And it is beingdone!
• There is still much work to do on the theory.• Can other models work better?
Ian Clarke - 2005 – p. 28
Conclusion
We believe very strongly that building a navigable,scalable Dark network is possible.And it is beingdone!
• There is still much work to do on the theory.• Can other models work better?• Can we find better selection functions for
switching?
Ian Clarke - 2005 – p. 28
Conclusion
We believe very strongly that building a navigable,scalable Dark network is possible.And it is beingdone!
• There is still much work to do on the theory.• Can other models work better?• Can we find better selection functions for
switching?• It needs to be tested on more data.
Ian Clarke - 2005 – p. 28
Conclusion, cont.
• We have learned the hard way that practice ismore difficult than theory.
Ian Clarke - 2005 – p. 29
Conclusion, cont.
• We have learned the hard way that practice ismore difficult than theory.• Security issues are very important.
Ian Clarke - 2005 – p. 29
Conclusion, cont.
• We have learned the hard way that practice ismore difficult than theory.• Security issues are very important.• How the network is deployed will affect how
well it works.
Ian Clarke - 2005 – p. 29
Conclusion, cont.
• We have learned the hard way that practice ismore difficult than theory.• Security issues are very important.• How the network is deployed will affect how
well it works.
People who are interested can join the discussion athttp://freenetproject.org/.
Ian Clarke - 2005 – p. 29