Data Mining using Fractals and Power laws

transcript

School of Computer ScienceCarnegie Mellon

Christos Faloutsos

Carnegie Mellon University

ICAST, June'09 C. Faloutsos 2

Thank you!

• Prof. Hsing-Kuo Kenneth PAO

• Prof. Yuh-Jye LEE

• Hsin Yeh

And also thanks to

Ching-Hao (Eric) MAO

Ming-Kung (Morgan) SUN

Yi-Ren (Ian) YEH

Lei LI

Leman AKOGLU

Ian ROLEWICZ

Overview

• Goals/ motivation: find patterns in large datasets:– (A) Sensor data– (B) network intrusion data

• Solutions: self-similarity and power laws

• Discussion

Applications of sensors/streams

• network monitoring

# alerts

Applications of sensors/streams

• Financial, sales, economic series

• Medical: ECGs +; blood pressure etc monitoring

Motivation - Applications• Scientific data: seismological;

astronomical; environment / anti-pollution; meteorological

Sunspot Data

Motivation - Applications (cont’d)

• Computer systems

– web servers (buffering, prefetching)

– ...

http://repository.cs.vt.edu/lbl-conn-7.tar.Z

Web traffic

• [Crovella Bestavros, SIGMETRICS’96]

survivable,self-managing storage

infrastructure

a storage brick(0.5–5 TB)~1 PB

“self-*” = self-managing, self-tuning, self-healing, …

Self-* Storage (Ganger+)

survivable,self-managing storage

infrastructure

a storage brick(0.5–5 TB)~1 PB

“self-*” = self-managing, self-tuning, self-healing, … Goal: 1 petabyte (PB) www.pdl.cmu.edu/SelfStar

Self-* Storage (Ganger+)

Problem definition

• Given: one or more sequences x1 , x2 , … , xt , …; (y1, y2, … , yt, …)

• Find – patterns; clusters; outliers; forecasts;

Problem

• Find patterns, in large datasets

# bytes

Problem

# bytes

Poisson indep., ident. distr

Problem

# bytes

Problem

# bytes

Q: Then, how to generatesuch bursty traffic?

Solutions

• New tools: power laws, self-similarity and ‘fractals’ work, where traditional assumptions fail

• Let’s see the details:

Overview

• Goals/ motivation: find patterns in large datasets:– (A) Sensor data– (B) network data

• Solutions: self-similarity and power laws

• Discussion

What is a fractal?

= self-similar point set, e.g., Sierpinski triangle:

...zero area: (3/4)^inf

infinite length!

(4/3)^inf

Q: What is its dimensionality??

What is a fractal?

= self-similar point set, e.g., Sierpinski triangle:

...zero area: (3/4)^inf

infinite length!

(4/3)^inf

Q: What is its dimensionality??A: log3 / log2 = 1.58 (!?!)

Intrinsic (‘fractal’) dimension

• Q: fractal dimension of a line?

• Q: fd of a plane?

Intrinsic (‘fractal’) dimension

• Q: fractal dimension of a line?

• A: nn ( <= r ) ~ r^1(‘power law’: y=x^a)

• Q: fd of a plane?• A: nn ( <= r ) ~ r^2fd== slope of (log(nn) vs..

log(r) )

Sierpinsky triangle

log( r )

log(#pairs within <=r )

== ‘correlation integral’

= CDF of pairwise distances

Observations: Fractals <-> power laws

Closely related:

• fractals <=>

• self-similarity <=>

• scale-free <=>

• power laws ( y= xa ; F=K r-2)

• (vs y=e-ax or y=xa+b)log( r )

Outline

• Problems

• Self-similarity and power laws

• Solutions to posed problems

• Discussion

#bytes

Solution #1: traffic

• disk traces: self-similar: (also: [Leland+94])• How to generate such traffic?

Solution #1: traffic

• disk traces (80-20 ‘law’) – ‘multifractals’

#bytes

20% 80%

80-20 / multifractals20 80

80-20 / multifractals20

• p ; (1-p) in general

• yes, there are dependencies

More on 80/20: PQRS

• Part of ‘self-* storage’ project

Overview

• Goals/ motivation: find patterns in large datasets:– (A) Sensor data

– (B) network data

• Solutions: self-similarity and power laws– sensor/traffic data

– network data

• Discussion

Problem dfn

<source-ip, target-ip, timestamp, alert-type>

eg., <192.168.2.5; 128.2.220.159; 3am june 6; ICMP-redirect-host>

goal: find patterns / anomalies

Power laws in intrusion data

human-like

robot-like

robot-like (bursty)

Q: Can we visuallysummarize / classifyour sequences?

Answer: yes!

two features:

• F1: how periodic (24h-cycle) is a sequence

• F2: how bursty it is

Q: how to measure burstiness?

A: Fractal dimension!

Burstiness & f.d.

uniform: fd = 1

@same time-tick: fd = 0

Burstiness & f.d.

uniform: fd = 1

@same time-tick: fd = 0

bursts within bursts within bursts:0<fd<1

6. Proposed Methods: The FDP Plot

• Notice: clustering wrt alert types!

human-like

robot-like

robot-like (bursty)

can we visuallysummarize / classifyour sequences?

Exampleshuman-like behavior

Outline

• problems

• Fractals

• Solutions

• Discussion – what else can they solve? – how frequent are fractals?

What else can they solve?

• separability [KDD’02]• forecasting [CIKM’02]• dimensionality reduction [SBBD’00]• non-linear axis scaling [KDD’02]• disk trace modeling [PEVA’02]• selectivity of spatial/multimedia queries

[PODS’94, VLDB’95, ICDE’00]• ...

Problem #3 - spatial d.m.

Galaxies (Sloan Digital Sky Survey w/ B. Nichol) - ‘spiral’ and ‘elliptical’

galaxies

- patterns? (not Gaussian; not uniform)

-attraction/repulsion?

- separability??

Solution#3: spatial d.m.

log(r)

spi-spi

spi-ell

ell-ell

- 1.8 slope

- plateau!

- repulsion!

CORRELATION INTEGRAL!

log(r)

spi-spi

spi-ell

ell-ell

- 1.8 slope

- plateau!

- repulsion!

[w/ Seeger, Traina, Traina, SIGMOD00]

Heuristic on choosing # of clusters

log(r)

spi-spi

spi-ell

ell-ell

- 1.8 slope

- plateau!

- repulsion!

Outline

• problems

• Fractals

• Solutions

• Discussion – what else can they solve? – how frequent are fractals?

Fractals & power laws:

appear in numerous settings:

• medical

• geographical / geological

• social

• computer-system related

• <and many-many more! see [Mandelbrot]>

Fractals: Brain scans

• brain-scans

octree levels

Log(#octants)

2.63 = fd

More fractals

• periphery of malignant tumors: ~1.5

• benign: ~1.3

• [Burdet+]

More fractals:

• cardiovascular system: 3 (!) lungs: ~2.9

• medical

• social

More fractals:

• Coastlines: 1.2-1.58

More fractals:

• the fractal dimension for the Amazon river is 1.85 (Nile: 1.4)

[ems.gphys.unc.edu/nonlinear/fractals/examples.html]

More fractals:

• the fractal dimension for the Amazon river is 1.85 (Nile: 1.4)

[ems.gphys.unc.edu/nonlinear/fractals/examples.html]

Cross-roads of Montgomery county:

•any rules?

GIS points

A: self-similarity:• intrinsic dim. = 1.51

log( r )

log(#pairs(within <= r))

Examples:LB county

• Long Beach county of CA (road end-points)

log(r)

log(#pairs)

More power laws: areas – Korcak’s law

Scandinavian lakes

Any pattern?

More power laws: areas – Korcak’s law

Scandinavian lakes area vs complementary cumulative count (log-log axes)

log(count( >= area))

log(area)

More power laws: Korcak

Japan islands;

area vs cumulative count (log-log axes) log(area)

log(count( >= area))

More power laws

• Energy of earthquakes (Gutenberg-Richter law) [simscience.org]

log(count)

Magnitude = log(energy)day

Energy released

• medical

• social

A famous power law: Zipf’s law

• Bible - rank vs. frequency (log-log)

log(rank)

log(freq)

“a”

“the”

“Rank/frequency plot”

TELCO data

# of service units

count ofcustomers

‘best customer’

SALES data – store#96

# units sold

count of products

“aspirin”

Olympic medals (Sidney’00, Athens’04):

log( rank)

log(#medals)

0 0.5 1 1.5 2

athens

sidney

Olympic medals (Sidney’00, Athens’04):

log( rank)

log(#medals)

0 0.5 1 1.5 2

athens

sidney

Even more power laws:

• Income distribution (Pareto’s law)• size of firms

• publication counts (Lotka’s law)

library science (Lotka’s law of publication count); and citation counts: (citeseer.nj.nec.com 6/2001)

log(#citations)

log(count)

Ullman

• web hit counts [w/ A. Montgomery]

Web Site Traffic

log(freq)

log(count)

Zipf“yahoo.com”

Autonomous systems• Power law in the degree distribution [SIGCOMM99]

internet domains

log(rank)

log(degree)

att.com

ibm.com

The Peer-to-Peer Topology

• Number of immediate peers (= degree), follows a power-law

[Jovanovic+]

degree

epinions.com

• who-trusts-whom [Richardson + Domingos, KDD 2001]

(out) degree

• medical

• social

Power laws, cont’d

• In- and out-degree distribution of web sites [Barabasi], [IBM-CLEVER]

log indegree

- log(freq)

from [Ravi Kumar, Prabhakar Raghavan, Sridhar Rajagopalan, Andrew Tomkins ]

log indegree

log(freq)

from [Ravi Kumar, Prabhakar Raghavan, Sridhar Rajagopalan, Andrew Tomkins ]

log indegree

log(freq)

Q: ‘how can we usethese power laws?’

“Foiled by power law”

• [Broder+, WWW’00]

(log) in-degree

(log) count

“Foiled by power law”

• [Broder+, WWW’00]

“The anomalous bump at 120on the x-axis is due a large clique formed by a single spammer”

(log) in-degree

(log) count

• length of file transfers [Crovella+Bestavros ‘96]

• duration of UNIX jobs

Conclusions

• Fascinating problems in Data Mining: find patterns in streams & network data

Conclusions - cont’d

New tools for Data Mining: self-similarity & power laws: appear in many cases

Bad news:

lead to skewed distributions

(no Gaussian, Poisson,

uniformity, independence,

mean, variance)

Good news:• ‘correlation integral’

for separability• rank/frequency plots• 80-20 (multifractals)• (Hurst exponent, • strange attractors,• renormalization theory, • ++)

Resources

• Manfred Schroeder “Chaos, Fractals and Power Laws”, 1991

References

• [vldb95] Alberto Belussi and Christos Faloutsos, Estimating the Selectivity of Spatial Queries Using the `Correlation' Fractal Dimension Proc. of VLDB, p. 299-310, 1995

• [Broder+’00] Andrei Broder, Ravi Kumar , Farzin Maghoul1, Prabhakar Raghavan , Sridhar Rajagopalan , Raymie Stata, Andrew Tomkins , Janet Wiener, Graph structure in the web , WWW’00

• M. Crovella and A. Bestavros, Self similarity in World wide web traffic: Evidence and possible causes , SIGMETRICS ’96.

References

• J. Considine, F. Li, G. Kollios and J. Byers, Approximate Aggregation Techniques for Sensor Databases (ICDE’04, best paper award).

• [pods94] Christos Faloutsos and Ibrahim Kamel, Beyond Uniformity and Independence: Analysis of R-trees Using the Concept of Fractal Dimension, PODS, Minneapolis, MN, May 24-26, 1994, pp. 4-13

References

• [vldb96] Christos Faloutsos, Yossi Matias and Avi Silberschatz, Modeling Skewed Distributions Using Multifractals and the `80-20 Law’ Conf. on Very Large Data Bases (VLDB), Bombay, India, Sept. 1996.

• [sigmod2000] Christos Faloutsos, Bernhard Seeger, Agma J. M. Traina and Caetano Traina Jr., Spatial Join Selectivity Using Power Laws, SIGMOD 2000

References

• [vldb96] Christos Faloutsos and Volker Gaede Analysis of the Z-Ordering Method Using the Hausdorff Fractal Dimension VLD, Bombay, India, Sept. 1996

• [sigcomm99] Michalis Faloutsos, Petros Faloutsos and Christos Faloutsos, What does the Internet look like? Empirical Laws of the Internet Topology, SIGCOMM 1999

References

• [Leskovec 05] Jure Leskovec, Jon M. Kleinberg, Christos Faloutsos: Graphs over time: densification laws, shrinking diameters and possible explanations. KDD 2005: 177-187

References

• [ieeeTN94] W. E. Leland, M.S. Taqqu, W. Willinger, D.V. Wilson, On the Self-Similar Nature of Ethernet Traffic, IEEE Transactions on Networking, 2, 1, pp 1-15, Feb. 1994.

• [brite] Alberto Medina, Anukool Lakhina, Ibrahim Matta, and John Byers. BRITE: An Approach to Universal Topology Generation. MASCOTS '01

References

• [icde99] Guido Proietti and Christos Faloutsos, I/O complexity for range queries on region data stored using an R-tree (ICDE’99)

• Stan Sclaroff, Leonid Taycher and Marco La Cascia , "ImageRover: A content-based image browser for the world wide web" Proc. IEEE Workshop on Content-based Access of Image and Video Libraries, pp 2-9, 1997.

References

• [kdd2001] Agma J. M. Traina, Caetano Traina Jr., Spiros Papadimitriou and Christos Faloutsos: Tri-plots: Scalable Tools for Multidimensional Data Mining, KDD 2001, San Francisco, CA.

Thank you!

Contact info:christos <at> cs.cmu.edu

www. cs.cmu.edu /~christos

(w/ papers, datasets, code for fractal dimension estimation, etc)

Data Mining using Fractals and Power laws

Documents