Gabriel Robins

Department of Computer Science

University of Virginiawww.cs.virginia.edu/robins

AlgorithmsCS4102– Fall 2015

Algorithms (C4102) Textbook

Textbook:Introduction to Algorithms

by Cormen et al (MIT)

Third Edition, 2009

Good Articles / videos:www.cs.virginia.edu/~robins/CS_readings.html

Theory of Computation (CS3102)

Supplemental reading:How to Solve It, by George Polya (MIT)

Princeton University Press, 1945

• A classic on problem solving

George Polya (1887-1985)

Algorithms Syllabus

Fundamentals:

• History of algorithms• Problem solving• Pigeon-hole principle• Occam's razor• Uncomputability• Universality• Asymptotic complexity• Set theory and logic

 

Algorithms Syllabus

Data structures:

• Arrays• Stacks and queues• Linked lists• Binary and general trees• Height-balanced trees• Heaps• Hash tables

Algorithms Syllabus

Sorting and searching: • Classical sorting methods• Specialized sorting techniques• Finding max & min• Median finding and Kth selection• Majority detection• Meta algorithms

Algorithms Syllabus

Computational geometry: • Convex hulls• Lower bounds • Line segment intersection• Planar subdivision search• Voronoi diagrams• Nearest neighbors• Geometric minimum spanning trees• Delaunay triangulations• Distance between convex polygons• Triangulation of polygons• Collinear subsets

Algorithms Syllabus

Graph algorithms:

• Depth-first search• Breadth-first search• Minimum spanning trees• Shortest paths trees• Radius-cost tradeoffs• Steiner trees• Degree-constrained trees

Algorithms Syllabus

NP-completeness: • Resource-constrained computation• Complexity classes• Intractability• Boolean satisfiability• Cook-Levin theorem• Transformations• Graph clique problem• Independent sets• Hamiltonian cycles• Colorability problems• Heuristics

PNP

NP-complete SAT

co-NP-complete TAUT

co-NP P-complete LP

Algorithms Syllabus

Other topics in algorithms:

• Linear programming• Matrix multiplication• String matching• Minimum matchings• Network flows• Distributed algorithms• Amortized analysis• Zero knowledge proofs

≈

• Focus on the “big picture” & “scientific method”

• Emphasis on problem solving & creativity

• Discuss applications & practice

• A primary objective: have fun!

Overarching Philosophy

Algorithms Throughout History

A brief history of computing:• Aristotle, Euclid, Archimedes, Eratosthenes

• Abu Ali al-Hasan ibn al-Haytham

• Fibonacci, Descartes, Fermat, Pascal

• Newton, Euler, Gauss, Hamilton

• Boole, De Morgan, Babbage, Ada Agusta

• Venn, Carroll, Cantor, Hilbert, Russell

• Hardy, Ramanujan, Ramsey

• Godel, Church, Turing, von Neumann

• Shannon, Kleene, Chomsky

An Ancient Computer: The Antikythera • Oldest known mechanical computer• Built around 150-100 BCE !• Calculates eclipses and astronomical positions of sun, moon, and planets• Very sophisticated for its era• Contains dozens of intricate gears• Comparable to 1700’s Swiss clocks• Has an attached “instructions manual”• Still the subject of ongoing research

Problem: Can 5 test tubes be spun simultaneously in a 12-hole centrifuge in a balanced way?

• What approaches fail?• What techniques work and why?• Lessons and generalizations

• Some discrete math & algorithms knowldege

• Ideally, should have taken CS2102

• Course will “bootstrap” (albeit quickly) from first principles

• Critical: Tenacity, patience

Prerequisites

• Exams: probably take home– Decide by vote– Flexible exam schedule

• Problem sets:– Lots of problem solving– Work in groups!– Not formally graded– Many exam questions will

come from homeworks!• Extra credit problems

– In class & take-home– Find mistakes in slides, handouts, etc.

• Course materials posted on Web sitewww.cs.virginia.edu/robins/theory

Course Organization

• Midterm 35%

• Final 35%

• Project 30%

• Extra credit 10%

Best strategy:

• Solve lots of problems!

Grading Scheme

Professor Gabriel RobinsOffice: 409 Rice HallPhone: (434) 982-2207Email: robins@cs.virginia.eduWeb: www.cs.virginia.edu/robins

www.cs.virginia.edu/robins

Office hours: after class• Any other time • By email (preferred)• By appointment• Q&A blog posted on class Web site

Contact Information

• Ask questions ASAP

• Do homeworks ASAP

• Work in study groups

• Do not fall behind

• “Cramming” won’t work

• Start on project early

• Attend every lecture

• Read Email often

• Solve lots of problems

Good Advice

Supplemental Readingswww.cs.virginia.edu/robins/CS_readings.html

• Great videos:– Randy Pausch's "Last Lecture”, 2007– Randy Pausch's "Time Management“, 2007– "Powers of Ten", Charles and Ray Eames, 1977

Supplemental Readingswww.cs.virginia.edu/robins/CS_readings.html

• Theory and Algorithms: – Who Can Name the Bigger Number, Scott Aaronson, 1999 – The Limits of Reason, Gregory Chaitin, Scientific American, March

2006, pp. 74-81. – Breaking Intractability, Joseph Traub and Henryk Wozniakowski,

Scientific American, January 1994, pp. 102-107. – Confronting Science's Logical Limits, John Casti, Scientific

American, October 1996, pp. 102-105. – Go Forth and Replicate, Moshe Sipper and James Reggia, Scientific

American, August 2001, pp. 34-43. – The Science Behind Sudoku, Jean-Paul Delahaye, Scientific

American, June 2006, pp. 80-87. – The Traveler's Dilemma, Kaushik Basu, Scientific American, June

2007, pp. 90-95.

Supplemental Readingswww.cs.virginia.edu/robins/CS_readings.html

• Biological Computing: – Computing with DNA, Leonard Adleman, Scientific American,

August 1998, pp. 54-61. – Bringing DNA Computing to Life, Ehud Shapiro and Yaakov

Benenson, Scientific American, May 2006, pp. 44-51. – Engineering Life: Building a FAB for Biology, David Baker et

al., Scientific American, June 2006, pp. 44-51. – Big Lab on a Tiny Chip, Charles Choi, Scientific American,

October 2007, pp. 100-103. – DNA Computers for Work and Play, Macdonald et al, Scientific

American, November 2007, pp. 84-91.

Supplemental Readingswww.cs.virginia.edu/robins/CS_readings.html

• Quantum Computing: – Quantum Mechanical Computers, Seth Lloyd, Scientific

American, 1997, pp. 98-104. – Quantum Computing with Molecules, Gershenfeld and Chuang,

Scientific American, June 1998, pp. 66-71. – Black Hole Computers, Seth Lloyd and Jack Ng, Scientific

American, November 2004, pp. 52-61. – Computing with Quantum Knots, Graham Collins, Scientific

American, April 2006, pp. 56-63. – The Limits of Quantum Computers, Scott Aaronson, Scientific

American, March 2008, pp. 62-69. – Quantum Computing with Ions, Monroe and Wineland,

Scientific American, August 2008, pp. 64-71.

Supplemental Readingswww.cs.virginia.edu/robins/CS_readings.html

• History of Computing: – Alan Turing's Forgotten Ideas, B. Jack Copeland and Diane

Proudfoot, Scientific American, May 1999, pp. 98-103. – Ada and the First Computer, Eugene Kim and Betty Toole,

Scientific American, April 1999, pp. 76-81.

• Security and Privacy: – Malware Goes Mobile, Mikko Hypponen, Scientific American,

November 2006, pp. 70-77. – RFID Powder, Tim Hornyak, Scientific American, February

2008, pp. 68-71. – Can Phishing be Foiled, Lorrie Cranor, Scientific American,

December 2008, pp. 104-110.

Supplemental Readingswww.cs.virginia.edu/robins/CS_readings.html

• Future of Computing: – Microprocessors in 2020, David Patterson, Scientific American, September

1995, pp. 62-67. – Computing Without Clocks, Ivan Sutherland and Jo Ebergen, Scientific

American, August 2002, pp. 62-69. – Making Silicon Lase, Bahram Jalali, Scientific American, February 2007,

pp. 58-65. – A Robot in Every Home, Bill Gates, Scientific Am, January 2007, pp. 58-65. – Ballbots, Ralph Hollis, Scientific American, October 2006, pp. 72-77. – Dependable Software by Design, Daniel Jackson, Scientific American, June

2006, pp. 68-75. – Not Tonight Dear - I Have to Reboot, Charles Choi, Scientific American,

March 2008, pp. 94-97. – Self-Powered Nanotech, Zhong Lin Wang, Scientific American, January

2008, pp. 82-87.

Supplemental Readingswww.cs.virginia.edu/robins/CS_readings.html

• The Web: – The Semantic Web in Action, Lee Feigenbaum et al., Scientific American,

December 2007, pp. 90-97. – Web Science Emerges, Nigel Shadbolt and Tim Berners-Lee, Scientific

American, October 2008, pp. 76-81.

• The Wikipedia Computer Science Portal: – Theory of computation and Automata theory – Formal languages and grammars – Chomsky hierarchy and the Complexity Zoo – Regular, context-free &Turing-decidable languages – Finite & pushdown automata; Turing machines – Computational complexity – List of data structures and algorithms

Supplemental Readingswww.cs.virginia.edu/robins/CS_readings.html

• The Wikipedia Math Portal: – Problem solving – List of Mathematical lists – Sets and Infinity – Discrete mathematics – Proof techniques and list of proofs – Information theory & randomness – Game theory

• Mathematica's “Math World”

Problem: Can 5 test tubes be spun simultaneously in a 12-hole centrifuge in a balanced way?

• What does “balanced” mean?• Why are 3 test tubes balanced?• Symmetry!• Can you merge solutions?• Superposition!• Linearity! ƒ(x + y) = ƒ(x) + ƒ(y)• Can you spin 7 test tubes?• Complementarity!• Empirical testing…

No vector calculus / trig

!

No equations!

Truth is guaranteed!

Fundamental principles exposed!

Easy to generalize!

High elegance / beauty!

Problem: 1 + 2 + 3 + 4 + …+ 100 = ?

Proof: Induction…

1 + 2 + 3 + … + 99 + 100

100 + 99 + 98 + … + 2 + 1

101 + 101 + 101 + … + 101 + 101 =

2

)1(

1

nni

n

i

n+1

n

100*101

= (100*101)/2 = 5050

• You must a priori know the formula / result• Easy to make mistakes in inductive proof• Mostly “mechanical” – ignores intuitions• Tedious to construct• Difficult to check• Hard to understand• Not very convincing• Generalizations not obvious• Does not “shed light on truth”• Obfuscates connections

Conclusion: only use induction as a last resort!I.e., almost never!

Drawbacks of Induction

Oh oh!induction

Problem: (1/4) + (1/4)2 + (1/4)3 + (1/4)4 + … = ?

?4

1

1

ii

Extra Credit:

Find a short, geometric, induction-free proof.

Problem: (1/4) + (1/4)2 + (1/4)3 + (1/4)4 + … = ?

Find a short, geometric, induction-free proof.

3

1

4

1

1

ii1

1

Problem: (1/8) + (1/8)2 + (1/8)3 + (1/8)4 + …= ?

?8

1

1

ii

Extra Credit:

Find a short, geometric, induction-free proof.

Problem: (1/8) + (1/8)2 + (1/8)3 + (1/8)4 + …= ?

Find a short, geometric, induction-free proof.

7

1

8

1

1

ii

Problem: 13 + 23 + 33 + 43 + …+ n3 = ?

?i1

3

n

i

Extra Credit:

find a short, geometric,

induction-free proof.

Problem: Can an 8x8 board with two opposite corners missing be tiles with 31 dominoes?

= 31 x ?

• What approaches fail?• What techniques work and why?• Lessons and generalizations

Problem: Given any five points in/on the unitsquare, is there always a pair with distance ≤ ?

1

1

2

1

• What approaches fail?• What techniques work and why?• Lessons and generalizations

Problem: Given any five points in/on the unit equilateral triangle, is there always a pair with distance ≤ ½ ?

1 1

1

• What approaches fail?• What techniques work and why?• Lessons and generalizations

• What approaches fail?• What techniques work and why?• Lessons and generalizations

Problem: Prove that there are an infinity of primes.

Extra Credit: Find a short, induction-free proof.

• What approaches fail?• What techniques work and why?• Lessons and generalizations

Problem: True or false: there are arbitrary long blocks of consecutive composite integers

(i.e., big “prime deserts”)

Extra Credit: find a short, induction-free proof.

• What approaches fail?• What techniques work and why?• Lessons and generalizations

Problem: Prove that is irrational.

Extra Credit: find a short, induction-free proof.

2

• What approaches fail?• What techniques work and why?• Lessons and generalizations

Problem: Does exponentiation preserve irrationality? i.e., are there two irrational numbers x and y such that xy is rational?

Extra Credit: find a short, induction-free proof.

X = 2X

XXX

…

Problem: Solve the following equation for X:

where the stack of exponentiated x’s extends forever.

• What approaches fail?• What techniques work and why?• Lessons and generalizations

Problem: Are the complex numbers closed under exponentiation ? E.g., what is the value of ii?

Theorem [Turing]: not all problems are solvable by algorithms.Theorem: not all functions are computable by algorithms.Theorem: not all Boolean functions are computable by algorithms.Theorem: most Boolean functions are not computable!

Q: Can we find a concrete example of an uncomputable function?A [Turing]: Yes, for example, the Halting Problem.

Definition: The Halting problem: given a program P and input I, will P ever halt if we ran it on I?

Define H:ℕ´ℕ®{0,1}H(P,I)=1 if program P halts on input IH(P,I)=0 otherwise

• Both P and I can be encoded as strings• P and I can also be encoded as integers (in some canonical order )• H is an everywhere-defined Boolean function on natural #’s

PI

yesno

Does P(I)halt?

H

Number of steps to termination for the first 10,000 numbers

Theorem [Turing]: the halting problem (H) is not computable.

Ex: the “3X+1” problem (the Ulam conjecture):• Start with any integer X>0• If X is even, then replace it with X/2• If X is odd then replace it with 3X+1• Repeat until X=1 (i.e., short cycle 4, 2, 1, ...)

Ex: 26 terminates after 10 steps 27 terminates after 111 stepsTermination verified for X<1018

Q: Does this terminate for every X>0 ?

A: Open since 1937!“Mathematics is not yet ready for such confusing, troubling, and hard problems." - Paul Erdős, who offered a $500 bounty for a solution to this problem

Observation: termination is in general difficult to detect!

Theorem [Turing]: the halting problem (H) is not computable.

Corollary: we can not algorithmically detect all infinite loops.

Q: Why not? E.g., do the following programs halt?

main(){ int k=3; }

main(){ while(1) {} }

Halts! Runs forever! ?

main(){ Find a Fermat triple an+bn=cn

with n>2 & stop}

Runs forever!Open from 1637-1995!

main(){ Find a Goldbach integer that is not a sum of two primes & stop}

?Still open since 1742!

Theorem: solving the halting problem is at least as hard as solving arbitrary open mathematical problems!

Theorem [Turing]: the halting problem (H) is not computable.Proof: Assume $ algorithm S that solves the halting problem H, that always stops with the correct answer for any P & I.

¥

PI

yes

noDoes

P(I) halt?

S

X

T

T(T) haltsQ Û ~Q Þ Contradiction!

PI

yes

noDoes

P(I) halt?

S

PI

yes

noDoes

P(I) halt?

S

Þ S cannot exist! (at least as an algorithm / program / TM)

Using S, construct algorithm / TM T:

Þ T(T) haltsÞ T(T) does not halt

T(T) does not halt

Diagon

aliza

tion

Non-existence proof!

Theorem: all computable numbers are finitely describable.Proof: A computable number can be outputted by a TM.

A TM is a (unique) finite description.

What the unsolvability of the Halting Problem means:

There is no single algorithm / program / TM that correctly solves all instances of the halting problem in finite time each.

This result does not necessarily apply if we allow:

• Incorrectness on some instances• Infinitely large algorithm / program• Infinite number of finite algorithms / programs• Some instances to not be solved• Infinite “running time” / steps• Powerful enough oracles

Q: When do we want to feed a program to itself in practice?A: When we build compilers.

Q: Why?A: To make them more efficient! To boot-strap the coding in the compiler’s own language!

Program Ccompiler

Executablecode

Theorem: virus detectionis not computable.

Theorem: Infinite loop detection is not computable.

Self-Replication• Biology / DNA• Nanotechnology• Computer viruses• Space exploration• Memetics / memes• “Gray goo”

Problem (extra credit): write a program thatprints out its own source code (no inputs of any kind are allowed).

Self-replicating cellular automata designed by von Neumann

Non-Existence Proofs• Must cover all possible (usually infinite) scenarios!• Examples / counter-examples are not convincing!• Not “symmetric” to existence proofs!

Ex: proof that you are a millionaire:

“Proof” that you are not a millionaire ?

Existence proofs

can be easy!

Non-existence proofs

are often hard!

P¹NP

Pigeon-Hole Principle• J. Dirichlet (1834)• “Drawer principle”• “Shelf Principle”• “Box principle”

Theorem (pigeon-hole): There is no injective (1-to-1) function from a finite set (domain) to a smaller finite set (range).

Generalization:N objects placed in M containers; then:

• at least 1 container must hold

• at least 1 container must hold

M

N

M

N

Naturals ℕ 6Integers ℤ -4

Rationals ℚ 2/9

Reals ℝ

Qua

tern

ions

ℍ 1

+i+

j+k Complex ℂ 7+3i

Surreal {

L|R

}

Sur

com

plex

A+B

i

Primes ℙ 5

Oct

onio

ns

1+

i+j+

k+E

+I+

J+K

Hypernumbers

Sed

enio

ns S

1+

i+j+

k+…

+e 15

+e 16

?

Boolean 1

Com

puta

ble

num

bers

Finitely describable numbers H

Alg

ebra

ic

2T

ranc

ende

ntal

pIr

rati

onal

s J

Theorem: some real numbers are not finitely describable!Theorem: some finitely describable real numbers are not computable!

Generalized Numbers

Algorithms

1. Existence

2. Efficiency

• Time

• Space

Worst case behavior analysis as a function of input size

 

Asymptotic growth: O W Q o wDonald Knuth

Upper BoundsDefinition: f(n) = O(g(n))

Û $ c,k > 0 ' 0 £ f(n) £ c×g(n) " n>k

Û Lim f(n) / g(n) exists n®¥

O(g(n))={f | $ c,k>0 ' 0 £ f(n) £ c×g(n) " n>k}

“f(n) is big-O of g(n)”

Ex: n = O(n2)33n+17 = O(n)n8+n7 = O(n12)n100 = O(2n)213 = O(1)

f(n)

g(n)

kn

Lower BoundsDefinition: f(n) = W(g(n))

Û g(n)=O(f(n))

Û Lim g(n) / f(n) exists n®¥

W(g(n))={f | $ c,k>0 ' 0 £ g(n) £ c×f(n) " n>k}

“f(n) is Omega of g(n)”

Ex: 100n = W(n)33n+17 = W(log n)n8-n7 = W(n8)213 = W(1/n)10100 = W(1)

g(n)

f(n)

kn

Tight BoundsDefinition: f(n) = Q(g(n))

Û f(n)=O(g(n)) and g(n)=O(f(n)) Û f(n)=O(g(n)) and f(n)=W(g(n))

Û Lim g(n)/f(n) and Lim f(n)/g(n) exist n®¥ n®¥

“f(n) is Theta of g(n)”

Ex: 99n = Q(n)n + log n = Q(n)n8-n7 = Q(n8)n2 + cos(n) = Q(n2)213 = Q(1)

c2×g(n)

f(n)

kn

c1×g(n)

Loose BoundsDefinition: f(n) = o(g1(n))

Û f(n)=O(g1(n)) and f(n)≠W(g1(n))“f(n) is little-o of g1(n)”Definition: f(n) = w(g2(n))

Û f(n)=W(g2(n)) and f(n)≠O(g2(n))“f(n) is little-omega of g2 (n)”

Ex: 8n = o(n log log n)n log n = w(n)n6 = o(n6.01)n2 + sin(n) = w(n)213 = o(log n)

g2(n)

f(n)

kn

g1(n)

Growth Laws

Let f1(n)=O(g1(n)) and f2(n)=O(g2(n)) 

Thm: f1(n) + f2(n) = O(max(g1(n) , g2(n))  

• Sequential code

Thm: f1(n) • f2(n) = O(g1(n) • g2(n))

• Nested loops & subroutine calls

Thm: nk = O(cn) " c,k>0 

Ex: n1000 = O(1.001n)

Solving Recurrences

T(n) = a • T(n/b) + f(n) 

a≥1, b>1, and let c = logba 

Thm: f(n)=O(nc-e) for some >e 0 Þ T(n)=Q(nc) f(n)=Q(nc) Þ T(n)=Q(nc log n) f(n)=W(nc+e) some >e 0 and a•f(n/b) ≤ d•f(n)

for some d<1 " n>n0 Þ T(n)=Q(f(n)) 

Ex: T(n) = 2T(n/2)+n Þ T(n)=Q(n log n) 

T(n) = 9T(n/3)+n Þ T(n)=Q(n2) 

T(n) = T(2n/3)+1 Þ T(n)=Q(log n)

Stirling’s Formula

Factorial: 

Theorem:

where e is Euler’s constant = 2.71828…

Theorem:  

Corollary: 

log(n!) = O(n log n)

• Useful in analyses and bounds

n1)-(n2)-(n . . .321 n!

n

1Θ1

e

nπn2n!

n

n

e

nπn2n!

n) log O(n

2

πn2log

e

nlogn

e

nπn2log)(n! log

n

Problem: Given any five points in/on the unitsquare, is there always a pair with distance ≤ ?

1

1

2

1

• What approaches fail?• What techniques work and why?• Lessons and generalizations

Problem: Given any five points in/on the unit equilateral triangle, is there always a pair with distance ≤ ½ ?

1 1

1

• What approaches fail?• What techniques work and why?• Lessons and generalizations

Problem: Given any ten points in/on the unitsquare, what is the maximum pairwise distance?

1

1

• What approaches fail?• What techniques work and why?• Lessons and generalizations

Data Structures

Data Structures• Techniques for organizing information effectively

• Allowed operations:• Initialize• Insert• Delete• Search• Min/max• Successor• Predecessor• Merge• Split• Revert

Primitive types:1. Boolean2. Character3. Floating-point4. Double5. Integer6. Enumerated type 

Abstract data types:7. Array8. Container9. Map10. Associative array11. Dictionary12. Multimap13. List14. Set15. Multiset / Bag16. Priority queue17. Queue18. Double-ended queue19. Stack20. String21. Tree22. Graph

Data StructuresComposite types:23. Array24. Record25. Union26. Tagged union

Arrays:27. Bit array28. Bit field29. Bitboard30. Bitmap31. Circular buffer32. Control table33. Image34. Dynamic array35. Gap buffer36. Hashed array tree37. Heightmap38. Lookup table39. Matrix40. Parallel array41. Sorted array42. Sparse array43. Sparse matrix44. Iliffe vector45. Variable-length array

Lists:46. Doubly linked list47. Array list48. Linked list49. Self-organizing list50. Skip list51. Unrolled linked list52. VList53. Xor linked list54. Zipper55. Doubly connected edge list56. Difference list57. Free list 

Binary trees:58. AA tree59. AVL tree60. Binary search tree61. Binary tree62. Cartesian tree63. Order statistic tree64. Pagoda65. Randomized binary search tree66. Red-black tree67. Rope

Binary trees (continued):68. Scapegoat tree69. Self-balancing search tree70. Splay tree71. T-tree72. Tango tree73. Threaded binary tree74. Top tree75. Treap76. Weight-balanced tree77. Binary data structure Trees:78. Trie79. Radix tree80. Suffix tree81. Suffix array82. Compressed suffix array83. FM-index84. Generalised suffix tree85. B-trie86. Judy array87. X-fast trie88. Y-fast trie89. Ctrie

Data StructuresB-trees:90. B-tree91. B+ tree92. B*-tree93. B sharp tree94. Dancing tree95. 2-3 tree96. 2-3-4 tree97. Queap98. Fusion tree99. Bx-tree100. AList 

Heaps:101. Heap102. Binary heap103. Weak heap104. Binomial heap105. Fibonacci heap106. AF-heap107. Leonardo Heap108. 2-3 heap109. Soft heap110. Pairing heap111. Leftist heap112. Treap

113. Beap114. Skew heap115. Ternary heap116. D-ary heap117. Brodal queue 

Multiway trees:118. Ternary tree119. K-ary tree120. And–or tree121. (a,b)-tree122. Link/cut tree123. SPQR-tree124. Spaghetti stack125. Disjoint-set data structure126. Fusion tree127. Enfilade128. Exponential tree129. Fenwick tree130. Van Emde Boas tree131. Rose tree 

Space-partitioning trees:132. Segment tree133. Interval tree134. Range tree

Space-partitioning trees (cont):135. Bin136. Kd-tree137. Implicit kd-tree138. Min/max kd-tree139. Adaptive k-d tree140. Quadtree141. Octree142. Linear octree143. Z-order144. UB-tree145. R-tree146. R+ tree147. R* tree148. Hilbert R-tree149. X-tree150. Metric tree151. Cover tree152. M-tree153. VP-tree154. BK-tree155. Bounding interval hierarchy156. BSP tree157. Rapidly exploring random tree 

Data StructuresApplication-specific trees:158. Abstract syntax tree159. Parse tree160. Decision tree161. Alternating decision tree162. Minimax tree163. Expectiminimax tree164. Finger tree165. Expression tree166. Log-structured merge-tree 

Hashes:167. Bloom filter168. Count-Min sketch169. Distributed hash table170. Double Hashing171. Dynamic perfect hash table172. Hash array mapped trie173. Hash list174. Hash table175. Hash tree176. Hash trie177. Koorde178. Prefix hash tree179. Rolling hash

180. MinHash181. Quotient filter182. Ctrie 

Graphs:183. Graph184. Adjacency list185. Adjacency matrix186. Graph-structured stack187. Scene graph188. Binary decision diagram189. 0-suppressed decision diagram190. And-inverter graph191. Directed graph192. Directed acyclic graph193. Propositional dir. acyclic graph194. Multigraph195. Hypergraph 

Other:196. Lightmap197. Winged edge198. Doubly connected edge list199. Quad-edge200. Routing table201. Symbol table

Arrays• Sequence of "indexible" locations 

• Unordered:• O(1) to add• O(n) to search / delete• O(n) for min / max

• Ordered:• O(n) to add / delete• O(log n) to (binary) search• O(1) for min / max

1 2 3 4 5 6 7 . . .

Stacks• LIFO (Last-In First-Out) 

• Operations: push / pop O(1) each• Can not access “middle”• Analogy: trays/plates at cafeteria • Applications:

• Recursion• Compiling / parsing• Dynamic binding• Web surfing

in

out

Queues• FIFO (First-In First-Out)   • Operations: push / pop O(1) each• Can not access “middle”• Analogy: line at the store• Applications:

• Simulations• Scheduling• Networks• Operating systems

in out

Linked Lists• Successor / predecessor pointers• Types:

• Single linked

• Double linked • Circular

• Operations:• Add: O(1) time• Search: O(n) time• Delete: O(1) time (given pointer)

• What approaches fail?• What techniques work and why?• Lessons and generalizations

Extra Credit Problem: Given a pointer to a read-only (unmodefiable) linked list containing an unknown number of nodes n, devise an O(n)-time and O(1) space algorithm that determines whether that list contains a cycle.

or

Trees• Parent/children pointers 

• Binary / N-ary 

• Ordered / unordered 

• Height-balanced:• AVL trees• B-trees• Red-black trees• 2-3 trees• add / delete / search in O(log n) time

c

b e

d fa

Height-Balanced AVL Trees

Self-Adjusting Trees

Tree Traversal• Pre-order:

1. process node2. visit children

Þ c b a e d f• Post-order:

1. visit children2. process node

Þ a b d f e c• In-order:

1. visit left-child2. process node3. visit right-child

Þ a b c d e f

c

b e

d fa

Heaps• A tree where all of each node’s children

have larger / smaller “keys”• Can be implemented

using binary tree or array• Operations:

• Find min/max: O(1) time• Add: O(log n) time• Delete: O(log n) time• Search: O(n) time

 

Hash Tables• Direct access• Hash function• Collision resolution:

• Chaining• Linear probing• Double hashing

• Universal hashing• O(1) average access• O(n) worst-case access

Q: Improve worst-case access to O(log n)? 

Hash Tables• Linear probing & searching 

SortingAlmost half of all CPU cycles are spent on sorting!• Input: array X[1..n] of integers• Output: sorted array (permutation of input)

In: 5,2,9,1,7,3,4,8,6 Out: 1,2,3,4,5,6,7,8,9

• Assume WLOG all input numbers are unique• Decision tree model Þ count comparisons “<”

Lower Bound for SortingTheorem: Sorting requires W(n log n) timeProof: Assume WLOG unique numbers Þ n! different permutations Þ comparison decision tree has n! leaves

Þ tree height is:

Þ W(n log n) decisions / time necessary to sort<

<

< <

<

< <

<

< <

<

< <

< <

n! permutations (i.e., distinct sorted outcomes )

W(n log n)

n) log (ne

nlogn

e

nlog)(n! log

n

Unique execution path

Non-existence proof!

1. AKS sort2. Bead sort3. Binary tree sort4. Bitonic sorter5. Block sort6. Bogosort7. Bozo sort8. Bubble sort9. Bucket sort10. Burstsort11. Cocktail sort12. Comb sort13. Counting sort14. Cubesort15. Cycle sort

Sorting Algorithms16. Flashsort 17. Franceschini's sort18. Gnome sort19. Heapsort20. In-place merge sort21. Insertion sort22. Introsort23. Library sort24. Merge sort25. Odd-even sort26. Patience sorting27. Pigeonhole sort28. Postman sort29. Quantum sort30. Quicksort

31. Radix Sort32. Shell sort33. Simple pancake sort34. Smoothsort35. Sorting network36. Spaghetti sort37. Spreadsort38. Stooge sort39. Strand sort40. Timsort41. Tournament sort42. UnShuffle Sort43. Selection sort44. Shaker sort

Q: Why so many sorting algorithms?A: Is there a “best” sorting algorithm?

• Worst case?• Average case?• In practice?• Input distribution?• Near-sorted data?• Stability?• In-Situ?

• Randomized?• Stack depth?• Internal vs. external?• Pipeline compatible?• Parallelizable?• Locality?

Sorting Algorithms

Animation of 15 Sorting Algorithms 

X = 2X

XXX

…

Problem: Solve the following equation for X:

where the stack of exponentiated x’s extends forever.

• What approaches fail?• What techniques work and why?• Lessons and generalizations

• What approaches fail?• What techniques work and why?• Lessons and generalizations

Problem: Prove that there are an infinity of primes.

Extra Credit: Find a short, induction-free proof.

• What approaches fail?• What techniques work and why?• Lessons and generalizations

Problem: True or false: there are arbitrary long blocks of consecutive composite integers

(i.e., big “prime deserts”)

Extra Credit: find a short, induction-free proof.

• What approaches fail?• What techniques work and why?• Lessons and generalizations

Problem: Prove that is irrational.

Extra Credit: find a short, induction-free proof.

2

• What approaches fail?• What techniques work and why?• Lessons and generalizations

Problem: Does exponentiation preserve irrationality? i.e., are there two irrational numbers x and y such that xy is rational?

Extra Credit: find a short, induction-free proof.

Problem: Can an 8x8 board with two opposite corners missing be tiles with 31 dominoes?

= 31 x ?

• What approaches fail?• What techniques work and why?• Lessons and generalizations

Problem: 13 + 23 + 33 + 43 + …+ n3 = ?

?i1

3

n

i

Extra Credit:

find a short, geometric,

induction-free proof.

Problem: Are the complex numbers closed under exponentiation ? E.g., what is the value of ii?