Plaintext
Computer Science III
Dr. Chris Bourke
Department of Computer Science & Engineering
University of Nebraska—Lincoln
Lincoln, NE 68588, USA
http://cse.unl.edu/~cbourke
cbourke@cse.unl.edu
2016/08/05 21:48:04
Version 1.2.0
These are lecture notes used in CSCE 310 (Data Structures & Algorithms) at the
University of Nebraska—Lincoln.
This work is licensed under a Creative Commons
Attribution-ShareAlike 4.0 International License
i
Contents
1 Introduction 1
2 Algorithm Analysis 3
2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2.1.1 Example: Computing a Sum . . . . . . . . . . . . . . . . . . . . . 6
2.1.2 Example: Computing a Mode . . . . . . . . . . . . . . . . . . . . 8
2.2 Pseudocode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.3 Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2.4 Asymptotics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.4.1 Big-O Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.4.2 Other Notations . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.4.3 Observations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
2.4.4 Limit Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
2.5 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
2.5.1 Linear Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
2.5.2 Set Operation: Symmetric Difference . . . . . . . . . . . . . . . . 29
2.5.3 Euclid’s GCD Algorithm . . . . . . . . . . . . . . . . . . . . . . . 30
2.5.4 Selection Sort . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
2.6 Other Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
2.6.1 Importance of Input Size . . . . . . . . . . . . . . . . . . . . . . . 33
2.6.2 Control Structures are Not Elementary Operations . . . . . . . . . 36
2.6.3 Average Case Analysis . . . . . . . . . . . . . . . . . . . . . . . . 37
2.6.4 Amortized Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . 38
2.7 Analysis of Recursive Algorithms . . . . . . . . . . . . . . . . . . . . . . . 39
2.7.1 The Master Theorem . . . . . . . . . . . . . . . . . . . . . . . . . 40
3 Storing Things 45
3.1 Lists . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
3.2 Sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
3.3 Hash-Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
3.3.1 Hash Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
3.3.2 Collisions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
3.3.3 Efficiency Rehashing . . . . . . . . . . . . . . . . . . . . . . . . . 47
3.3.4 Other Applications . . . . . . . . . . . . . . . . . . . . . . . . . . 48
3.3.5 Java Implementations . . . . . . . . . . . . . . . . . . . . . . . . . 48
iii
Contents
3.4 Bloom Filters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
3.5 Disjoint Sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
3.6 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
4 Brute Force Style Algorithms 51
4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
4.1.1 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
4.1.2 Backtracking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
4.2 Generating Combinatorial Objects . . . . . . . . . . . . . . . . . . . . . . 52
4.2.1 Generating Combinations (Subsets) . . . . . . . . . . . . . . . . . 52
4.2.2 Generating Permutations . . . . . . . . . . . . . . . . . . . . . . . 53
4.2.3 Permutations with Repetition . . . . . . . . . . . . . . . . . . . . 54
4.2.4 Set Partitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
4.3 Problems & Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
4.3.1 Satisfiability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
4.3.2 Hamiltonian Path/Cycle . . . . . . . . . . . . . . . . . . . . . . . 59
4.3.3 0-1 Knapsack . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
4.3.4 Closest Pair . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
4.3.5 Convex Hull . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
4.3.6 Assignment Problem . . . . . . . . . . . . . . . . . . . . . . . . . 67
4.3.7 Subset Sum . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
4.4 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
5 Divide & Conquer Style Algorithms 71
5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
5.2 Problems & Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
5.3 Repeated Squaring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
5.4 Euclid’s GCD Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
5.5 Peasant Multiplication . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
5.6 Karatsuba Multiplication . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
5.7 Strassen’s Matrix Multiplication . . . . . . . . . . . . . . . . . . . . . . . 78
5.8 Closest Pair Revisited . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
5.9 Convex Hull Revisited . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
5.10 Fast Fourier Transform . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
5.11 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
6 Linear Systems 83
6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
6.2 Solving Linear Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
6.2.1 LU Decomposition . . . . . . . . . . . . . . . . . . . . . . . . . . 87
6.2.2 Matrix Inverse . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
6.2.3 Determinants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
6.3 Linear Programming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
6.3.1 Formulations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
iv
Contents
6.4 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
7 Trees 97
7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
7.2 Definitions & Terminology . . . . . . . . . . . . . . . . . . . . . . . . . . 97
7.3 Tree Traversal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
7.3.1 Preorder Traversal . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
7.3.2 Inorder Traversal . . . . . . . . . . . . . . . . . . . . . . . . . . 100
7.3.3 Postorder Traversal . . . . . . . . . . . . . . . . . . . . . . . . . 101
7.3.4 Breadth-First Search Traversal . . . . . . . . . . . . . . . . . . . 104
7.3.5 Implementations & Data Structures . . . . . . . . . . . . . . . . 104
7.3.6 Operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
7.4 Binary Search Trees . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
7.4.1 Basic Operations . . . . . . . . . . . . . . . . . . . . . . . . . . 110
7.5 Balanced Binary Search Trees . . . . . . . . . . . . . . . . . . . . . . . 112
7.5.1 AVL Trees . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
7.5.2 B-Trees . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120
7.5.3 Red-Black Trees . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
7.6 Heaps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
7.6.1 Operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
7.6.2 Implementations . . . . . . . . . . . . . . . . . . . . . . . . . . . 124
7.6.3 Java Collections Framework . . . . . . . . . . . . . . . . . . . . 128
7.6.4 Other Operations . . . . . . . . . . . . . . . . . . . . . . . . . . 128
7.6.5 Variations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128
7.7 Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128
7.7.1 Heap Sort . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128
7.7.2 Huffman Coding . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
8 Graph Algorithms 141
8.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
8.2 Depth First Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142
8.2.1 DFS Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144
8.2.2 DFS Artifacts . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
8.2.3 Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146
8.3 Breadth First Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148
8.4 DFS/BFS Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . 150
8.4.1 Connectivity & Path Finding . . . . . . . . . . . . . . . . . . . . 151
8.4.2 Topological Sorting . . . . . . . . . . . . . . . . . . . . . . . . . 151
8.4.3 Shortest Path . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151
8.4.4 Cycle Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . 152
8.4.5 Bipartite Testing . . . . . . . . . . . . . . . . . . . . . . . . . . 153
8.4.6 Condensation Graphs . . . . . . . . . . . . . . . . . . . . . . . . 153
8.5 Minimum Spanning Tree Algorithms . . . . . . . . . . . . . . . . . . . . 154
8.5.1 Greedy Algorithmic Strategy . . . . . . . . . . . . . . . . . . . . 154
v
Contents
8.5.2 Kruskal’s Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . 155
8.5.3 Prim’s Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . 157
8.6 Minimum Distance Algorithms . . . . . . . . . . . . . . . . . . . . . . . 160
8.6.1 Dijkstra’s Algorithm . . . . . . . . . . . . . . . . . . . . . . . . 160
8.6.2 Floyd-Warshall Algorithm . . . . . . . . . . . . . . . . . . . . . 162
8.6.3 Huffman Coding . . . . . . . . . . . . . . . . . . . . . . . . . . . 168
8.7 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168
9 Dynamic Programming 171
9.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171
9.1.1 Optimal Substructure Property . . . . . . . . . . . . . . . . . . 171
9.2 Binomial Coefficients . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172
9.3 Optimal Binary Search Trees . . . . . . . . . . . . . . . . . . . . . . . . 174
9.3.1 Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178
9.4 Dynamic Knapsack . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179
9.4.1 Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182
9.4.2 Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182
9.5 Coin Change Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182
9.5.1 Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183
9.6 Matrix Chain Multiplication . . . . . . . . . . . . . . . . . . . . . . . . 184
9.6.1 Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186
9.7 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 188
10 Computational Models 191
10.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191
10.1.1 Languages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192
10.2 Computational Models . . . . . . . . . . . . . . . . . . . . . . . . . . . 195
10.2.1 Finite-State Automata . . . . . . . . . . . . . . . . . . . . . . . 195
10.2.2 Turing Machines . . . . . . . . . . . . . . . . . . . . . . . . . . . 197
10.2.3 Church-Turing Thesis . . . . . . . . . . . . . . . . . . . . . . . . 199
10.2.4 Halting Problem & Decidability . . . . . . . . . . . . . . . . . . 202
10.3 Complexity Classes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204
10.3.1 Deterministic Polynomial Time . . . . . . . . . . . . . . . . . . 205
10.3.2 Nondeterminism . . . . . . . . . . . . . . . . . . . . . . . . . . . 205
10.4 Reductions & NP-Completeness . . . . . . . . . . . . . . . . . . . . . . 206
10.4.1 Satisfiability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 208
10.4.2 Satisfiability to Clique . . . . . . . . . . . . . . . . . . . . . . . 210
10.4.3 Clique to Vertex Cover . . . . . . . . . . . . . . . . . . . . . . . 211
10.5 Beyond P and NP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213
10.6 Misc . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214
10.7 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 216
11 More Algorithms 219
11.1 A∗ Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219
vi
Contents
11.2 Jump Point Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219
11.3 Minimax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219
Glossary 221
Acronyms 223
Index 227
References 228
vii
List of Algorithms
1 Computing the Mean . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2 Computing the Mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
3 Trivial Sorting (Bad Pseudocode) . . . . . . . . . . . . . . . . . . . . . . . . . 14
4 Trivially Finding the Minimal Element . . . . . . . . . . . . . . . . . . . . . . 14
5 Finding the Minimal Element . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
6 Linear Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
7 Symmetric Difference of Two Sets . . . . . . . . . . . . . . . . . . . . . . . . 29
8 Euclid’s GCD Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
9 Selection Sort . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
10 Sieve of Eratosthenes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
11 Fibonacci(n) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
12 Binary Search – Recursive . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
13 Merge Sort . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
14 Next k-Combination . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
15 Next Lexicographic Permutation . . . . . . . . . . . . . . . . . . . . . . . . . 55
16 Next Repeated Permutation Generator . . . . . . . . . . . . . . . . . . . . . . 56
17 Base Conversion Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
18 Set Partition Generator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
19 Brute Force Iterative Algorithm for Satisfiability . . . . . . . . . . . . . . . . 59
20 Brute Force Recursive Algorithm for Satisfiability . . . . . . . . . . . . . . . . 59
21 Brute Force Iterative Algorithm for Hamiltonian Cycle . . . . . . . . . . . . . 61
22 Hamiltonian DFS Cycle Walk . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
23 Hamiltonian DFS Path Walk–Main Algorithm . . . . . . . . . . . . . . . . . . 62
24 Walk(G, p) – Hamiltonian DFS Path Walk . . . . . . . . . . . . . . . . . . . 62
25 Knapsack(K, S) – Backtracking Brute Force 0-1 Knapsack . . . . . . . . . . 65
26 Brute Force Convex Hull . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
27 Brute Force Subset Sum . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
28 Binary Exponentiation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
29 Euclid’s Simple GCD Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . 74
30 ExtendedEuclideanAlgorithm . . . . . . . . . . . . . . . . . . . . . . . 75
31 (Better) Gaussian Elimination . . . . . . . . . . . . . . . . . . . . . . . . . . 86
32 Back Solving . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
33 Recursive Preorder Tree Traversal . . . . . . . . . . . . . . . . . . . . . . . 104
ix
LIST OF ALGORITHMS
34 Stack-based Preorder Tree Traversal . . . . . . . . . . . . . . . . . . . . . . 105
35 Stack-based Inorder Tree Traversal . . . . . . . . . . . . . . . . . . . . . . . 106
36 Stack-based Postorder Tree Traversal . . . . . . . . . . . . . . . . . . . . . . 107
37 Queue-based BFS Tree Traversal . . . . . . . . . . . . . . . . . . . . . . . . 108
38 Tree Walk based Tree Traversal . . . . . . . . . . . . . . . . . . . . . . . . . 109
39 Search algorithm for a binary search tree . . . . . . . . . . . . . . . . . . . . 111
40 Find Next Open Spot - Numerical Technique . . . . . . . . . . . . . . . . . 126
41 Find Next Open Spot - Walk Technique . . . . . . . . . . . . . . . . . . . . 127
42 Heap Sort . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
43 Huffman Coding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
44 Recursive Depth First Search, Main Algorithm Dfs(G) . . . . . . . . . . . . 143
45 Recursive Depth First Search, Subroutine dfs(G) . . . . . . . . . . . . . . . 143
46 Stack-Based Depth First Search . . . . . . . . . . . . . . . . . . . . . . . . . 144
47 Breadth First Search, Main Algorithm Bfs(G) . . . . . . . . . . . . . . . . 149
48 Breadth First Search, Subroutine bfs(G, v) . . . . . . . . . . . . . . . . . . 149
49 Kruskal’s Minimum Spanning Tree Algorithm . . . . . . . . . . . . . . . . . 156
50 Prim’s Minimum Spanning Tree Algorithm . . . . . . . . . . . . . . . . . . 158
51 Dijkstra’s Single-Source Minimum Distance Algorithm . . . . . . . . . . . . 161
52 Floyd’s All Pair Minimum Distance Algorithm . . . . . . . . . . . . . . . . 164
53 Construct Shortest Path Algorithm . . . . . . . . . . . . . . . . . . . . . . . 165
54 Binomial Coefficient – Dynamic Programming Solution . . . . . . . . . . . . 173
55 Optimal Binary Search Tree . . . . . . . . . . . . . . . . . . . . . . . . . . . 177
56 OBST Tree Construction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178
57 0-1 Knapsack Generator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181
58 Optimal Matrix Chain Multiplication . . . . . . . . . . . . . . . . . . . . . . 186
x
List of Code Samples
2.1 Summation Algorithm 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.2 Summation Algorithm 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.3 Summation Algorithm 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.4 Mode Finding Algorithm 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.5 Mode Finding Algorithm 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.6 Mode Finding Algorithm 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.7 Naive Exponentiation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
2.8 Computing an Average . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
9.1 Recursive Binomial Computation . . . . . . . . . . . . . . . . . . . . . . . 172
xi
List of Figures
2.1 Plot of two functions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.2 Expected number of comparisons for various success probabilities p. . . . 38
4.1 Two combination sequence examples . . . . . . . . . . . . . . . . . . . . . 54
4.2 Computation Tree for Satisfiability Backtracking Algorithm . . . . . . . . 60
4.3 A small Hamiltonian Graph . . . . . . . . . . . . . . . . . . . . . . . . . . 63
4.4 Brute Force Backtracing Hamiltonian Path Traversal . . . . . . . . . . . . 64
4.5 Example Knapsack Input with W = 8 . . . . . . . . . . . . . . . . . . . . 65
4.6 Knapsack Computation Tree for n = 4 . . . . . . . . . . . . . . . . . . . . 66
6.1 Visualization of a Linear Program with two sub-optimal solution lines
and the optimal one. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
6.2 Binary search tree and its inversion. . . . . . . . . . . . . . . . . . . . . . 96
7.1 A Binary Tree . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
7.2 A Binary Search Tree . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
7.3 Binary Search Tree Operations . . . . . . . . . . . . . . . . . . . . . . . 113
7.4 Balance Factor example on a Binary Search Tree . . . . . . . . . . . . . 115
7.5 Simple AVL L Rotation . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
7.6 Simple AVL R Rotation . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
7.7 Simple AVL LR Rotation . . . . . . . . . . . . . . . . . . . . . . . . . . 117
7.8 Simple AVL RL Rotation . . . . . . . . . . . . . . . . . . . . . . . . . . 117
7.9 Generalized AVL L Rotation . . . . . . . . . . . . . . . . . . . . . . . . 118
7.10 Generalized AVL L Rotation . . . . . . . . . . . . . . . . . . . . . . . . 118
7.11 Generalized AVL LR Rotation . . . . . . . . . . . . . . . . . . . . . . . 119
7.12 Generalized AVL RL Rotation . . . . . . . . . . . . . . . . . . . . . . . 119
7.13 AVL Tree Insertion Sequence . . . . . . . . . . . . . . . . . . . . . . . . 133
7.14 Worst-case Example of Rebalancing Following a Deletion. . . . . . . . . 134
7.15 2-3 Tree Insertion Sequence . . . . . . . . . . . . . . . . . . . . . . . . . 135
7.16 2-3 Tree Deletion Operations A . . . . . . . . . . . . . . . . . . . . . . . 136
7.17 2-3 Tree Deletion Operations B . . . . . . . . . . . . . . . . . . . . . . . 137
7.18 A min-heap . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138
7.19 Tree-based Heap Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . 138
7.20 Huffman Tree . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
8.1 A small graph. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
8.2 A larger graph. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146
xiii
List of Figures
8.3 DFS Forrest with initial vertex i. Dashed edges indicate back edges. . . 147
8.4 BFS Tree with initial vertex i. Dashed edges indicate cross edges. . . . . 150
8.5 BFS Tree with cross edges (dashed) involved in a cycle. . . . . . . . . . 152
8.6 A Directed Graph. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154
8.7 Condensation Graph . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154
8.8 Minimum Spanning Tree example. . . . . . . . . . . . . . . . . . . . . . 157
8.9 Illustration of Tree, Fringe, and Unseen vertex sets. . . . . . . . . . . . 159
8.10 Prim’s Algorithm after the second iteration. . . . . . . . . . . . . . . . . 159
8.11 Weighted directed graph. . . . . . . . . . . . . . . . . . . . . . . . . . . 161
8.12 Result of Dijsktra’s Algorithm with source vertex e. . . . . . . . . . . . 162
8.13 Basic idea behind Floyd-Warshal: Supposing that a path from vi vj
(k−1)
has already been found with a distance of di,j , the consideration of vk
(k−1) (k−1)
as a new intermediate node may have shorter distance, di,k + dk,j . . 164
8.14 Floyd’s Algorithm Demonstration. . . . . . . . . . . . . . . . . . . . . . 166
8.15 Another Demonstration of Floyd-Warshall’s Algorithm. . . . . . . . . . 167
9.1 Several valid Binary Search Trees . . . . . . . . . . . . . . . . . . . . . 175
9.2 Optimal Binary Search Tree split . . . . . . . . . . . . . . . . . . . . . . 176
9.3 Visualization of the OBST tableau. . . . . . . . . . . . . . . . . . . . . 177
9.4 Final Optimal Binary Search Tree. . . . . . . . . . . . . . . . . . . . . . 180
9.5 Dymamic Knapsack Backtracking . . . . . . . . . . . . . . . . . . . . . 183
10.1 A Finite State Automaton. . . . . . . . . . . . . . . . . . . . . . . . . . 196
10.2 Visualization of a Turing Machine . . . . . . . . . . . . . . . . . . . . . 198
10.3 Turing Machine Finite State Transitions . . . . . . . . . . . . . . . . . . 201
10.4 Reduction Visualization. . . . . . . . . . . . . . . . . . . . . . . . . . . 207
10.5 The P and NP landscape . . . . . . . . . . . . . . . . . . . . . . . . . . 208
10.6 Clique Reduction Visualization . . . . . . . . . . . . . . . . . . . . . . . 212
10.7 Reduction Idea for Clique/Vertex Cover. . . . . . . . . . . . . . . . . . 213
10.8 Graph and Complement: Clique and Vertex Cover. . . . . . . . . . . . . 214
10.9 Complexity Hierarchy . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215
10.10 Alternating and Augmenting Paths . . . . . . . . . . . . . . . . . . . . 215
xiv
1 Introduction
These lecture notes assume that you are already familiar with the following topics:
• Mastery over at least one high-level programming language
• Working knowledge of algorithm design and analysis
• Familiarity with design and analysis of recursive algorithms
• Working knowledge of Big-O notation and proof techniques involving asymptotics
• Familiarity with basic data structures such as lists, stacks, queues, binary search
trees
Nevertheless, this section serves as a high-level review of these topics.
1
2 Algorithm Analysis
2.1 Introduction
An algorithm is a procedure or description of a procedure for solving a problem. An
algorithm is a step-by-step specification of operations to be performed in order to compute
an output, process data, or perform a function. An algorithm must always be correct (it
must always produce a valid output) and it must be finite (it must terminate after a
finite number of steps).
Algorithms are not code. Programs and code in a particular language are implementations
of algorithms. The word, “algorithm” itself is derived from the latinization of Abū
‘Abdalāh Muhammad ibn Mūsā al-Khwārizmı̄, a Persian mathematician (c. 780 – 850).
The concept of algorithms predates modern computers by several thousands of years.
Euclid’s algorithm for computing the greatest common denominator (see Section 2.5.3)
is 2,300 years old.
Often, to be useful an algorithm must also be feasible: given its input, it must execute
in a reasonable amount of time using a reasonable amount of resources. Depending on
the application requirements our tolerance may be on the order of a few milliseconds
to several days. An algorithm that takes years or centuries to execute is certainly not
considered feasible.
Deterministic An algorithm is deterministic if, when given a particular input, will always
go through the exact same computational process and produce the same output.
Most of the algorithms you’ve used up to this point are deterministic.
Randomized An algorithm is randomized is an algorithm that involves some form of
random input. The random source can be used to make decisions such as random
selections or to generate random state in a program as candidate solutions. There
are many types of randomized algorithms including Monte-Carlo algorithms (that
may have some error with low probability), Las Vagas algorithms (whose results
are always correct, but may fail with a certain probability to produce any results),
etc.
Optimization Many algorithms seek not only to find a solution to a problem, but to
find the best, optimal solution. Many of these type of algorithms are heuristics:
rather than finding the actual best solution (which may be infeasible), they can
approximate a solution (Approximation algorithms). Other algorithms simulate
3
2 Algorithm Analysis
biological processes (Genetic algorithms, Ant Colony algorithms, etc.) to search
for an optimal solution.
Parallel Most modern processors are multicore, meaning that they have more than one
processor on a chip. Many servers have dozens of processors that work together.
Multiple processors can be utilized by designing parallel algorithms that can split
work across multiple processes or threads which can be executed in parallel to each
other, improving overall performance.
Distributed Computation can also be distributed among completely separate devices that
may be located half way across the globe. Massive distributed computation networks
have been built for research such as simulating protein folding (Folding@Home).
An algorithm is a more abstract, generalization of what you might be used to in a
typical programming language. In an actual program, you may have functions/methods,
subroutines or procedures, etc. Each one of these pieces of code could be considered an
algorithm in and of itself. The combination of these smaller pieces create more complex
algorithms, etc. A program is essentially a concrete implementation of a more general,
theoretical algorithm.
When a program executes, it expends some amount of resources. For example:
Time The most obvious resource an algorithm takes is time: how long the algorithm
takes to finish its computation (measured in seconds, minutes, etc.). Alternatively,
time can be measured in how many CPU cycles or floating-point operations a
particular piece of hardware takes to execute the algorithm.
Memory The second major resource in a computer is memory. An algorithm requires
memory to store the input, output, and possibly extra memory during its execution.
How much memory an algorithm uses in its execution may be even more of
an important consideration than time in certain environments or systems where
memory is extremely limited such as embedded systems.
Power The amount of power a device consumes is an important consideration when you
have limited capacity such as a battery in a mobile device. From a consumer’s
perspective, a slower phone that offered twice the batter life may be preferable. In
certain applications such as wireless sensor networks or autonomous systems power
may be more of a concern than either time or memory.
Bandwidth In computer networks, efficiency is measured by how much data you can
transmit from one computer to another, called throughput. Throughput is generally
limited by a network’s bandwidth: how much a network connection can transmit
under ideal circumstances (no data loss, no retransmission, etc.)
Circuitry When designing hardware, resources are typically measured in the number
of gates or wires are required to implement the hardware. Fewer gates and wires
means you can fit more chips on a silicon die which results in cheaper hardware.
Fewer wires and gates also means faster processing.
4
2.1 Introduction
Idleness Even when a computer isn’t computing anything, it can still be “costing” you
something. Consider purchasing hardware that runs a web server for a small user
base. There is a substantial investment in the hardware which requires maintenance
and eventually must be replaced. However, since the user base is small, most of
the time it sits idle, consuming power. A better solution may be to use the same
hardware to serve multiple virtual machines (VMs). Now several small web serves
can be served with the same hardware, increasing our utilization of the hardware.
In scenarios like this, the lack of work being performed is the resource.
Load Somewhat the opposite of idleness, sometimes an application or service may have
occasional periods of high demand. The ability of a system to service such high
loads may be considered a resource, even if the capacity to handle them goes unused
most of the time.
These are all very important engineering and business considerations when designing
systems, code, and algorithms. However, we’ll want to consider the complexity of
algorithms in a more abstract manner.
Suppose we have to different programs (or algorithms) A and B. Both of those algorithms
are correct, but A uses fewer of the above resources than B. Clearly, algorithm A is the
better, more efficient solution. However, how can we better quantify this efficiency?
List Operations
To give a concrete example, consider a typical list ADT. The list could be implemented
as an array-based list (where the class owns a static array that is resized/copied when
full) or a linked list (with nodes containing elements and linking to the next node in the
list). Some operations are “cheap” on one type of list while other operations may be
more “expensive.”
Consider the problem of inserting a new element into the list at the beginning (index 0).
For a linked list this involves creating a new node and shuffling a couple of references.
The number of operations in this case is not contingent on the size of the the list. In
contrast, for an array-based list, if the list contains n elements, each element will need to
be shifted over one position in the array in order to make room for the element to be
inserted. The number of shifts is proportional to the number of elements in the array, n.
Clearly for this operation, a linked list is better (more efficient).
Now consider a different operation: given an index i, retrieve the i-th element in the list.
For an array-based list we have the advantage of random access to the array. When we
index an element, arr[i], it only takes one memory address computation to “jump” to
the memory location containing the i-th element. In contrast here, a linked list would
require us to start at the head, and traverse the list until we reach the i-th node. This
requires i traversal operations. In the worst case, retrieving the last element, the n-th
element, would require n such operations. A summary of these operations can be found
5
2 Algorithm Analysis
List Type Insert at start Index-based Retrieve
Array-based List n 1
Linked List 2 i≈n
Table 2.1: Summary of the Complexity of List Operations
in Table 2.1.
Already we have a good understanding of the relative performance of algorithms based on
the type of data structure we choose. In this example we saw constant time algorithms
and linear algorithms. Constant time algorithms execute a constant number of operations
regardless of the size of the input (in this case, the size of the list n). Linear algorithms
perform a number of operations linearly related to the size of the input.
In the following examples, we’ll begin to be a little bit more formal about this type of
analysis.
2.1.1 Example: Computing a Sum
The following is a toy example, but its easy to understand and straightforward. Consider
the following problem: given an integer n ≥ 0, we want to compute the arithmetic series,
n
X
i = 1 + 2 + 3 + · · · + (n − 1) + n
i=1
As a naive approach, consider the algorithm in Code Sample 2.1. In this algorithm, we
iterate over each possible number i in the series. For each number i, we count 1 through
i and add one to a result variable.
1 int result = 0;
2 for(int i=1; i<=n; i++) {
3 for(int j=1; j<=i; j++) {
4 result = result + 1;
5 }
6 }
Code Sample 2.1: Summation Algorithm 1
As an improvement, consider the algorithm in Code Sample 2.2. Instead of just adding
one on each iteration of the inner loop, we omit the loop entirely and simply just add
the index variable i to the result.
Can we do even better? Yes. The arithmetic series actually has a closed-form solution
6
2.1 Introduction
1 int result = 0;
2 for(int i=1; i<=n; i++) {
3 result = result + i;
4 }
Code Sample 2.2: Summation Algorithm 2
Algorithm Number of Input Size
Additions
10 100 1,000 10,000 100,000 1,000,000
1 ≈ n2 0.003ms 0.088ms 1.562ms 2.097ms 102.846ms 9466.489ms
2 n 0.002ms 0.003ms 0.020ms 0.213ms 0.872ms 1.120ms
3 1 0.002ms 0.001ms 0.001ms 0.001ms 0.001ms 0.000ms
Table 2.2: Empirical Performance of the Three Summation Algorithms
(usually referred to as Gauss’s Formula):
n
X n(n + 1)
i=
i=1
2
Code Sample 2.3 uses this formula to directly compute the sum without any loops.
1 int result = n * (n + 1) / 2;
Code Sample 2.3: Summation Algorithm 3
All three of these algorithms were run on a laptop computer for various values of n from
10 up to 1,000,000. Table 2.2 contains the resulting run times (in milliseconds) for each
of these three algorithms on the various input sizes.
With small input sizes, there is almost no difference between the three algorithms.
However, that would be a naive way of analyzing them. We are more interested in how
each algorithm performs as the input size, n increases. In this case, as n gets larger, the
differences become very stark. The first algorithm has two nested for loops. On average,
the inner loop will run about n2 times while the outer loop runs n times. Since the loops
are nested, the inner loop executes about n2 times for each iteration of the outer loop.
Thus, the total number of iterations, and consequently the total number of additions is
about
n
n × ≈ n2
2
The second algorithm saves the inner for loop and thus only makes n additions. The
final algorithm only performs a constant number of operations.
Observe how the running time grows as the input size grows. For Algorithm 1, increasing
n from 100,000 to 1,000,000 (10 times as large) results in a running time that is about
100 times as slow. This is because it is performing n2 operations. To see this, consider
7
2 Algorithm Analysis
the following. Let t(n) be the time that Algorithm 1 takes for an input size of n. From
before we know that
t(n) ≈ n2
Observe what happens when we increase the input size from n to 10n:
t(10n) ≈ (10n)2 = 100n2
which is 100 times as large as t(n). The running time of Algorithm 1 will grow quadratically
with respect to the input size n.
Similarly, Algorithm 2 grows linearly,
t(n) ≈ n
Thus, a 10 fold increase in the input,
t(10n) ≈ 10n
leads to a 10 fold increase in the running time. Algorithm 3’s runtime does not depend
on the input size, and so its runtime does not grow as the input size grows. It essentially
remains flat–constant.
Of course, the numbers in Table 2.2 don’t follow this trend exactly, but they are pretty
close. The actual experiment involves a lot more variables than just the algorithms: the
laptop may have been performing other operations, the compiler and language may have
optimizations that change the algorithms, etc. Empirical results only provide general
evidence as to the runtime of an algorithm. If we moved the code to a different, faster
machine or used a different language, etc. we would get different numbers. However, the
general trends in the rate of growth would hold. Those rates of growth will be what we
want to analyze.
2.1.2 Example: Computing a Mode
As another example, consider the problem of computing the mode of a collection of
numbers. The mode is the most common element in a set of data.1
Consider the strategy as illustrated in Code Sample 2.4. For each element in the array,
we iterate through all the other elements and count how many times it appears (its
multiplicity). If we find a number that appears more times than the candidate mode
we’ve found so far, we update our variables and continue. As with the previous algorithm,
the nested nature of our loops leads to an algorithm that performs about n2 operations
(in this case, the comparison on line 9).
1
In general there may be more than one mode, for example in the set {10, 20, 10, 20, 50}, 10 and 20 are
both modes. The problem will simply focus on finding a mode, not all modes.
8
2.1 Introduction
1 public static int mode01(int arr[]) {
2
3 int maxCount = 0;
4 int modeIndex = 0;
5 for(int i=0; i<arr.length; i++) {
6 int count = 0;
7 int candidate = arr[i];
8 for(int j=0; j<arr.length; j++) {
9 if(arr[j] == candidate) {
10 count++;
11 }
12 }
13 if(count > maxCount) {
14 modeIndex = i;
15 maxCount = count;
16 }
17 }
18 return arr[modeIndex];
19 }
Code Sample 2.4: Mode Finding Algorithm 1
Now consider the following variation in Code Sample 2.5. In this algorithm, the first
thing we do is sort the array. This means that all equal elements will be contiguous. We
can exploit this to do less work. Rather than going through the list a second time for
each possible mode, we can count up contiguous runs of the same element. This means
that we need only examine each element exactly once, giving us n comparison operations
(line 8).
We can’t, however, ignore the fact that to exploit the ordering, we needed to first “invest”
some work upfront by sorting the array. Using a typical sorting algorithm, we would
expect that it would take about n log (n) comparisons. Since the sorting phase and mode
finding phase were separate, the total number of comparisons is about
n log (n) + n
The highest order term here is the n log (n) term for sorting. However, this is still lower
than the n2 algorithm. In this case, the investment to sort the array pays off! To compare
with our previous analysis, what happens when we increase the input size 10 fold? For
simplicity, let’s only consider the highest order term:
t(n) = n log (n)
Then
t(10n) = 10n log (10n) = 10n log (n) + 10 log (10)
9
2 Algorithm Analysis
1 public static int mode02(int arr[]) {
2 Arrays.sort(arr);
3 int i=0;
4 int modeIndex = 0;
5 int maxCount = 0;
6 while(i < arr.length-1) {
7 int count=0;
8 while(i < arr.length-1 && arr[i] == arr[i+1]) {
9 count++;
10 i++;
11 }
12 if(count > maxCount) {
13 modeIndex = i;
14 maxCount = count;
15 }
16 i++;
17 }
18 return arr[modeIndex];
19 }
Code Sample 2.5: Mode Finding Algorithm 2
The second term is a constant additive term. The increase in running time is essentially
linear! We cannot discount the additive term in general, but it is so close to linear that
terms like n log (n) are sometimes referred to as quasilinear.
Yet another solution, presented in Code Sample 2.6, utilizes a map data structure to
compute the mode. A map is a data structure that allows you to store key-value pairs.
In this case, we map elements in the array to a counter that represents the element’s
multiplicity. The algorithm works by iterating over the array and entering/updating the
elements and counters.
There is some cost associated with inserting and retrieving elements from the map,
but this particular implementation offers amortized constant running time for these
operations. That is, some particular entries/retrievals may be more expensive (say
linear), but when averaged over the life of the algorithm/data structure, each operation
only takes a constant amount of time.
Once built, we need only go through the elements in the map (at most n) and find the
one with the largest counter. This algorithm, too, offers essentially linear runtime for all
inputs. Similar experimental results can be found in Table 2.3.
The difference in performance is even more dramatic than in the previous example. For an
input size of 1,000,000 elements, the n2 algorithm took nearly 8 minutes! This is certainly
unacceptable performance for most applications. If we were to extend the experiment to
10
2.2 Pseudocode
1 public static int mode03(int arr[]) {
2 Map<Integer, Integer> counts = new HashMap<Integer, Integer>();
3 for(int i=0; i<arr.length; i++) {
4 Integer count = counts.get(arr[i]);
5 if(count == null) {
6 count = 0;
7 }
8 count++;
9 counts.put(arr[i], count);
10 }
11 int maxCount = 0;
12 int mode = 0;
13 for(Entry<Integer, Integer> e : counts.entrySet()) {
14 if(e.getValue() > maxCount) {
15 maxCount = e.getValue();
16 mode = e.getKey();
17 }
18 }
19 return mode;
20 }
Code Sample 2.6: Mode Finding Algorithm 3
Algorithm Number of Input Size
Additions
10 100 1,000 10,000 100,000 1,000,000
1 ≈ n2 0.007ms 0.155ms 11.982ms 45.619ms 3565.570ms 468086.566ms
2 n 0.143ms 0.521ms 2.304ms 19.588ms 40.038ms 735.351ms
3 n 0.040ms 0.135ms 0.703ms 10.386ms 21.593ms 121.273ms
Table 2.3: Empirical Performance of the Three Mode Finding Algorithms
n = 10,000,000, we would expect the running time to increase to about 13 hours! For
perspective, input sizes in the millions are small by today’s standards. Algorithms whose
runtime is quadratic are not considered feasible for today’s applications.
2.2 Pseudocode
We will want to analyze algorithms in an abstract, general way independent of any
particular hardware, framework, or programming language. In order to do this, we need
a way to specify algorithms that is also independent of any particular language. For that
purpose, we will use pseudocode.
11
2 Algorithm Analysis
Pseudocode (“fake” code) is similar to some programming languages that you’re familiar
with, but does not have any particular syntax rules. Instead, it is a higher-level description
of a process. You may use familiar control structures such as loops and conditionals, but
you can also utilize natural language descriptions of operations.
There are no established rules for pseudocode, but in general, good pseudocode:
• Clearly labels the algorithm
• Identifies the input and output at the top of the algorithm
• Does not involve any language or framework-specific syntax–no semicolons, decla-
ration of variables or their types, etc.
• Makes liberal use of mathematical notation and natural language for clarity
Good pseudocode abstracts the algorithm by giving enough details necessary to under-
stand the algorithm and subsequently implement it in an actual programming language.
Let’s look at some examples.
Input : A collection of numbers, A = {a1 , . . . , an }
Output : The mean, µ of the values in A
1 sum ← 0
2 foreach ai ∈ A do
3 sum ← sum + ai
4 end
5 µ ← sum
n
6 output µ
Algorithm 1: Computing the Mean
Algorithm 1 describes a way to compute the average of a collection of numbers. Observe:
• The input does not have a specific type (such as int or double), it uses set notation
which also indicates how large the collection is.
• There is no language-specific syntax such as semicolons, variable declarations, etc.
• The loop construct doesn’t specify the details of incrementing a variable, instead
using a “foreach” statement with some set notation2
• Code blocks are not denoted by curly brackets, but are clearly delineated by using
indentation and vertical lines.
• Assignment and compound assignment operators do not use the usual syntax from
C-style languages, instead using a left-oriented arrow to indicate a value is assigned
2
To review, ai ∈ A is a predicate meaning the element ai is in the set A.
12
2.2 Pseudocode
to a variable.3
Consider another example of computing the mode, similar to the second approach in a
previous example.
Input : A collection of numbers, A = {a1 , . . . , an }
Output : A mode of A
1 Sort the elements in A in non-decreasing order
2 multiplicity ← −∞
3 foreach run of contiguous equal elements a do
4 m ← count up the number of times a appears
5 if m > multiplicity then
6 mode ← a
7 multiplicity ← m
8 end
9 end
10 output m
Algorithm 2: Computing the Mode
Some more observations about Algorithm 2:
• The use of natural language to specify that the collection should be sorted and how
• The usage of −∞ as a placeholder so that any other value would be greater than it
• The use of natural language to specify that an iteration takes place over contiguous
elements (line 3) or that a sub-operation such as a count/summation (line 4) is
performed
In contrast, bad pseudocode would be have the opposite elements. Writing a full program
or code snippet in Java for example. Bad pseudocode may be unclear or it may overly
simplify the process to the point that the description is trivial. For example, suppose we
wanted to specify a sorting algorithm, and we did so using the pseudocode in Algorithm
3. This trivializes the process. There are many possible sorting algorithms (insertion
sort, quick sort, etc.) but this algorithm doesn’t specify any details for how to go about
sorting it.
On the other hand, in Algorithm 2, we did essentially do this. In that case it was perfectly
fine: sorting was a side operation that could be achieved by a separate algorithm. The
point of the algorithm was not to specify how to sort, but instead how sorting could be
used to solve another problem, finding the mode.
3
Not all languages use the familiar single equals sign = for the assignment operator. The statistical
programming language R uses the left-arrow operator, <- and Maple uses := for example.
13
2 Algorithm Analysis
Input : A collection of numbers, A = {a1 , . . . , an }
Output : A0 , sorted in non-decreasing order
1 A0 ← Sort the elements in A in non-decreasing order
2 output A0
Algorithm 3: Trivial Sorting (Bad Pseudocode)
Another example would be if we need to find a minimal element in a collection. Trivial
pseudocode may be like that found in Algorithm 4. No details are presented on how to
find the element. However, if finding the minimal element were an operation used in
a larger algorithm (such as selection sort), then this terseness is perfectly fine. If the
primary purpose of the algorithm is to find the minimal element, then details must be
presented as in Algorithm 5.
Input : A collection of numbers, A = {a1 , . . . , an }
Output : The minimal element of A
1 m ← minimal element of A
2 output m
Algorithm 4: Trivially Finding the Minimal Element
Input : A collection of numbers, A = {a1 , . . . , an }
Output : The minimal element of A
1 m←∞
2 foreach ai ∈ A do
3 if ai < m then
4 m ← ai
5 end
6 end
7 output m
Algorithm 5: Finding the Minimal Element
2.3 Analysis
Given two competing algorithms, we could empirically analyze them like we did in
previous examples. However, it may be infeasible to implement both just to determine
14
2.3 Analysis
which is better. Moreover, by analyzing them from a more abstract, theoretical approach,
we have a better more mathematically-based proof of the relative complexity of two
algorithm.
Given an algorithm, we can analyze it by following this step-by-step process.
1. Identify the input
2. Identify the input size, n
3. Identify the elementary operation
4. Analyze how many times the elementary operation is executed with respect to the
input size n
5. Characterize the algorithm’s complexity by providing an asymptotic (Big-O, or
Theta) analysis
Identifying the Input
This step is pretty straightforward. If the algorithm is described with good pseudocode,
then the input will already be identified. Common types of inputs are single numbers,
collections of elements (lists, arrays, sets, etc.), data structures such as graphs, matrices,
etc.
However, there may be some algorithms that have multiple inputs: two numbers or a
collection and a key, etc. In such cases, it simplifies the process if you can, without loss
of generality, restrict attention to a single input value, usually the one that has the most
relevance to the elementary operation you choose.
Identifying the Input Size
Once the input has been identified, we need to identify its size. We’ll eventually want to
characterize the algorithm as a function f (n): given an input size, how many resources
does it take. Thus, it is important to identify the number corresponding to the domain
of this function.
This step is also pretty straightforward, but may be dependent on the type of input or
even its representation. Examples:
• For collections (sets, lists, arrays), the most natural is to use the number of elements
in the collection (cardinality, size, etc.). The size of individual elements is not as
important as number of elements since the size of the collection is likely to grow
more than individual elements do.
• An n × m matrix input could be measured by one or both nm of its dimensions.
• For graphs, you could count either the number of vertices or the number of edges
15
2 Algorithm Analysis
in the graph (or both!). How the graph is represented may also affect its input size
(an adjacency matrix vs. an adjacency list).
• If the input is a number x, the input size is typically the number of bits required
to represent x. That is,
n = dlog2 (x)e
To see why, recall that if you have n bits, the maximum number you can represent
is 2n − 1. Inverting this expression gives us dlog2 (x)e.
Some algorithms may have multiple inputs. For example, a collection and a number (for
searching) or two integers as in Euclid’s algorithm. The general approach to analyzing
such algorithms to simplify things by only considering one input. If one of the inputs is
larger, such as a collection vs. a single element, the larger one is used in the analysis.
Even if it is not clear which one is larger, it may be possible to assume, without loss of
generality, that one is larger than the other (and if not, the inputs may be switched).
The input size can then be limited to one variable to simplify the analysis.
Identifying the Elementary Operation
We also need to identify what part of the algorithm does the actual work (where the most
resources will be expended). Again, we want to keep the analysis simple, so we generally
only identify one elementary operation. There may be several reasonable candidates
for the elementary operation, but in general it should be the most common or most
expensive operation performed in the algorithm. For example:
• When performing numeric computations, arithmetic operations such as additions,
divisions, etc.
• When sorting or searching, comparisons are the most natural elementary operations.
Swaps may also be a reasonable choice depending on how you want to analyze the
algorithm.
• When traversing a data structure such as a linked list, tree, or graph a node traversal
(visiting or processing a node) may be considered the elementary operation.
In general, operations that are necessary to control structures (such as loops, assignment
operators, etc.) are not considered good candidates for the elementary operation. An
extended discussion of this can be found in Section 2.6.2.
Analysis
Once the elementary operation has been identified, the algorithm must be analyzed to
count the number of times it is executed with respect to the input size. That is, we
analyze the algorithm to find a function f (n) where n is the input size and f (n) gives
the number of times the elementary operation is executed.
16
2.3 Analysis
The analysis may involve deriving and solving a summation. For example, if the
elementary operation is performed within a for loop and the loop runs a number of times
that depends on the input size n.
If there are multiple loops in which the elementary operation is performed, it may be
necessary to setup multiple summations. If two loops are separate and independent (one
executes after the other), then the sum rule applies. The total number of operations is
the sum of the operations of each loop.
If two loops are nested, then the product rule applies. The inner loop will execute fully
for each iteration of the outer loop. Thus, the number of operations are multiplied with
each other.
Sometimes the analysis will not be so clear cut. For example, a while loop may execute
until some condition is satisfied that does not directly depend on the input size but also
on the nature of the input. In such cases, we can simplify our analysis by considering
the worst-case scenario. In the while loop, what is the maximum possible number of
iterations for any input?
Asymptotic Characterization
As computers get faster and faster and resources become cheaper, they can process more
and more information in the same amount of time. However, the characterization of
an algorithm should be invariant with respect to the underlying hardware. If we run
an algorithm on a machine that is twice as fast, that doesn’t mean that the algorithm
has improved. It still takes the same number of operations to execute. Faster hardware
simply means that the time it takes to execute those operations is half as much as it was
before.
To put it in another perspective, performing Euclid’s algorithm to find the GCD of two
integers took the same number of steps 2,300 years ago when he performed them on
paper as it does today when they are executed on a digital computer. A computer is
obviously faster than Euclid would have been, but both Euclid and the computer are
performing the same number of steps when executing the same algorithm.
For this reason, we characterize the number of operations performed by an algorithm
using asymptotic analysis. Improving the hardware by a factor of two only affects the
“hidden constant” sitting outside of the function produced by the analysis in the previous
step. We want our characterization to be invariant of those constants.
Moreover, we are really more interested in how our algorithm performs for larger and
larger input sizes. To illustrate, suppose that we have two algorithms, one that performs
f (n) = 100n2 + 5n
operations and one that performs
g(n) = n3
17
2 Algorithm Analysis
·106
1.5
1
f (n) = 100n2 + 5n
0.5
g(n) = n3
0
0 20 40 60 80 100 120
n
Figure 2.1: Plot of two functions.
operations. These functions are graphed in Figure 2.1. For inputs of size less than 100,
the first algorithm performs worse than the second (the graph is higher indicating “more”
resources). However, for inputs of size greater than 100, the first algorithm is better. For
small inputs, the second algorithm may be better, but small inputs are not the norm
for any “real” problems.4 In any case, on modern computers, we would expect small
inputs to execute fast anyway as they did in our empirical experiments in Section 2.1.1
and 2.1.2. There was essentially no discernible difference in the three algorithms for
sufficiently small inputs.
We can rigorously quantify this by providing an asymptotic characterization of these
functions. An asymptotic characterization essentially characterizes the rate of growth of
a function or the relative rate of growth of functions. In this case, n3 grows much faster
than 100n2 + 5n as n grows (tends toward infinity). We formally define these concepts
in the next section.
2.4 Asymptotics
2.4.1 Big-O Analysis
We want to capture the notion that one function grows faster than (or at least as fast as)
another. Categorizing functions according to their growth rate has been done for a long
4
There are problems where we can apply a “hybrid” approach: we can check for the input size and
choose one algorithm for small inputs and another for larger inputs. This is typically done in hybrid
sorting algorithms such as when merge sort is performed for “large” inputs but switches over to
insertion sort for smaller arrays.
18
2.4 Asymptotics
time in mathematics using big-O notation.5
Definition 1. Let f and g be two functions, f, g : N → R+ . We say that
f (n) ∈ O(g(n))
read as “f is big-O of g,” if there exist constants c ∈ R+ and n0 ∈ N such that for every
integer n ≥ n0 ,
f (n) ≤ cg(n)
First, let’s make some observations about this definition.
• The “O” originally stood for “order of”, Donald Knuth referred to it as the capital
greek letter omicron, but since it is indistinguishable from the Latin letter “O” it
makes little difference.
• Some definitions are more general about the nature of the functions f, g. However,
since we’re looking at these functions as characterizing the resources that an
algorithm takes to execute, we’ve restricted the domain and codomain of the
functions. The domain is restricted to non-negative integers since there is little
sense in negative or factional input sizes. The codomain is restricted to nonnegative
reals as it doesn’t make sense that an algorithm would potentially consume a
negative amount of resources.
• We’ve used the set notation f (n) ∈ O(g(n)) because, strictly speaking, O(g(n)) is
a class of functions: the set of all functions that are asymptotically bounded by
g(n). Thus the set notation is the most appropriate. However, you will find many
sources and papers using notation similar to
f (n) = O(g(n))
This is a slight abuse of notation, but common nonetheless.
The intuition behind the definition of big-O is that f is asymptotically less than or equal
to g. That is, the rate of growth of g is at least as fast as the growth rate of f . Big-O
provides a means to express that one function is an asymptotic upper bound to another
function.
The definition essentially states that f (n) ∈ O(g(n)) if, after some point (for all n ≥ n0 ),
the value of the function g(n) will always be larger than f (n). The constant c possibly
serves to “stretch” or “compress” the function, but has no effect on the growth rate of
the function.
5
The original notation and definition are attributed to Paul Bachmann in 1894 [4]. Definitions and
notation have been refined and introduced/reintroduced over the years. Their use in algorithm
analysis was first suggested by Donald Knuth in 1976 [10].
19
2 Algorithm Analysis
Example
Let’s revisit the example from before where f (n) = 100n2 + 5n and g(n) = n3 . We want
to show that f (n) ∈ O(g(n)). By the definition, we need to show that there exists a c
and n0 such that
f (n) ≤ cg(n)
As we observed in the graph in Figure 2.1, the functions “crossed over” somewhere
around n = 100. Let’s be more precise about that. The two functions cross over when
they are equal, so we setup an equality,
100n2 + 5n = n3
Collecting terms and factoring out an n (that is, the functions have one crossover point
at n = 0), we have
n2 − 100n − 5 = 0
The values of n satisfying this inequality can be found by applying the quadratic formula,
and so √
100 ± 10000 + 20
n=
2
Which is −0.049975 . . . and 100.0499 . . .. The first root is negative and so irrelevant.
The second is our cross over point. The next largest integer is 101. Thus, for c = 1 and
n0 = 101, the inequality is satisfied.
In this example, it was easy to find the intersection because we could employ the quadratic
equation to find roots. This is much more difficult with higher degree polynomials. Throw
in some logarithmic functions, exponential functions, etc. and this approach can be
difficult.
Revisit the definition of big-O: the inequality doesn’t have to be tight or precise. In the
previous example we essentially fixed c and tried to find n0 such that the inequality held.
Alternatively, we could fix n0 to be small and then find the c (essentially compressing
the function) such that the inequality holds. Observe:
100n2 + 5n ≤ 100n2 + 5n2 since n ≤ n2 for all n ≥ 0
= 105n2
≤ 105n3 since n2 ≤ n3 for all n ≥ 0
= 105g(n)
By adding positive values, we make the equation larger until it looks like what we want,
in this case g(n) = n3 . By the end we’ve got our constants: for c = 105 and n0 = 0, the
inequality holds. There is nothing special about this c, c = 1000000 would work too.
The point is we need only find at least one c, n0 pair that the inequality holds (there are
an infinite number of possibilities).
20
2.4 Asymptotics
2.4.2 Other Notations
Big-O provides an asymptotic upper bound characterization of two functions. There are
several other notations that provide similar characterizations.
Big-Omega
Definition 2. Let f and g be two functions, f, g : N → R+ . We say that
f (n) ∈ Ω(g(n))
read as “f is big-Omega of g,” if there exist constants c ∈ R+ and n0 ∈ N such that for
every integer n ≥ n0 ,
f (n) ≥ cg(n)
Big-Omega provides an asymptotic lower bound on a function. The only difference is the
inequality has been reversed. Intuitively f has a growth rate that is bounded below by g.
Big-Theta
Yet another characterization can be used to show that two functions have the same order
of growth.
Definition 3. Let f and g be two functions f, g : N → R+ . We say that
f (n) ∈ Θ(g(n))
read as “f is Big-Theta of g,” if there exist constants c1 , c2 ∈ R+ and n0 ∈ N such that
for every integer n ≥ n0 ,
c1 g(n) ≤ f (n) ≤ c2 g(n)
Big-Θ essentially provides an asymptotic equivalence between two functions. The function
f is bounded above and below by g. As such, both functions have the same rate of
growth.
Soft-O Notation
Logarithmic factors contribute very little to a function’s rate of growth especially com-
pared to larger order terms. For example, we called n log (n) quasi linear since it was
nearly linear. Soft-O notation allows us to simplify terms by removing logarithmic factors.
Definition 4. Let f, g be functions such that f (n) ∈ O(g(n) · logk (n)). Then we say
that f (n) is soft-O of g(n) and write
f (n) ∈ Õ(g(n))
21
2 Algorithm Analysis
For example,
n log (n) ∈ Õ(n)
Little Asymptotics
Related to big-O and big-Ω are their corresponding “little” asymptotic notations, little-o
and little-ω.
Definition 5. Let f and g be two functions f, g : N → R+ . We say that
f (n) ∈ o(g(n))
read as “f is little-o of g,” if
f (n)
lim =0
n→∞ g(n)
The little-o is sometimes defined as for every > 0 there exists a constant N such that
|f (n)| ≤ |g(n)| ∀n ≥ N
but given the restriction that g(n) is positive, the two definitions are essentially equivalent.
Little-o is a much stronger characterization of the relation of two functions. If f (n) ∈
o(g(n)) then not only is g an asymptotic upper bound on f , but they are not asymptoti-
cally equivalent. Intuitively, this is similar to the difference between saying that a ≤ b
and a < b. The second is a stronger statement as it implies the first, but the first does
not imply the second. Analogous to this example, little-o provides a “strict” asymptotic
upper bound. The growth rate of g is strictly greater than the growth rate of f .
Similarly, a little-ω notation can be used to provide a strict lower bound characterization.
Definition 6. Let f and g be two functions f, g : N → R+ . We say that
f (n) ∈ ω(g(n))
read as “f is little-omega of g,” if
f (n)
lim =∞
n→∞ g(n)
2.4.3 Observations
As you might have surmised, big-O and big-Ω are duals of each other, thus we have the
following.
Lemma 1. Let f, g be functions. Then
f (n) ∈ O(g(n)) ⇐⇒ g(n) ∈ Ω(f (n))
22
2.4 Asymptotics
Because big-Θ provides an asymptotic equivalence, both functions are big-O and big-Θ
of each other.
Lemma 2. Let f, g be functions. Then
f (n) ∈ Θ(g(n)) ⇐⇒ f (n) ∈ O(g(n)) and f (n) ∈ Ω(g(n))
Equivalently,
f (n) ∈ Θ(g(n)) ⇐⇒ g(n) ∈ O(f (n)) and g(n) ∈ Ω(f (n))
With respect to the relationship between little-o and little-ω to big-O and big-Ω, as
previously mentioned, little asymptotics provide a stronger characterization of the growth
rate of functions. We have the following as a consequence.
Lemma 3. Let f, g be functions. Then
f (n) ∈ o(g(n)) ⇒ f (n) ∈ O(g(n))
and
f (n) ∈ ω(g(n)) ⇒ f (n) ∈ Ω(g(n))
Of course, the converses of these statements do not hold.
Common Identities
As a direct consequence of the definition, constant coefficients in a function can be
ignored.
Lemma 4. For any constant c,
c · f (n) ∈ O(f (n))
In particular, for c = 1, we have that
f (n) ∈ O(f (n))
and so any function is an upper bound on itself.
In addition, when considering the sum of two functions, f1 (n), f2 (n), it suffices to consider
the one with a larger rate of growth.
Lemma 5. Let f1 (n), f2 (n) be functions such that f1 (n) ∈ O(f2 (n)). Then
f1 (n) + f2 (n) ∈ O(f2 (n))
23
2 Algorithm Analysis
In particular, when analyzing algorithms with independent operations (say, loops), we
only need to consider the operation with a higher complexity. For example, when we
presorted an array to compute the mode, the presort phase was O(n log (n)) and the
mode finding phase was O(n). Thus the total complexity was
n log (n) + n ∈ O(n log (n))
When dealing with a polynomial of degree k,
ck nk + ck−1 nk−1 + ck−2 nk−2 + · · · + c1 n + c0
The previous results can be combined to conclude the following lemma.
Lemma 6. Let p(n) be a polynomial of degree k,
p(n) = ck nk + ck−1 nk−1 + ck−2 nk−2 + · · · + c1 n + c0
then
p(n) ∈ Θ(nk )
Logarithms
When working with logarithmic functions, it suffices to consider a single base. As
Computer Scientists, we always work in base-2 (binary). Thus when we write log (n), we
implicitly mean log2 (n) (base-2). It doesn’t really matter though because all logarithms
are the same to within a constant as a consequence of the change of base formula:
loga (n)
logb (n) =
loga (b)
That means that for any valid bases a, b,
logb (n) ∈ Θ(loga (n))
Another way of looking at it is that an algorithm’s complexity is the same regardless of
whether or not it is performed by hand in base-10 numbers or on a computer in binary.
Other logarithmic identities that you may find useful remembering include the following:
log (nk ) = k log (n)
log (n1 n2 ) = log (n1 ) + log (n2 )
Classes of Functions
Table 2.4 summarizes some of the complexity functions that are common when doing
algorithm analysis. Note that these classes of functions form a hierarchy. For example,
linear and quasilinear functions are also O(nk ) and so are polynomial.
24
2.4 Asymptotics
Class Name Asymptotic Characterization Algorithm Examples
Constant O(1) Evaluating a formula
Logarithmic O(log (n)) Binary Search
Polylogarithmic O(logk (n))
Linear O(n) Linear Search
Quasilinear O(n log (n)) Mergesort
Quadratic O(n2 ) Insertion Sort
Cubic O(n3 )
Polynomial O(nk ) for any k > 0
Exponential O(2n ) Computing a powerset
Super-Exponential O(2f (n) ) for f (n) ∈ Ω(n) Computing permutations
For example, n!
Table 2.4: Common Algorithmic Efficiency Classes
2.4.4 Limit Method
The method used in previous examples directly used the definition to find constants c, n0
that satisfied an inequality to show that one function was big-O of another. This can get
quite tedious when there are many terms involved. A much more elegant proof technique
borrows concepts from calculus.
Let f (n), g(n) be functions. Suppose we examine the limit, as n → ∞ of the ratio of
these two functions.
f (n)
lim
n→∞ g(n)
One of three things could happen with this limit.
The limit could converge to 0. If this happens, then by Definition 5 we have that
f (n) ∈ o(g(n)) and so by Lemma 3 we know that f (n) ∈ O(g(n)). This makes sense: if
the limit converges to zero that means that g(n) is growing much faster than f (n) and
so f is big-O of g.
The limit could diverge to infinity. If this happens, then by Definition 6 we have that
f (n) ∈ ω(g(n)) and so again by Lemma 3 we have f (n) ∈ Ω(g(n)). This also makes
sense: if the limit diverges, f (n) is growing much faster than g(n) and so f (n) is big-Ω
of g.
Finally, the limit could converge to some positive constant (recall that both functions are
restricted to positive codomains). This means that both functions have essentially the
same order of growth. That is, f (n) ∈ Θ(g(n). As a consequence, we have the following
Theorem.
25
2 Algorithm Analysis
Theorem 1 (Limit Method). Let f (n) and g(n) be functions. Then if
0 then f (n) ∈ O(g(n))
f (n)
lim = c > 0 then f (n) ∈ Θ(g(n))
n→∞ g(n)
∞ then f (n) ∈ Ω(g(n))
Examples
Let’s reuse the example from before where f (n) = 100n2 + 5n and g(n) = n3 . Setting up
our limit,
f (n) 100n2 + 5n
lim = lim
n→∞ g(n) n→∞ n3
100n + 5
= lim
n→∞ n2
100n 5
= lim 2
+ lim 2
n→∞ n n→∞ n
100
= lim +0
n→∞ n
=0
And so by Theorem 1, we conclude that
f (n) ∈ O(g(n))
Consider the following example: let f (n) = log2 n and g(n) = log3 (n2 ). Setting up our
limit we have
f (n) log2 n
lim =
n→∞ g(n) log3 n2
log n
= 2 log2 n
2
log2 3
log2 3
=
2
= .7924 . . . > 0
And so we conclude that log2 (n) ∈ Θ(log3 (n2 )).
As another example, let f (n) = log (n) and g(n) = n. Setting up the limit gives us
log (n)
lim
n→∞ n
The rate of growth might seem obvious here, but we still need to be mathematically
rigorous. Both the denominator and numerator are monotone increasing functions. To
solve this problem, we can apply l’Hôpital’s Rule:
26
2.4 Asymptotics
Theorem 2 (l’Hôpital’s Rule). Let f and g be functions. If the limit of the quotient fg(n)
(n)
exists, it is equal to the limit of the derivative of the denominator and the numerator.
That is,
f (n) f 0 (n)
lim = lim 0
n→∞ g(n) n→∞ g (n)
Applying this to our limit, the denominator drops out, but what about the numerator?
Recall that log (n) is the logarithm base-2. The derivative of the natural logarithm is well
known, ln0 (n) = n1 . We can use the change of base formula to transform log (n) = ln (n)
ln (2)
and then take the derivative. That is,
1
log0 (n) =
ln (2)n
Thus,
log (n) log0 (n)
lim = lim
n→∞ n n→∞ n0
1
= lim
n→∞ ln (2)n
=0
Concluding that log (n) ∈ O(n).
Pitfalls
l’Hôpital’s Rule is not always the most appropriate tool to use. Consider the following
example: let f (n) = 2n and g(n) = 3n . Setting up our limit and applying l’Hôpital’s
Rule we have
2n (2n )0
lim = lim n 0
n→∞ 3n n→∞ (3 )
(ln 2)2n
= lim
n→∞ (ln 3)3n
which doesn’t get us anywhere. In general, we should look for algebraic simplifications
first. Doing so we would have realized that
n
2n 2
lim n = lim
n→∞ 3 n→∞ 3
2
Since 3
< 1, the limit of its exponent converges to zero and we have that 2n ∈ O(3n ).
27
2 Algorithm Analysis
2.5 Examples
2.5.1 Linear Search
As a simple example, consider the problem of searching a collection for a particular
element. The straightforward solution is known as Linear Search and is featured as
Algorithm 6
Input : A collection A = {a1 , . . . , an }, a key k
Output : The first i such that ai = k, φ otherwise
1 for i = 1, . . . , n do
2 if ai = k then
3 output i
4 end
5 end
6 output φ
Algorithm 6: Linear Search
Let’s follow the prescribed outline above to analyze Linear Search.
1. Input: this is clearly indicated in the pseudocode of the algorithm. The input is
the collection A.
2. Input Size: the most natural measure of the size of a collection is its cardinality; in
this case, n
3. Elementary Operation: the most common operation is the comparison in line 2
(assignments and iterations necessary for the control flow of the algorithm are not
good candidates).
4. How many times is the elementary operation executed with respect to the input
size, n? The situation here actually depends not only on n but also the contents of
the array.
• Suppose we get lucky and find k in the first element, a1 : we’ve only made one
comparison.
• Suppose we are unlucky and find it as the last element (or don’t find it at all).
In this case we’ve made n comparisons
• We could also look at the average number of comparisons (see Section 2.6.3)
In general, algorithm analysis considers at the worst case scenario unless otherwise
stated. In this case, there are C(n) = n comparisons.
28
2.5 Examples
5. This is clearly linear, which is why the algorithm is called Linear Search,
Θ(n)
2.5.2 Set Operation: Symmetric Difference
Recall that the symmetric difference of two sets, A ⊕ B consists of all elements in A or
B but not both. Algorithm 7 computes the symmetric difference.
Input : Two sets, A = {a1 , . . . , an }, B = {b1 , . . . , bm }
Output : The symmetric difference, A ⊕ B
1 C←∅
2 foreach ai ∈ A do
3 if ai 6∈ B then
4 C ← C ∪ {ai }
5 end
6 end
7 foreach bj ∈ B do
8 if bj 6∈ A then
9 C ← C ∪ {bj }
10 end
11 end
12 output C
Algorithm 7: Symmetric Difference of Two Sets
Again, following the step-by-step process for analyzing this algorithm,
1. Input: In this case, there are two sets as part of the input, A, B.
2. Input Size: As specified, each set has cardinality n, m respectively. We could
analyze the algorithm with respect to both input sizes, namely the input size could
be n + m. For simplicity, to work with a single variable, we could also define
N = n + m.
Alternatively, we could make the following observation: without loss of generality,
we can assume that n ≥ m (if not, switch the sets). If one input parameter is
bounded by the other, then
n + m ≤ 2n ∈ O(n)
That is, we could simplify the analysis by only considering n as the input size.
There will be no difference in the final asymptotic characterization as the constants
will be ignored.
29
2 Algorithm Analysis
3. Elementary Operation: In this algorithm, the most common operation is the
set membership query (6∈). Strictly speaking, this operation may not be trivial
depending on the type of data structure used to represent the set (it may entail a
series of O(n) comparisons for example). However, as our pseudocode is concerned,
it is sufficient to consider it as our elementary operation.
4. How many times is the elementary operation executed with respect to the input
size, n? In the first for-loop (lines 2–6) the membership query is performed n times.
In the second loop (lines 7–11), it is again performed m times. Since each of these
loops is independent of each other, we would add these operations together to get
n+m
total membership query operations.
5. Whether or not we consider N = n + m or n to be our input size, the algorithm is
clearly linear with respect to the input size. Thus it is a Θ(n)-time algorithm.
2.5.3 Euclid’s GCD Algorithm
The greatest common divisor (or GCD) of two integers a, b is the largest positive integer
that divides both a and b. Finding a GCD has many useful applications and the problem
has one of the oldest known algorithmic solutions: Euclid’s Algorithm (due to the Greek
mathematician Euclid c. 300 BCE).
Input : Two integers, a, b
Output : The greatest common divisor, gcd(a, b)
1 while b 6= 0 do
2 t←b
3 b ← a mod b
4 a←t
5 end
6 Output a
Algorithm 8: Euclid’s GCD Algorithm
The algorithm relies on the following observation: any number that divides a, b must
also divide the remainder of a since we can write b = a · k + r where r is the remainder.
This suggests the following strategy: iteratively divide a by b and retain the remainder
r, then consider the GCD of b and r. Progress is made by observing that b and r are
necessarily smaller than a, b. Repeating this process until we have a remainder of zero
gives us the GCD because once we have that one evenly divides the other, the larger
must be the GCD. Pseudocode for Euclid’s Algorithm is provided in Algorithm 8.
30
2.5 Examples
The analysis of Euclid’s algorithm is seemingly straightforward. It is easy to identify the
division in line 3 as the elementary operation. But how many times is it executed with
respect to the input size? What is the input size?
When considering algorithms that primarily execute numerical operations the input is
usually a number (or in this case a pair of numbers). How big is the input of a number?
The input size of 12,142 is not 12,142. The number 12,142 has a compact representation
when we write it: it requires 5 digits to express it (in base 10). That is, the input size of
a number is the number of symbols required to represent its magnitude. Considering a
number’s input size to be equal to the number would be like considering its representation
in unary where a single symbol is used and repeated for as many times as is equal to the
number (like a prisoner marking off the days of his sentence).
Computers don’t “speak” in base-10, they speak in binary, base-2. Therefore, the input
size of a numerical input is the number of bits required to represent the number. This is
easily expressed using the base-2 logarithm function:
dlog (n)e
But in the end it doesn’t really matter if we think of computers as speaking in base-10,
base-2, or any other integer base greater than or equal to 2 because as we’ve observed that
all logarithms are equivalent to within a constant factor using the change of base formula.
In fact this again demonstrates again that algorithms are an abstraction independent
of any particular platform: that the same algorithm will have the same (asymptotic)
performance whether it is performed on paper in base-10 or in a computer using binary!
Back to the analysis of Euclid’s Algorithm: how many times does the while loop get
executed? We can first observe that each iteration reduces the value of b, but by how
much? The exact number depends on the input: some inputs would only require a single
division, other inputs reduce b by a different amount on each iteration. The important
thing to to realize is that we want a general characterization of this algorithm: it suffices
to consider the worst case. That is, at maximum, how many iterations are performed?
The number of iterations is maximized when the reduction in the value of b is minimized
at each iteration. We further observe that b is reduced by at least half at each iteration.
Thus, the number of iterations is maximized if we reduce b by at most half on each
iteration. So how many iterations i are required to reduce n down to 1 (ignoring the
last iteration when it is reduced to zero for the moment)? This can be expressed by the
equation: i
1
n =1
2
Solving for i (taking the log on either side), we get that
i = log (n)
Recall that the size of the input is log (n). Thus, Euclid’s Algorithm is linear with respect
to the input size.
31
2 Algorithm Analysis
2.5.4 Selection Sort
Recall that Selection Sort is a sorting algorithm that sorts a collection of elements by
first finding the smallest element and placing it at the beginning of the collection. It
continues by finding the smallest among the remaining n − 1 and placing it second in
the collection. It repeats until the “first” n − 1 elements are sorted, which by definition
means that the last element is where it needs to be.
The Pseudocode is presented as Algorithm 9.
Input : A collection A = {a1 , . . . , an }
Output : A sorted in non-decreasing order
1 for i = 1, . . . , n − 1 do
2 min ← ai
3 for j = (i + 1), . . . , n do
4 if min < aj then
5 min ← aj
6 end
7 swap min, ai
8 end
9 end
10 output A
Algorithm 9: Selection Sort
Let’s follow the prescribed outline above to analyze Selection Sort.
1. Input: this is clearly indicated in the pseudocode of the algorithm. The input is
the collection A.
2. Input Size: the most natural measure of the size of a collection is its cardinality; in
this case, n
3. Elementary Operation: the most common operation is the comparison in line 4
(assignments and iterations necessary for the control flow of the algorithm are not
good candidates). Alternatively, we could have considered swaps on line 7 which
would lead to a different characterization of the algorithm.
4. How many times is the elementary operation executed with respect to the input
size, n?
• Line 4 does one comparison each time it is executed
• Line 4 itself is executed multiple times for each iteration of the for loop in line
3 (for j running from i + 1 up to n inclusive.
32
2.6 Other Considerations
• line 3 (and subsequent blocks of code) are executed multiple times for each
iteration of the for loop in line 1 (for i running from 1 up to n − 1
This gives us the following summation:
n−1 X
X n
1
i=1 j=i+1 |{z}
| {zline }4
| {zline 3 }
line 1
Solving this summation gives us:
n−1 X
X n n−1
X
1= n−i
i=1 j=i+1 i=1
n−1
X n−1
X
= n− i
i=1 i=1
n(n − 1)
= n(n − 1) −
2
n(n − 1)
=
2
Thus for a collection of size n, Selection Sort makes
n(n − 1)
2
comparisons.
5. Provide an asymptotic characterization: the function determined is clearly Θ(n2 )
2.6 Other Considerations
2.6.1 Importance of Input Size
The second step in our algorithm analysis outline is to identify the input size. Usually this
is pretty straightforward. If the input is an array or collection of elements, a reasonable
input size is the cardinality (the number of elements in the collection). This is usually
the case when one is analyzing a sorting algorithm operating on a list of elements.
Though seemingly simple, sometimes identifying the appropriate input size depends on
the nature of the input. For example, if the input is a data structure such as a graph,
33
2 Algorithm Analysis
the input size could be either the number of vertices or number of edges. The most
appropriate measure then depends on the algorithm and the details of its analysis. It
may even depend on how the input is represented. Some graph algorithms have different
efficiency measures if they are represented as adjacency lists or adjacency matrices.
Yet another subtle difficulty is when the input is a single numerical value, n. In such
instances, the input size is not also n, but instead the number of symbols that would be
needed to represent n. That is, the number of digits in n or the number of bits required
to represent n. We’ll illustrate this case with a few examples.
Sieve of Eratosthenes
A common beginner mistake is made when analyzing the Sieve of Eratosthenes (named
for Eratosthenes of Cyrene, 276 BCE – 195 BCE). The Sieve is an ancient method for
prime number factorization. A brute-force algorithm, it simply tries every integer up to
a point to see if it is a factor of a given number.
Input : An integer n
Output : Whether n is prime or composite
1 for i = 2, . . . , n do
2 if i divides n then
3 Output composite
4 end
5 end
6 Output prime
Algorithm 10: Sieve of Eratosthenes
√ √
The for-loop only needs to check integers up to √ n because any factor greater than n
would necessarily have a corresponding
√ factor
√ n. A naive approach would observe that
the for-loop gets executed n − 1 ∈ O( n) times which would lead to the (incorrect)
impression that the Sieve is a polynomial-time (in fact sub-linear!) running time algorithm.
Amazing that we had a primality testing algorithm over 2,000 years ago! In fact, primality
testing was a problem that was not known to have a deterministic polynomial time
running algorithm until 2001 (the AKS Algorithm [3]).
The careful observer would realize that though n is the input, the actual input size is
again log (n), the number of bits required to represent n. Let N = log (n) be a placeholder
for our actual input size (and so n = 2N ). Then the running time of the Sieve is actually
√ √
O( n) = O( 2N )
which is exponential with respect to the input size N .
34
2.6 Other Considerations
This distinction is subtle but crucial: the difference between a polynomial-time algorithm
and an exponential algorithm is huge even for modestly sized inputs. What may take
a few milliseconds using a polynomial time algorithm may take billions and billions of
years with an exponential time algorithm as we’ll see with our next example.
Computing an Exponent
As a final example, consider the problem of computing a modular exponent. That is,
given integers a, n, and m, we want to compute
an mod m
A naive (but common!) solution might be similar to the Java code snippet in Code
Sample 2.7.
Whether one chooses to treat multiplication or integer division as the elementary operation,
the for-loop executes exactly n times. For “small” values of n this may not present a
problem. However, for even moderately large values of n, say n ≈ 2256 , the performance
of this code will be terrible.
1 int a = 45, m = 67;
2 int result = 1;
3 for(int i=1; i<=n; i++) {
4 result = (result * a % m);
5 }
Code Sample 2.7: Naive Exponentiation
To illustrate, suppose that we run this code on a 14.561 petaFLOP (14 quadrillion floating
point operations per second) super computer cluster (this throughput was achieved by
the Folding@Home distributed computing project in 2013). Even with this power, to
make 2256 floating point operations would take
2256
≈ 2.5199 × 1053
14.561 × 1015 · 60 · 60 · 24 · 365.25
or 252 sexdecilliion years to compute!
For context, this sort of operation is performed by millions of computers around the
world every second of the day. A 256-bit number is not really all that “large”. The
problem again lies in the failure to recognize the difference between an input, n, and
the input’s size. We will explore the better solution to this problem when we examine
Repeated Squaring in Section 5.3.
35
2 Algorithm Analysis
2.6.2 Control Structures are Not Elementary Operations
When considering which operation to select as the elementary operation, we usually
do not count operations that are necessary to the control structure of the algorithm.
For example, assignment operations, or operations that are necessary to execute a loop
(incrementing an index variable, a comparison to check the termination condition).
To see why, consider method in Code Sample 2.8. This method computes a simple
average of an array of double variables. Consider some of the minute operations that
are performed in this method:
• An assignment of a value to a variable (lines 2, 3, and 4)
• An increment of the index variable i (line 3)
• A comparison (line 3) to determine if the loop should terminate or continue
• The addition (line 4) and division (line 6)
1 public static double average(double arr[]) {
2 double sum = 0.0;
3 for(int i=0; i<arr.length; i++) {
4 sum = sum + arr[i];
5 }
6 return sum / arr.length;
7 }
Code Sample 2.8: Computing an Average
A proper analysis would use the addition in line 4 as the elementary operation, leading
to a Θ(n) algorithm. However, for the sake of argument, let’s perform a detailed analysis
with respect to each of these operations. Assume that there are n values in the array,
thus the for loop executes n times. This gives us
• n + 2 total assignment operations
• n increment operations
• n comparisons
• n additions and
• 1 division
Now, suppose that each of these operations take time t1 , t2 , t3 , t4 , and t5 milliseconds
each (or whatever time scale you like). In total, we have a running time of
t1 (n + 2) + t2 n + t3 n + t4 n + t5 = (t1 + t2 + t3 + t4 )n + (2t1 + t5 ) = cn + d
Which doesn’t change the asymptotic complexity of the algorithm: considering the
36
2.6 Other Considerations
additional operations necessary for the control structures only changed the constant
sitting out front (as well as some additive terms).
The amount of resources (in this case time) that are expended for the assignment,
increment and comparison for the loop control structure are proportional to the true
elementary operation (addition). Thus, it is sufficient to simply consider the most
common or most expensive operation in an algorithm. The extra resources for the control
structures end up only contributing constants which are ultimately ignored when an
asymptotic analysis is performed.
2.6.3 Average Case Analysis
The behavior of some algorithms may depend on the nature of the input rather than
simply the input size. From this perspective we can analyze an algorithm with respect
to its best-, average-, and worst-case running time.
In general, we prefer to consider the worst-case running time when comparing algorithms.
Example: searching an array
Consider the problem of searching an array for a particular element (the array is unsorted,
contains n elements). You could get lucky (best-case) and find it immediately in the first
index, requiring only a single comparison. On the other hand, you could be unlucky and
find the element in the last position or not at all. In either case n comparisons would be
required (worst-case).
What about the average case? A naive approach would be to average the worst and best
case to get an average of n+1
2
comparisons. A more rigorous approach would be to define
a probability of a successful search p and the probability of an unsuccessful search, 1 − p.
For a successful search, we further define a uniform probability distribution on finding
the element in each of the n indices. That is, we will find the element at index i with
probability np . Finding the element at index i requires i comparisons. Thus the total
number of expected comparisons in a successful search is
n
X p p(n + 1)
i· =
i=1
n 2
For an unsuccessful search, we would require n comparisons, thus the number of expected
comparisons would be
n(1 − p)
Since these are mutually exclusive events, we sum the probabilities:
p(n + 1) p − pn + 2n
+ n(1 − p) =
2 2
37
2 Algorithm Analysis
p C
0 n
1 7 1
n+
4 8 8
1 3 1
n+
2 4 4
3 5 3
n+
4 8 8
n+1
1
2
Table 2.5: Expected number of comparisons C for various values of the probability of a
successful search p.
We cannot remove the p terms, but we can make some observations for various values
(see Table 2.6.3). When p = 1 for example, we have the same conclusion as the naive
approach. As p decreases, the expected number of comparisons grows to n.
Figure 2.2: Expected number of comparisons for various success probabilities p.
2.6.4 Amortized Analysis
Sometimes it is useful to analyze an algorithm not based on its worst-case running time,
but on its expected running time. That is, on average, how many resources does an
algorithm perform? This is a more practical approach to analysis. Worst-case analysis
assumes that all inputs will be difficult, but in practice, difficult inputs may be rare. The
average input may require fewer resources.
38
2.7 Analysis of Recursive Algorithms
Amortized algorithm analysis is similar to the average-case analysis we performed before,
but focuses on how the cost of operations may change over the course of an algorithm.
This is similar to a loan from a bank. Suppose the loan is taken out for $1,000 at a
5% interest rate. You don’t actually end up paying $50 the first year. You pay pay
slightly less than that since you are (presumably) making monthly payments, reducing
the balance and thus reducing the amount of interest accrued each month.
We will not go into great detail here, but as an example, consider Heap Sort. This
algorithm uses a Heap data structure. It essentially works by inserting elements into the
heap and then removing them one by one. Due to the nature of a heap data structure,
the elements will come out in the desired order.
An amortized analysis of Heap Sort may work as follows. First, a heap requires about
log (n) comparisons to insert an element (as well as remove an element, though we’ll
focus on insertion), where n is the size of the heap (the number of elements currently in
the heap. As the heap grows (and eventually shrinks), the cost of inserts will change.
With an initially empty heap, there will only be 0 comparisons. When the heap has
1 element, there will be about log (1) comparisons, then log (2) comparisons and so on
until we insert the last element. This gives us
n−1
X
C(n) = log (i)
i=1
total comparisons.
2.7 Analysis of Recursive Algorithms
Recursive algorithms can be analyzed with the same basic 5 step process. However,
because they are recursive, the analysis (step 4) may involve setting up a recurrence
relation in order to characterize how many times the elementary operation is executed.
Though cliche, as a simple example consider a naive recursive algorithm to compute the
n-th Fibonacci number. Recall that the Fibonacci sequence is defined as
Fn = Fn−1 + Fn−2
with initial conditions F0 = F1 = 1. That is, the n-th Fibonacci number is defined as the
sum of the two previous numbers in the sequence. This defines the sequence
1, 1, 2, 3, 5, 8, 13, 21, 34, 55, . . .
The typical recursive solution is presented as Algorithm 11
The elementary operation is clearly the addition in line 4. However, how do we analyze
the number of additions performed by a call to Fibonacci(n)? To do this, we setup
39
2 Algorithm Analysis
Input : An integer n
Output : The n-th Fibonacci Number, Fn
1 if n = 0 or n = 1 then
2 output 1
3 else
4 output Fibonacci(n − 1) + Fibonacci(n − 2)
5 end
Algorithm 11: Fibonacci(n)
a recurrence relation. Let A(n) be a function that counts the number of additions
performed by Fibonacci(n). If n is zero or one, the number of additions is zero (the base
case of our recursion performs no additions). That is, A(0) = A(1) = 0. But what about
n > 1?
When n > 1, line 4 executes. Line 4 contains one addition. However, it also contains
two recursive calls. How many additions are performed by a call to Fibonacci(n − 1)
and Fibonacci(n − 2)? We defined A(n) to be the number of additions on a call to
Fibonacci(n), so we can reuse this function: the number of additions is A(n − 1) and
A(n − 2) respectively. Thus, the total number of additions is
A(n) = A(n − 1) + A(n − 2) + 1
This is a recurrence relation,6 in particular, it is a second-order linear non-homogeneous
recurrence relation. This particular relation can be solved. That is, A(n) can be expressed
as a non-recursive closed-form solution.
The techniques required for solving these type of recurrence relations are beyond the
scope of this text. However, for many common recursive algorithms, we can use a simple
tool to characterize their running time, the “Master Theorem.”
2.7.1 The Master Theorem
Suppose that we have a recursive algorithm that takes an input of size n. The recursion
may work by dividing the problem into a subproblems each of size nb (where a, b are
constants). The algorithm may also perform some amount of work before or after the
recursion. Suppose that we can characterize this amount of work by a polynomial
function, f (n) ∈ Θ(nd ).
This kind of recursive algorithm is common among “divide-and-conquer” style algorithms
that divide a problem into subproblems and conquers each subproblem. Depending on
6
Also called a difference equation, which are sort of discrete analogs of differential equations.
40
2.7 Analysis of Recursive Algorithms
the values of a, b, d we can categorize the runtime of this algorithm using the Master
Theorem.
Theorem 3 (Master Theorem). Let T (n) be a monotonically increasing function that
satisfies
T (n) = aT ( nb ) + f (n)
T (1) = c
where a ≥ 1, b ≥ 2, c > 0. If f (n) ∈ Θ(nd ) where d ≥ 0, then
Θ(nd ) if a < bd
T (n) = Θ(n log n) if a = bd
d
Θ(nlogb a ) if a > bd
“Master” is a bit of a misnomer. The Master Theorem can only be applied to recurrence
relations of the particular form described. It can’t, for example, be applied to the previous
recurrence relation that we derived for the Fibonacci algorithm. However, it can be used
for several common recursive algorithms.
Example: Binary Search
As an example, consider the Binary Search algorithm, presented as Algorithm 12. Recall
that binary search takes a sorted array (random access is required) and searches for an
element by checking the middle element m. If the element being searched for is larger
than m, a recursive search on the upper half of the list is performed, otherwise a recursive
search on the lower half is performed.
Let’s analyze this algorithm with respect to the number of comparisons it performs. To
simplify, though we technically make two comparisons in the pseudocode (lines 5 and 7),
let’s count it as a single comparison. In practice a comparator pattern would be used and
logic would branch based on the result. However, there would still only be one invocation
of the comparator function/object. Let C(n) be a function that equals the number of
comparisons made by Algorithm 12 while searching an array of size n in the worst case
(that is, we never find the element or it ends up being the last element we check).
As mentioned, we make one comparison (line 5, 7) and then make one recursive call. The
recursion roughly cuts the size of the array in half, resulting in an array of size n2 . Thus,
n
C(n) = C +1
2
Applying the master theorem, we find that a = 1, b = 2. In this case, f (n) = 1 which is
bounded by a polynomial: a polynomial of degree d = 0. Since
1 = a = bd = 20
by case 2 of the Master Theorem applies and we have
C(n) ∈ Θ(log (n))
41
2 Algorithm Analysis
Input : A sorted collection of elements A = {a1 , . . . , an }, bounds 1 ≤ l, h ≤ n,
and a key ek
Output : An element a ∈ A such that a = ek according to some criteria; φ if no
such element exists
1 if l > h then
2 output φ
3 end
4 m ← b h+l
2
c
5 if am = ek then
6 output am
7 else if am < ek then
8 BinarySearch(A, m + 1, h, e)
9 else
10 BinarySearch(A, l, m − 1, e)
11 end
Algorithm 12: Binary Search – Recursive
Example: Merge Sort
Recall that Merge Sort is an algorithm that works by recursively splitting an array of
size n into two equal parts of size roughly n2 . The recursion continues until the array
is trivially sorted (size 0 or 1). Following the recursion back up, each subarray half is
sorted and they need to be merged. This basic divide and conquer approach is presented
in Algorithm 13. We omit the details of the merge operation on line 4, but observe that
it can be achieved with roughly n − 1 comparisons in the worst case.
Input : An array, sub-indices 1 ≤ l, r ≤ n
Output : An array A0 such that A[l, . . . , r] is sorted
1 if l < r then
2 MergeSort(A, l, b r+l
2
c)
3 MergeSort(A, d r+l
2
e, r)
4 Merge sorted lists A[l, . . . , b r+l
2
c] andA[d r+l
2
e, . . . , r]
5 end
Algorithm 13: Merge Sort
Let C(n) be the number of comparisons made by Merge Sort. The main algorithm makes
two recursive calls on subarrays of size n2 . There are also n + 1 comparisons made after
the recursion. Thus we have
42
2.7 Analysis of Recursive Algorithms
n
C(n) = 2C + (n − 1)
2
Again, applying the Master Theorem we have that a = 2, b = 2 and that f (n) = (n + 1) ∈
Θ(n) and so d = 1. Thus,
2 = a = bd = 21
and by case 2,
C(n) ∈ Θ(n log (n))
We will apply the Master Theorem to several other recursive algorithms later on.
43
3 Storing Things
3.1 Lists
3.2 Sets
3.3 Hash-Tables
Motivation:
• Linked lists provide O(n), O(1) access/insert efficiency
• Array-based lists provide O(1), O(n) access/insert efficiency
• Balanced binary search trees provide O(log n) operations
• Hash tables provide amortized O(1) operations
Basics:
• Elements are stored in a fixed-size array
• The location of an element (its index) is computed by computing a hash value of
the element
• We can restrict attention to integer keys (elements/objects can be mapped to an
integer key based on their state)
• Computing the hash value of a key provides quick, random access to the associated
bucket containing the element (value)
• Insertion and retrieval are achieved by computing the hash value and accessing the
array
3.3.1 Hash Functions
• Simply put, a hash function is a function that maps a large domain to a small
co-domain
45
3 Storing Things
• For our purposes:
h : Z → Zm
where Zm = {0, 1, 2, . . . , m − 1}
• Hash functions should be
1. Easy to compute
2. Distribute values as uniformly as possible among Zm
Example. Consider the hash function:
h(k) = 7k + 13 mod 8
Insertion of the following keys into a hash table with the above has function results in
the following configuration.
10, 16, 4, 13, 86
Index 0 1 2 3 4 5 6 7
Value 13 4 10 16 86
Table 3.1: Resulting Hash Table
But, now suppose we attempt to insert the key 100: h(100) = 1. The array cell is already
occupied by 4. This is known as a collision
3.3.2 Collisions
• By definition, hash functions map large sets to small sets; they cannot be one-to-one
• Some elements k1 , k2 will necessarily map to the same hash value h(k1 ) = h(k2 )
• When two elements map to the same hash value, it is called a collision
• Collisions can be resolved using various strategies; we examine two:
1. Open addressing
2. Chaining
Open Addressing
Open addressing involves probing the hash table to find an open cell in the table. Several
strategies are available.
• Linear probing involves probing the table using a linear function:
h(k, i) = h(k) + c1 · i mod m
46
3.3 Hash-Tables
for i = 1, . . . , m − 1 and c1 ∈ Zm
• Quadratic probing involves probing the table using a quadratic function:
h(k, i) = h(k) + c1 · i + c2 · i2 mod m
for i = 0, 1, . . . , m
• In general, c1 , c2 ∈ Zm
• Probing wraps around the table (modulo m)
• Double Hashing involves using a second hash function s(k) to probe:
h(k, i) = h(k) + i · s(k) mod m
for i = 0, 1, . . . , m
Retrieval and Deletion
• Retrieval now involves hashing and resolving collisions until either the element is
found or a free cell is found (item is not in the table)
• If deletions are allowed, additional bookkeeping is needed to indicate if a cell has
ever been occupied; sometimes referred to as a “dirty bit”
Chaining
• No probing, instead each bucket in the hash table represents a linked list
• Alternatively: another efficient collection (BST, etc.)
• Each element is added to the list associated with the hash value
• Retrieval now may involve traversing another data structure
• Retrieval may now also involve another mechanism to identify each object in the
collection
• No more issues with deletion
• Rehashing may still be necessary to prevent each list/collection from getting too
large
3.3.3 Efficiency Rehashing
Efficiency of a hash table depends on how full it is. Let n be the number of elements in
the table, then the load factor of the table is
n
α=
m
47
3 Storing Things
For successful searches, the expected number of key comparisons is
α
S =1+
2
The fuller a table gets, the more its performance degrades to linear search:
• Smaller table → more collisions → slower access/insert (but less wasted space)
• Larger table → fewer collisions → faster access/insert (but more wasted space)
This represents a fundamental time/space trade-off.
• As a hash table fills up, more collisions slow down access of elements
• If deletion is allowed, more cells become dirty and slow down access of elements
• To accommodate more elements and to ensure that insert/retrieval remains fast a
rehashing of the tables
1. A new, larger table is created with a new hash function using a larger m
2. Each element is reinserted into the new hash table, an O(n) operation.
• Rehashing is done infrequently (when the load factor is exceeded): when averaged
over the lifecycle of the data structure, it leads to amortized constant time.
3.3.4 Other Applications
• Pseudorandom number generators
• Data Integrity (check sums)
• Cryptography: message authentication codes, data integrity, password storage (ex:
MD4, 5, SHA-1, 2, 3)
• Derandomization
• Maps, Associative Arrays, Sets, Caches
3.3.5 Java Implementations
The Java Collections library provides several data structures that are backed by hash
tables that utilize an object’s hashCode() method and rely on several assumptions. For
this reason, it is best practice to always override the hashCode() and equals() methods
for every user-defined object so that they depend on an object’s entire state.
• java.util.HashSet<E>
– Implements the Set interface
48
3.4 Bloom Filters
– Permits a single null element
• java.util.Hashtable<K,V>
– A chained hash table implementation
– Documentation labels this implementation as “open” (actually refers to “open
hashing”)
– Allows you to specify an initial capacity and load factor with defaults (.75)
– Automatically grows/rehashes (rehash())
– Implements the Map interface
– Synchronized for multithreaded applications
– Does not allow null keys nor values
• java.util.HashMap<K,V>
– Implements the Map interface
– Allows at most one null key, any number of null values
– Subclass: LinkedHashMap (guarantees iteration order, can be used as an LRU
cache)
ol
3.4 Bloom Filters
3.5 Disjoint Sets
3.6 Exercises
Exercise 3.1. Consider the following simple hash function.
h(k) = (5k + 3) mod 11
and consider the following elements:
4, 42, 26, 5, 8, 18, 14
(a) Insert the above elements into a hash table, using linear probing to resolve collisions.
Show what the hash table would look like after inserting the elements in the order
above.
49
3 Storing Things
(b) Insert the above elements into a hash table using chaining to resolve collisions. Show
what the hash table would look like after inserting the elements in the order above.
50
4 Brute Force Style Algorithms
4.1 Introduction
• Brute Force style algorithms are simple a “just do it” approach
• Usually an “obvious”, straightforward solution
• Not ideal and may be completely infeasible for even small inputs
• May be necessary (ex: sequential search, max/min finding)
4.1.1 Examples
Shuffling Cards
• There are 52! ≈ 8.065817 × 1067 possible shuffles
• If 5 billion people shuffled once per second for the last 1,000 years, only:
1.95 × 10−48 = 0. 00 . . 00} 195%
| .{z
45 zeros
have been examined so far!
• Even at 5 billion shuffles per second, it would take
5.1118 × 1050
years to enumerate all possibilities
Greatest Common Divisor
• Given two integers, a, b ∈ Z+ , the greatest common divisor :
gcd(a, b)
is the largest integer that divides both a and b
• Naive solution: factor a, b:
a = pk11 pk21 · · · pknn
b = p`11 p`21 · · · p`nn
51
4 Brute Force Style Algorithms
• Then
min{k1 ,`1 } min{k2 ,`2 } n ,`n }
gcd(a, b) = p1 p2 · · · pmin{k
n
• Factoring is not known (or believed) to be in P (deterministic polynomial time)
• Better solution: Euclid’s GCD algorithm
4.1.2 Backtracking
• We iteratively (or recursively) build partial solutions such that solutions can be
“rolled back” to a prior state
• Once a full solution is generated, we process/test it then roll it back to a (previous
partial) solution
• Advantage: depending on the nature/structure of the problem, entire branches of
the computation tree may be “pruned” out
• Avoiding infeasible solutions can greatly speed up computation in practice
• Heuristic speed up only: in the worst case and in general, the computation may
still be exponential
4.2 Generating Combinatorial Objects
4.2.1 Generating Combinations (Subsets)
Recall that combinations are simply all possible subsets of size k. For our purposes, we
will consider generating subsets of
{1, 2, 3, . . . , n}
The algorithm works as follows.
• Start with {1, . . . , k}
• Assume that we have a1 a2 · · · ak , we want the next combination
• Locate the last element ai such that ai 6= n − k + i
• Replace ai with ai + 1
• Replace aj with ai + j − i for j = i + 1, i + 2, . . . , k
52
4.2 Generating Combinatorial Objects
Input : A set of n elements and an k-combination, a1 · · · ak .
Output : The next k-combination.
1 i=k
2 while ai = n − k + i do
3 i=i−1
4 end
5 ai = ai + 1
6 for j = (i + 1) . . . r do
7 aj = ai + j − i
8 end
Algorithm 14: Next k-Combination
Example: Find the next 3-combination of the set {1, 2, 3, 4, 5} after {1, 4, 5}
Here, n = 5, k = 3, a1 = 1, a2 = 4, a3 = 5.
The last i such that ai 6= 5 − 3 + i is 1.
Thus, we set
a1 = a1 + 1 = 2
a2 = a1 + 2 − 1 = 3
a3 = a1 + 3 − 1 = 4
So the next k-combination is {2, 3, 4}.
Two full examples are shown in Figure 4.1.
4.2.2 Generating Permutations
• A permutation of n elements, a1 , . . . , an is an ordered arrangement of these elements
• There are n!
• The Johnson-Trotter algorithm generates all permutations with the least change
property (at most two elements are swapped in each permutation).
Lexicographically Ordered Permutations:
• Start with permutation 1234 · · · n
• Given a current permutation, a1 , a2 , . . . , an
• Find the right-most pair ai , ai+1 such that ai < ai+1
• Find the smallest element (to the right, “in the tail”) larger than ai , label it a0
• Swap a0 and ai
53
4 Brute Force Style Algorithms
Iteration Combination
1 1234
Iteration Combination 2 1235
3 1236
1 123
4 1245
2 124
5 1246
3 125
6 1256
4 134
7 1345
5 135
8 1346
6 145
9 1356
7 234
10 1456
8 235
11 2345
9 245
12 2346
10 345
5
13 2356
(a) Sequence for 3
14 2456
15 3456
6
(b) Sequence for 4
Figure 4.1: Two combination sequence examples
• Order (sort) elements to the right of a0
• Example: 163542 → 164235
• 35 is the last pair; 4 is the minimal element (among 5, 4, 2) greater than ai = 3;
swap 3, 4 giving 164532; sort 532 to 235 giving the result
• Example: 12345 → 12354 → 12435 → · · ·
Generalization:
• In general, permutations can be formed from subsets of a set
• The number ordered permutations of k elements from a set of n distinct elements
is:
n!
P (n, k) =
(n − k)!
• We can adapt the two previous algorithms to generate these permutations: use
the combinations algorithm to generate all distinct subsets of size k; then run the
permutation algorithm on each subset to generate ordered permutations.
4.2.3 Permutations with Repetition
• Let A = {a1 , . . . , an } be a set with n objects
54
4.2 Generating Combinatorial Objects
Input : A permutation a1 a2 . . . an of elements 1, 2, . . . , n
Output : The lexicographically next permutation
1 i←n−1
2 while ai > ai+1 do
3 i ← (i − 1)
4 end
5 a0 ← ai+1
6 for j = (i + 2) . . . n do
7 if aj > ai ∧ aj < a0 then
8 a0 ← aj
9 end
10 end
11 swap a0 , ai
12 sort elements ai+1 · · · an
Algorithm 15: Next Lexicographic Permutation
• We wish to form an ordered permutation of length k of elements from A but we
are also allowed to take as many “copies” of each element as we wish
• Number of such permutations: nk
• Use case: generating DNA sequences from amino acid bases, {A, G, C, T }
• Straightforward idea: count in base b = n and associate each number in base b
with an element in A
• Count from 0 to nk − 1 and generate elements
• A direct counting algorithm is presented as Algorithm 16
• A general purpose algorithm for base-conversion is presented as Algorithm 17
55
4 Brute Force Style Algorithms
Input : A base set A = {a1 , . . . , an } and a current permutation (base-n
number) x1 · · · xk
Output : The next permutation
1 i←k
2 carry ← true
3 while carry ∧ i ≥ 1 do
4 xi ← xi + 1
5 if xi ≥ n then
6 xi ← 0
7 i←i+1
8 else
9 carry ← f alse
10 end
11 end
12 output x1 · · · xk
Algorithm 16: Next Repeated Permutation Generator
Input : n ∈ N, an ordered symbol set, Σ = {σ0 , σ1 , . . . , σb−1 }, a base b
Output : The base-b representation of n, xk . . . x1 x0 using the symbol set Σ
1 i←0
2 while n > 0 do
3 j ← n mod b
4 n ← b nb c
5 xi ← σ j
6 i ← (i + 1)
7 end
8 optionally: pad out x with leading σ0 symbols (leading zeros) until it has the
desired length
9 output xk · · · x1 x0
Algorithm 17: Base Conversion Algorithm
4.2.4 Set Partitions
Let A = {a1 , . . . an } be a set. A partition of A is a collection of disjoint subsets of A
whose union is equal to A. That is, a partition splits up the set A into a collection of
subsets. The number of ways to partition a set corresponds to the Bell numbers:
56
4.2 Generating Combinatorial Objects
n
X n
Bn+1 = Bk
k=0
k
with B0 = 1.
Input : A set A = {a1 , . . . , an }
Output : A set of all set partitions of A
1 partitions ← {{a1 }}
2 for i = 2, . . . , |A| do
3 newP artitions ← ∅
4 foreach partition part ∈ partitions do
5 foreach set s ∈ part do
6 newP art ← copyof part
7 add ai to s in newP art
8 add newP art to newP artitions
9 end
10 newP art ← copyof part
11 add {ai } to newP art
12 end
13 partitions ← newP artitions
14 end
15 output partitions
Algorithm 18: Set Partition Generator
Demonstration: Starting with {{a, b}, {{a}, {b}}} (which partitions the set A = {a, b}
into two possible partitions), we can produce all partitions of the set A0 = {a, b, c} by
adding c to each of the possible subsets of subsets (or in a set by itself):
{{a}, {b}, {c}},
{{a}, {b, c}},
{{b}, {a, c}},
{{c}, {a, b}},
{{a, b, c}}
57
4 Brute Force Style Algorithms
4.3 Problems & Algorithms
4.3.1 Satisfiability
• Recall that a predicate is a boolean function on n variables:
P (~x) = P (x1 , . . . , xn )
• A predicate is satisfiable if there exists an assignment of variables that makes P
evaluate to true:
∃x1 , . . . , xn [P (x1 , . . . , xn )]
• Example:
(x1 ∨ ¬x2 ) ∧ (¬x1 ∨ x2 )
is satisfiable (there are two assignments for which the exclusive-or evaluates to
true)
• Example:
(x1 ∨ x2 ∨ x3 ) ∧ (x1 ∨ ¬x2 ) ∧ (x2 ∨ ¬x3 ) ∧ (x3 ∨ ¬x1 ) ∧ (¬x1 ∨ ¬x2 ∨ ¬x3 )
is not satisfiable: any setting of the three variables evaluates to false
• Deciding if a given predicate is satisfiable or not is a fundamental NP-complete
problem
Problem 1 (Satisfiability).
Given: A boolean predicate P on n variables
Output: True if P is satisfiable, false otherwise
58
4.3 Problems & Algorithms
Input : A predicate P (x1 , . . . , xn )
Output : true if P is satisfiable, false otherwise
1 foreach truth assignment ~t do
2 if P (~t) = 1 then
3 return True
4 end
5 return False
6 end
Algorithm 19: Brute Force Iterative Algorithm for Satisfiability
Input : A predicate P (x1 , . . . , xn ), a partial truth assignment ~t = t1 , . . . , tk
Output : true if P is satisfiable, false otherwise
1 if k = n then
2 return P (t1 , . . . , tn )
3 else
4 tk+1 ← 0
5 if Sat(P, ~t) then
6 return True;
7 else
8 tk+1 ← 1
9 return Sat(P, ~t)
10 end
11 end
Algorithm 20: Brute Force Recursive Algorithm for Satisfiability
4.3.2 Hamiltonian Path/Cycle
Definition 7. Let G = (V, E) be an undirected graph and let v ∈ V be a vertex. The
neighborhood of v, N (v) is the set of vertices connected to v.
N (v) = {x|(u, v) ∈ E}
Problem 2 (Hamiltonian Path).
Given: An undirected graph G = (V, E)
Output: True if G contains a Hamiltonian Path, false otherwise
Variations:
59
4 Brute Force Style Algorithms
x1
0 1
x2 x2
0 1 0 1
x3 x3 x3 x3 d=n
xn xn
2n
Figure 4.2: Computation Tree for Satisfiability Backtracking Algorithm
• Functional version: output a Hamiltonian path if one exists
• Hamiltonian Cycle (Circuit) problem
• Weighted Hamiltonian Path/Cycle
• Traveling Salesperson Problem
Problem 3 (Traveling Salesperson (TSP)).
Given: A collection of n cities, C = {c1 , . . . , cn }
Output: A permutation of cities π(C) = (c01 , c02 , . . . , c0n ) such that the total distance,
n−1
!
X
δ(c0i , c0i+1 ) + δ(c0n , c01 )
i=1
is minimized.
Discussion: is one variation “harder” than the other?
• Contrast two brute force strategies
• Backtracking enables pruning of recurrence tree
• potential for huge heuristic speed up
60
4.3 Problems & Algorithms
Input : A graph, G = (V, E), a path p = v1 , . . . , vk
Output : true if G contains a Hamiltonian cycle, false otherwise
1 foreach permutation of vertices pi = v1 , . . . , vn do
2 isHam ← true
3 for i = 1, . . . , n − 1 do
4 if (vi , vi+1 ) 6∈ E then
5 isHam ← f alse
6 end
7 end
8 if (vn , v1 ) 6∈ E then
9 isHam ← f alse
10 end
11 if isHam then
12 return True
13 end
14 end
15 return False
Algorithm 21: Brute Force Iterative Algorithm for Hamiltonian Cycle
Input : A graph, G = (V, E), a path p = v1 , . . . , vk
Output : true if G contains a Hamiltonian cycle, false otherwise
1 if |p| = n then
2 return true
3 end
4 foreach x ∈ N (vk ) do
5 if x 6∈ p then
6 if Walk(G, p + x) then
7 return true
8 end
9 end
10 end
11 return true
Algorithm 22: Hamiltonian DFS Cycle Walk
61
4 Brute Force Style Algorithms
Input : A graph, G = (V, E)
Output : True if G contains a Hamiltonian path, False otherwise
1 foreach v ∈ V do
2 path ← v
3 if Walk(G, p) then
4 output True
5 end
6 output False
7 end
8 return True
Algorithm 23: Hamiltonian DFS Path Walk–Main Algorithm
Input : A graph, G = (V, E), a path p = v1 · · · vk
Output : True if G contains a Hamiltonian path, False otherwise
1 if k = n then
2 return True
3 end
4 foreach v ∈ N (vk ) do
5 if v 6∈ p then
6 Walk(G, p + v)
7 end
8 end
9 return False
Algorithm 24: Walk(G, p) – Hamiltonian DFS Path Walk
• To see how the walk approach can provide a substantial speed up observe the graph
if Figure 4.3
• Any permutation of vertices that involves edges that are not present need not be
processed
• Backtracking computation is illustrated in Figure 4.4
4.3.3 0-1 Knapsack
Problem 4 (0-1 Knapsack).
Given: A collection of n items, A = {a1 , . . . , an }, a weight function wt : A → R+, a
value function val : A → R+ and a knapsack capacity W
62
4.3 Problems & Algorithms
b c
a f
d e
Figure 4.3: A small Hamiltonian Graph
Output: A subset S ⊆ A such that
X
wt(S) = wt(s) ≤ W
s∈S
and X
val(S) = val(s)
s∈S
is maximized.
An example input can be found in Figure 4.5. The optimal solution is to take items
a1 , a3 , a4 for a value of 29. Another example can be found in Table 9.7 in a later section
where we look at a dynamic programming solution for the same problem.
• We wish to maximize the value of items (in general, an “objective function”) taken
subject to some constraint
• 0-1 refers to the fact that we can an item or leave it
• Variation: the dual of the problem would be to minimize some loss or cost
• Variation: fractional knapsack
• A greedy strategy fails in general; a small example: three items with weights 1, 2, 3
and values 6, 10, 12 respectively; and a total weight capacity of 4. The ratios would
be 6, 5, 4 respectively meaning that a greedy strategy would have us select the first
two items for a total weight of 3 and value of 16. However, the optimal solution
would be to select the first and third item for a total weight of 4 and value of 18.
• A brute-force backtracking algorithm is presented as Algorithm 14; note that you
would invoke this algorithm with j = 0 and S = ∅ initially.
63
4 Brute Force Style Algorithms
a
b d
b c d b
a c d e e f c e
d e f a c f f e e f c f
f e f c f e f c
(a) Hamiltonian path computation tree (b) Hamiltonian path computation tree starting from
starting from b. a.
Figure 4.4: Brute Force Backtracing Hamiltonian Path Traversal. The first traversal
starts from b and finds no Hamiltonian Path. The second traversal finds
several and would terminate prior to exploring the entire computation tree.
The entire tree is presented for completeness.
64
4.3 Problems & Algorithms
Item Value Weight
a1 15 1
a2 10 5
a3 9 3
a4 5 4
Figure 4.5: Example Knapsack Input with W = 8
Input : An instance of the knapsack problem K = (A, wt, val, W ), an index j,
and a partial solution S ⊆ A consisting of elements not indexed more
than j
Output : A solution S 0 that is at least as good as S
1 if j = n then
2 return S
3 end
4 Sbest ← S
5 for k = j + 1, . . . , n do
6 S 0 ← S ∪ {ak }
7 if wt(S 0 ) ≤ W then
8 T ← Knapsack(K, k, S 0 )
9 if val(T ) > val(Sbest ) then
10 Sbest ← T
11 end
12 end
13 end
14 return Sbest
Algorithm 25: Knapsack(K, S) – Backtracking Brute Force 0-1 Knapsack
4.3.4 Closest Pair
Problem 5 (Closest Pair).
Given: A collection of n points, P = {p1 = (x1 , y2 ), . . . , pn = (xn , yn )}
Output: A pair of points (pa , pb ) that are the closest according to Euclidean distance.
4.3.5 Convex Hull
• A set of point P in the plane is convex if for any pair of points p, q ∈ P every point
in the line segment pq is contained in P
65
4 Brute Force Style Algorithms
∅
{a1 } {a2 } {a3 } {a4 }
{a1 , a2 } {a1 , a3 } {a1 , a4 } {a2 , a3 } {a2 , a4 } {a3 , a4 }
{a1 , a2 , a3 } {a1 , a2 , a4 } {a1 , a3 , a4 } {a2 , a3 , a4 }
{a1 , a2 , a3 , a4 }
Figure 4.6: Knapsack Computation Tree for n = 4
• A convex hull of a set of points P is the smallest convex set containing P
Problem 6 (Convex Hull).
Given: A set P of n points
Output: A convex hull H of P
• The idea is to choose extreme points: points that when connected form the border
of the convex hull
• A pair of points p1 = (x1 , y1 ), p2 = (x2 , y2 ) are extreme points if all other p ∈
P \ {p1 , p2 } lie on the same side of the line defined by p1 , p2
66
4.3 Problems & Algorithms
Input : A set of points P ⊆ R2
Output : A convex hull H of P
1 H←∅
2 foreach pair of points pi , pj do
y −y
3 m ← xjj −xii
4 b ← yi − m · xi //m, b define the line pi pj
5 pk ← an arbitrary point in P \ {pi , pj }
6 s ← sgn(yk − m · xk − b) //s indicates which side of the line pk
lies on: positive for above, negative for below
7 isExtreme ← true
8 foreach p = (x, y) ∈ P \ {pi , pj } do
9 if s 6= sgn(y − m · x − b) then
10 isExtreme ← f alse
11 end
12 end
13 if isExtreme then
14 H ← H ∪ {p1 , p2 }
15 end
16 end
17 output H
Algorithm 26: Brute Force Convex Hull
4.3.6 Assignment Problem
Problem 7 (Assignment).
Given: A collection of n tasks T = {t1 , . . . , tn } and n persons P = {p1 , . . . , pn } and an
n × n matrix C with entries ci,j representing the cost of assigning person pi to task tj .
Output: A bijection f : P → T such that
X
C[i, f (pi )]
pi ∈P
is minimized.
A brute force approach: try all bijections f : T → P , equivalent to all permutations of
one of these sets.
This is actually efficiently solvable via the Hungarian Method or by a Linear Program
formulation.
67
4 Brute Force Style Algorithms
4.3.7 Subset Sum
Problem 8 (Subset Sum).
Given: A set of n integers, A = {a1 , . . . , an } and an integer k
Output: true if there exists a subset S ⊆ A such that
X
s=k
s∈S
• Solution: Generate all possible subsets of A and check their sum
• Backtracking solution: use the same algorithm as Knapsack; backtrack if sum
exceeds k
• Variation: does there exist a partitioning into two sets whose sums are equal?
Input : A set of n integers, A = {a1 , . . . , an }, an integer k, a partial solution
S = {a` , . . . , aj } ⊆ A
Output : A subset S 0 ⊆ A whose elements sum to k if one exists; nil otherwise
1 if sum(S) = k then
2 return S
3 end
4 if sum(S) > k then
5 return nil
6 end
7 for i = j + 1, . . . n do
8 T ← SubsetSum(S ∪ {ai }, k)
9 if T 6= nil then
10 return T
11 end
12 end
Algorithm 27: Brute Force Subset Sum
68
4.4 Exercises
4.4 Exercises
Exercise 4.1. Implement the brute-force solution for the Hamiltonian Path problem in
the high-level programming language of your choice.
Exercise 4.2. Implement the brute-force solution for the 0-1 Knapsack problem in the
high-level programming language of your choice.
Exercise 4.3. Consider the following frequent itemset problem: we are given a set of
items A = {a1 , a2 , . . . , an } and a set of baskets B1 , B2 , . . . , Bm such that each basket is
a subset containing a certain number of items, Bi ⊆ A. Given a support s, frequent
itemsets are subsets S ⊂ A such that S is a subset of t baskets (that is, the items in S
all appear in at least s baskets).
For example, suppose that we have the following baskets:
B1 = {beer, screws, hammer}
B2 = {saw, lugnuts}
B3 = {beer, screws, hammer, lugnuts}
B4 = {beer, screws, hammer, router}
If our support s = 2 then we’re looking for all subsets of items that appear in at least 2
of the baskets. In particular:
• {lugnuts} as it appears in B2 , B3
• {beer} as it appears in B1 , B3 , B4
• {hammer} as it appears in B1 , B3 , B4
• {screws} as it appears in B1 , B3 , B4
• {beer, screws} as it appears in B1 , B3 , B4
• {hammer, screws} as it appears in B1 , B3 , B4
• {hammer, beer} as it appears in B1 , B3 , B4
This problem is widely seen in commerce, language analysis, search, and financial
applications to learn association rules. For example, a customer analysis may show that
people often buy nails when they buy a hammer (the pair has high support), so run a
sale on hammers and jack up the price of nails!
For this exercise, you will write a program that takes a brute force approach (with
pruning) and finds the support for each possible subset S ⊆ A. If there is zero support,
you will omit it from your output. Essentially you are solving the problem for s = 1, but
we’re not only interested in the actual sets, but the sets and the number of baskets that
they appear in.
Write a solution to this problem in the high-level programming language of your choice.
69
5 Divide & Conquer Style Algorithms
5.1 Introduction
Divide & conquer approach already seen in other algorithms: Merge Sort, Quick Sort,
Binary Search, etc.
Google’s MapReduce
5.2 Problems & Algorithms
5.3 Repeated Squaring
• Wish to compute an mod m
• Brute force: n multiplications
• Basic idea: square and repeat to reduce total number of multiplications
Example:
a77 = a64 · a13
= a64 · a8 · a5
= a64 · a8 · a4 · a1
Observe that we can compute:
a2 = a·a
a4 = a2 · a2
a8 = a4 · a4
a16 = a8 · a8
a32 = a16 · a16
a64 = a32 · a32
Which only requires 6 multiplications, then a77 can be computed with an additional 3
multiplications (instead of 277 = 1.511 × 1023 ).
71
5 Divide & Conquer Style Algorithms
Formally, we note that
k k−1
αn = αbk 2 +bk−1 2 +···+b1 2+b0
k k−1
= αbk 2 × αbk−1 2 × · · · × α2b1 × αb0
So we can compute αn by evaluating each term as
i
bi 2i α2 if bi = 1
α =
1 if bi = 0
We can save computation because we can simply square previous values:
i i−1
α2 = (α2 )2
We still evaluate each term independently however, since we will need it in the next term
(though the accumulated value is only multiplied by 1).
Input : Integers α, m and n = (bk bk−1 . . . b1 b0 ) in binary.
Output : αn mod m
1 term = α
2 if (b0 = 1) then
3 product ← α
4 end
5 else
6 product ← 1
7 end
8 for i = 1 . . . k do
9 term ← term × term mod m
10 if (bi = 1) then
11 product ← product × term mod m
12 end
13 end
14 output product
Algorithm 28: Binary Exponentiation
Example: Compute 1226 mod 17 using Modular Exponentiation.
1 1 0 1 0 = (26)2
4 3 2 1 - i
1 16 13 8 12 term
9 9 8 8 1 product
72
5.4 Euclid’s GCD Algorithm
Thus,
1226 mod 17 = 9
5.4 Euclid’s GCD Algorithm
TODO: remove this (redundancy from intro)?
Recall that we can find the gcd (and thus lcm) by finding the prime factorization of the
two integers.
However, the only algorithms known for doing this are exponential (indeed, computer
security depends on this).
We can, however, compute the gcd in polynomial time using Euclid’s Algorithm.
Consider finding the gcd(184, 1768). Dividing the large by the smaller, we get that
1768 = 184 · 9 + 112
Using algebra, we can reason that any divisor of 184 and 1768 must also be a divisor of
the remainder, 112. Thus,
gcd(184, 1768) = gcd(184, 112)
Continuing with our division we eventually get that
gcd(184, 1768) = gcd(184, 112)
= gcd(112, 72)
= gcd(72, 40)
= gcd(40, 32)
= gcd(32, 8) = 8
This concept is formally stated in the following Lemma.
Lemma 7. Let a = bq + r, a, b, q, r ∈ Z, then
gcd(a, b) = gcd(b, r)
73
5 Divide & Conquer Style Algorithms
Input : Integers, a, b ∈ Z+
Output : gcd(a, b)
1 while b 6= 0 do
2 t←b
3 b ← a mod t
4 a←t
5 end
6 output a
Algorithm 29: Euclid’s Simple GCD Algorithm
Analysis:
• Number of iterations is dependent on the nature of the input, not just the input
size
• Generally, we’re interested in the worst case behavior
• Number of iterations is maximized when the reduction in b (line 3) is minimized
• Reduction is minimized when b is minimal; i.e. b = 2
• Thus, after at most n iterations, b is reduced to 1 (0 on the next iteration), so:
b
=1
2n
• The number of iterations, n = log b
The algorithm we present here is actually the Extended Euclidean Algorithm. It keeps
track of more information to find integers such that the gcd can be expressed as a linear
combination.
Theorem 4. If a and b are positive integers, then there exist integers s, t such that
gcd(a, b) = sa + tb
74
5.4 Euclid’s GCD Algorithm
Input : Two positive integers a, b.
Output : r = gcd(a, b) and s, t such that sa + tb = gcd(a, b).
1 a0 = a, b0 = b
2 t0 = 0, t = 1
3 s0 = 1, s = 0
4 q = b ab00 c
5 r = a0 − qb0
6 while r > 0 do
7 temp = t0 − qt
8 t0 = t, t = temp
9 temp = s0 − qs
10 s0 = s, s = temp
11 a0 = b 0 , b 0 = r
12 q = b ab00 c, r = a0 − qb0
13 if r > 0 then
14 gcd = r
15 end
16 end
17 output gcd, s, t
Algorithm 30: ExtendedEuclideanAlgorithm
a0 b0 t0 t s0 s q r
27 58 0 1 1 0 0 27
58 27 1 0 0 1 2 4
27 4 0 1 1 -2 6 3
4 3 1 -6 -2 13 1 1
3 1 -6 7 13 -15 3 0
Therefore,
gcd(27, 58) = 1 = (−15)27 + (7)58
Example:
Compute gcd(25480, 26775) and find s, t such that
gcd(25480, 26775) = 25480s + 26775t
75
5 Divide & Conquer Style Algorithms
a0 b0 t0 t s0 s q r
25480 26775 0 1 1 0 0 25480
26775 25480 1 0 0 1 1 1295
25480 1295 0 1 1 -1 19 875
1295 875 1 -19 -1 20 1 420
875 420 -19 20 20 -21 2 35
420 35 20 -59 -21 62 12 0
Therefore,
gcd(25480, 26775) = 35 = (62)25480 + (−59)26775
5.5 Peasant Multiplication
5.6 Karatsuba Multiplication
Straightforward multiplication of two n-bit integers, a, b requires O(n2 ) multiplications
and O(n) additions.
Observe:
47 · 51 = (4 × 101 + 7 × 100 ) · (5 × 101 + 1 × 100 )
Requires 4 multiplications; we can rewrite using the FOIL rule:
47 · 51 = (4 · 5) × 102 + (4 · 1 + 7 · 5) × 101 + (7 · 1) × 100
But observe, the inner multiplication:
(4 · 1 + 7 · 5) = (4 + 7) · (1 + 5) − (4 · 5) − (7 · 1)
In general we have that
(a + b)(c + d) = ac + ad + bc + bd
But we’re interested in ad + bc:
ad + bc = (a + b)(c + d) − ac − bd
which only requires 3 multiplications.
In general given a, b, split them into four sub numbers:
a = a1 · 10n/2 + a0
b = b1 · 10n/2 + b0
76
5.6 Karatsuba Multiplication
That is, a1 are the higher order bits and a0 are the lower order bits.
a · b = a1 · 10n/2 + a0 · b1 · 10n/2 + b0
= a1 · b1 · 10n + (a n/2
1 · b0 + a0 · b1 ) · 10 + a0 · b0
= a1 · b1 · 10 + (a1 + a0 )(b1 + b0 ) − a1 · b1 − a0 · b0 · 10n/2 + a0 · b0
n
Observations:
• Due to Karatsuba (1960)
• Powers of 10 are a simple shift
• If a, b are not powers of two, we can “pad” them out with leading zeros
• Three recursive calls to compute the multiplication
• Overhead in recursion/book keeping only makes this better for large numbers
(thousands of digits)
n
M (n) = 3 · M +0
2
By the Master Theorem:
M (n) ∈ Θ(nlog2 3 )
As log2 (3) = 1.5849625 . . ., this is an asymptotic improvement over the naive multiplica-
tion.
Example:
a · b = 72, 893, 424 · 33, 219, 492
a1 = 7, 289
a0 = 3, 424
b1 = 3, 321
b0 = 9, 492
Then (after recursion):
a1 · b 1 = 24, 206, 769
a0 · b 0 = 32, 500, 608
x = a1 + a0 = 10, 719
y = b1 + b0 = 12, 813
x·y = 137, 342, 547
Altogether:
a · b = 24, 206, 769 · 108 + (137, 342, 547 − 24, 206, 769 − 32, 500, 608) · 104 + 32, 500, 608
= 2, 421, 483, 284, 200, 608
Other, similar schemes:
• Toom-Cook (1963): O(n1.465 )
77
5 Divide & Conquer Style Algorithms
• Schönhage-Strassen (1968, FFT): O(n log n log log n)
∗
• Fürer (2007): O(n log n2log n ) (iterated log function)
5.7 Strassen’s Matrix Multiplication
Matrix multiplication is well defined (we restrict our examination to square n × n
matrices); C = A · B where
n−1
X
ci,j = ai,k · bk,j
k=0
This constitutes Θ(n3 ) additions/multiplications.
Rather than the brute-force approach, we can use the following formulas:
c00 c01 a00 a01 b00 b01
=
c10 c11 a10 a11 b10 b11
m1 + m4 − m5 + m7 m3 + m5
=
m2 + m4 m1 + m3 − m2 + m6
where
m1 = (a00 + a11 ) · (b00 + b11 )
m2 = (a10 + a11 ) · b00
m3 = a00 · (b01 − b11 )
m4 = a11 · (b10 − b00 )
m5 = (a00 + a01 ) · b11
m6 = (a10 − a00 ) · (b00 + b01 )
m7 = (a01 − a11 ) · (b10 + b11 )
The number of multiplications and additions for a 2 × 2 matrix using Strassen’s formulas
are 7 and 18 respectively. Contrast with 8 and 4 operations for the brute force method.
We’re doing a lot more in terms of additions, and only slightly less in terms of multiplications–
but it turns out that this is still asymptotically better than the brute force approach.
Strassen’s formulas can be generalized to matrices rather than just scalars
C00 C01 A00 A01 B00 B01
=
C10 C11 A10 A11 B10 B11
78
5.7 Strassen’s Matrix Multiplication
M1 + M4 − M5 + M7 M3 + M5
=
M2 + M4 M1 + M3 − M2 + M6
Where Mi are matrices defined just as the scalars were.
There is an implicit assumption in the general formulas: that a matrix can be evenly
divided into four equal parts, forcing us to consider matrices of size 2k × 2k .
In fact, any sized matrix can be evaluated using the formulas–the trick is to “pad-out”
the matrix with zero entries to get a 2k × 2k matrix.
Say we have an n × n matrix. We have to make 7 recursive calls to Strassen’s algorithm
to evaluate a term in each of the M1 , . . . , M7 formulas. Each call halves the size of the
2
input ( n4 = n2 ). When a matrix reaches a size of 1, no multiplications are necessary. This
suggests the following recurrence:
n
M (n) = 7M , M (1) = 0
2
Working this recurrence relation gives us
M (n) = nlog2 7 ≈ n2.807
For additions, the same 7 matrices of size n/2 must be recursively dealt with while 18
additions must be made on matrices of size n/2, so
A(n) = 7A(n/2) + 18(n/2)2 , A(1) = 0
By the Master Theorem, A(n) ∈ Θ(nlog2 7 ).
Is this better than the brute force method?
Multiplications Additions
Brute-Force Θ(n3 ) Θ(n3 )
Strassen’s Θ(nlog2 7 ) Θ(nlog2 7 )
Table 5.1: Relative complexity for two matrix multiplication algorithms
Many other algorithms have been discovered with the best being Winogard’s algorithm
at O(n2.375477 ).
More recently this was improved (analytically) to O(n2.3727 ) (Williams 2012 [15])
However, the overhead and multiplicative constants for these algorithms make them
unfeasible for even moderately sized matrices.
79
5 Divide & Conquer Style Algorithms
5.8 Closest Pair Revisited
Without loss of generality, we can assume that the given point set, P = {p1 =
(x1 , y1 ), . . . , pn = (xn , yn )} are sorted in ascending order with respect to x coordinates,
x1 ≤ x2 ≤ . . . ≤ xn .
Then we have the following divide and conquer strategy:
• Divide: Partition the points into two subsets:
P1 = {(x1 , y1 ), (x2 , y2 ), . . . (xn/2 , yn/2 )}
P2 = {(xn/2+1 , yn/2+1 ), . . . (xn , yn )}
• Conquer: Find the two closest pair of points in each set P1 , P2 recursively
• Combine: We take the minimum distance between the two subproblems, d1 , d2 ;
however, we must also check points with distances that cross our dividing line
From the dividing line, we need only check points in two regions: [y − d, y + d] where
d = min{d1 , d2 }. For each point, one needs only check, at most 6 points (see figure 4.6,
p149).
Such a check takes linear time–there are n/2 points (we need only check C1 ) and up to 6
distance calculations each.
This suggests the usual recurrence,
n
T (n) = 2T + Θ(n)
2
which by the Master Theorem Θ(n log n).
5.9 Convex Hull Revisited
Levitin describes a “Quick Hull” algorithm. Here we present a different divide & conquer
approach. Yet another approach is the Graham Scan (a “walk” approach).
Basic Idea:
1. Divide: partition into two sets A, B according to a lower/upper half with respect
to the x coordinate
2. Conquer: Compute the convex hull of A and B
3. Combine: Combine into a larger convex hull by computing the upper/lower
tangents of each and throwing out all internal points (O(n) operation)
80
5.10 Fast Fourier Transform
Combine procedure:
• Starting with the right-most point a of the lower hull and the left-most point of
the upper hull
• Iterate through the hull points clockwise in the lower hull and counter clockwise on
the upper partition, alternating between the two
• Repeat until the tangent ab has been found
• Tangent has been found when the “next” point in each hull is contained in the new
hull
• Only required to look at the immediate next point (if the next point is above or
below, stop)
• Repeat the process for the upper tangent
5.10 Fast Fourier Transform
• Article: http://blog.revolutionanalytics.com/2014/01/the-fourier-transform-explained-i
html
• Nice Visualization: http://bbrennan.info/fourier-work.html
5.11 Exercises
Exercise 5.1. Implement Strassen’s algorithm in the high-level programming language
of your choice.
Exercise 5.2. Consider the following problem: Given an array of sorted integers and
two values l, u (lower and upper bounds), find the indices i, j such that l ≤ A[k] ≤ u for
all k, i ≤ k ≤ j. Solve this problem by developing an O(log n) algorithm. Give good
pseudocode and analyze your algorithm.
Exercise 5.3. Adaptive Quadrature is a numerical method for approximating the definite
integral of a function f (x):
Z b
f (x) dx
a
It works as follows: given a, b and a tolerance τ , it approximates the integral using some
method. It then computes an error bound for that method and if the error is less than τ
then the estimation is used. If the error is greater than τ , then the midpoint m = a+b 2
is
τ
used to recursively integrate on the intervals [a, m] and [m, b], updating τ to 2 .
81
5 Divide & Conquer Style Algorithms
For our purposes, use Simpson’s rule to approximate:
Z b
b−a a+b
f (x) dx ≈ f (a) + 4f + f (b)
a 6 2
which has a known error bound of:
5
1 b−a
f (4) (ξ) ,
90 2
Write good pseudocode for this algorithm and analyze its complexity.
Exercise 5.4. Let A be a zero-index array that is sorted. Now suppose that someone
comes along and cyclicly “shifts” the contents of A so that the contents are moved some
number of positions, with those at the end being shifted back to the beginning. For
example, if the array contained the elements 2, 6, 10, 20, 30, 40 and it were shifted by
3 positions, the resulting array would be 20, 30, 40, 2, 6, 10 (in general, you will not be
given how many positions the array has been shifted).
Design an efficient (that is an O(log n)) algorithm that can compute the median element
of A. Give good pseudocode and a short explanation. Analyze your algorithm.
82
6 Linear Systems
6.1 Introduction
TODO
6.2 Solving Linear Systems
A system of linear equations is a collection of n equalities in n unknowns. An example:
3x + 2y = 35
x + 3y = 27
We are interested in solving such systems algorithmically. A generalized form:
a11 x1 + a12 x2 + · · · + a1n xn = b1
a21 x1 + a22 x2 + · · · + a2n xn = b2
..
.
an1 x1 + an2 x2 + · · · + ann xn = bn
In general, we want to eliminate variables until we have 1 equation in 1 unknown.
The goal of Guassian Elimination is to transform a system of linear equations in n
unknowns to an equivalent, upper-triangular system.
An upper triangular system is one in which everything below and to the left of the
diagonal on the matrix is zero. In general we would want something like
a011 x1 + a012 x2 + . . . + a01n xn = b01
a022 x2 + . . . + a02n xn = b02
..
.
a0nn xn = b0n
A system of linear equations can be thought of as an n × n matrix with a variable
vector x = [x1 , x2 , . . . xn ] and a solution vector b = [b01 , b02 , . . . b0n ]. In other words, we can
83
6 Linear Systems
equivalently solve the matrix equation
Ax = b → A0 x = b0
An upper-triangular system is easier to solve because we can easily back-track (from the
bottom right) and solve each unknown. This is known as backward substitution (not to
be confused with the method for solving recurrence relations).
Now that we know how to back-solve, how do we take a given system and transform it
to an upper-triangular system? We use several elementary row operations:
• We can exchange any two rows of a system
1
• We can exchange any two columns of a system
• We can multiply an equation by a non-zero scalar
• We can add or subtract one equation with another (or a multiple thereof)
Such operations do not change the solution to the system.
Example
Consider the following system of linear equations:
3x1 − x2 + 2x3 = 1
4x1 + 2x2 − x3 = 4
x1 + 3x2 + 3x3 = 5
3 −1 2 1
4 2 −1 4
1 3 3 5
3 −1 2 1 ⇒
4 2 −1 4 subtract 4 times row 1
3
1 3 3 5 subtract 31 times row 1
3 −1 2 1 ⇒
0 10 − 11 8
3 3 3
0 10
3
7
3
14
3
subtract 1 times row 2
3 −1 2 1
0 10 − 11 8
3 3 3
0 0 6 2
1
If we do, however, we must make note that the variables have also been exchanged. For instance if we
exchange column i and column j then xi and xj have exchanged.
84
6.2 Solving Linear Systems
Back solving gives us:
1 7 1
~x = (x1 , x2 , x3 ) = , ,
2 6 3
Using elementary row operations, we can proceed as follows. Using a11 as a pivot, we
subtract a21 /a11 times the first equation from the second equation, and continue with
a31 /a11 from the third, etc. In general, we subtract ai1 /a11 times row 1 from the i-th row
for 2 ≤ i ≤ n. We repeat for a22 as the next pivot, etc.
Issues:
• It may be the case that we attempt to divide by zero, which is invalid. To solve
this problem, we can exchange a row!
• Though a divisor may not be zero, it could be so small that our quotient is too
large. We can solve this one of two ways– either scale the current row so that it
becomes 1 or we can again exchange rows. Specifically, the row with the largest
absolute value in the i-th row.
• The system may be indeterminant: there may be one or more free-variables (an
infinite number of solutions). Visually, both equations correspond to the same line
(intersecting at an infinite number of points); example:
x + 7y = 8
2x + 14y = 16
• The system may be inconsistent (there are no solutions). Visually the equations
correspond to two parallel lines that never intersect. Example:
x + 7y = 8
2x + 14y = 20
85
6 Linear Systems
Input : An n × n matrix A and a 1 × n vector b
Output : An augmented, upper triangluar matrix A0 preserving the solution to
A · ~x = ~b
1 A0 ← A|~b //augment
2 for i = 1, . . . n − 1 do
3 pivotrow ← i
4 for j = (i + 1), . . . , n do
5 if |A0 [j, i]| > |A0 [pivotrow, i]| then
6 pivotrow ← j
7 end
8 end
9 for k = i, . . . , (n + 1) do
10 swap(A0 [i, k], A0 [pivotrow, k])
11 end
12 for j = (i + 1), . . . , n do
0 [j,i]
13 t← AA0 [i,i]
14 for k = i, . . . , (n + 1) do
15 A0 [j, k] ← A0 [j, k] − A0 [i, k] · t
16 end
17 end
18 end
19 output A0
Algorithm 31: (Better) Gaussian Elimination
Input : An augmented, upper triangular (n × n + 1) matrix A
Output : A solution vector ~x = (x1 , . . . , xn )
1 for i = n, . . . 1 do
2 t ← A[i, n + 1] //set to b0i
3 for j = (i + 1), . . . , n do
4 t ← t − xj · A[i, j]
5 end
t
6 xi ← A[i,i]
7 end
8 output ~x
Algorithm 32: Back Solving
86
6.2 Solving Linear Systems
Analysis
The multiplication is our elementary operation. We thus have the following summation:
n−1 X
n X
n+1
X 1
C(n) = 1 ≈ n3 ∈ Θ(n3 )
i=1 j=i+1 k=i
3
n(n−1)
Backward substitution will work on the remaining entries making about 2
multipli-
cations and n divisions, so the asymptotic classification doesn’t change.
6.2.1 LU Decomposition
Performing Gaussian Elimination on a matrix A can give us to additional matrices:
• L – A lower-triangular matrix with 1’s on the diagonal and lower entries corre-
sponding to the row multiples used in Gaussian elimination.
• U – The upper-triangular matrix as a result of Gaussian elimination on A.
LU-Decomposition of the previous example:
1 0 0 3 −1 2
L = 43 1 0 U = 0 103
− 11
3
1
3
1 1 0 0 6
It is not hard to prove that
A=L·U
where L · U is the usual matrix multiplication. We observe then, that solving the system
Ax = b is equivalent to solving
LU~x = ~b
We do this by performing Gaussian Elimination, then we solve the system Ly = b where
y = U x then we solve U x = y.
Though this is essentially Gaussian elimination with a bit more work, it has the advantage
that once we have the decomposition L and U , we can solve for any vector b—a huge
advantage if we were to make changes to the ~b vector.
6.2.2 Matrix Inverse
The inverse of an n × n matrix A is another matrix A−1 such that
AA−1 = I
87
6 Linear Systems
where I is the identity matrix (1’s on the diagonal, 0’s everywhere else). The inverse of a
matrix is always unique. However not every matrix is invertible–such matrices are called
singular.
Recall that when using Gaussian elimination, the system may be inconsistent. That
is, no valid solution exists. One can prove that this is the case if and only if A is not
invertible which in turn is the case if and only if one of the rows is a linear combination
of another row.
Such a linear combination can be detected by running Gaussian Elimination on A. If
one of the diagonal entries is 0 then A is singular. Otherwise A is consistent.
Finding an inverse is important because it allows us an implicit division operation among
matrices2 :
A−1 Ax = A−1 b
x = A−1 b
We can use Gaussian Elimination to find a matrix’s inverse by solving
AX = I
where X are n2 unknowns in the inverse matrix. We do this by solving n systems of
linear equations with the same coefficient matrix A and a vector of unknowns xj (the
j-th column of X and a solution vector ej (the j-th column of the identity matrix):
Axj = ej
for 1 ≤ j ≤ n
Alternatively, you can adapt Gaussian Elimination to compute an inverse as follows.
Given A, you create a “super augmented” matrix,
AI
Where I is the n × n identity matrix, resulting in an n × 2n. Then you perform Gaussian
Elimination to make the left part upper triangular, then perform a Gaussian Elimination
for the upper part as well, then normalize each row so that each entry on the diagonal is
1 (this is actually Gauss-Jordan elimination). The resulting augmented matrix represents
I A−1
2
be careful: remember, matrix multiplication is not commutative in general
88
6.2 Solving Linear Systems
6.2.3 Determinants
The determinant of an n × n square matrix is a measure of its coefficients which has
several interpretations. With respect to linear transformations, it can quantify “area” or
“volume”.
The determinant has several equivalent definitions.
" n
#
X Y
det A = sgn(σ) Ai,σi
σ∈Sn i=1
where Sn is the symmetric group on n elements (the set of all permutations).
Alternatively, it can be recursively defined as:
n
X
det A = sj a1j det Aj
j=1
where
1 if j is odd
sj =
−1 if j is even
• a1j is the element in row 1 and column j
• Aj is the (n − 1) × (n − 1) matrix obtained by deleting row 1 and column j from A
• If n = 1 then det A = a11
Determinants are easy to compute for 2 × 2 matrices and even 3 × 3 matrices. There is
also an easy to remember visual way to do it.
If you can easily compute a determinant you can easily tell if a matrix is invertible or
not:
Theorem 5. An n × n matrix is invertible if and only if det A 6= 0
If you were to use the definition to compute a determinant, you would be summing n!
terms! Instead we can again use Gaussian elimination.
The determinant of an upper-triangular matrix is simply the product of the elements on its
diagonal. Thus we can use Gaussian elimination to transform A into an upper-triangular
matrix A0 and simply compute
Yn
det A0 = a0ii
i=1
Some things to keep in mind about determinants when using Gaussian elimination:
• Realize that det (A + B) 6= det A + det B
89
6 Linear Systems
• However, it is true that det AB = det A det B
• Also, det A = det AT
• If we multiply a row by a constant c, then det A = det A0 /c
• The value of det A does not change if we add a constant multiple of another row.
• If we exchange rows, the sign of det A flips
• The above hold for columns as well
Though not algorithmically sound (it is far too complex, cumbersome to program and
certainly not stable3 ) Cramer’s Rule does give us an explicit solution for a system of
linear equations:
det A1 det A2 det An
x1 = , x2 = , . . . xn =
det A det A det A
where Aj is the matrix obtained by replacing the j-th column of A by the vector b.
6.3 Linear Programming
A linear program is mathematical model for linear optimization problems.
• There is some objective function which you wish to maximize (or minimize, called
the dual )
• There are constraints that cannot be violated
• Both must be linear functions
• In general there can be n variables
• Objective: output the value of n variables that is both feasible and optimizes the
outcome
A linear program in standard form:
maximize ~c T · ~x
subject to Ax ≤ ~b
~x ≥ 0
Simplex Method: Example
The simplex method (Dantzig, 1947) is an old way of solving linear programs
• Start at an arbitrary point
3
stable in the sense that floating point operations are prone to error and that error propagates
90
6.3 Linear Programming
• Travel along the feasible region to the next vertex (in n-space)
n
• In general, the optimal solution lies on the polytope with m extreme points (n
variables, m constraints)
• May have an exponential number of such critical points
• Choice of direction: that which improves the solution the most (the direction in
which the variable has the largest partial derivative)
• In general, pivot choices may lead to aberrant (exponential) behavior or could even
cycle
• Very fast in practice
• Polynomial-time algorithm (Khachiyan, 1979) and better interior-point method
(Karmarkar, 1984) among other improvements
maximize 4x + 3y
subject to x ≥ 0
y ≥ 0
x ≤ 9
2y + x ≤ 12
2y − x ≤ 8
(2, 5) y = − 43 x + 13.5
y ≤ 12 x + 4
y≥0 y = − 43 x + 7
(0, 4)
y ≤ − 21 x + 6
(9, 1.5) - optimal solution
y = − 43 x + 2
x≤9
(0, 0) (9, 0)
x≥0
Figure 6.1: Visualization of a Linear Program with two sub-optimal solution lines and
the optimal one.
91
6 Linear Systems
Point Value
(0, 0) 0
(0, 4) 12
(2, 5) 23
(9, 1.5) 40.5
(9, 0) 36
Table 6.1: Function values at several critical points
6.3.1 Formulations
We can transform and conquer several previously examined problems as follows: we can
reformulate the problem as an optimization problem (a linear program) and solve it with
the simplex method or other algorithm for LP.
Knapsack
The Knapsack problem can be formulated as a linear program as follows.
• We want to maximize the sum of the value of all items
• Subject to the total weight constraint
• But we want to constrain the fact that we can take an item or not
• We establish n indicator variables, x1 , . . . , xn ,
1 if item i is taken
xi
0 otherwise
n
X
maximize v i · xi
i=1
Xn
subject to w i · xi ≤ W
i=1
xi ∈ {0, 1} 1 ≤ i ≤ n
• This is known as an Integer Linear Program
• Some constraints are further required to be integers
• In general, ILP is NP-complete
• A linear program can be relaxed :
• Integer constraints are relaxed and allowed to be continuous
92
6.3 Linear Programming
An example: the fractional knapsack problem:
n
X
maximize v i · xi
i=1
Xn
subject to w i · xi ≤ W
i=1
0 ≤ xi ≤ 1 1 ≤ i ≤ n
• This formulation doesn’t solve the original 0-1 Knapsack problem, it could come
up with a solution where a fraction of an item is taken
Assignment Problem & Unimodularity
Recall the assignment problem; we can formulate an ILP as follows
XX
minimize ci,j · xi,j
i j
X
subject to xi,j = 1 ∀j
i
xi,j ∈ {0, 1} ∀i, j
Where xi,j is an indicator variable for assigning task i to person j (with associated cost
ci,j )
This program has integer constraints (ILP) and is, in general, NP-complete. We could
relax it:
XX
minimize ci,j · xi,j
i j
X
subject to xi,j = 1 ∀j
i
0 ≤ xi,j ≤ 1 ∀i, j
• It doesn’t make much sense to assign half a task to a quarter of a person (or maybe
it does!)
• However, it turns out that certain problems are unimodular
• A matrix is unimodular if it is:
93
6 Linear Systems
– square
– integer entries
– has a determinant that is −1, 1
• Further it is totally unimodular if every square non-singular submatrix is unimodular
(thus determinant is 0, −1, 1)
• Fact: if an ILP’s constraint matrix is totally unimodular, then its optimal solution
is integral
• If an optimum exists, it is composed of integers: ~x = (x1 , . . . , xn ), xi ∈ Z
• Fact: The assignment problem is totally unimodular
• Consequence: the relaxation of the assignment problem’s ILP will have the same
optimal solution as the original ILP!
94
6.4 Exercises
6.4 Exercises
Exercise 6.1. Let T be a binary search tree and let v1 , v2 , . . . , vn be a preorder traversal
of its nodes. Design an algorithm that reconstructs the tree T given v1 , . . . , vn .
Exercise 6.2. Diagram each step of inserting the following keys into an initially empty
AVL tree. Clearly indicate which type of rotations occur.
10, 15, 30, 25, 5, 20, 35, 40, 45, 100, 50
Exercise 6.3. Diagram each step of inserting the following keys into an initially empty
2-3 tree. Clearly indicate which type of promotions and node transformations occur.
10, 15, 30, 25, 5, 20, 35, 40, 45, 100, 50
Exercise 6.4. Consider the following questions regarding AVL trees.
(a) Considering adding the keys a, b, c to an empty AVL tree. How many possible orders
are there to insert them and how many of those do not result in any rotations?
(b) Considering adding keys a, b, c, d, e, f, g into an AVL tree. How many ways could
you insert these keys into an empty AVL tree such that no rotations would result?
(c) Can you generalize this for keys k1 , . . . , km where m = 2n − 1?
Exercise 6.5. Suppose we insert 1, 2, . . . , 25 into an initially empty AVL tree. What
will be the depth of the resulting tree?
Exercise 6.6. Suppose we insert 1, 2, . . . , 25 into an initially empty 2-3 tree. What will
be the depth of the resulting tree?
Exercise 6.7. Write an algorithm (give good pseudocode and a complete analysis) that
inverts a binary search tree so that for each node with key k, all keys in the left-sub-tree
are greater than k and all nodes in the right-sub-tree are less than k. Thus, an in-order
traversal of an inverted tree will produce a key ordering in non-increasing order instead.
An example of a binary search tree and its inversion is presented in Figure 6.2. Your
algorithm must be efficient and should not create a new tree.
Exercise 6.8. Implement Gaussian Elimination in the high-level programming language
of your choice. Use your code to solve matrix problems: finding a matrix inverse, solving
a system of linear equations, etc.
95
6 Linear Systems
20 20
10 30 30 10
5 15 25 35 35 25 15 5
(a) Binary Search Tree (b) Inverted Binary Search Tree
Figure 6.2: Binary search tree and its inversion.
96
7 Trees
7.1 Introduction
Motivation: we want a data structure to store elements that offers efficient, arbitrary
retrieval (search), insertion, and deletion.
• Array-based Lists
– O(n) insertion and deletion
– Fast index-based retrieval
– Efficient binary search if sorted
• Linked Lists
– Efficient, O(1) insert/delete for head/tail
– Inefficient, O(n) arbitrary search/insert/delete
– Efficient binary search not possible without random access
• Stacks and queues are efficient, but are restricted access data structures
• Possible alternative: Trees
• Trees have the potential to provide O(log n) efficiency for all operations
7.2 Definitions & Terminology
• A tree is an acyclic graph
• For our purposes: a tree is a collection of nodes (that can hold keys, data, etc.)
that are connected by edges
• Trees are also oriented : each node has a parent and children
• A node with no parents is the root of the tree, all child nodes are oriented downward
• Nodes not immediately connected can have an ancestor, descendant or cousin
relationship
97
7 Trees
• A node with no children is a leaf
• A tree such that all nodes have at most two children is called a binary tree
• A binary tree is also oriented horizontally: each node may have a left and/or a
right child
• Example: see Figure 7.1
• A path in a tree is a sequence nodes connected by edges
• The length of a path in a tree is the number of edges in the path (which equals the
number of nodes in the path minus one)
• A path is simple if it does not traverse nodes more than once (this is the default
type of path)
• The depth of a node u is the length of the (unique) path from the root to u
• The depth of the root is 0
• The depth of a tree is the maximal depth of any node in the tree (sometimes the
term height is used)
• All nodes of the same depth are considered to be at the same level
• A binary tree is complete (also called full or perfect) if all nodes are present at all
levels 0 up to its depth d
• A sub-tree rooted at a node u is the tree consisting of all descendants with u
oriented as the root
a
b c
d e f
g h i j k
l m n o p q
r
Figure 7.1: A Binary Tree
Properties:
• In a tree, all nodes are connected by exactly one unique path
• The maximum number of nodes at any level k is 2k
98
7.3 Tree Traversal
• Thus, the maximum number of nodes n for any binary tree of depth d is:
d
X
n = 20 + 21 + 22 + · · · + 2d−1 + 2d = 2k = 2d+1 − 1
k=0
• Given a full binary tree with n nodes in it has depth:
d = log (n + 1) − 1
• That is, d = O(log n)
Motivation: if we can create a tree-based data structure with operations proportional to
its depth, then we could potentially have a data structure that allows retrieval/search,
insertion, and deletion in O(log n)-time.
7.3 Tree Traversal
• Given a tree, we need a way to enumerate elements in a tree
• Many algorithms exist to iterate over the elements in a tree
• We’ll look at several variations on a depth-first-search
7.3.1 Preorder Traversal
• A preorder traversal strategy visits nodes in the following order: root; left-sub-tree;
right-sub-tree
• An example traversal on the tree in Figure 7.1:
a, b, d, g, l, m, r, h, n, e, i, o, c, f, j, p, q, k
• Applications:
– Building a tree, traversing a tree, copying a tree, etc.
– Efficient stack-based implementation
– Used in prefix notation (polish notation); used in languages such as Lisp/Scheme
Simulation of a stack-based preorder traversal of the binary tree in Figure 7.1.
99
7 Trees
push a push m (enter loop) print i push q
push l pop, node = h push p
(enter loop) print g push n (enter loop) print j
pop, node = a (no push) pop, node = o
push c (enter loop) print h (no push) (enter loop)
push b pop, node = l (no push) pop, node = p
print a (no push) (enter loop) print o (no push)
(no push) pop, node = n (no push)
(enter loop) print l (no push) (enter loop) print p
pop, node = b (no push) pop, node = c
push e (enter loop) print n push f (enter loop)
push d pop, node = m (no push) pop, node = q
print b (no push) (enter loop) print c (no push)
push r pop, node = e (no push)
(enter loop) print m push i (enter loop) print q
pop, node = d (no push) pop, node = f
push h (enter loop) print e push k (enter loop)
push g pop, node = r push j pop, node = k
print d (no push) (enter loop) print f (no push)
(no push) pop, node = i (no push)
(enter loop) print r (no push) (enter loop) print k
pop, node = g push o pop, node = j
7.3.2 Inorder Traversal
• An inorder traversal strategy visits nodes in the following order: left-sub-tree; root;
right-sub-tree
• An example traversal on the tree in Figure 7.1:
l, g, r, m, d, h, n, b, e, o, i, a, c, p, j, q, f, k
• Applications:
– Enumerating elements in order in a binary search tree
– Expression trees
Simulation of a stack-based inorder traversal of the binary tree in Figure 7.1.
(enter loop, u = a) update u = g (enter loop, u=g update u = null
push a u = null) process g
update u = b (enter loop, u = g) pop l, update update u = m (enter loop,
push g u=l u = null)
(enter loop, u = b) update u = l process l (enter loop, u = m) pop r, update
push b update u = null push m u=r
update u = d (enter loop, u = l) update u = r process r
push l (enter loop, update u = null
(enter loop, u = d) update u = null u = null) (enter loop, u = r)
push d pop g, update push r (enter loop,
100
7.3 Tree Traversal
u = null) process n u=o u=c update u = null
pop m, update update u = null process o process c
u=m update u = null update u = f (enter loop,
process m (enter loop, u = null)
update u = null u = null) (enter loop, (enter loop, u = f ) pop q, update
pop b, update u = null) push f u=q
(enter loop, u=b pop i, update update u = j process q
u = null) process b u=i update u = null
pop d, update update u = e process i (enter loop, u = j)
u=d update u = null push j (enter loop,
process d (enter loop, u = e) update u = p u = null)
update u = h push e (enter loop, pop f , update
update u = null u = null) (enter loop, u = p) u=f
(enter loop, u = h) pop i, update push p process f
push h (enter loop, u=i update u = null update u = k
update u = null u = null) process i
pop e, update update u = null (enter loop, (enter loop, u = k)
(enter loop, u=e u = null) push k
u = null) process e (enter loop, pop p, update update u = null
pop h, update update u = i u = null) u=p
u=h pop a, update process p (enter loop,
process h (enter loop, u = i) u=a update u = null u = null)
update u = n push i process a pop k, update
update u = o update u = c (enter loop, u=k
(enter loop, u = n) u = null) process k
push n (enter loop, u = o) (enter loop, u = c) pop j, update update u = null
update u = null push o push c u=j
update u = null update u = null process j (done)
(enter loop, update u = q
u = null) (enter loop, (enter loop,
pop n, update u = null) u = null) (enter loop, u = q)
u=n pop o, update pop c, update push q
7.3.3 Postorder Traversal
• A postorder traversal strategy visits nodes in the following order: left-sub-tree;
right-sub-tree; root
• An example traversal on the tree in Figure 7.1:
l, r, m, g, n, h, d, o, i, e, b, p, q, j, k, f, c, a
• Applications:
– Topological sorting
– Destroying a tree when manual memory management is necessary (roots are
the last thing that get cleaned up)
– Reverse polish notation (operand-operand-operator, unambiguous, used in old
HP calculators)
101
7 Trees
– PostScript (Page Description Language)
Simulation of a stack-based postorder traversal of the binary tree in Figure 7.1:
prev = null update prev = l (enter loop) (null.rightChild = (n))
push a update curr = (m) process n
(enter loop) check: update prev = n
(enter loop) update curr = (g) prev.rightChild = curr
update curr = (a) check: (null.rightChild = (m)) (enter loop)
check: prev = null prev.rightChild = curr process m update curr = (h)
push (b) (null.rightChild = (g)) update prev = m check:
update prev = a check: prev.rightChild = curr
curr.lef tChild = prev (enter loop) (null.rightChild = (h))
(enter loop) ((l).lef tChild = (l)) update curr = (g) process h
update curr = (b) push (m) check: update prev = h
check: update prev = g prev.rightChild = curr
prev.lef tChild = curr (null.rightChild = (g)) (enter loop)
((b).lef tChild = (b)) (enter loop) process g update curr = (d)
push (d) update curr = (m) update prev = g check:
update prev = b check: prev.rightChild = curr
prev.rightChild = curr (enter loop) ((n).rightChild = (d))
(enter loop) ((m).rightChild = (m)) update curr = (d) process d
update curr = (d) push (r) check: update prev = d
check: update prev = m prev.rightChild = curr
prev.lef tChild = curr ((m).rightChild = (d)) (enter loop)
((d).lef tChild = (d)) (enter loop) check: update curr = (b)
push (g) update curr = (r) curr.lef tChild = prev check:
update prev = d check: ((g).lef tChild = (g)) prev.rightChild = curr
prev.lef tChild = curr push (h) ((h).rightChild = (b))
(enter loop) ((r).lef tChild = (r)) update prev = d check:
update curr = (g) (noop) curr.lef tChild = prev
check: update prev = r (enter loop) ((d).lef tChild = (d))
prev.lef tChild = curr update curr = (h) push (e)
((g).lef tChild = (g)) (enter loop) check: update prev = b
push (l) update curr = (r) prev.rightChild = curr
update prev = g check: ((h).rightChild = (h)) (enter loop)
prev.rightChild = curr push (n) update curr = (e)
(enter loop) (null.rightChild = (r)) update prev = h check:
update curr = (l) process r prev.rightChild = curr
check: update prev = r (enter loop) ((e).rightChild = (e))
prev.lef tChild = curr update curr = (n) push (i)
((l).lef tChild = (l)) (enter loop) check: update prev = e
(noop) update curr = (m) prev.rightChild = curr
update prev = l check: ((n).rightChild = (n)) (enter loop)
prev.rightChild = curr (noop) update curr = (i)
(enter loop) (null.rightChild = (m)) update prev = n check:
update curr = (l) check: prev.rightChild = curr
check: curr.lef tChild = prev (enter loop) ((i).rightChild = (i))
prev.rightChild = curr ((r).lef tChild = (r)) update curr = (n) push (o)
(null.rightChild = (l)) update prev = m check: update prev = i
process l prev.rightChild = curr
102
7.3 Tree Traversal
(enter loop) process b update curr = (p) check:
update curr = (o) update prev = b check: curr.lef tChild = prev
check: prev.rightChild = curr ((j).lef tChild = (j))
prev.lef tChild = curr (enter loop) (null.rightChild = (p)) push (k)
((o).lef tChild = (o)) update curr = (a) process p update prev = f
(noop) check: update prev = p
update prev = o prev.rightChild = curr (enter loop)
((e).rightChild = (a)) (enter loop) update curr = (k)
(enter loop) check: update curr = (j) check:
update curr = (o) curr.lef tChild = prev check: prev.rightChild = curr
check: ((b).lef tChild = (b)) prev.rightChild = curr ((k).rightChild = (k))
prev.rightChild = curr push (c) (null.rightChild = (j)) (noop)
(null.rightChild = (o)) update prev = a check: update prev = k
process o curr.lef tChild = prev
update prev = o (enter loop) ((p).lef tChild = (p)) (enter loop)
update curr = (c) push (q) update curr = (k)
(enter loop) check: update prev = j check:
update curr = (i) prev.rightChild = curr prev.rightChild = curr
check: ((c).rightChild = (c)) (enter loop) (null.rightChild = (k))
prev.rightChild = curr push (f ) update curr = (q) process k
(null.rightChild = (i)) update prev = c check: update prev = k
check: prev.rightChild = curr
curr.lef tChild = prev (enter loop) ((q).rightChild = (q)) (enter loop)
((o).lef tChild = (o)) update curr = (f ) (noop) update curr = (f )
update prev = i check: update prev = q check:
prev.rightChild = curr prev.rightChild = curr
(enter loop) ((f ).rightChild = (f )) (enter loop) (null.rightChild = (f ))
update curr = (i) push (j) update curr = (q) process f
check: update prev = f check: update prev = f
prev.rightChild = curr prev.rightChild = curr
(null.rightChild = (i)) (enter loop) (null.rightChild = (q)) (enter loop)
process i update curr = (j) process q update curr = (c)
update prev = i check: update prev = q check:
prev.lef tChild = curr prev.rightChild = curr
(enter loop) ((j).lef tChild = (j)) (enter loop) ((k).rightChild = (c))
update curr = (e) push (p) update curr = (j) process c
check: update prev = j check: update prev = c
prev.rightChild = curr prev.rightChild = curr
(null.rightChild = (e)) (enter loop) (null.rightChild = (j)) (enter loop)
process e update curr = (p) process j update curr = (a)
update prev = e check: update prev = j check:
prev.lef tChild = curr prev.rightChild = curr
(enter loop) ((p).lef tChild = (p)) (enter loop) ((f ).rightChild = (a))
update curr = (b) (noop) update curr = (f ) process a
check: update prev = p check: update prev = a
prev.rightChild = curr prev.rightChild = curr
((i).rightChild = (b)) (enter loop) ((q).rightChild = (f ))
103
7 Trees
7.3.4 Breadth-First Search Traversal
• Breadth-First Search (BFS) traversal is a general graph traversal strategy that
explores local or close nodes first before traversing “deeper” into the graph
• When applied to an oriented binary tree, BFS explores the tree level-by-level
(top-to-bottom, left-to-right)
7.3.5 Implementations & Data Structures
• Reference based implementation: TreeNode<T>
– Owns (through composition) references to: leftChild, rightChild, parent
– Can use either sentinel nodes or null to indicate missing children and parent
• BinaryTree<T> owns a root
• SVN examples: unl.cse.bst
Preorder Implementations
Input : A binary tree node u
Output : A preorder traversal of the nodes in the subtree rooted at u
1 print u
2 preOrderTraversal(u → lef tChild)
3 preOrderTraversal(u → rightChild)
Algorithm 33: Recursive Preorder Tree Traversal
Stack-based implementation:
• Initially, we push the tree’s root into the stack
• Within a loop, we pop the top of the stack and process it
• We need to push the node’s children for future processing
• Since a stack is LIFO, we push the right child first.
104
7.3 Tree Traversal
Input : A binary tree, T
Output : A preorder traversal of the nodes in T
1 S ← empty stack
2 push T ’s root onto S
3 while S is not empty do
4 node ← S.pop
5 push node’s right-child onto S
6 push node’s left-child onto S
7 process node
8 end
Algorithm 34: Stack-based Preorder Tree Traversal
Inorder Implementation
Stack-based implementation:
• The same basic idea: push nodes onto the stack as you visit them
• However, we want to delay processing the node until we’ve explored the left-sub-tree
• We need a way to tell if we are visiting the node for the first time or returning
from the left-tree exploration
• To achieve this, we allow the node to be null
• If null, then we are returning from a left-tree exploration, pop the top of the stack
and process (then push the right child)
• If not null, then we push it to the stack for later processing, explore the left child
105
7 Trees
Input : A binary tree, T
Output : An inorder traversal of the nodes in T
1 S ← empty stack
2 u ← root
3 while S is not empty Or u 6= null do
4 if u 6= null then
5 push u onto S
6 u ← u.leftChild
7 else
8 u ← S.pop
9 process u
10 u ← u.rightChild
11 end
12 end
Algorithm 35: Stack-based Inorder Tree Traversal
Postorder Implementation
Stack-based implementation:
• Same basic ideas, except that we need to distinguish if we’re visiting the node for
the first time, second time or last (so that we can process it)
• To achieve this, we keep track of where we came from: a parent, left, or right node
• We keep track of a previous and a current node
106
7.3 Tree Traversal
Input : A binary tree, T
Output : A postorder traversal of the nodes in T
1 S ← empty stack
2 prev ← null
3 push root onto S
4 while S is not empty do
5 curr ← S.peek
6 if prev = null Or prev.leftChild = curr Or prev.rightChild = curr
then
7 if curr.leftChild 6= null then
8 push curr.leftChild onto S
9 else if curr.rightChild 6= null then
10 push curr.rightChild onto S
11 end
12 else if curr.leftChild = prev then
13 if curr.rightChild 6= null then
14 push curr.rightChild onto S
15 end
16 else
17 process curr
18 S.pop
19 end
20 prev ← curr
21 end
Algorithm 36: Stack-based Postorder Tree Traversal
107
7 Trees
BFS Implementation
Input : A binary tree, T
Output : A BFS traversal of the nodes in T
1 Q ← empty queue
2 enqueue T ’s root into Q
3 while Q is not empty do
4 node ← Q.dequeue
5 enqueue node’s left-child onto Q
6 enqueue node’s right-child onto Q
7 print node
8 end
Algorithm 37: Queue-based BFS Tree Traversal
Tree Walk Implementations
• Simple rules based on local information: where you are and where you came from
• No additional data structures required
• Traversal is a “walk” around the perimeter of the tree
• Can use similar rules to determine when the current node should be processed to
achieve pre, in, and postorder traversals
• Need to take care with corner cases (when current node is the root or children are
missing)
• Pseudocode presented Algorithm 38
7.3.6 Operations
Basic Operations:
• Search for a particular element/key
• Adding an element
– Add at most shallow available spot
– Add at a random leaf
– Add internally, shift nodes down by some criteria
108
7.3 Tree Traversal
Input : A binary tree, T
Output : A Tree Walk around T
1 curr ← root
2 prevT ype ← parent
3 while curr 6=null do
4 if prevT ype = parent then
//preorder: process curr here
5 if curr.lef tChild exists then
//Go to the left child:
6 curr ← curr.lef tChild
7 prevT ype ← parent
8 else
9 curr ← curr
10 prevT ype ← lef t
11 end
12 else if prevT ype = lef t then
//inorder: process curr here
13 if curr.rightChild exists then
//Go to the right child:
14 curr ← curr.rightChild
15 prevT ype ← parent
16 else
17 curr ← curr
18 prevT ype ← right
19 end
20 else if prevT ype = right then
//postorder: process curr here
21 if curr.parent = null then
//root has no parent, we’re done traversing
22 curr ← curr.parent
//are we at the parent’s left or right child?
23 else if curr = curr.parent.lef tChild then
24 curr ← curr.parent
25 prevT ype ← lef t
26 else
27 curr ← curr.parent
28 prevT ype ← right
29 end
30 end
31 end
109
Algorithm 38: Tree Walk based Tree Traversal
7 Trees
• Removing elements
– Removing leaves
– Removing elements with one child
– Removing elements with two children
Other Operations:
• Compute the total number of nodes in a tree
• Compute the total number of leaves in a tree
• Given an item or node, compute its depth
• Compute the depth of a tree
7.4 Binary Search Trees
Regular binary search trees have little structure to their elements; search, insert, delete
operations are still linear with respect to the number of tree nodes, O(n). We want a
data structure with operations proportional to its depth, O(d). To this end, we add
structure and order to tree nodes.
• Each node has an associated key
• Binary Search Tree Property: For every node u with key uk in T
1. Every node in the left-sub-tree of u has keys less than uk
2. Every node in the right-sub-tree of u has keys greater than uk
• Duplicate keys can be handled, but you must be consistent and not guaranteed to
be contiguous
• Alternatively: do not allow duplicate keys or define a key scheme that ensures a
total order
• Inductive property: all sub-trees are also binary search trees
• A full example can be found in Figure 7.2
7.4.1 Basic Operations
Observation: a binary search tree has more structure: the key in each node provides
information on where a node is not located. We will exploit this structure to achieve
O(log n) operations.
Search/retrieve
110
7.4 Binary Search Trees
50
40 60
20 45 90
10 24 49 80 95
2 15 28 48 75 85
12
Figure 7.2: A Binary Search Tree
• Goal: find a node (and its data) that matches a given key k
• Start at the node
• At each node u, compare k to u’s key, uk :
– If equal, element found, stop and return
– If k < uk , traverse to u’s left-child
– If k > uk , traverse to u’s right-child
• Traverse until the sub-tree is empty (element not found)
• Analysis: number of comparisons is bounded by the depth of the tree, O(d)
Input : A binary search tree, T , a key k
Output : The tree node u ∈ T whose key, uk matches k
1 u ← T ’s root
2 while u 6= φ do
3 if uk = k then
4 output u
5 end
6 else if uk > k then
7 u ← u’s left-child
8 else if uk < k then
9 u ← u’s left-child
10 end
11 output φ
Algorithm 39: Search algorithm for a binary search tree
111
7 Trees
Insert
• Insert new nodes as leaves
• To determine where it should be inserted: traverse the tree as above
• Insert at the first available spot (first missing child node)
• Analysis: finding the available location is O(d), inserting is just reference juggling,
O(1)
Delete
• Need to first find the node u to delete, traverse as above, O(d)
• If u is a leaf (no children): its safe to simply delete it
• If u has one child, then we can “promote” it to u’s spot (u’s parent will now point
to u’s child)
• If u has two children, we need to find a way to preserve the BST property
– Want to minimally change the tree’s structure
– Need the operation to be efficient
– Find the minimum element of the greater nodes (right sub-tree) or the maximal
element of the lesser nodes (left sub-tree)
– Such an element will have at most one child (which we know how to delete)
– Delete it and store off the key/data
– Replace u’s key/data with the contents of the minimum/maximum element
• Analysis:
– Search/Find: O(d)
– Finding the min/max: O(d)
– Swapping: O(1)
– In total: O(d)
• Examples illustrated in Figure 7.3
7.5 Balanced Binary Search Trees
In a Binary Search Tree (BST), insert, delete, and search operations are proportional to
the depth of the tree.
• The depth d of a tree is the length of the maximal path from the root to any leaf
112
7.5 Balanced Binary Search Trees
15 15
7 30 7 30
4 9 20 40 4 9 20 40
1 5 17 23 1 5 17 23
16
(a) A Binary Search Tree (b) Insertion of a new node (16) into a Binary
Search Tree
15 9
7 30 7 30
4 9 20 40 4 20 40
1 5 17 23 1 5 17 23
(c) Deletion of a node with two children (15). (d) Node 15 is replaced with the extremal node,
First step: find the maximum node in the left- preserving the BST property
sub-tree (lesser elements).
9 9
7 30 4 30
4 20 40 1 5 20 40
1 5 17 23 17 23
(e) Deletion a node with only one child (7). (f) Removal is achieved by simply promoting
the single child/subtree.
Figure 7.3: Binary Search Tree Operations. Figure 7.3(b) depicts the insertion of (16)
into the tree in Figure 7.3(a). Figures 7.3(c) and 7.3(d) depict the deletion of
a node (15) with two children. Figures 7.3(e) and 7.3(f) depict the deletion
of a node with only one child (7).
113
7 Trees
• Ideally, in a full/complete binary tree, d ∈ Θ(log n)
• In the worst case though, d = n − 1 ∈ Θ(n)
• Our goal is to maintain a BST’s balance so that the depth is always O(log n)
• Many different types of self-balancing binary search trees: 2-3 trees, 2-3-4 trees,
B-trees, AA trees, AVL trees, Red-Black Trees, Splay Trees, Scapegoat Trees,
Treaps, T-Trees,
7.5.1 AVL Trees
Let T be a binary tree with u a node in T . The height of u is the length of the longest
path from u to any descendant leaves in its left or right subtree. For example, in the
tree in Figure 7.4, the node containing 6 has height 2 (as the longest path is from 6 to
4 which has length 2). The node containing 8 has height 3 and the root has height 4.
Whereas the depth of a node is defined with respect to the root, the height of a node is
defined with respect to leaves. The height of a tree (or subtree) is the height of its root
node. By convention, the height of a single node tree is 0 and the height of an empty
tree is -1.
The balance factor of a node u is equal to the height of u’s left subtree minus the height
of its right subtree.
balance factor(u) = h(TL ) − h(TR )
A balance factor is a measure of how skewed or unbalanced a tree or subtree is. A
balance factor of 0 indicates that a node is balanced between its left/right subtree. A
“large” positive balance factor indicates a tree that is unbalanced to the left while a large
negative balance factor indicates a tree that is unbalanced to the right. A node’s balance
factor only quantifies how well balanced a tree is at that particular node, not overall.
That is, its a local property. Figure 7.4 depicts a BST with the balance factor indicated
for each node. Observe that for the node containing 4, the left and right subtree both
have a height of −1, so the balance factor is −1 − (−1) = 0. The balance factor of 8 is 3
since its left subtree has a height of 2 and its right subtree has a height of −1.
Definition 8. AVL Tree An AVL Tree is a binary search tree in which the balance factor
of every node’s left and right subtrees is 0, 1, or -1.
AVL represents the developer’s names (Adelson-Velsky and Landis [8]).
Insertion into an AVL tree is done in the same way as a standard BST. However, an
insertion may result in unbalanced nodes. First, we observe that since we only insert
one node at a time, a node’s balance factor can only ever become +2 or −2 (assuming it
was an AVL tree to begin with). If several nodes become unbalanced, it is enough to
consider the unbalanced node closest to the new node we just inserted. Correcting the
imbalance at that node should correct the imbalance at other nodes.
114
7.5 Balanced Binary Search Trees
1
10
3 −1
8 40
1 0 1
6 30 80
1 0 0
5 7 50
0
4
Figure 7.4: Balance Factor example on a Binary Search Tree
We need to re-balance the tree to ensure the AVL Tree properties are preserved after
insertion. To do this, we use one of several rotations depending on the type of imbalance.
There are four basic rotations:
• R – A right rotation is used when three nodes are skewed all the way to the right.
We rotate such that the middle one becomes the new root.
• L – A left rotation is used when three nodes are skewed all the way to the left. We
rotate such that the middle node becomes the new root.
• LR – A double left-right rotation is done when the middle node is skewed to the
left and its child is skewed to the right. We left-rotate at the middle node and then
right rotate at the top node.
• RL – A double right-left rotation is done when the middle node is skewed to the
right and its child is skewed to the left. We right-rotate at the middle node and
then left rotate at the top node.
These rotations are depicted in simple trees in Figure 7.5 through 7.8 and the generalized
rotations with subtrees present are presented in Figures 7.9 through 7.12.
To illustrate some of these operations, consider the example depicted Figure 7.13 where
we insert the keys, 8, 4, 7, 3, 2, 5, 1, 10, 6 into an initially empty tree.
Deleting from an AVL tree is the same as a normal BST. Leaves can be straightforwardly
deleted, nodes with a single child can have the child promoted, and nodes with both
children can have their key replaced with either the maximal element in the left subtree
(containing the lesser elements) or the minimal element in the right subtree (containing
the greater elements). However, doing so may unbalance the tree. Deleting a node may
reduce the height of some subtree which will affect the balance factor of its ancestors.
The same rotations may be necessary to rebalance ancestor nodes. Moreover, rebalancing
may also reduce the height of subtrees. Thus, the rebalancing may propagate all the way
back to the root node. An example of a worst case deletion is depicted in Figure 7.14.
115
7 Trees
−2
a
−1 0
b b
0 0 0
c a c
(a) Upon insertion of a (b) AVL tree is rebal-
new node c (a < b < c), anced: b becomes the
the AVL tree becomes new root and the BST
unbalanced at the root property is preserved.
with a balance factor of
−2.
Figure 7.5: Simple AVL L Rotation.
+2
c
+1
0
b b
0 0 0
a a c
(a) Upon insertion of a (b) AVL tree is rebal-
new node a (a < b < c), anced: b becomes the
the AVL tree becomes new root and the BST
unbalanced at the root property is preserved.
with a balance factor of
+2.
Figure 7.6: Simple AVL R Rotation.
116
7.5 Balanced Binary Search Trees
+2
c
−1
0
a b
0 0 0
b a c
(a) Upon insertion of (b) AVL tree is re-
a new node b (a < balanced after a left
b < c), the AVL tree rotation about a fol-
becomes unbalanced lowed by a right rota-
at the root with a tion about c.
balance factor of +2,
but its left-subtree is
skewed to the right.
Figure 7.7: Simple AVL LR Rotation.
−2
a
+1 0
c b
0 0 0
b a c
(a) Upon insertion of (b) AVL tree is re-
a new node b (a < balanced after a right
b < c), the AVL tree rotation about c fol-
becomes unbalanced lowed by a left rota-
at the root with a tion about a.
balance factor of −2,
but its right-subtree
is skewed to the left.
Figure 7.8: Simple AVL RL Rotation.
117
7 Trees
−2
r c
−1
c r
T1 T3
T2 T3 T1 T2
a
a
(a) Upon insertion of a new node a, the AVL (b) AVL tree is rebalanced. The node c be-
Tree becomes unbalanced at r (which may be comes the new root of the (sub)tree, r becomes
a subtree) with a balance factor of −2. c’s left-child.
Figure 7.9: Generalized AVL L Rotation. Subtree T2 “swings” over to become r’s new
right subtree.
2
r c
1
c r
T3 T1
T1 T2 T2 T3
a
a
(a) Upon insertion of a new node a, the AVL (b) AVL tree is rebalanced. The node c be-
Tree becomes unbalanced at r (which may be comes the new root of the (sub)tree, r becomes
a subtree) with a balance factor of +2. c’s right-child.
Figure 7.10: Generalized AVL R Rotation. Subtree T2 “swings” over to become r’s new
left subtree.
118
7.5 Balanced Binary Search Trees
2
r g
−1
c c r
g
T4
T1 T1 T2 T3 T4
T2 T3
a a
a a
(a) Upon insertion of a new node a in either subtree, (b) AVL tree is rebalanced after a left rotation about c
the AVL Tree becomes unbalanced at r (which may followed by a right rotation about r making g the new
be a subtree) with a balance factor of +2. But the root. T3 “swings” over to become the left-subtree of r.
left-subtree rooted at c is skewed to the right. The node c becomes the new root of the (sub)tree, r
becomes c’s right-child.
Figure 7.11: Generalized AVL LR Rotation. Subtree T3 “swings” over to become r’s new
left subtree.
−2
r g
1
c r c
g
T1
T4 T1 T2 T3 T4
T2 T3
a a
a a
(a) Upon insertion of a new node a in either subtree, (b) AVL tree is rebalanced after a right rotation about
the AVL Tree becomes unbalanced at r (which may c followed by a left rotation about r making g the new
be a subtree) with a balance factor of −2. But the root. T2 “swings” over to become the right-subtree of
right-subtree rooted at c is skewed to the left. r. The node g becomes the new root of the (sub)tree, r
becomes g’s left-child.
Figure 7.12: Generalized AVL RL Rotation. Subtree T2 “swings” over to become r’s new
right subtree.
119
7 Trees
Analysis
The height h of an AVL tree is bounded above and below:
blog2 nc ≤ h ≤ 1.4405 log2 (n + 2) − 0.3277
Thus, no matter what the specific value of h is for a given tree, we can conclude that
h ∈ Θ(log n)
where n is the number of nodes in the tree. Furthermore, rotations are simply a
matter of switching out pointers (O(1)) thus searching, insertion and deletion are all
O(h) = Θ(log n) for AVL Trees.
You can get some practice with AVL trees by trying out one of the many online simulations
and animation applications. For example:
• https://www.cs.usfca.edu/~galles/visualization/AVLtree.html
• http://www.qmatica.com/DataStructures/Trees/AVL/AVLTree.html
7.5.2 B-Trees
• Every node has at most m children
• Every node (except the root) has at least d m2 e children
• Root has at least 2 children unless it is a leaf
• Non-leaf nodes with k children contain k − 1 keys
• All leaves appear in the same level and are non-empty
• Commonly used in databases
• Special case: 2-3 Trees (m = 3)
• 2-3 Trees credited to Hopcroft (1970) and B-Trees credited to Bayer & McCreight
(1972) [5]
Rather than a single key per node, a 2-3 tree may have 1 or 2 keys per node.
• 2-node – A node containing a single key k with two children, i.e. the usual type
of node in a regular BST.
• 3-node – A node containing two keys, k1 , k2 such that k1 < k2 . A 3-node has three
children:
– Left-Most-Child represents all keys less than k1
– Middle-Child represents all keys between k1 , k2
120
7.5 Balanced Binary Search Trees
– Right-Most-Child represents all keys greater than k2
Another requirement or property of a 2-3 Tree is that all of its leaves are on the same
level. Every path from the root to any leaf will be have the same length. This ensures
that the height h is uniform and balanced.
Basic Operations
• Searching is done straight forward, but slightly different than regular BSTs
• Insertion is always done in a leaf.
– If the leaf is a 2-node, we insert by making it a 3-node and ordering the keys.
– If the leaf is a 3-node, we split it up–the minimal key becomes the left-child,
the maximal key becomes the right child and the middle key is promoted to
the parent node
The promotion of a node to the parent can cause the parent to overflow (if the parent
was a 3-node), and thus the process may continue upwards to the node’s ancestors.
Note that if the promotion is at the root, a new node is created that becomes the new
root.
Analysis
Let T be a 2-3 tree of height h. The smallest number of keys T could have is when all
nodes are 2-nodes, which is essentially a regular binary search tree. As the tree is full,
there would be at least
Xh
n≥ 2i = 2h+1 − 1
i=0
keys. Conversely, the largest number of keys would be when all nodes are 3-nodes and
every node has 2 keys and three children. Thus,
h
X
3i = 2 3h+1 − 1
n≤2
i=0
Combining these two bounds and solving for h yields
n
dlog3 + 1 e − 1 ≤ h ≤ dlog2 (n + 1)e − 1
2
Thus, h ∈ Θ(log n).
As before, these bounds show that searching, insertion and deletion are all Θ(log n) in
the worst and average case.
Deleting: deleting from an internal node: exchange it with the maximal value in the left
subtree (or minimal value in the right subtree) as is standard. Such a value will always
be in a leaf.
121
7 Trees
Deletion may propagate to the root, merging siblings, etc.
7.5.3 Red-Black Trees
Definition 9. A Red-Black Tree is a binary search tree in which every node has a single
(unique) key and a color, either red or black.
• The root is black and every leaf is black (this can be made trivial true by adding
sentinels).
• Every red node has only black children.
• Every path from the root to a leaf has the same number of black nodes in it.
The black height of a node x is the number of black nodes in any path from x to a root
(not counting x), denoted bh(x).
Lemma 8. In a Red-Black Tree with n internal nodes satisfies the height property
h ≤ 2 log2 (n + 1)
You can prove this by induction on the height h with the base case being a leaf.
As with AVL Trees, insertion and deletion of nodes may violate the Red-Black Tree
properties. Thus, we insert just like a regular BST, but we also perform rotations if
necessary.
In fact, we use the exact same Left and Right rotations as are used in AVL Trees.
However, LR and RL rotations are unnecessary.
The first step to inserting a new node is to insert it as a regular binary search tree and
color it red.
There are three cases for insertion.
• Case 1 – If the inserted node z has a red parent and a red uncle (the cousin node
of z’s parent). In this case we recolor z’s parent, uncle, and grandfather. We then
recurse back up and see if any properties are violated at z’s grandfather.
• Case 2 – z’s parent is red but z’s uncle is black. If z is the right-sub-child of its
parent, then we perform a Left rotation. The new z becomes the new left-sub-child
of z.
• Case 3 – z’s parent is red but z’s uncle is black. z is the left-sub-child of its parent,
so we perform a Right rotation. The new z becomes the new left-sub-child of z.
• http://mathpost.la.asu.edu/~daniel/redblack.html
• http://www.ececs.uc.edu/~franco/C321/html/RedBlack/redblack.html
122
7.6 Heaps
7.6 Heaps
Definition 10. A heap is a binary tree that satisfies the following properties.
1. It is a full or complete binary tree: all nodes are present except possibly the last
row
2. If the last row is not full, all nodes are full-to-the-left
3. It satisfies the Heap Property: every node has a key that is greater than both of its
children (max-heap)
• As a consequence of the Heap Property, the maximal element is always at the root
• Alternatively, we can define a min-heap
• Variations: 2-3 heaps, fibonacci heaps, etc.
• A min-heap example can be found in Figure 7.18
Applications
• Heaps are an optimal implementation of a priority queue
• Used extensively in other algorithms (Heap Sort, Prim’s, Dijkstra’s, Huffman
Coding, etc.) to ensure efficient operation
7.6.1 Operations
Insert
• Want to preserve the full-ness property and the Heap Property
• Preserve full-ness: add new element at the end of the heap (last row, first free spot
on the left)
• Insertion at the end may violate the Heap Property
• Heapify/fix: bubble up inserted element until Heap Property is satisfied
• Analysis: insert is O(1); heapify: O(d)
Remove from top
• Want to preserve the full-ness property and the Heap Property
• Preserve full-ness: swap root element with the last element in the heap (lowest row,
right-most element)
• Heap property may be violated
• Heapify/fix: bubble new root element down until Heap Property is satisfied
123
7 Trees
• Analysis: Swap is O(1); heapify: O(d)
Others
• Arbitrary remove
• Find
• Possible, but not ideal: Heaps are restricted-access data structures
Analysis
• All operations are O(d)
• Since Heaps are full, d = O(log n)
• Thus, all operations are O(log n)
7.6.2 Implementations
Array-Based
• Root is located at index 1
• If a node u is at index i, u’s left-child is at 2i, its right-child is at 2i + 1
• If node u is at index j, its parent is at index b 2j c
• Alternatively: 0-index array left/right children/parent are at 2n + 1, 2n + 2, and
b j−1
2
c
• Advantage: easy implementation, all items are contiguous in the array (in fact, a
BFS ordering!)
• Disadvantage: Insert operation may force a reallocation, but this can be done in
amortized-constant time (though may still have wasted space)
Tree-Based
• Reference-based tree (nodes which contain references to children/parent)
• Parent reference is now required for efficiency
• For efficiency, we need to keep track of the last element in the tree
• For deletes/inserts: we need a way to find the last element and first “open spot”
• We’ll focus on finding the first available open spot as the same technique can be
used to find the last element with minor modifications
124
7.6 Heaps
Finding the first available open spot in a Tree-based Heap
Technique A: numerical technique
• WLOG: assume we keep track of the number of nodes in the heap, n and thus the
depth d = blog nc
• If n = 2d+1 − 1 then the tree is full, the last element is all the way to the right, the
first available spot is all the way to the left
• Otherwise n < 2d+1 − 1 and the heap is not full (the first available spot is located
at level d, root is at level 0)
• Starting at the root, we want to know if the last element is in the left-subtree or
the right subtree
• Let m = n − (2d − 1), the number of nodes present in level d
d
• If m ≥ 22 then the left-sub tree is full at the last level and so the next open spot
would be in the right-sub tree
d
• Otherwise if m < 22 then the left-sub tree is not full at the last level and so the
next open spot is in the left-sub tree
• Traverse down to the left or right respectively and repeat: the resulting sub-tree
d
will have depth d − 1 with m = m (if traversing left) or m = m − 22 (if traversing
right)
• Repeat until we’ve found the first available spot
• Analysis: in any case, its O(d) = O(log n) to traverse from the root to the first
open spot
125
7 Trees
Input : A tree-based heap H with n nodes
Output : The node whose child is the next available open spot in the heap
1 curr ← T.head
2 d ← blog nc
3 m←n
4 while curr has both children do
5 if m = 2d+1 − 1 then
//remaining tree is full, traverse all the way left
6 while curr has both children do
7 curr ← curr.lef tChild
8 end
9 else
//remaining tree is not full, determine if the next open
spot is in the left or right sub-tree
d
10 if m ≥ 22 then
//left sub-tree is full
11 d ← (d − 1)
d
12 m ← (m − 22 )
13 curr ← curr.rightChild
14 else
//left sub-tree is not full
15 d ← (d − 1)
16 m←m
17 curr ← curr.lef tChild
18 end
19 end
20 end
21 output curr
Algorithm 40: Find Next Open Spot - Numerical Technique
Technique B: Walk Technique
• Alternatively, we can adapt the idea behind the tree walk algorithm to find the
next available open spot
• We’ll assume that we’ve kept track of the last node
• If the tree is full, we simply traverse all the way to the left and insert, O(d)
• If the last node is a left-child then its parent’s right child is the next available spot,
126
7.6 Heaps
finding it is O(1)
• Otherwise, we’ll need to traverse around the perimeter of the tree until we reach
the next open slot
Input : A tree-based heap H with n nodes
Output : The node whose (missing) child is the next available open spot in the
heap
1 d ← blog nc
2 if n = 2d+1 − 1 then
//The tree is full, traverse all the way to the left
3 curr ← root
4 while curr.lef tChild 6= null do
5 curr ← curr.lef tChild
6 end
7 else if last is a left-child then
//parent’s right child is open
8 curr ← last.parent
9 else
//The open spot lies in a subtree to the right of the last node
//Walk the tree until we reach it
10 curr ← last.parent
11 while curr is a right-child do
12 curr ← curr.parent
13 end
//"turn" right
14 curr ← curr.parent
15 curr ← curr.rightChild
//traverse all the way left
16 while curr.lef tChild 6= null do
17 curr ← curr.lef tChild
18 end
19 end
//current node’s missing child is the open spot
20 output curr
Algorithm 41: Find Next Open Spot - Walk Technique
127
7 Trees
7.6.3 Java Collections Framework
Java has support for several data structures supported by underlying tree structures.
• java.util.PriorityQueue<E> is a binary-heap based priority queue
– Priority (keys) based on either natural ordering or a provided Comparator
– Guaranteed O(log n) time for insert (offer) and get top (poll)
– Supports O(n) arbitrary remove(Object) and search (contains) methods
• java.util.TreeSet<E>
– Implements the SortedSet interface; makes use of a Comparator
– Backed by TreeMap, a red-black tree balanced binary tree implementation
– Guaranteed O(log n) time for add, remove, contains operations
– Default iterator is an in-order traversal
7.6.4 Other Operations
include decrease key operation here
7.6.5 Variations
Binomial, Fibonacci, etc.
7.7 Applications
7.7.1 Heap Sort
• If min/max element is always at the top; simply insert all elements, then remove
them all!
• Perfect illustration of “Smart data structures and dumb code are a lot better than
the other way around”
128
7.7 Applications
Input : A collection of elements A = {a1 , . . . , an }
Output : A collection, A0 of elements in A, sorted
1 H ← empty heap
2 A0 ← empty collection
3 foreach x ∈ A do
4 insert x into H
5 end
6 while H is not empty do
7 y ← remove top from H
8 Add y to the end of A0
9 end
10 output A0
Algorithm 42: Heap Sort
Analysis
• Amortized analysis: insert/remove operations are not constant throughout the
algorithm
• On first iteration: insert is d = O(1); on the i-th iteration, d = O(log i); only on
the last iteration is insertion O(log n)
• In total, the insert phase is:
n
X
log i = O(n log n)
i=1
• A similar lower bound can be shown
• Same analysis applies to the remove phase:
1
X
log i
i=n
• In total, O(n log n)
7.7.2 Huffman Coding
Overview
• Coding Theory is the study and theory of codes—schemes for transmitting data
• Coding theory involves efficiently padding out data with redundant information to
increase reliability (detect or even correct errors) over a noisy channel
129
7 Trees
• Coding theory also involves compressing data to save space
– MP3s (uses a form of Huffman coding, but is information lossy)
– jpegs, mpegs, even DVDs
– pack (straight Huffman coding)
– zip, gzip (uses a Ziv-Lempel and Huffman compression algorithm)
Basics
• Let Σ be a fixed alphabet of size n
• A coding is a mapping of this alphabet to a collection of binary codewords,
Σ → {0, 1}∗
• A block encoding is a fixed length encoding scheme where all codewords have the
same length (example: ASCII); requires dlog2 ne length codes
• Not all symbols have the same frequency, alternative: variable length encoding
• Intuitively: assign shorter codewords to more frequent symbols, longer to less
frequent symbols
• Reduction in the overall average codeword length
• Variable length encodings must be unambiguous
• Solution: prefix free codes: a code in which no whole codeword is the prefix of
another (other than itself of course).
• Examples:
– {0, 01, 101, 010} is not a prefix free code.
– {10, 010, 110, 0110} is a prefix free code.
• A simple way of building a prefix free code is to associate codewords with the leaves
of a binary tree (not necessarily full).
• Each edge corresponds to a bit, 0 if it is to the left sub-child and 1 to the right
sub-child.
• Since no simple path from the root to any leaf can continue to another leaf, then
we are guaranteed a prefix free coding.
• Using this idea along with a greedy encoding forms the basis of Huffman Coding
Steps
• Consider a precomputed relative frequency function:
freq : Σ → [0, 1]
130
7.7 Applications
• Build a collection of weighted trees Tx for each symbol x ∈ Sigma with wt(Tx ) =
freq(x)
• Combine the two least weighted trees via a new node; associate a new weight (the
sum of the weights of the two subtrees)
• Keep combining until only one tree remains
• The tree constructed in Huffman’s algorithm is known as a Huffman Tree and it
defines a Huffman Code
Input : An alphabet of symbols, Σ with relative frequencies, freq(x)
Output : A Huffman Tree
1 H ← new min-heap
2 foreach x ∈ Σ do
3 Tx ← single node tree
4 wt(Tx ) ← freq(x)
5 insert Tx into H
6 end
7 while size of H > 1 do
8 Tr ← new tree root node
9 Ta ← H.getM in
10 Tb ← H.getM in
11 Tr .lef tChild ← Ta
12 Tr .rightChild ← Tb
13 wt(r) ← wt(Ta ) + wt(Tb )
14 insert Tr into H
15 end
16 output H.getM in
Algorithm 43: Huffman Coding
Example
Construct the Huffman Tree and Huffman Code for a file with the following content.
character A B C D E F G
frequency 0.10 0.15 0.34 .05 .12 .21 .03
• Average codeword length:
.10 · 3 + .15 · 3 + .34 · 2 + .05 · 4 + .12 · 3 + .21 · 2 + .03 · 4 = 2.53
131
7 Trees
• Compression ratio:
(3 − 2.53)
= 15.67%
3
• In general, for text files, pack (Huffman Coding), claims an average compression
ratio of 25-40%.
• Degenerative cases:
– When the probability distribution is uniform: p(x) = p(y) for all x, y ∈ Σ
– When the probability distribution follows a fibonacci sequence (the sum of
each of the two smallest probabilities is less than equal to the next highest
probability for all probabilities)
132
7.7 Applications
2
7
2 2 0
8 4 8
1 1
0
4 7 3
0 0 0
0
7 4 8 2
(a) Insertion of 8, 4, 7 (b) AVL tree is rebal- (c) Insertion of 3, 2 unbalances
causes the tree to be- anced after an LR Ro- the tree at node 4
come unbalanced. tation
1
7
0 0
3 8
0 0
2 4
(d) AVL tree is rebalanced
after an R Rotation
2
7 7
−1
0
3 8 4 8
4
0 −1
2 4 3 5 3 7
0
5 2 2 5 8
(e) Insertion of 5 unbalances (f) Step 1: A left rotation (g) Step 2: A right rota-
the tree at 7, an LR rotation at node 3 tion at node 7; node 5
will be performed at node 7 swings over to become
7’s left-child.
4 −1
4
0 0
3 7 4
2 7
0 0−1 −1
2 5 8 2 7 1 3 5 8
0 0
1 1 3 5 8 6 10
(h) Insertion of 1 unbal- (i) The right rotation re- (j) Insertion of 10, 6 does
ances the tree at 3, a right sults in a complete bi- not unbalance the AVL tree.
rotation about 3 is per- nary tree. The balance
formed. factor of all nodes is
zero.
Figure 7.13: AVL Tree Insertion Sequence. Insertion of elements 8, 4, 7, 3, 2, 5, 1, 10, 6.
133
7 Trees
50 50
−2 −1
20 60 20 60
−2
10 30 12 30
0
3 12 25 40 10 14 25 40
14 27 35 47 T1 T2 27 35 47 T1 T2
48 48
(a) Deletion of 3 causes node 10 to become unbal- (b) The rotation reduces the height of the subtree at
anced, prompting a left rotation about 10. node 12 causing an unbalance at 20. A left rotation
is performed about 20; the subtree rooted at 25
swings over to become the right subtree of 20.
60
−2 50
50
−1 30
30 60
20 40 T2
20 40
12 25 35 47 T1
12 25 35 47
10 14 27 48 T1 T2
10 14 27 48
(c) The rotation reduces the height of the subtree at (d) The rotation at the root rebalances the
node 30 causing an unbalance at the root, 50. A left tree.
rotation is performed about 50; the subtree T1 swings
over to become the right subtree of 50.
Figure 7.14: Worst-case Example of Rebalancing Following a Deletion. Deletion of 3
causes an imbalance at its parent. Each subsequent rotation correction
induces another imbalance to the parent all the way up to the root. Deletion
can potentially result in Θ(log n) correction rotations.
134
7.7 Applications
7 7 3, 7
8, 4, 7 4 8 2, 3, 4 7 2 4 8
(a) In- (b) The tree (c) Insertion of (d) The middle key 3 is
sertion splits into two 3, 2 causes the left promoted to the parent
of 8, 4, 7 children and node to become (root) node while 2, 4 get
causes the creates a new overfull split into 2 subtrees
root node root node.
to become
overfull
3, 7 3, 7
1, 2 4, 5 8, 10 1, 2 4, 5, 6 8, 10
(e) Insertion of 5, 1, 10 does (f) Insertion of 6 causes the
not cause any disruption, but middle node to become over-
all nodes are full at this point. full
5
3, 5, 7 3 7
1, 2 4 6 8, 10 1, 2 4 6 8, 10
(g) 5 is promoted while 2 children are (h) However, the promo-
created. tion of 5 caueses the root
to become overfull, the
split propagates up the
tree.
Figure 7.15: 2-3 Tree Insertion Sequence. Insertion of elements 8, 4, 7, 3, 2, 5, 1, 10, 6.
135
7 Trees
c b a, d b, d
a, b φ a c φ b, c e, f a c e, f
(a) The left sibling node can spare (b) The right sibling can spare a key; b, a are
a key, b, c are rotated. rotated.
a, c b, d c, d b, e
φ b d, e a c e a, b φ e, f a, b d f
(c) The right sibling does not have a key to (d) The immediate right sibling has a key to
spare, but the far right one does, a rotation spare, e is promoted, d fulfills the empty child.
involves all three children.
b φ a, c c
a φ a, b T φ b d a, b d
(e) The left sibling does not have a (f) The right sibling does not have a key
key to spare; instead the parent is to spare; instead one of the parent keys
merged down. The resulting parent is merged down, reducing the number of
is now empty and has only one child. children nodes.
We deal with it recursively.
f d
b, c c b, d φ b f
a φ d a, b d a c e g a c e g
(g) Neither sibling has a key to spare, so the (h) A generalized rotation, b is promoted, f
two left children are merged, b is demoted. is demoted; e swings over to become f ’s new
left child.
Figure 7.16: 2-3 Tree Deletion Operations Part I. Different configurations result in redis-
tributions or merging of nodes. Symmetric operations are omitted.
136
7.7 Applications
b, h d, h
φ d, f j, l b f j, l
a c e g i k m a c e g i k m
(a) Another generalized rotation. b is demoted, d promoted while c swings over to be b’s right
child.
d φ
b φ b, d φ
a c e a c e
(b) A generalized merge procedure: d is merged
down; e swings over to become the right-most
child.
b, f f
φ d h b, d h
a c e g i a c e g i
(c) Another generalized merge procedure; b is merged with
d; a becomes their left-most child.
φ
b, d b, d
a c e a c e
(d) Deleting an empty root. The root’s
child becomes the new root.
Figure 7.17: 2-3 Tree Deletion Operations Part II. Different configurations result in
redistributions or merging of nodes. Symmetric operations are omitted.
137
7 Trees
1
5 30
50 55 40 80
60 53 72 70 65 45 90 95
92 63
Figure 7.18: A min-heap
r
d−1
X
2i = 2d − 1
k=0
L R Number of
nodes at level
level d level d d is
2d 2d
m = n − (2d − 1)
at most 2
at most 2
d d
If m < 22 , the If m ≥ 22 , the
open slot is in open slot is in
the left subtree the right subtree
Figure 7.19: Tree-based Heap Analysis. Because of the fullness property, we can determine
which subtree (left or right) the “open” spot in a heap’s tree is by keeping
track of the number of nodes, n. This can be inductively extended to each
subtree until the open spot is found.
138
7.7 Applications
.08
0 1
.39 .61
0 1 0 1
.18 F : .21 .27 C : .34
0 1 0 1
.08 A : .10 E : .12 B : .15
0 1
G : .03 D : .05
Figure 7.20: Huffman Tree
139
8 Graph Algorithms
8.1 Introduction
Definition 11. An undirected graph G = (V, E) is a 2-tuple with
• V = {v1 , . . . , vn } a set of vertices.
• E = {e1 , . . . , em } a set of edges where ei = (u, v) for u, v ∈ V
Graph variations:
• Directed: each edge is oriented and (u, v) is an ordered pair
• Weighted: there is a weight function defined on the edge set. It usually suffices to
consider:
wt : E → Z
Representations:
• Adjacency List – An adjacency list representation of a graph G = (V, E) main-
tains |V | linked lists. For each vertex v ∈ V , the head of the list is v and subsequent
entries correspond to adjacent vertices v 0 ∈ V .
– Advantage: less storage
– Disadvantage: adjacency look up is O(|V |), extra work to maintain vertex
ordering (lexicographic)
• Adjacency Matrix – An adjacency matrix representation maintains an n × n
sized matrix such that
0 if (vi , vj ) 6∈ E
A[i, j] =
1 if (vi , vj ) ∈ E
for 0 ≤ i, j ≤ (n − 1)
– Advantages: adjacency/Weight look up is constant
– Disadvantage: extra storage
• In either case, weights can be stored as extra data in nodes or as entries in the
matrix
141
8 Graph Algorithms
Libraries:
• JGraphT (Java) – http://jgrapht.org/
• Boost (C++ header only library) – http://www.boost.org/doc/libs/1_54_0/
libs/graph/doc/index.html
• LEMON (C++ Graph library) – http://lemon.cs.elte.hu/trac/lemon
• Python: NetworkX (http://networkx.github.io/), igraph (http://igraph.org/)
• Ruby: https://github.com/bruce/graphy
• JavaScript: https://github.com/devenbhooshan/graph.js
8.2 Depth First Search
Depth First Search (DFS) traverses a graph by visiting “deeper” vertices first, before
processing the vertices. That is, it explores a graph as deeply as possible before back
tracking and visiting other vertices.
At each iteration of DFS we can choose to visit an unvisited vertex according to several
criteria:
• Lexicographic order – visit the next vertex according to the label ordering
• Weight – in a weighted graph, we can visit the vertex whose edge is the least (or
greatest) weight
In general, we can keep track of the state of nodes by two measures, a vertex’s color and
the discovery/processing “time”.
At various points in the algorithm, a vertex is colored:
• White – indicating that the vertex is unvisited (all vertices are initially white)
• Gray – indicating that the vertex has been discovered, but not yet processed (we
go deeper in the graph if possible before processing gray vertices)
• Black – indicating that the vertex has been visited and processed. By the end of
the algorithm, all vertices are black
Discovery and processing time stamps can be associated with each vertex by keeping a
global counter that increments each time a vertex is initially visited and again when it is
processed. The sequence of time stamps indicates visit/process order.
A straightforward implementation would be a recursive algorithm (Algorithms 44 and
45). The main algorithm is necessary to accommodate:
• Directed graphs - we may need to restart the search if some vertices are unreachable
from the initial vertex
142
8.2 Depth First Search
• Disconnected graphs – we may need to restart at another connected component
Input : A graph, G = (V, E)
1 foreach vertex v ∈ V do
2 color v white (unvisited)
3 end
4 count ← 0
5 foreach v ∈ V do
6 if v is white then
7 dfs(G, v)
8 end
9 end
Algorithm 44: Recursive Depth First Search, Main Algorithm Dfs(G)
Input : A graph, G = (V, E), a vertex v ∈ V
1 count ← count + 1
2 mark v with count (discovery time)
3 color v gray (discovered, not processed)
4 foreach w ∈ N (v) do
5 if w is white then
6 dfs(G, w)
7 end
8 end
9 count ← count + 1
10 mark v with count (processing time)
11 process v
12 color v black
Algorithm 45: Recursive Depth First Search, Subroutine dfs(G)
A better implementation would be to use a smarter data structure, namely a stack
143
8 Graph Algorithms
(Algorithm 46).
Input : A graph, G = (V, E)
1 foreach vertex v ∈ V do
2 color v white (unvisited)
3 end
4 count ← 1
5 S ← empty stack
6 push start vertex v onto S
7 mark v with count (discovery time)
8 color v gray
9 while S is not empty do
10 count ← count + 1
11 x ← S.peek
12 y ← next white vertex in N (x)
//If there is a gray vertex in N (y) not equal to x, it
constitutes a cycle
13 if y is nil then
//Neighborhood of x has been exhausted, start backtracking
14 S.pop (x is popped off of S)
15 process x
16 color x black
17 mark x with count (processing time)
18 end
19 else
20 S.push y
21 color y gray
22 mark y with count (discovery time)
23 end
24 end
Algorithm 46: Stack-Based Depth First Search
8.2.1 DFS Example
Consider the small graph in Figure 8.1; the discovery and processing (finishing) time
stamps are presented in Table 8.1.
144
8.2 Depth First Search
a b
e
c d
Figure 8.1: A small graph.
Vertex discovery time process time
a 1 10
b 2 9
c 4 5
d 3 6
e 7 8
Table 8.1: Depth First Search Time Stamps
8.2.2 DFS Artifacts
Depth-First Search can produce a depth-first forest (or tree if the graph is connected).
Edges in a DFS forest can be classified as:
• Tree Edges – Edges that are part of the search path, whenever an unvisited vertex
v 0 is discovered from the current vertex, v, (v, v 0 ) is a tree edge.
• Back Edge – From the current vertex, v, if an edge is found pointing to a vertex
that has already been discovered, the edge (v, v 0 ) is a back edge. Back edges connect
vertices to their ancestors in the DFS forest
• Forward Edge – If, after backtracking, the current vertex v points to a visited
vertex v 0 , (v, v 0 ) is a forward edge.
• Cross Edge – All other edges, they can be edges between trees in the forest or
edges between vertices in the same tree with a common ancestor
Note that:
• For undirected graphs forward and back edges are the same (no orientation)
• For undirected graphs, cross edges do not exist
• For directed graphs, cross edges may connect between components (created by
restarting the DFS subroutine) or within components
Edges can be classified as follows. From the current vertex, the color of the adjacent
vertex determines the type of edge:
• White indicates a tree edge
145
8 Graph Algorithms
i a
c b h
d g
f e
Figure 8.2: A larger graph.
• Gray indicates a back edge
• Black indicates a forward or cross edge (extra work is required to determine if the
vertex is in the same component or not)
Consider the larger graph in Figure 8.2. A DFS performed starting at vertex i would
lead to the following.
Visitation order (using lexicographic next choice):
i, a, b, c, d, e, f, g, h
Processing order:
f, e, h, g, d, c, b, a, i
The DFS Tree produced by this traversal can be found in Figure 8.3. Note that only
forward edges are present as this is a connected, undirected graph.
8.2.3 Analysis
• Each vertex is examined once when it is first visited (pushed) and when it is finished
(popped)
• Each vertex is processed exactly once
• Each edge is examined once (twice for undirected graphs): each time it appears in
a neighborhood
• For adjacency matrices, this is O(n2 ) as each traversal will examine every entry in
the matrix (to to determine the neighborhood of each vertex)
146
8.2 Depth First Search
i
a
b
c
d
e g
f h
Figure 8.3: DFS Forrest with initial vertex i. Dashed edges indicate back edges.
147
8 Graph Algorithms
• For adjacency matrices this is O(n + m): for each vertex, its entire adjacency list
will have to be examined, but need not be computed
8.3 Breadth First Search
Breadth First Search (BFS) is a “shallow search.” BFS explores the nearest vertices
first. At any given vertex v, the entire neighborhood, N (v) is explored before progressing
further in the graph.
As with DFS, a BFS may need to be restarted (disconnected graphs or unreachable
vertices in a directed graph), see Algorithms 47 and 48.
148
8.3 Breadth First Search
Note the contrast: BFS uses a queue data structure.
Input : A graph, G = (V, E)
1 foreach vertex v ∈ V do
2 color v white (unvisited)
3 end
4 count ← 0
5 foreach v ∈ V do
6 if v is white then
7 bfs(G, v)
8 end
9 end
Algorithm 47: Breadth First Search, Main Algorithm Bfs(G)
Input : A graph, G = (V, E), an initial vertex v ∈ V
1 count ← count + 1
2 Q ← empty queue
3 mark v with count (discovery time)
4 color v gray (discovered, not processed)
5 Q.enqueue(v)
6 while Q is not empty do
7 x ← Q.peek
8 foreach y ∈ N (x) do
9 if y is white then
10 count ← count + 1
11 mark y with count (discovery time)
12 Q.enqueue(y)
13 end
14 end
15 z ← Q.dequeue
16 count ← count + 1
17 mark z with count (processing time)
18 process z
19 color z black
20 end
Algorithm 48: Breadth First Search, Subroutine bfs(G, v)
149
8 Graph Algorithms
Vertex discovery time process time
a 1 2
b 3 5
c 4 8
d 6 9
e 7 10
Table 8.2: Breadth First Search Time Stamps
i
a f
b c h d e
g
Figure 8.4: BFS Tree with initial vertex i. Dashed edges indicate cross edges.
An example BFS run on the same small example can be found in Table 8.2.
BFS can also produce a BFS Forest (or tree) with similar types of edges (Tree, Cross,
Forward, Back). The BFS Tree of the graph in Figure 8.2 can be seen in Figure 8.4.
• For undirected graphs only tree and cross edges are possible (no back/forward
edges: why?)
• For directed graphs, back and cross edges are possible; forward edges are not
8.4 DFS/BFS Applications
Some problems can be solved by either DFS or BFS, others are more appropriately solved
by one more than the other.
150
8.4 DFS/BFS Applications
8.4.1 Connectivity & Path Finding
Problem 9 (Connectivity).
Given: A graph G = (V, E), two vertices s, t ∈ V
Output: true if there is a path p : s t
Variations:
• Directed or undirected paths
• A functional version: not only do we want to know if a path exists, but if one does,
we want the algorithm to output one
• Does there exist a path involving a particular vertex (or edge)?
8.4.2 Topological Sorting
Recall that a partial ordering is a relation on a set that is reflexive, antisymmetric and
transitive. Posets can model:
• Tasks to be performed with certain prerequisites
• Entities with constrained relations, etc.
In general, consider a directed acyclic graph (DAG). A topological ordering (or sorting)
is an arrangement of vertices that is consistent with the connectivity defined by the
underlying graph.
That is, if (u, v) ∈ E then u precedes v in the topological ordering.
There may be many valid topological orders
DFS Solution: Run DFS on the DAG and ordering the vertices in descending order
according to their finishing timestamps.
8.4.3 Shortest Path
The length of a path p : s t is equal to the number of edges in p.
Problem 10 (Shortest Path).
Given: A graph G = (V, E), two vertices s, t ∈ V
Output: The shortest length path p : s t
Variations:
• Directed or undirected versions
• Output the shortest weighted path p
151
8 Graph Algorithms
a
b c
d ye
x g
Figure 8.5: BFS Tree with cross edges (dashed) involved in a cycle.
• Output the shortest path s v for all vertices v
• Output the shortest path for all pairs of vertices
Observation: for directed graphs, the shortest path s t is not necessarily the shortest
(or even a valid) path for t s.
BFS Solution: For unweighted graphs (directed or undirected), the BFS tree provides
all shortest paths s → v for all v where s is the start vertex of the BFS.
Observation: contrast this “easy” problem with the opposite problem of finding the
longest path in a graph. Finding the longest path is in fact equivalent to the Hamiltonian
Path problem, a much more difficult, NP-complete problem.
8.4.4 Cycle Detection
Problem 11 (Cycle Detection).
Given: A graph G = (V, E)
Output: true if G contains a cycle, false otherwise
For undirected graphs:
• If DFS encounters a back edge, then a cycle exists
• If BFS encounters a cross edge, then a cycle exists
Directed graphs are more complex:
• If DFS or BFS encounters a back edge, then a cycle exists
• A single cross edge or forward edge does not necessarily imply a cycle (see Figure
8.5)
• A cycle may consist of a series or cross, tree, and forward edges
• A lot more work may be required to distinguish true cycles.
152
8.4 DFS/BFS Applications
Alternatives: make two runs of DFS or BFS to answer the connectivity question:
∃u, v ∈ V ∃ [p1 : u v ∧ p2 : v u]
For undirected graphs, this does not necessarily work: the path could be the same.
Instead we could take the following strategy: use a functional version of DFS/BFS to
actually find a path p from u to v, then remove each of the vertices (and corresponding
edges) in the path (except for u, v) and run DFS/BFS again to see if there is a different
path p0 which would constitute a cycle.
8.4.5 Bipartite Testing
Recall that a bipartite graph G = (R, L, E) is a graph with two disjoint vertex sets, R
and L and edges between vertices in R and L. That is, for v, v 0 ∈ R, (v, v 0 ) is never an
edge (L likewise).
Problem 12 (Bipartite Testing).
Given: A graph G = (V, E)
Output: true if G is a bipartite graph, false otherwise
Theorem 6. An undirected graph G = (V, E) is bipartite if and only if G contains no
odd-length cycles.
We already have an algorithm for cycle detection, how can we modify it to detect
odd-cycles?
8.4.6 Condensation Graphs
A directed graph G = (V, E) is called strongly connected if for any pair of vertices,
u, v ∈ V there exists a directed path p : u v. That is, every vertex is reachable from
every other vertex by some path.
More generally, a directed graph’s vertices can be partitioned into maximal, disjoint
subsets, V1 , V2 , . . . , Vk such that each induced subgraph Gi = (Vi , Ei ) is a strongly
connected graph. These are known as the strongly connected components of G.
A condensation graph Gc = (V 0 , E 0 ) of a directed graph G is defined as
• V 0 = {V1 , . . . , Vk }
• E 0 = {(Vi , Vj | for some u ∈ Vi , v ∈ Vj , (u, v) ∈ E}
That is, the vertices are the strongly connected components of G and there are edges
between two components if some vertex is reachable between them in G.
A condensation graph is always a directed, acyclic graph that is never strongly connected
(unless G was strongly connected–what would Gc look like then?). The following is the
Kosaraju-Sharir Algorithm.
153
8 Graph Algorithms
a b c d
e f g
Figure 8.6: A Directed Graph.
a d, g
b, c, e
f
Figure 8.7: Condensation Graph
1. Do a Depth-First-Search traversal of G and note the finish time stamps
2. Compute the transpose graph GT (all edges are reversed)
3. Do a Depth-First-Search on GT in descending order of the ending time stamps in
step 1.
• Each time you start over is a new strongly connected component
• Edges in Gc can be created on the fly–there will be at most one out edge and
at most one in edge for each vertex in Gc .
4. Output Gc
8.5 Minimum Spanning Tree Algorithms
8.5.1 Greedy Algorithmic Strategy
The greedy algorithmic design approach involves iteratively building a solution to a
problem using the next immediately available best choice.
As we iteratively build up a solution to a problem, each greedy choice we make has to
154
8.5 Minimum Spanning Tree Algorithms
satisfy a few properties:
• It must be feasible – it has to satisfy the constraints of the problem (like a total
weight constraint)
• It must be locally optimal – it should be the best, immediately available choice we
have.
• It must be irrevocable – once a choice has been made, we are not allowed to back
track and undo it.
Greedy solutions are efficient, but make decisions based only on local information which
does not necessarily guarantee a globally optimal solution. Often greedy algorithms can
provide an approximations.
One such example is Kruskal’s algorithm for the minimum spanning tree problem.
In a network topology there may be many redundant connections. We wish to identify
the “backbone” of the network. That is, we want to find a subgraph with the minimal
number of edges of minimal weight that keep the network connected.
More generally, given a weighted graph, we wish to find a subgraph that keeps the graph
connected, namely a tree.
Definition 12. Let G = (V, E) be a connected, weighted graph. A spanning tree of G
is a subgraph, T = (V, E 0 ) where E 0 ⊆ E is a set of edges that span the vertex set V
without inducing a cycle. A minimum spanning tree is a spanning tree of minimal weight.
That is, X
wt(e)
e∈E 0
is minimal.
Problem 13 (Minimum Spanning Tree).
Given: A weighted graph G = (V, E)
Output: A minimum spanning tree T of G
In general, there may be many unique minimum spanning trees. In fact, the number
of possible spanning trees is exponential and generating them is very difficult. Rather,
solving the minimum spanning tree problem can be solved exactly by a greedy strategy.
8.5.2 Kruskal’s Algorithm
Kruskal’s algorithm takes a greedy approach
• presorts edges in nondecreasing order according to their weights
• it considers each edge in this order
155
8 Graph Algorithms
Edge Weight Result
(a, d) 5 include
(c, e) 5 include
(d, f ) 6 include
(a, b) 7 include
(b, e) 7 include
(b, c) 8 exclude
(e, f ) 8 exclude
(b, d) 9 exclude
(e, g) 9 include
Table 8.3: Sequence of edges considered by Kruskal’s Algorithm and results.
• if the inclusion of the edge would induce a cycle in the tree created so far, it is
ignored, otherwise it takes it
• the algorithm terminates when it has taken n − 1 edges
Input : A weighted graph, G = (V, E)
Output : A minimum spanning tree of G
1 sort edges in nondecreasing order with respect to their weights
2 ET ← ∅
3 k←1
4 while |ET | < (n − 1) do
5 if (V, ET ∪ {ek }) is acyclic then
6 ET ← ET ∪ {ek }
7 end
8 k ← (k + 1)
9 end
10 output (V, ET )
Algorithm 49: Kruskal’s Minimum Spanning Tree Algorithm
An example run of Kruskal’s algorithm on the weighted graph in Figure 8.8(a) of Kruskal’s
can be found in Table 8.3. The resulting graph can be seen in Figure 8.8(b).
The presorting is simply Θ(m log m) with any good sorting algorithm.
The acyclicity check can be made using either DFS or BFS. If both of the edge’s endpoints
are in the same connected component created thus far, reject it.
In the worst case, the while loop will run for Θ(m) iterations. Each intermediate collection
of (disjoint) trees will be Θ(n) for BFS/DFS (why?) so in all,
Θ(m log m) + Θ(m) · Θ(n) ∈ O(n3 )
156
8.5 Minimum Spanning Tree Algorithms
a c a c
7 8 7
5 b 5 5 b 5
9 7 7
15
d e d e
6 8 6
f 9 f 9
11
g g
(a) A weighted, undirected graph. (b) Minimum Spanning Tree with total weight
39.
Figure 8.8: Minimum Spanning Tree example. For this particular example, Kruskal’s
and Prim’s algorithms result in the same tree. In fact, for this example there
is only one unique MST.
which can be improved to Θ(m log m) using an appropriate data structure (that removes
the need to perform a DFS/BFS).
8.5.3 Prim’s Algorithm
Prim’s algorithm is also greedy but works differently. It may result in a different, but
equivalent minimum spanning tree.
Starting at an arbitrary vertex, it builds subtrees by adding a single vertex on each
iteration. The vertex it chooses is based on a greedy choice: add the vertex whose edge
157
8 Graph Algorithms
connects the current subtree with the minimum weight.
Input : A weighted graph, G = (V, E)
Output : A minimum spanning tree of G
1 ET ← ∅
2 VT ← {v1 }
3 E ∗ ← N (v1 )
4 for i = 1, . . . , n − 1 do
5 e ← minimum weighted edge in E ∗
6 u ← the endpoint in e that is contained in VT
7 v ← the end point in e not contained in VT
8 VT ← VT ∪ {v}
9 ET ← ET ∪ {e}
10 E ∗ ← (E ∗ ∪ N (v)) \ {e = (x, y)|x ∈ VT ∧ y ∈ VT }
11 end
12 output (VT , ET )
Algorithm 50: Prim’s Minimum Spanning Tree Algorithm
During the execution of the algorithm, vertices can be partitioned into one of the following
sets:
• Tree vertices – vertices that have already been added to the tree
• Fringe vertices – vertices in the neighborhood of the set of tree vertices
• Unseen vertices – all other vertices that would not affect the next-vertex greedy
choice
On each iteration, we move the closest fringe vertex v to the set of tree vertices, add the
relevant edge and update the fringe vertex set by adding the neighborhood of v, N (v).
An example run on of Prim’s algorithm on the weighted graph in Figure 8.8(a) can be
found in Table 8.4. The resulting graph can be seen in Figure 8.8(b) (which is the same
MST produced by Kruskal’s). A visual snapshot after the second iteration can be found
in Figure 8.10, detailing the tree, fringe, and unseen vertices/edges.
The performance depends on the data structures we use to maintain fringe vertices/edges.
If the priority queue is implemented as a min-heap, deletion and insertion are at most
O(log m). Another important observation is that Prim’s does not involve any cycle
detection subroutine. Instead edges can be excluded based on whether or not their end
points are in the tree vertex set VT which can be tested in amortized constant time with
an appropriate data structure.
158
8.5 Minimum Spanning Tree Algorithms
Tree Vertices Fringe Vertices Unseen Vertices
Figure 8.9: Illustration of Tree, Fringe, and Unseen vertex sets.
a c
7 8
5 b 5
9 7
15
d e
6 8
f 9
11
g
Figure 8.10: Prim’s Algorithm after the second iteration. Tree vertices (green), Fringe
vertices (yellow), and unseen vertices (grey) are highlighted. Fringe edges
(dark green) connect tree vertices and fringe vertices. The next edge to be
added, (a, b) is highlighted in red.
159
8 Graph Algorithms
Iteration Tree Vertices Fringe Edges Vertex added Edge Weight
– ∅ – a –
1 {a} (a, d, 5), (a, b, 7) d 5
2 {a, d} (d, f, 6), (a, b, 7), f 6
(d, b, 9), (d, b, 15)
3 {a, d, f } (a, b, 7), (f, e, 8), (d, b, 9), b 7
(f, g, 11), (d, e, 15)
4 {a, b, d, f } (b, e, 7), (b, c, 8), (f, e, 8), e 7
(f, g, 11), (d, e, 15)
5 {a, b, d, e, f } (e, c, 5), (b, c, 8), c 5
(e, g, 9), (f, g, 11)
6 {a, b, c, d, e, f } (e, g, 9), (f, g, 11) g 9
Table 8.4: Sequence of edges considered by Prim’s Algorithm starting at vertex a and
results.
8.6 Minimum Distance Algorithms
Recall the Shortest Path problem (Problem 10). We have seen a BFS solution that works
for several special cases. We now consider two solutions that work for weighted graphs
(either directed or undirected).
8.6.1 Dijkstra’s Algorithm
Dijkstra’s algorithm works making the following greedy choice: from a source vertex s,
it chooses a path (edge) to the its nearest neighbor. Then it chooses its second nearest
neighbor and so on. By the i-th iteration, it has chosen the shortest paths to i − 1 other
vertices.
The idea is similar to Prim’s in that we make our selection from a set of fringe vertices.
Each vertex is given two labels: the tree vertex by which the current shortest path can
be reached and the length of the shortest path. To choose the next vertex to be added
to the tree, we simply choose the minimal length among all fringe vertices, breaking ties
arbitrarily.
Once a vertex u has been moved from the fringe to part of the tree, we must update each
fringe edge that is adjacent to u∗ . If, via u a vertex v can be reached by a shorter path,
160
8.6 Minimum Distance Algorithms
20
c f
5 5
20
a 20 10 i
5
5
10 g 15
7 d 15
15
20 5
b 5 15 j
10
5 20
e h
Figure 8.11: Weighted directed graph.
then we update the labels of v to the new shortest path and the vertex label u.
Input : A weighted graph, G = (V, E), a source vertex s ∈ V
Output : For each v ∈ V , the minimal weighted path dv for p : s v
1 Q ← empty min-priority queue
2 foreach v ∈ V \ {s} do
3 dv ← ∞
4 pv ← φ //the predecessor to v in the shortest path from s
5 Q.enqueue(v, dv )
6 end
7 ds ← 0
8 pv ← φ
9 Q.enqueue(s, ds )
10 VT ← ∅
11 for i = 1, . . . , n do
12 u ← Q.dequeue
13 VT ← VT ∪ {u}
14 foreach v ∈ N (u) \ VT do
15 if du + wt(u, v) < dv then
16 dv ← du + wt(u, v)
17 pv ← u
18 Q.DecreaseP riority(v, dv )
19 end
20 end
21 end
22 output dv1 , . . . , dvn , pv1 , . . . , pvn ,
161
Algorithm 51: Dijkstra’s Single-Source Minimum Distance Algorithm
8 Graph Algorithms
c f
20
a 20 i
5
5
10 g
d
b 5 j
5
e h
Figure 8.12: Result of Dijsktra’s Algorithm with source vertex e.
An example run of Dijkstra’s Algorithm on the graph in Figure 8.11 can be found in
Table 8.5. The resulting shortest path tree is in Figure 8.12.
If you wanted to actually build a shortest path from the source vertex to any other
vertex v, you could “back track” from v using the previous vertex values, pv computed
by Dijkstra’s algorithm. Starting at v, you look up pv = u, then you look up pu and so
on until you arrive at the source vertex s.
As presented, using a min-heap implementation (or even a balanced binary search tree
implementation) for the priority queue will lead to a loose O(n2 log n) analysis. However,
using a more advanced heap implementation such as a Fibonacci heap can improve the
running time to O(m + n log n).
8.6.2 Floyd-Warshall Algorithm
Recall that by using BFS, we could determine the shortest path to all vertices from a
given source vertex. By running BFS for each vertex we can get all pairs shortest paths.
Floyd’s algorithm works equally well on directed and undirected graphs, but is particularly
interesting for weighted graphs of either type.
Pitfall: Floyd’s algorithm does not work on a graph with a negatively weighted cycle.
Why?
Floyd’s starts with the usual adjacency matrix with entries that are the weights of the
edges. Much like Warshall’s, Floyd’s algorithm computes intermediate distance matrices,
D(0) , . . . , D(k−1) , D(k) , . . . , D(n)
Each intermediate distance matrix D(k) contains entries di,j which correspond to the
162
8.6 Minimum Distance Algorithms
(a) Prior to the first it- (b) First iteration, N (e) (c) Second iteration, (d) Iteration 3, N (d) is
eration. is explored. N (b) is explored explored
Vertex dv p v Vertex dv pv Vertex dv pv Vertex dv pv
a ∞ φ a ∞ φ a 12 b a 10 d
b ∞ φ b 5 e c ∞ φ c 25 d
c ∞ φ c ∞ φ d 5 e f 25 d
d ∞ φ d 5 e f ∞ φ g 15 d
e 0 φ f ∞ φ g 20 e h ∞ φ
f ∞ φ g 20 e h ∞ φ i ∞ φ
g ∞ φ h ∞ φ i ∞ φ j ∞ φ
h ∞ φ i ∞ φ j ∞ φ
i ∞ φ j ∞ φ
j ∞ φ
(h) Final Shortest
(e) Iteration 4, N (a) is (f) Iteration 5, N (g) is (g) Iteration 6, N (i) is Paths from e and
explored explored explored Precesessors
Vertex dv p v Vertex dv pv Vertex dv pv Vertex dv pv
c 25 d c 25 d c 25 d a 10 d
f 25 d f 25 d f 25 d b 5 e
g 15 d h ∞ φ h ∞ φ c 25 d
h ∞ φ i 20 g j ∞ φ d 5 e
i ∞ φ j ∞ φ e 0 φ
j ∞ φ f 25 d
g 15 d
h ∞ φ
i 20 g
j ∞ φ
Table 8.5: Walkthrough of Dijkstra’s Algorithm for source vertex e . Iterations 7 – 9 are
omitted as they do not result in any further changes.
163
8 Graph Algorithms
vk
(k−1) (k−1)
di,k dk,j
vi vj
(k−1)
di,j
Figure 8.13: Basic idea behind Floyd-Warshal: Supposing that a path from vi vj has
(k−1)
already been found with a distance of di,j , the consideration of vk as a
(k−1) (k−1)
new intermediate node may have shorter distance, di,k + dk,j .
total weight of the shortest path from vi to vj not using any vertex numbered higher
than k.
Also as before, we can compute each intermediate distance matrix by using its immediate
predecessor. The same reasoning as before yields a similar recurrence:
n o
(k) (k−1) (k−1) (k−1) (0)
di,j ← min di,j , di,k + dk,j , di,j = wi,j
The idea behind this algorithm is illustrated in Figure 8.13.
Input : A weighted graph, G = (V, E)
Output : A shortest distance matrix D of G
1 D ← (n × n) matrix
2 for 1 ≤ i, j ≤ n do
3 di,j ← wt(vi , vj ) //∞ if no such edge
//Initialize the successor matrix S: set si,j = j for all edges
(vi , vj )
4 end
5 for k = 1, . . . , n do
6 for i = 1, . . . , n do
7 for j = 1, . . . , n do
8 di,j = min{di,j , di,k + dk,j }
//if di,j is updated, set the successor matrix value
si,j = si,k
9 end
10 end
11 end
12 output D
Algorithm 52: Floyd’s All Pair Minimum Distance Algorithm
164
8.6 Minimum Distance Algorithms
Observations:
• We can keep track of which k corresponds to an update to reconstruct the minimum
paths
• Clearly the algorithm is O(n3 ).
• An example of a full run of Floyd’s can be found in Figure 8.14
A mentioned, we can keep track of which value(s) of k caused the matrix entries to
change. This results in a successor matrix S as depicted in Figure 8.14(h). We now
describe how we can use this successor matrix to construct the shortest path for any pair.
Suppose that we wanted to construct the shortest path vi vj . We start at vi ; to find
the next vertex, we reference the successor matrix S by looking at the i-th row and j-th
column which gives us the next vertex in the shortest path. Suppose this is v` . Then to
find the next vertex, we want the entry in the `-th row, j-th column. We continue until
the entry is equal to vj . The complete pseudocode is found in Algorithm 53. For the
example, the shortest path a d is
p:a→b→e→d
Input : A successor matrix S produced by Floyd-Warshall for a graph
G = (V, E), two vertices vi , vj ∈ V
Output : The shortest path p : vi vj in G
1 p ← vi
2 x ← vi
3 while x 6= vj do
4 x ← Sx,vj
5 p←p+x
6 end
7 output p
Algorithm 53: Construct Shortest Path Algorithm
We present another full example. Consider the graph in Figure 8.15(a) which represents
a tournament graph (a complete Directed Acyclic Graph (DAG)). Each iteration of
Floyd-Warshall updates entries in the upper diagonal because a new, shorter path found
each time. The matrices at each iteration are presented in Figure 8.15 along with the
successor matrix. Consider constructing the shortest path a e using this matrix. We
first look at entry Sa,e which is b so b is the next vertex in the shortest path. Then we
look at the entry Sb,e which is c. Proceeding in this manner until Sd,e = e we construct
the path:
p:a→b→c→d→e
165
8 Graph Algorithms
1
a b 2
2 4 8 e
c 3
d
5
(a) A weighted directed graph.
0 1 2 ∞ ∞ 0 1 2 ∞ ∞ 0 1 2 9 3
∞ 0 ∞ 8 2
∞ 0 ∞ 8 2
∞ 0 ∞ 8 2
∞ ∞ 0 5 ∞
∞ ∞ 0 5 ∞
∞ ∞ 0 5 ∞
∞ 4 ∞ 0 ∞ ∞ 4 ∞ 0 ∞ ∞ 4 ∞ 0 6
∞ ∞ ∞ 3 0 ∞ ∞ ∞ 3 0 ∞ ∞ ∞ 3 0
(b) Initial distance matrix (c) Distance matrix after itera- (d) Distance matrix after iter-
tion k = 1 ation k = 2
0 1 2 7 3 0 1 2 7 3 0 1 2 6 3
∞ 0 ∞ 8 2
∞ 0 ∞ 8 2
∞ 0 ∞ 5 2
∞ ∞ 0 5 ∞
∞ 9 0 5 11
∞ 9 0 5 11
∞ 4 ∞ 0 6 ∞ 4 ∞ 0 6 ∞ 4 ∞ 0 6
∞ ∞ ∞ 3 0 ∞ 7 ∞ 3 0 ∞ 7 ∞ 3 0
(e) Distance matrix after iter- (f) Distance matrix after it- (g) Distance matrix after it-
ation k = 3 eration k = 4 eration k = 5
− b c b b
− − − e e
− d − d d
− b − − b
− d − d −
(h) Successor matrix pro-
duced by Floyd’s Algorithm
Figure 8.14: Floyd’s Algorithm Demonstration.
166
8.6 Minimum Distance Algorithms
15
10
5
1 1 1 1
a b c d e
5 5
10
(a) A weighted directed graph.
0 1 5 10 15 a b c d e
∞ 0 1 5 10 − b c
d e
∞ ∞ 0 1 5 − − c
d e
∞ ∞ ∞ 0 1 − − − d e
∞ ∞ ∞ ∞ 0 − − − − e
(b) Initial distance and successor matrices
0 1 5 10 15 a b c d e
∞ 0 1 5 10 − b c
d e
∞ ∞ 0 1 5 − − c
d e
∞ ∞ ∞ 0 1 − − − d e
∞ ∞ ∞ ∞ 0 − − − − e
(c) After iteration k = 1
0 1 2 6 11 a b b b b
∞ 0 1 5 10 − b c
d e
∞ ∞ 0 1 5 − − c
d e
∞ ∞ ∞ 0 1 − − − d e
∞ ∞ ∞ ∞ 0 − − − − e
(d) After iteration k = 2
0 1 2 3 7 a b b b b
∞ 0 1 2 6
− b c c c
∞ ∞ 0 1 5
− − c d e
∞ ∞ ∞ 0 1 − − − d e
∞ ∞ ∞ ∞ 0 − − − − e
(e) After iteration k = 3
0 1 2 3 4 a b b b b
∞ 0 1 2 3
− b c c c
∞ ∞ 0 1 2
− − c d d
∞ ∞ ∞ 0 1 − − − d e
∞ ∞ ∞ ∞ 0 − − − − e
(f) After iteration k = 4
0 1 2 3 4 a b b b b
∞ 0 1 2 3
− b c c c
∞ ∞ 0 1 2
− − c d d
∞ ∞ ∞ 0 1 − − − d e
∞ ∞ ∞ ∞ 0 − − − − e 167
(g) After iteration k = 5
Figure 8.15: Another Demonstration of Floyd-Warshall’s Algorithm.
8 Graph Algorithms
8.6.3 Huffman Coding
See Tree Notes
8.7 Exercises
Exercise 8.1. Consider a weighted, complete graph Kn (where all vertices are connected
by edges). Further, assume that we are given G as an adjacency list where the list for
each vertex is ordered in increasing order with respect the edge weights. Describe an
O(|V |)-time algorithm to compute a minimum spanning tree for such an input.
Exercise 8.2. The Food Bucket, a regional chain of restaurants wants you to develop a
program that will generate mazes for its children’s menu. Rather than create one maze,
they want a program that will generate a random maze on an n × n grid such that there
is only one solution (that is, one path from the lower left to the upper right corners).
The algorithm should produce a random maze with various paths connecting grid points
(representing a traversable path) with only a single path leading from grid point (0, 0) to
grid point (n − 1, n − 1). There should also not be any cycles. The algorithm should be
correct and efficient.
Exercise 8.3. Let G = (V, E) be an undirected, unweighted graph. Let δ(u, v) be the
minimum distance between vertices u, v ∈ V . The eccentricity of a vertex v ∈ V is
defined as the maximal minimum distance between v and any other vertex in G:
(v) = max δ(v, u)
u∈V
The diameter d of G is then defined as the maximal eccentricity of any vertex in G:
d = max (v)
v∈V
The radius of a graph is the minimum eccentricity of any vertex in G,
r = min (v)
v∈V
(a) Design an algorithm that utilizes either DFS, BFS, or some variation/application of
DFS/BFS to compute the radius of a graph G. Provide good pseudocode and give a
brief analysis of your algorithm.
(b) Design an algorithm that utilizes either DFS, BFS, or some variation/application
of DFS/BFS to compute the diameter of a graph G. Provide good pseudocode and
give a brief analysis of your algorithm.
Exercise 8.4. Give an example of weighted (directed or undirected) in which the BFS
Tree does not provide the shortest distance path.
168
8.7 Exercises
Exercise 8.5. Reconsider the condensation graph Gc of a directed graph G.
(a) Suppose that G is strongly connected. What does Gc look like?
(b) Suppose that G is a DAG, what does Gc look like?
Exercise 8.6. Give an algorithm to solve the following problem: given a graph G = (V, E)
and vertices u, v, x, determine if there exists a path from u to v that involves x
Exercise 8.7. Provide a small example of a weighted undirected graph whose MST
produced from Kruskal’s algorithm would be different from that produced by Prim’s
algorithm.
Exercise 8.8. Suppose that we restrict the MST problem to weighted graphs whose
weights are all positive integers; that is wt : E → Z+ . Show that each of the following
variations are equivalent to this formulation by showing a transformation between them.
That is, describe a transformation from the variation below to the positive weighted
version and describe how a solution to the positive weighted version can be transformed
back to a solution for the variation.
1. Let wt : E → Z (that is we allow negative and zero weighted edges)
2. Rather than finding the minimum spanning tree, suppose that we wish to find the
maximum spanning tree
Exercise 8.9. Let G = (V, E) be an undirected graph (unweighted). Prove or disprove:
the minimum spanning tree T formed by Kruskal’s algorithm also provides a minimum
distance tree: that is, for any two vertices x, y, the unique path between them in the
MST T is also the shortest path in the original graph G.
Exercise 8.10. Let G = (V, E) be an undirected graph (unweighted) with x ∈ V . Prove
or disprove: the minimum spanning tree T formed by Prim’s algorithm by starting at x
also provides a minimum distance tree: that is the unique path from x to any other node
y in T is the shortest path from x to y in G.
Exercise 8.11. Each iteration of Kruskal’s algorithm considers adding an edge e = (u, v)
to the current minimum spanning tree T by checking whether or not its inclusion in T
would induce a cycle. Gomer thinks he’s found a better way: rather than checking for a
cycle, just check if the end points of e are already in T : if they are, then do not include
the edge as they would not add any vertex to the minimum spanning tree. If either end
point (u or v) is outside the current tree then do add the edge as it would connect the
tree further. Show that Gomer is wrong by providing an example of a tree where using
this criteria instead would fail. Briefly explain why Gomer’s criteria is wrong.
Exercise 8.12. Recall that a graph is a tree if it contains no cycles. Adapt either DFS
or BFS to develop an algorithm that determines if a given graph G = (V, E) is a tree or
not. Provide good pseudocode and fully analyze your algorithm.
169
8 Graph Algorithms
Exercise 8.13. Kruskal’s algorithm works by checking connectivity as a sub routine.
Connectivity questions are usually solved using a DFS or BFS which will require a data
structure such as a stack or a queue which may need to hold up to O(n) references
to vertices. Assume that we have the following function (oracle, subroutine, etc.):
isConnected(G, u, v), that, given an undirected graph G = (V, E) and two vertices
u, v ∈ V outputs true if there is a path between u, v in G and false otherwise.
Show that Kruskal’s algorithm can be performed without the use of DFS/BFS or any
O(n) extra space (that is, show that it can be performed with only constant extra space.
Exercise 8.14. Implement DFS in the high-level programming language of your choice.
Exercise 8.15. Implement BFS in the high-level programming language of your choice.
Exercise 8.16. Implement Kruskal’s in the high-level programming language of your
choice.
Exercise 8.17. Implement Prim’s in the high-level programming language of your choice.
Exercise 8.18. Implement Dijkstra’s Algorithm in the high-level programming language
of your choice.
Exercise 8.19. Implement Floyd’s Algorithm in the high-level programming language
of your choice.
170
9 Dynamic Programming
9.1 Introduction
One pitfall of recursive solutions is unnecessarily recomputing solutions. A classic example
is the Fibonacci sequence.
Fn = Fn−1 + Fn−2 , F0 = 0, F1 = 1
Observing the computation tree of a recursive algorithm to compute a fibonacci number
we realize that many of the computations are repeated an exponential number of times.
One method for avoiding recomputing values is memoization: once a value has been
computed it is placed into a table. Then, instead of recomputing it we check the table; if
it has been computed already, we use the value stored in the table, if not then we go
through the effort of computing it.
Both of these approaches are fundamentally top-down solutions. That is, they start with
the larger problem, then attempt to solve it by computing solutions to sub problems.
Dynamic Programming is an algorithmic technique that seeks bottom-up solutions. Rather
than dividing the problem up into sub-problems, dynamic programming attempts to
solve the sub-problems first, then uses them to solve the larger problem.
This approach typically works by defining a tableau (or several) and filling them out
using previously computed values in the tableau. In general, recursion is avoided and the
algorithm is achieved with just a few loops.
9.1.1 Optimal Substructure Property
In general, for a problem to have a dynamic programming solution, it must posses the
optimal substructure property. This is a formalization of the idea that the optimal solution
to a problem can be formulated by (efficiently) finding the optimal solution to each of its
sub-problems. This property is also integral to greedy algorithmic solutions. Obviously,
not all problems poses this property.
171
9 Dynamic Programming
9.2 Binomial Coefficients
Recall the choose function:
n n!
=
k (n − k)!k!
which is usually referred to as a binomial coefficient as it represents the k-th coefficient
in the expansion of a binomial:
n n n n n−k k n n
(x + y) = x + ··· + x y + ··· y
0 k n
Pascal’s identity allows one to express binomial coefficients as the sum of other binomial
coefficients:
n n−1 n−1
= +
k k−1 k
with
n n
= =1
0 n
as base cases to the recursive definition.
However, such a solution turns out to be exponential as many of the sub-solutions are
recomputed many times. To illustrate, consider Code Snippet 9.1. A recent run of
this program on the cse server to compute binomial(38,12) resulted in 1, 709, 984, 303
recursive calls and took about 20 seconds. In contrast, using memoization to avoid
repeated computations resulted in only 573 function calls and took less than 1ms.
1 long binomial(int n, int k) {
2
3 if(k < 0 || n < 0) {
4 return -1;
5 } else if(k == 0 || n == k) {
6 return 1;
7 } else {
8 return binomial(n-1, k-1) + binomial(n-1, k);
9 }
10 }
Code Sample 9.1: Recursive Binomial Computation
Memoization is still a top-down solution that utilizes recursion. Dynamic programming
is a bottom-up solution that does not make function calls, but rather fills out a tableau
of values.
In the case of binomial coefficients, we define a (n + 1) × (k + 1) sized table (see Table
9.1).
172
9.2 Binomial Coefficients
0 1 2 3 4 ··· k−1 k
0 1
1 1 1
2 1 2 1
3 1 3 3 1
4 1 4 6 4 1
.. ..
. .
k 1 ··· 1
..
.
n−1 n−1
n−1 1 ··· k−1 k
n
n 1 ··· k
Table 9.1: Tableau for Binomial Coefficients. Astute readers will recognize the tableau
as Pascal’s Triangle.
The tableau is filled out top-to-bottom, left-to-right with the trivial base cases pre-filled.
Input : Integers, n, k, 0 ≤ k ≤ n
n
Output : The binomial coefficient k
1 for i = 0, . . . , n do
2 for j = 0, . . . , min i, k do
3 if j = 0 ∨ j = k then
4 Ci,j ← 1
5 else
6 Ci,j ← Ci−1,j−1 + Ci−1,j
7 end
8 end
9 end
10 output Cn,k
Algorithm 54: Binomial Coefficient – Dynamic Programming Solution
173
9 Dynamic Programming
Analysis: clearly the elementary operation is addition and in total, the algorithm is
Θ(nk). This matches the complexity of a memoization approach, but avoids any additional
function calls.
9.3 Optimal Binary Search Trees
We’ve considered Binary Search Trees and Balanced BSTs. Consider the following
variation: suppose we know in advance the keys to be searched along with a known (or
estimated) probability distribution of searches on these keys.
We don’t necessarily want a balanced binary search tree, rather we want a BST that
will minimize the overall expected (average) number of key comparisons. Intuitively,
keys with higher probabilities should be shallower in the tree while those with lower
probabilities should be deeper. Specifically, average number of comparisons made would
be: n
X
h(ki ) · p(ki )
i=1
where h(ki ) is the level in which the key ki lies and p(ki ) is the probability of searching
for ki . Our goal will be to compute the Optimal Binary Search Tree (OBST).
As an example, consider the key distribution in Table 9.2 and the trees in Figure 9.1.
There are several valid BSTs, but the expected number of comparisons given the key
probabilities are different.
k p(k)
a 0.6
b 0.3
c 0.1
Table 9.2: Probability distribution for keys
The number of possible BSTs with n keys corresponds to the Catalan numbers.
Definition 13. The Catalan numbers are a sequence of natural numbers defined by
1 2n (2n)!
Cn = =
n+1 n (n + 1)!n!
• The first few numbers in the sequence: 1, 1, 2, 5, 14, 42, 132, 429, . . . ,
• Corresponds to the number of valid, balanced parenthesization of operations:
((ab)c)d, (a(bc))d, (ab)(cd), a((bc)d), a(b(cd))
174
9.3 Optimal Binary Search Trees
a a
b
x c x b
a c b c
(a) A balanced tree, but (b) Another tree, but (c) The optimal config-
the expected number of the expected number of uration, the expected
comparisons is 1.7 comparisons is still 1.7 number of comparisons
is 1.5
Figure 9.1: Several valid Binary Search Trees
• Corresponds to the number of “full” (every node has 0 or 2 children) binary trees
with n + 1 leaves. This is the same a binary tree with n key nodes (which will
necessarily have n + 1 sentinel leaves).
• Many other interpretations
• They have an exponential growth:
4n
Cn ∈ O
n1.5
OBSTs are not ideal for applications that perform a lot of insertion and deletion operations.
The introduction of a new key or the elimination of a key will necessarily change the
probability distribution and a new OBST may have to be generated from scratch. The
use cases are for more static, indexed collections that do not change frequently.
For a dynamic programming solution (due to Knuth [9]), we need a recurrence. Let Ci,j
be defined as the smallest average number of comparisons made in a successful search
in a binary tree Ti,j involving keys ki through kj for 1 ≤ i, j ≤ n. Ultimately, we are
interested in finding C1,n —the optimal BST involving all keys.
To define a recurrence, we need to consider which key should be the root of the intermediate
tree Ti,j , say kl . Such a tree will have kl as the root and a left subtree, Ti,l−1 containing keys
ki , . . . kl−1 optimally arranged and the right-sub-tree, Tl+1,j containing keys kl+1 , . . . , j.
j
X
Ci,j = min Ci,k−1 + Ck+1,j + p(ks )
i≤k≤j
s=i
for 1 ≤ i ≤ j ≤ n. Note that the sum is invariant over all values of k: it represents the
fact that building the tree necessarily adds 1 comparison to the depth of all the keys. A
visualization of the split can be found in Figure 9.2.
Some obvious corner cases:
175
9 Dynamic Programming
k`
ki , . . . , k`−1 k`+1 , . . . , kj
Figure 9.2: Optimal Binary Search Tree split
HH
j
HH 0 1 2 ··· n
i HH
1 0 p(k1 ) C1,n
2 0 0 p(k2 )
.. ..
. .
n 0 0 ··· 0 p(kn )
n+1 0 0 ··· 0 0
Table 9.3: Optimal Binary Search Tree Tableau
• Ci,i−1 = 0 for 1 ≤ i ≤ n + 1 since no comparisons are made in an empty tree.
• Ci,i = p(ki ) for 1 ≤ i ≤ n since an optimal tree of size 1 is the item itself.
With these values, we can define an (n + 1) × (n + 1) tableau (see Table 9.3).
Each entry in the tableau requires previous values in the same row to the left and in the
same column below as depicted in Figure 9.3. Thus, the tableau is filled out along the
diagonals from top-left to bottom-right. The first diagonal is initialized to zero and the
second with each of the respective probabilities of each key. Then each successive table
entry is calculated according to our recurrence.
The final value C1,n is what we seek. However, this gives us the average number of key
comparisons (minimized), not the actual tree. To build the tree, we also need to maintain
a root table that keeps track of the values of k for which we choose the minimum (thus
kk is the root and we can build top down).
176
9.3 Optimal Binary Search Trees
..
.
· · · Ci,i−1 Ci,i Ci,i+1 · · · Ci,j−2 Ci,j−1 Ci,j ···
Ci+1,j
Ci+2,j
Ci+3,j
..
.
Cj,j
Cj+1,j
..
.
Figure 9.3: Visualization of the OBST tableau. Computing the minimal value for Ci,j
involves examining potential splits of keys, giving pairs of sub-solutions. This
indicates that we need elements in the same row to the left and in the same
column below. Thus, the algorithm needs to complete the tableau top-left to
bottom-right diagonally.
A stack-based algorithm for constructing the OBST is presented in Algorithm 56.
Input : A set of keys k1 , . . . , kn and a probability distribution p on the keys
Output : An optimal Binary Search Tree
1 for i = 1, . . . , n do
2 Ci,i−1 ← 0
3 Ci,i ← p(ki )
4 Ri,i ← i
5 end
6 Cn+1,n ← 0
7 for d = 1, . . . , (n − 1) do
8 for i = 1, . . . , (n − d) do
9 j ←i+d
10 min ← ∞
11 for k = i, . . . , j do
12 q ← Ci,k−1 + Ck+1,j
13 if q < min then
14 min ← q
15 Ri,j ← k
16 end
17 end
Pj
18 Ci,j ← min + s=i p(ks )
19 end
20 end 177
21 output C1,n , R
Algorithm 55: Optimal Binary Search Tree
9 Dynamic Programming
Input : A set of keys k1 , . . . , kn and root Table R
Output : The root node of the OBST
1 root ← new root node
2 r.key ← R1,n
3 S ← empty stack
4 S.push(r, 1, n) //In general, we push a node u, and indices i, j
5 while S is not empty do
6 (u, i, j) ← S.pop
7 k ← Ri,j //this is the key corresponding to u
8 if k < j then
//create the right child and push it
9 v ← new node
10 v.key ← Rk+1,j
11 u.rightChild ← v
12 S.push(v, k + 1, j)
13 end
14 if i < k then
//create the left child and push it
15 v ← new node
16 v.key ← Ri,k−1
17 u.lef tChild ← v
18 S.push(v, i, k − 1)
19 end
20 end
21 output root
Algorithm 56: OBST Tree Construction
9.3.1 Example
The full tableaus can be found in Table 9.5. As an example computation:
k = 1 : C1,0 + C2,2 + 2s=1 p(ks ) = 0 + .02 + (2.13 + .02) = .253
P
C1,2 = min
k = 2 : C1,1 + C3,2 + 2s=1 p(ks ) = 0.213 + .02 + (2.13 + .02) = .446
P
1≤k≤2
These two options correspond to splitting the sub-tree involving keys A, B at A, B
respectively. That is, A as the root with B as its right-child or B as the root with A as
its left-child. The values indicate that k = 1 is the better option.
178
9.4 Dynamic Knapsack
Key Probability
A .213
B .020
C .547
D .100
E .120
Table 9.4: Optimal Binary Search Tree Example Input
(b) Root array indicating which keys
(a) Cost tableau for OBST resulted in the optimal split.
HH j HH j 1 2 3 4 5
HH HH
0 1 2 3 4 5
i HH i HH
1 0 0.213 0.253 1.033 1.233 1.573 1 1 1 3 3 3
2 0 0.020 0.587 0.787 1.127 2 2 3 3 3
3 0 0.547 0.747 1.087 3 3 3 3
4 0 0.100 0.320 4 4 5
5 0 0.120 5 5
6 0
Table 9.5: Tableaus resulting from the Optimal Binary Search Tree example
= 2 : C2,1 + C3,5 + 5s=2 p(ks ) = 1.874
P
k
= 3 : C2,2 + C4,5 + 5s=2 p(ks ) = 1.127
P
k
C2,5 = min P5
k
2≤k≤5 = 4 : C2,3 + C5,5 + s=2 p(ks ) = 1.494
= 5 : C2,4 + C6,5 + 5s=2 p(ks ) = 1.573
P
k
The final result can be seen in Figure 9.4. The expected number of key comparisons
for any search is 1.573. Though this particular example resulted in a balanced BST, in
general, OBSTs need not be balanced.
9.4 Dynamic Knapsack
Recall the 0-1 Knapsack Problem (see Problem 4). This problem lends itself to an
“efficient” dynamic programming solution.
Let Vi,j be the optimal solution (a subset of items) involving the first i objects subject to
an intermediate weight constraint j. Clearly, 1 ≤ i ≤ n and 1 ≤ j ≤ W .
For a given Vi,j we can divide all the possible subsets of the first i items that fit a
knapsack capacity of j into two groups: those that include item i and those that do not.
179
9 Dynamic Programming
C
A E
x B D y
Figure 9.4: Final Optimal Binary Search Tree.
• Among subsets that do not include the i-th element, the value of the optimal subset
is Vi−1,j .
• Among the subsets that do include the i-th element, the optimal subset will be
made up of the i-th item and the optimal subset of the first i − 1 items that fit
into a knapsack of capacity j − wi . This is exactly
vi + Vi−1,j−wi
.
We are interested in maximizing our total value, so we take the max of these two solutions
if feasible.
max Vi−1,j , vi + Vi−1,j−wi if j − wi ≥ 0
Vi,j =
Vi−1,j if j − wi < 0
In addition, we define the initial conditions
• V0,j = 0 for j ≥ 0 (taking no items has no value)
• Vi,0 = 0 for i ≥ 0 (no capacity means we can take no items)
The tableau that we fill out will be a (n + 1) × (W + 1) (rows, columns) sized table
numbered 0 . . . n and 0 . . . W respectively.
The tableau can be filled out as follows. For each entry, we can take the maximum of:
• the entry in the previous row of the same column and
• the sum of vi and the entry in the previous row and wi columns to the left.
This enables us to calculate row-by-row (left to right) or column-by-column (top to
bottom).
We can also build the optimal solution using the same table by working backwards. For
6 Vi−1,w . If this is
each intermediate capacity w, item i is in the optimal subset if Vi,w =
the case, we can update the intermediate capacity, w − wi and look at the corresponding
column to continue.
180
9.4 Dynamic Knapsack
H
HH j
0 ··· j − wi ··· ··· W
i H
HH
0 0 ··· 0 ··· 0 ··· 0
1 0 ··· ··· ···
..
.
i−1 0 ··· Vi−1,j−wi ··· Vi−1,j ···
i 0 ··· ··· Vi,j ···
..
.
n 0 ··· ··· Vi,j ··· Vn,W
Table 9.6: Tableau for Dynamic Knapsack. The value of an entry Vi,j depends on the
values in the previous row and columns.
Visually, we start with entry Vn,W . We then scan upwards in this column until the table
value changes. A change from row i to row i − 1 corresponds to taking item i with weight
wi . We then jump left in the table on row i − 1 to column j − wi (where j is the column
we started out) and repeat the process until we have reached the border of the tableau.
Input : Completed tableau V of a Dynamic Programming 0-1 knapsack
solution
Output : The optimal knapsack S
1 S←∅
2 i←n
3 j←W
4 while i ≥ 1 ∧ j ≥ 1 do
5 while i ≥ 1 ∧ Vi,j = Vi−1,j do
6 i ← (i − 1)
7 end
8 S ← S ∪ {ai }
9 j ← (j − wi )
10 i ← (i − 1)
11 end
12 output S
Algorithm 57: 0-1 Knapsack Generator
181
9 Dynamic Programming
Item Weight Value
a1 5 10
a2 2 5
a3 4 8
a4 2 7
a5 3 7
Table 9.7: Example input for 0-1 Knapsack
H
HH j
0 1 2 3 4 5 6 7
i H
HH
0 0 0 0 0 0 0 0 0
1 0 0 0 0 0 10 10 10
2 0 0 5 5 5 10 10 15
3 0 0 5 5 8 10 13 15
4 0 0 7 7 12 12 15 17
5 0 0 7 7 12 14 15 19
Table 9.8: Resulting tableau for the 0-1 Knapsack Example
9.4.1 Example
The resulting tableau can be found in figure 9.8. The corresponding optimal knapsack
would be {a2 , a4 , a5 }; the backtracking is illustrated in Figure 9.5.
9.4.2 Analysis
Clearly, the running time is equivalent to the number of entries that we compute, Θ(nW ).
Unfortunately this fails to give a complete picture. Recall that W is part of the input,
thus the input size is log W . If W ∈ O(2n ) for example, this is clearly not polynomial.
Algorithms with analysis like this are called pseudopolynomial (or more specifically,
pseudolinear in this case). This is because another parameter of the input “hides”
the true complexity of the algorithm. If we considered W to be constant or even
W ∈ O(nk ) then the algorithm runs in polynomial time. In general, such restrictions are
not reasonable and the problem remains NP-complete.
9.5 Coin Change Problem
The Coin Change problem is the problem of giving the minimum number of coins adding
up to a total L in change using a given set of denominations {c1 , . . . , cn }. For certain
182
9.5 Coin Change Problem
0 1 2 3 4 5 6 7
0 0 0 0 0 0 0 0 0
a1 0 0 0 0 0 10 10 10
a2 0 0 5 5 5 10 10 15
a3 0 0 5 5 8 10 13 15
a4 0 0 7 7 12 12 15 17
a5 0 0 7 7 12 14 15 19
Figure 9.5: Dymamic Knapsack Backtracking
denominations (like US currency, how fortunate!), a simple greedy strategy works.
However, for other denominations, this approach doesn’t necessarily work. A dynamic
programming solution, however, guarantees an optimal solution. Optimal here means
that we are minimizing the number of coins used to make L change.
Let Ci,j be defined as the minimum number of coins from denominations {c1 , . . . , ci } that
add up to j for 1 ≤ i ≤ n and 0 ≤ j ≤ L. To build this array, we require two base cases.
• Ci,0 = 0 for all i (no coins needed for a total of zero)
• C0,j = ∞ for j 6= 0 (change is not possible with no coins)
Now, we can define a recurrence based on the intuitive subproblem split:
Ci−1,j for j < ci
Ci,j =
min{Ci−1,j , Ci,j−ci + 1} for j ≥ ci
The corresponding tableau can be found in Table 9.9.
The tableau is filled row-by-row (left-to-right) from top-to-bottom. The final solution
(the optimal number of coins) is found in entry Cn,L .
9.5.1 Example
Suppose that we have a set of four coin denominations (see Table 9.10).
The resulting tableau can be found in Table 9.11.
183
9 Dynamic Programming
H
HH j
0 1 2 ··· C
i H
HH
0 ∞ ∞ ∞ ··· ∞
..
1 0 .
2 0
.. ..
. .
n 0
Table 9.9: Tableau for the Coin Change Problem
Denomination Value
c1 5
c2 2
c3 4
c4 1
Table 9.10: Coin Change Example Input
9.6 Matrix Chain Multiplication
Suppose we have three matrices, A, B, C of dimensions (5 × 10), (10 × 15), (15 × 10)
respectively. If we were to perform the multiplication ABC, we could do it one of two
ways; multiply AB first then by C or BC first then by A. That is, either
(AB)C
or
A(BC)
Using straightforward matrix multiplication, the first way would result in
(5 · 10 · 15) + (5 · 15 · 10) = 1, 500
multiplications while the second would result in
(5 · 10 · 10) + (10 · 15 · 10) = 2, 000
multiplications.
In general, suppose that we are given a chain of matrices A1 , . . . An and we wanted to
multiply them all but minimize the total number of multiplications. This is the matrix
chain multiplication problem. More generally, what parenthesization of the associative
operations minimizes the overall number of multiplication made to compute the product?
Clearly, each product, Ai Ai+1 has to be a valid operation. Further, we will assume that
each matrix has different dimensions (d1 , . . . , dn+1 )—if they were all square there would
be no point in finding the optimal order (they would all be the same).
184
9.6 Matrix Chain Multiplication
H
HH j
ci 0 1 2 3 4 5 6 7 8
i H
HH
- 0 - ∞ ∞ ∞ ∞ ∞ ∞ ∞ ∞
5 1 0 ∞ ∞ ∞ ∞ 1 ∞ ∞ ∞
2 2 0 ∞ 1 ∞ 2 1 3 2 4
4 3 0 ∞ 1 ∞ 1 1 2 2 2
1 4 0 1 1 2 1 1 2 2 2
Table 9.11: Coin Change Example Results
HH j
HH
1 2 ··· n
i HH
1 0
2 φ 0
.. .. ..
. . .
n φ φ ··· 0
Table 9.12: Matrix Chain Multiplication Tableau.
For this problem we define a (lower-triangular) table of size n × n. Each entry represents
the minimum number of multiplications it takes to evaluate the matrices from Ai . . . Aj .
The tableau can be filled in using the following recurrence relation:
(
0 if i = j
C(i, j) = min {C(i, k) + C(k + 1, j) + di−1 dk dj } otherwise
i≤k<j
The order in which the tableau is filled in begins with the base cases, down the diagonal
where i = j. Depending on how you orient your tableau, you will then work downward
in a diagonal fashion, see Table 9.12.
In addition, at each calculation, we fill in an S-array that keeps track of which value of k
was chosen for each entry. This value can be used to construct the optimal solution. The
S-array ranges 2 ≤ i ≤ n and 1 ≤ j ≤ n − 1; see Table 9.13.
HH
j
HH 1 2 ··· n−1
i HH
2 0
3 φ 0
.. .. ..
. . .
n φ φ ··· 0
Table 9.13: Matrix Chain Multiplication S-array.
185
9 Dynamic Programming
Matrix Dimensions
A1 30 × 35
A2 35 × 15
A3 15 × 5
A4 5 × 10
A5 10 × 20
A6 20 × 25
Table 9.14: Example Matrix Chain Multiplication Input
Input : n matrices, A1 , . . . , An with dimensions d0 , . . . , dn+1
Output : The number of multiplications in the optimal multiplication order
1 for i = 1, . . . , n do
2 Ci,j ← 0
3 Si+1,j ← i
4 end
5 for l = 2, . . . , n do
6 for i = 1, . . . , n − l + 1 do
7 j ←i+l−1
8 Ci,j ← ∞
9 for k = i, . . . , (j − 1) do
10 q ← Ci,k + Ck+1,j + di−1 dk dj
11 if q < Ci,j then
12 Ci,j ← q
13 Si,j ← k
14 end
15 end
16 end
17 end
18 output C1,n , S
Algorithm 58: Optimal Matrix Chain Multiplication
9.6.1 Example
Consider the following 6 matrices:
The final tableau can be found in Table 9.15.
For an example of the computation, consider the index at C2,5 :
186
9.6 Matrix Chain Multiplication
H
HH j
1 2 3 4 5 6
i H
HH
1 0 15,750 7,875 9,375 11,875 15,125
2 φ 0 2,625 4,375 7,125 10,500
3 φ φ 0 750 2,500 5,375
4 φ φ φ 0 1,000 3,500
5 φ φ φ φ 0 5,000
6 φ φ φ φ φ 0
Table 9.15: Final Tableau for the Matrix Chain Example
H
HH j
2 3 4 5 6
i H
HH
1 1 1 3 3 3
2 2 3 3 3
3 3 3 3
4 4 5
5 5
Table 9.16: Final S-array for the Matrix Chain Example
C2,2 + C3,5 + d1 d2 d5 = 0 + 2500 + 35 · 15 · 20 = 13, 000
C2,5 = min C2,3 + C4,5 + d1 d3 d5 = 2625 + 1000 + 35 · 5 · 20 = 7, 125
2≤k≤4
C2,4 + C5,5 + d1 d4 d5 = 4375 + 0 + 35 · 10 · 20 = 11, 375
Clearly, the minimum among these is 7,125. The k value that gave this result was 3,
consequently we place this value into our S-array, S2,5 = 3. The full S-array can be
found in Table 9.16.
Each entry in the S-array records the value of k such that the optimal parenthesization of
Ai Ai+1 Ai+2 . . . Aj splits the product between Ak and Ak+1 . Thus we know that the final
matrix multiplication in computing A1...n optimally is A1...s[1,n] As[1,n]+1...n . Multiplications
in each sub-chain can be calculated recursively; the intuitive base case is when you only
have two matrices to multiply.
From the S-array above, the initial split would be at S1,6 = 3. The full parenthesization
would be:
((A1 )(A2 A3 )) ((A4 A5 )(A6 ))
187
9 Dynamic Programming
9.7 Exercises
Exercise 9.1. Construct the C tableau, S-array and final parenthesization for the
following matrices:
Matrix Dimensions
A1 5 × 10
A2 10 × 3
A3 3 × 12
A4 12 × 5
A5 5 × 50
A6 50 × 6
Table 9.17: Matrix Chain Multiplication Exercise
Exercise 9.2. Gomer thinks he’s found a better solution to constructing an Optimal
Binary Search Tree. He thinks a greedy strategy would work just as well: Sort the keys
and choose the one with the highest probability as the root of the tree. Then, repeat this
process with the left/right keys. Show that Gomer’s strategy will not work by providing
an example input that will not be optimal with his strategy.
Exercise 9.3. Implement the dynamic programming solution to the Optimal Binary
Search Tree problem in the high-level programming language of your choice.
Exercise 9.4. Implement the dynamic programming solution to the Matrix Chain
Multiplication problem in the high-level programming language of your choice.
Exercise 9.5. Implement the dynamic programming solution to the 0-1 knapsack problem
in the high-level programming language of your choice.
Exercise 9.6. Implement the dynamic programming solution to the Coin Change
problem in the high-level programming language of your choice.
Exercise 9.7. Consider the following “game”: n items of values v1 , v2 , . . . , vn are placed
in a line. You are allowed to take an item only at the beginning or the end of the line.
Once you take an item, the second player is allowed to take an item at the beginning or
the end of the line. You take turns like this until all items have been taken. The winner
is the player whose item values sum to the greater amount.
Prove or disprove: a greedy-choice strategy will work; that is, always take the item with
the higher value.
Exercise 9.8. Recall that a palindrome is a string that is the same backwards and
forwards (“abba” or “deified”).
(a) Write pseudocode that, given a string s, determines the longest contiguous palin-
dromic subsequence in s. Analyze your algorithm.
188
9.7 Exercises
(b) A palindromic subsequence may not necessarily be contiguous. For example,
BBABCBCAB
has palindromic subsequences “BBBBB”, “BBCBB” and “BABCBAB” among others,
but are made from non-contiguous elements. Devise a dynamic programming solution
to find the length of the longest palindromic subsequence of a given string s. In
addition to finding the length, devise a scheme to output a palindromic subsequence
of maximal length. Give pseudocode and analyze your solution. Hint: consider
defining a tableau P such that Pi,j represents the length of the maximal palindromic
subsequence within the substring si , . . . , sj (where s = s1 s2 . . . sn ).
(c) Implement your dynamic programming solution in in the high-level programming
language of your choice.
189
10 Computational Models
10.1 Introduction
At the foundation of computer science, we have questions like:
• What are the minimum amount of resources required to solve a given problem?
• Given enough time or memory, can an algorithm be designed to solve any problem?
• Can algorithms solve any problem?
These notes will take a breadth approach to computability and complexity touching on
the following topics.
• Language Theory
• Computational Models
• Turing Machines
• Computability Theory
• Complexity Classes P & NP
• NP-Completeness
• Reductions
We will first establish a few fundamentals. As a motivation, consider the following
statement:
“Algorithms solve problems”
each part of this statement is imprecise from a mathematical point of view; each has an
intuitive and dictionary meaning, but we will want to give a more rigorous, quantifiable
mathematical equivalent. In particular,
“Turing Machines decide languages”
191
10 Computational Models
10.1.1 Languages
We’ve looked at several different types of problems:
• Numerical
• Graph
• Sorting
• Logic
We need a unified model that captures all different types of problems; for this we use
languages.
Definition 14. • An alphabet, Σ is a finite, nonempty set of symbols. Examples:
[A-Za-z], [0–9], {0, 1}.
• A string (or word ) over Σ is a finite combination of symbols from Σ. Example: any
integer is a string over Σ = [0 − 9].
• The empty string, denoted (sometimes λ) is a string containing no symbols from
Σ.
• The set of all finite strings over Σ is denoted Σ∗ .
• A language over Σ is a set (finite or infinite), L ⊆ Σ∗ .
It suffices to consider a binary alphabet Σ = {0, 1}
• Often for convenience we may include a delimiter in our alphabet
• Any alphabet can be converted to a binary alphabet using a reasonable encoding
• Example: a prefix-free encoding generated by a Huffman code
Language Operations
Operations can be performed on languages:
• Union – L1 ∪ L2
• Concatenation – L1 ◦ L2 (or just L1 L2 )
• Kleene Star – L∗
• Many others
The concatenation of two languages is the concatenation of all strings in each language.
L1 L2 = {xy | x ∈ L1 , y ∈ L2 }
192
10.1 Introduction
Examples: Let L1 = {0, 10}, L2 = {1, 01}. Then
L1 L2 = {01, 001, 101, 1001}
and
L1 L1 = {00, 010, 100, 1010}
The Kleene Star is a recursively defined operation. For a given language L, L0 = {λ}
and for n > 0, we define
Ln = L(n−1) L
For example, let L = {0, 10} then,
L0 = {λ}
L1 = {0, 10}
L2 = {00, 010, 100, 1010}
L3 = {000, 0010, 0100, 01010, 1000, 10010, 10100, 101010}
The Kleene Star operation is then defined as the union of all such concatenations.
[
L∗ = Ln
n≥0
For the alphabet Σ itself, Σ∗ is the set of all binary strings.
Sometimes it is useful to use the following notation to consider only nonempty strings.
[
L+ = Ln = L∗ − {λ}
n≥1
Regular Languages
We say that R is a regular expression if
• R = b for some bit b ∈ Σ
• R=λ
• R=∅
• R = (R1 ∪ R2 ) where R1 , R2 are regular expressions.
• R = (R1 ◦ R2 ) where R1 , R2 are regular expressions.
• R = (R1∗ ) where R1 is a regular expression.
Regular expressions are used in grep, sed, vi, Java, Perl, and most other scripting
languages.
Regular languages are those that can be generated by a regular expression.
Examples:
193
10 Computational Models
• 0∗ ∪ 1∗ is the language consisting of all strings with either all 1s or all 0s (plus the
empty string).
• 0∗ 10∗ is the language consisting of all strings with a single 1 in them.
• (ΣΣ)∗ the set of all even length strings
• 1Σ∗ 0 the set of all canonical representation of even integers.
Exercise: Give a regular expression for the set of all strings where every 0 appears before
any occurrence of a 1.
Languages are Equivalent to Problems
An instance of a decision problem involves a given configuration of data.
An algorithm answers
• yes if the data conforms to or has some property, and
• no if it does not.
Though many natural problems (optimization, functional) are not decision problems, we
can usually formulate the decision version of them.
An optimization problem of the form “what is the maximum (minimum) number of x
such that property P holds?” can be reformulated as, “Does property P hold for all
x ≥ k?”
Languages are are equivalent to problems: given a problem, you can define a language
that represents that problem.
Problem 14 (Sorting). Given elements x0 , . . . , xn−1 (properly encoded ) and an ordering
.
Question: is xi xi+1 for 0 ≤ i ≤ n − 2?
The language model is robust. Any problem P can be equivalently stated as a language
L where
• (Encodings) x of yes instances are members of the language; x ∈ L.
• (Encodings) x of no instances are not members of the language; x 6∈ L.
The key is that we establish a proper encoding scheme.
A proper encoding of graphs, for example, may be a string that consists of a binary
representation of n, the number of vertices.
Using some delimiter (which can also be in binary), we can specify connectivity by listing
pairs of connected vertices.
hGi = 11:00:01:01:10
194
10.2 Computational Models
We can then define a language,
L = {hGi | G is a connected graph}
Graph connectivity is now a language problem;
• hGi ∈ L if G is a (properly encoded) graph that is connected.
• hGi 6∈ L if G is not connected.
Instead of asking if a given graph G is connected, we instead ask, is hGi ∈ L?
10.2 Computational Models
There are many different computational models corresponding to many classes of lan-
guages.
Some are provably more powerful than others. Here, we give a brief introduction to
• Finite State Automata
• Grammars
• Turing Machines
10.2.1 Finite-State Automata
Definition 15. A finite automaton is a 5-tuple, A = (Q, Σ, δ, q0 , F ) where
• Q is a nonempty finite set of states
• Σ is our alphabet
• δ : Q × Σ → Q is the transition function
• q0 ∈ Q is an initial state
• F ⊆ Q is the set of accept states
An example of an FSM can be found in Figure 10.1.
• Q = {q0 , q1 , q2 }
• Σ = {0, 1}
• q0 is our initial state
• F = {q2 }
195
10 Computational Models
1 0
0 1
q0 q1 q2
0
1
Figure 10.1: A Finite State Automaton.
The transition function is specified by the labeled arrows.
δ(q0 , 0) = q1
δ(q0 , 1) = q0
δ(q1 , 0) = q1
δ(q1 , 1) = q2
δ(q2 , 0) = q1
δ(q2 , 1) = q0
This FSM accepts is any string that ends in 01. An equivalent regular expression is
simply Σ∗ 01.
The set of strings that a finite-state automaton A accepts is its language:
L(A) = {x ∈ Σ∗ | A(x) accepts}
Conversely, any string that ends in a non-accept state is rejected. This also defines a
language–the complement language:
L(M) = Σ∗ − L(M)
Finite-state automata are a simple computation model, but very restrictive. The only
types of languages it can recognize are those that can be defined by regular expressions.
Theorem 7. Finite-State Languages = Regular Languages = Regular Expressions
Recognition here means that a machine, given a finite string x ∈ Σ∗ can tell if x ∈ L(M).
Examples of regular languages:
• The language containing all strings, Σ∗
• The language consisting of all strings that are all 0s or all 1s
• The language consisting of all strings with an equal number of 0s and 1s
L = {w ∈ Σ∗ | w has an equal number of 0s and 1s}
196
10.2 Computational Models
• The language consisting of all even parity strings (an even number of 1s)
Not all languages are regular. Even simple languages such as
L = {0n 1n | ∀n ≥ 0}
10.2.2 Turing Machines
A more general computational model is a Turing Machines.
Definition 16. A Turing Machine is a 7-tuple, M = (Q, Σ, Γ, δ, q0 , qaccept , qreject ) where
• Q is a nonempty, finite set of states
• Σ is the input alphabet
• Γ is the work tape alphabet
• δ : Q × Σ × Γ → Q × Γ × {L, R, −}2 is the transition function
• q0 ∈ Q is the initial state
• qaccept is the accept state
• qreject is the reject state
A Turing Machine is a basic computational model which has an input tape that can be
read from, an output tape that can be written on and a set of states.
• A tape head moves left and right along the input/output tape and performs reads
and writes according to what symbols it encounters. A special empty symbol, t is
used to indicate the end of the input in the input tape and unwritten/uninitialized
cells in the work tape.
• A definition of a given Turing Machine can be made precise by enumerating every
possible transition on every possible input and output symbol for every state.
• A state diagram similar to automatons can visualize this transition. However, it is
much easier to simply describe a Turing Machine in high level English.
• A visualization can be seen in Figure 10.2
• In our definition:
– There are separate input and work tapes; the input tape is read-only and
cannot be changed/written to
– We allow left/right transitions for both tapes as well as a “no transition”, −
Other definitions (such as those in [14]) may omit these conveniences, but it can
be shown to be equivalent
197
10 Computational Models
Input x0 x1 x2 x3 x4 x5 x6 x7 · · · xn t
Finite State Control
Q, δ : Q × Σ × Γ → Q × Γ × {L, R, −}2
Work Tape γ0 γ1 γ2 γ3 γ4 γ5 ··· t t ···
Figure 10.2: Visualization of a Turing Machine
There are many variations of Turing Machines:
• Multi-tape
• Multi-head
• Multi-work tape
• Randomized
• Random access
However, it can be shown that, from a computability point of view, all Turing machines
are equivalent.
Example: consider the following language:
L = {x#x | x ∈ Σ∗ }
The following high-level description of a Turing Machine decides this language.
M(x) (read: on input x)
1. Scan the input to be sure that it contains a single #, if not reject.
2. Zig Zag across the tape to corresponding positions on each side of #. If symbols do
not match, reject, otherwise cross them off (write a blank symbol, t) and continue.
3. After all symbols to the left of # have been crossed off, check to the right of #, if
any symbols remain, reject otherwise, accept.
198
10.2 Computational Models
Extended Example
Designing Turing Machines can be a very complex process (and more arduous than
programming with 1950s era punchcards). We’re essentially programming using machine
language.
As an example, let’s design a full Turing Machine to decide the non-regular language,
{0n 1n | ∀n ≥ 1}. Let’s start with a high-level description:
1. Start by writing a special end symbol, # to the work tape. Moreover, if we see a 1
in the input tape or it is empty, t immediately halt and reject.
2. Start reading 0s in the input tape, for each 0, write a zero to the worktape (the
work tape will act as a stack that we push symbols onto).
3. When the first 1 is encountered, continue reading 1s; for each 1, erase the work
tape symbols.
• If we ever encounter another 0, reject
• If we encounter # with 1s still in the input tape, reject
• If we encounter the end of the input but there are still 0s on the work tape,
reject
• Otherwise, accept.
The transition function is defined by in Table 10.1 and a visual depiction of the finite
state control is depicted in Figure 10.3. This Turing Machine construction is actually
equivalent to a push-down automaton (essentially a finite state machine with access to
an infinite-capacity stack), which define Context-Free Languages.
10.2.3 Church-Turing Thesis
The Church-Turing Thesis gives a formal (though debatably not rigorous) definition of
what an algorithm is. It states that the intuitive notion of an algorithmic process is
equivalent to the computational model of Turing Machines.
This means that any rigorous computational model can be simulated by a Turing Machine.
Moreover, no other computational model is more powerful (in terms of the types of
languages it can accept) than a Turing Machine.
A programming language is Turing complete if it can do anything that a Turing machine
can do.
As a consequence, any two Turing complete programming languages are equivalent.
Intuitively, anything that you can do in Java, you can do in C++ (algorithmically, we’re
not talking about specific libraries), Perl, Python, PHP, etc.
199
10 Computational Models
Table 10.1: Turing Machine Transitions. Some states are not possible (NA).
State Input Output Transition
0 NA
0 1 NA
# NA
t q1 , #, −, R
q0 0 NA
1 1 NA
# NA
t reject
0 NA
t 1 NA
# NA
t reject
0 NA
0 1 NA
# NA
t q1 , 0, R, R
q1 0 NA
1 1 NA
# NA
t q2 , t, −, −
0 NA
t 1 NA
# NA
t reject
0 NA
0 1 NA
# reject
t reject
q2 0 NA
1 1 NA
# reject
t q1 , t, R, L
0 reject
t 1 NA
# accept
t NA
200
10.2 Computational Models
(0, t) → (0, R, R) (1, 0) → (t, R, L)
(0, ·) → (#, −, R) (1, t) → (t, −, −)
start q0 q1 q2
(1, #)
(0, ·)
(t, ·) (1, ·) (t, #)
(t, 0)
qreject qaccept
Figure 10.3: Turing Machine Finite State Transitions. Invalid or unnecessary transitions
have been omitted. The transition (a, b) → (x, y, z) indicates a is the symbol
on the input tape, b is the symbol on the work tape, x is the symbol we
write to the work tape, and y, z are the tape head transitions for the input
tape and work tape respectively. We have used the notation · as in (0, ·) to
indicate that the symbol on the work tape is irrelevant. Transitions to a
final halting state omit the tape write/transitions as they are final states.
201
10 Computational Models
The statement is a thesis since an algorithm is a dictionary definition, not a mathematical
definition. There are other notions of computability (Lambda calculus) that do have
mathematical definitions can can be proven to be equivalent to Turing Machines.
As with many programs, a Turing Machine may not always terminate or halt. On some
inputs, it may get caught in an infinite loop. We require a machine to halt in order to
accept (or reject) an input.
Definition 17. Let M be a Turing Machine and let L be a language. We say that M
decides L if for all x ∈ Σ∗ , M (s) halts and:
• accepts if and only if x ∈ L
• rejects if and only if x 6∈ L
The Church-Turing thesis can then be stated as follows.
“There exists a Turing Machine M that decides a language L”
=
“There exists an Algorithm A that solves a problem P”
10.2.4 Halting Problem & Decidability
As previously noted, a machine M on an input x may not halt. A less restrictive property
is language recognition.
We say that a Turing machine M recognizes a language L if for every x ∈ L, M (x) halts
and accepts.
A language L is in RE if some Turing machine recognizes it.
RE is the class of recursively enumerable languages (also called computably enumerable).
For a language in L ∈ RE, if x ∈ L, then some machine will eventually halt and accept it.
If x 6∈ L then the machine may or may not halt.
A language L is in R if some Turing machine decides it.
R is the class of recursive languages (also called computable).
Here, if L ∈ R, then there is some machine that will halt on all inputs and is guaranteed
to accept or reject.
Its not hard to see that if a language is decidable, it is also recognizable by definition,
thus
R ⊆ RE
202
10.2 Computational Models
There are problems (languages) that are not Turing Decidable: languages L ∈ RE, L 6∈ R.
We take as our first example the halting problem.
Problem 15 (Halting Problem). Given: A Turing Machine M and an input x.
Question: does M (x) halt?
This indeed would be a very useful program—once you’ve compiled a program, you may
want to determine if you’ve screwed up and caused an infinite loop somewhere.
We will show that the halting problem is undecidable.
That is, no algorithm, program or Turing Machine exists that could ever tell if another
Turing Machine halts on a given input or not.
Proof. By way of contradiction assume that there exists a Turing Machine H that decides
the halting problem:
halt and output 1 if M halts on x
H(hM, xi) =
halt and output 0 if M does not halt on x
We now consider P as an input to itself.
In case you may think this is invalid, it happens all the time. A text editor may open
itself up, allowing you to look at its binary code. The compiler for C was itself written in
C and may be called on to compile itself. An emulator opens machine code intended for
another machine and simulates that machine.
From the encoding hM, M i we construct another Turing Machine, Q as follows:
halts if H(hM, M i) = 0
Q(hM i) =
does not halt if H(hM, M i) = 1
It is easy to construct Q: we run H(hM, M i), if it outputs 0, then we halt and accept; if
it outputs 1 then we go into a trivial loop state, never halting. Now that Q is constructed,
we can run Q on itself:
halts if H(hQ, Qi) = 0
Q(hQi) =
does not halt if H(hQ, Qi) = 1
Which is a contradiction because Q(hQi) will halt if and only if Q(hQi) doesn’t halt and
vice versa.
Therefore, no such H can exist.
Many other problems, some of even practical interest have been shown to be undecidable.
This means that no matter how hard you try, you can never solve these problems with
any algorithm.
203
10 Computational Models
• Hilbert’s 10th problem: Given a multivariate polynomial, does it have integral
roots?
• Post’s Correspondence Problem: Given a set of “dominos” in which the top has a
finite string and the bottom has another finite string, can you produce a sequence
of dominos that is a match—where the top sequence is the same as the bottom?
• Rice’s Theorem: In general, given a Turing Machine, M , answering any question
about any non-trivial property of the language which it defines, L(M ) is undecidable.
To show that a problem is undecidable, you need to show a reduction to the halting
problem (or to any other problem that is known to be undecidable). That is, given a
problem P that you wish to show undecidable, you proceed by contradiction:
1. Assume that P is decidable by a Turing Machine M .
2. Construct a machine R that uses M to decide the Halting Problem.
3. Contradiction – such a machine M cannot exist.
Intuitive example: we can categorize all statements into two sets: lies and truths. How
then can we categorize the sentence,
I am lying
The key to this seeming paradox is self-reference. This is where we get the terms recursive
and recursively enumerable.
10.3 Complexity Classes
Now that we have a concrete model to work from: Problems as languages and Algorithms
as Turing Machines, we can further delineate complexity classes within R (all decidable
problems) by considering Turing Machines with respect to resource bounds.
In the computation of a Turing Machine M , the amount of memory M uses can be
quantified by how many tape cells are required in the computation of an input x. The
amount of time M uses can be quantified by the number of transitions M makes in the
computation of x.
Of course, just as before, we are interested in how much time and memory are used as a
function of the input size. In this case,
T (|x|)
and
M (|x|)
respectively where x ∈ Σ∗ . Again, the restriction to decisional versions of problems is
perfectly fine—we could just consider languages and Turing Machines themselves.
204
10.3 Complexity Classes
10.3.1 Deterministic Polynomial Time
Definition 18. The complexity class P consists of all languages that are decidable by a
Turing Machine running in polynomial time with respect to the input |x|. Alternatively,
P is the class of all decision problems that are solvable by a polynomial time running
algorithm.
10.3.2 Nondeterminism
A nondeterministic algorithm (or Turing Machine) is an algorithm that works in two
stages:
1. It guesses a solution to a given instance of a problem. This set of data corresponding
to an instance of a decision problem is called a certificate.
2. It verifies whether or not the guessed solution is valid or not.
3. It accepts if the certificate is a valid witness.
As an example, recall the HamiltonianCycle problem:
• A nondeterministic algorithm would guess a solution by forming a permutation π
of each of the vertices.
• It would then verify that (vi , vi+1 ) ∈ E for 0 ≤ i ≤ n − 1.
• It accepts if π is a Hamiltonian Cycle, otherwise it rejects.
• An instance is in the language if there exists a computation path that accepts.
Therein lies the nondeterminism – such an algorithm does not determine an actual
answer.
Alternatively, a nondeterministic algorithm solves a decision problem if and only if for
every yes instance of the problem it returns yes on some execution.
This is the same as saying that there exists a certificate for an instance.
A certificate can be used as a proof that a given instance is a yes instance of a decision
problem. In such a case, we say that the certificate is valid.
If a nondeterministic algorithm produces an invalid certificate, it does NOT necessarily
mean that the given instance is a no instance.
We can now define the class NP.
Definition 19. NP (“Nondeterministic Polynomial Time”) is the class of all languages
(problems) that can be decided (solved) by a Nondeterministic Turing Machine (algorithm)
running in polynomial time with respect to the size of the input |x|.
205
10 Computational Models
That is, each stage, guessing and verification, can be done in polynomial time. HamiltonianCycle
NP since a random permutation can be guessed in O(n) time and the verification process
can be done in O(n2 ) time.
It is not hard to see that
P ⊆ NP
since any problem that can be deterministically solved in polynomial time can certainly
be solved in nondeterministic polynomial time.
The most famous unanswered question so far then is
?
P = NP
If the answer is yes (very unlikely), then every problem in NP could be solved in
polynomial time. If the answer is no, then the hardest problems in NP could never be
solved by a polynomial time algorithm. Such problems will forever remain intractable.
To understand this more fully, we need to explore the notion of NP-Completeness.
10.4 Reductions & NP-Completeness
Reductions between problems establish a relative complexity. Reductions define a
structure and order to problems and a hierarchy of complexity.
Definition 20. A decision problem P1 is said to be polynomial time reducible to a
decision problem P2 if there exists a function f such that
• f maps all yes instances of P1 to yes instances of P2 . no instances likewise.
• f is computable by a polynomial time algorithm
In such a case we write
P1 ≤P P2
In general, there are other notions of reductions
• Adaptive reductions
• Truth-table reductions
• Projections
• Reductions that use more or less resources (time, space) than polynomial-time
However, polynomial-time reductions suffice for problems within NP. However, we want
to take care that we do not “cheat” by leveraging the power of the reduction to solve the
problem. Suppose we allowed exponential time reductions: we could solve SAT using the
reduction, then make a trivial map to another, easier problem.
206
10.4 Reductions & NP-Completeness
A0 (x)
yes yes
x f≤P (x) y A(y)
no no
Figure 10.4: Reduction Visualization. An algorithm A0 for problem X is formed by first
reducing instances x into instances y of problem Y . A known algorithm A
for Y is run on the reduced instance y and the output is used directly as
output for A0 .
An equivalent interpretation is that a language A is reducible to language B if there
exists a polynomial time computable function f : Σ∗ → Σ∗ such that
x ∈ A ⇐⇒ f (x) ∈ B
A reduction establishes a relative complexity between problems as follows. Suppose that
• P1 ≤P P2 via a function f and
• there is an algorithm A for P2 that runs in time O(g(n))
• we then have an algorithm A0 for L1 :
1. On input x, run f (x) to get y
2. Run A(y) and map the answer appropriately
• This algorithm runs in O(g(n) + nk )
• If g(n) ∈ Ω(nk ) then A0 runs in O(g(n))
• A visualization of this interpretation is presented in Figure 10.4.
Reductions establish a relative complexity. If P1 ≤P P2 then
• P2 is at least as difficult/complex as P1 .
• If we have a polynomial time solution to P2 then we have a polynomial time solution
to P1 via its reduction
Definition 21. A problem P is said to be NP-Complete if
1. P ∈ NP and
2. For every problem P 0 ∈ NP, P 0 ≤P P
207
10 Computational Models
NPC NP
f≤P
P
Figure 10.5: All problems in NP reduce to any NP-complete problem.
Intuitively, NP-Complete problems are the hardest (most difficult computationally speak-
ing) problems in NP (of course there are provably harder problems in classes such as
EXP). The “landscape” is depicted in Figure 10.5.
There are 5 basic steps to show that a given problem P is NP-Complete.
1. Prove that P ∈ NP by giving an algorithm that guesses a certificate and an
algorithm that verifies a solution in polynomial time.
2. Select a known NP-Complete problem P 0 that we will reduce to P (P 0 ≤P P)
0
3. Give an algorithm that computes f : Pyes 7→ Pyes for every instance x ∈ {0, 1}∗ .
4. Prove that f satisfies x ∈ P 0 if and only if f (x) ∈ P
5. Prove that the algorithm in step 3 runs in polynomial time
10.4.1 Satisfiability
To establish NP completeness we need a starting point: a “first problem” to reduce from.
In fact, the computation of a generic NP Turing Machine is the canonically complete
language for NP:
M is a non-deterministic Turing Machine that
LNP = hM, xi
accepts x in a polynomial number of steps
However, LNP is unnatural from a “problem” point of view. However, the key is that
208
10.4 Reductions & NP-Completeness
polynomial time reductions are transitive:
P1 ≤P P2 ≤P P3 ⇒ P1 ≤P P3
Thus, we need only show a reduction from a known NP-Complete problem to P to
show that P ∈ NPC. In 1971, Stephen Cook [6] independently (Levin published similar
concepts in 1973 [12]) defined the notions of NP and NP-Completeness showing the first
NP-Complete problem ever by showing a reduction to Satisfiability, LNP ≤P SAT
Recall the following notations:
• A literal is a boolean variable that can be set to 0 or 1
• ∨ denotes the logical or of 2 boolean variables
• ∧ denotes the logical and of 2 boolean variables
• ¬ denotes the negation of a boolean variable
• A clause is the is the logical disjunction (or -ing) of a set of boolean variables. Ex:
(x1 ∨ ¬x2 ∨ x5 )
• The conjunction of a collection of clauses is the logical and of all their values. The
value is true only if every clause is true.
Satisfiability or simply, SAT is the following
Problem 16 (Satisfiability). Given: a set of boolean variables, V = {x1 , x2 , . . . xn } and
a set of clauses, C = {C1 , C2 , . . . Cm }.
Question: Does there exist a satisfying assignment of boolean values to each literal xi ,
1 ≤ i ≤ n such that
^m
C i = C1 ∧ C2 ∧ . . . ∧ Cm = 1
i=1
Let n = 4 and consider the following conjunction:
C = (x1 ∨ x2 ∨ ¬x4 ) ∧ (¬x1 ∨ ¬x2 ∨ x3 ) ∧ (x1 ∨ ¬x3 ∨ ¬x4 )
This conjunction is satisfiable and is therefore a yes instance of SAT since if we set
x1 = x4 = 0 and x2 = x3 = 1, C = 1.
Let n = 3 and consider the following conjunction:
C = (x1 ∨ x2 ) ∧ (¬x1 ∨ ¬x2 )∧
(x1 ∨ x3 ) ∨ (¬x1 ∨ ¬x3 )∧
(x2 ∨ x3 ) ∧ (¬x2 ∨ ¬x3 )
This conjunction is not satisfiable since none of the 2n = 8 possible boolean assignments
will ever make C = 1.
209
10 Computational Models
Another way to visualize nondeterministic computation is to think of all possible compu-
tation paths being explored at once in parallel (refer back to Figure 4.2): suppose that
we explored all paths in this tree and accepted if any of the leaf computations accepted.
We would be able to do this in O(n) nondeterministic time.
This illustrates the intuition behind the NP class. Yes instances are easily solved if we
are lucky enough to guess a satisfying assignment. No instances require an exhaustive
search of all possible assignments.
Theorem 8 (Cook, 1971). The problem SAT is NP-Complete.
Other standard NP-complete problems:
• 3-CNF – A restricted version of SAT where each clause is a disjunction of exactly
3 literals. CNF stands for Conjunctive Normal Form. Note that 2-CNF ∈ P.
• HamiltonianCycle – Determine if a given undirected graph G contains a cycle
which passes through every vertex exactly once.
• TravelingSalesman – Find the least weighted cycle in a graph G that visits
each vertex exactly once.
• SubsetSum – Given a collection of integers, can you form a subset S such that
the sum of all items in S is exactly p.
• GraphColoring – For a given graph G, find its chromatic number χ(G) which
is the smallest number of colors that are required to color the vertices of G so that
no two adjacent vertices have the same color.
Some good resources on the subject can be found in:
• Computers and Intractability – A Guide to the Theory of NP Completeness 1979
[7]
• Annotated list of about 90 NP-Complete Problems: http://www.csc.liv.ac.uk/
~ped/teachadmin/COMP202/annotated_np.html
10.4.2 Satisfiability to Clique
Problem 17 (Clique).
Given: An undirected graph G = (V, E)
Output: A subset C ⊆ V of maximal cardinality such that all vertices in C are connected
A “clique” is a group that is strongly connected (a clique of friends). In terms of graphs,
a clique is a complete subgraph.
In terms of languages we can define
Clique = {hG, ki | G is a graph with a clique of size k}
210
10.4 Reductions & NP-Completeness
We want to prove that Clique is NP-Complete. To do this we will go by our 5 step
process.
1. Clique ∈ NP. We can nondeterministically guess k vertices from a given graph’s
vertex set V in O(|V |) time. Further, we can check if, for each pair of vertices
v, v 0 ∈ V 0 if (v, v 0 ) ∈ E in O(|V |2 ) time.
2. We select the 3-CNF-SAT problem, a known NP-Complete problem for our reduc-
tion: 3-CNF-SAT ≤P Clique.
3. We define the following function. Let φ = C1 ∧ . . . ∧ Ck be a 3-CNF formula. We
will construct a graph G that has a clique of size k if and only if φ is satisfiable.
For each clause Ci = (xi1 ∨ xi2 ∨ xi3 ) we define vertices v1i , v2i , v3i ∈ V . Edges are
defined such that (vαi , vβj ) ∈ E if both of the following hold:
a) If the vertices are in different clauses, i.e. i 6= j
b) Their corresponding literals are consistent: vαi is not the negation of vβj
4. We need to show that yes instances of the 3-CNF function φ are preserved with
this function. That is we want to show that φ is satisfiable if and only if f (φ) = G
has a clique of size k (see proof below)
5. The computation of the function described in step 3 is clearly polynomial. The
input size is something like O(nk) so building G takes at most O(2n2 k) time.
As an example, consider the following clause:
C = (x1 ∨ ¬x2 ∨ x3 ) ∧ (x1 ∨ ¬x2 ∨ ¬x3 ) ∧ (x1 ∨ x2 ∨ x3 )
Proof. (⇒) : Suppose that φ has a satisfying assignment. This implies that each clause
Ci contains at least one true literal. Remember that each literal corresponds to some
vertex in G. Choosing a true literal from each clause yields a set V 0 of size k. To see
that V 0 is a clique we look at our two requirements from before: vαi and vβj are consistent
and both are true, thus (vαi , vβj ) ∈ E.
(⇐): Suppose that G has a clique of size k. No edges in G connect vertices in the same
triple corresponding to a clause so V 0 contains exactly one vertex per triple. Without fear
of causing inconsistencies, we can assign a 1 to each literal (0 if a negation) corresponding
to some vertex in each triple thus each clause is satisfied and so φ is satisfied.
10.4.3 Clique to Vertex Cover
Problem 18 (VertexCover).
Given: An undirected graph G = (V, E)
Output: A subset V 0 ⊆ V of minimal cardinality such that each edge e ∈ V is incident
on some v ∈ V 0
211
10 Computational Models
C2
x1 x2 x3
x1 x1
C1 ¬x2 ¬x2 C3
x3 ¬x3
Figure 10.6: Clique Reduction Visualization
The vertices “cover” all the edges. Clearly V is a trivial vertex cover, but we are interested
in vertex covers of minimal size.
Theorem 9. Vertex Cover is NP-complete.
Proof. We first observe that the verification of the Vertex Cover problem can be achieved
with an NP algorithm, establishing that VertexCover ∈ NP. We first convert this problem
to a decision version by adding a parameter k and ask the question, does there exist a
subset V 0 ⊆ V of cardinality k that represents a vertex cover. We then nondetermin-
istically guess a subset V 0 ⊆ V of size k and verify in deterministic O(km) time that
it constitutes a vertex cover by iterating over vertices in the cover and removing edges
incident on them. If all edges have been removed, it is a vertex cover (accept) otherwise
there are uncovered edges and we reject.
We complete the proof by showing a reduction from the clique problem to vertex cover.
The fact that the clique problem is NP-complete was established previously. Given an
encoding of the (decision version) of the clique problem, hG, ki, we transform it to an
instance of the vertex cover problem by taking the complement graph G and substituting
|V | − k for our cardinality parameter. That is:
hG, ki → hG, |V | − ki
We now make the following claim:
C ⊆ V is a clique of size k in G if and only if V \ C is a vertex cover of size |V | − k in G.
The proof is below; the idea is visualized in Figure 10.7.
212
10.5 Beyond P and NP
G G
C C
u u
v v
C =V \C C =V \C
(a) Suppose that a (b) In G, u, v are con-
clique C of size k ex- nected and C vertex
isted in G with u, v cover of size n − k
unconnected.
Figure 10.7: Reduction Idea for Clique/Vertex Cover. The complement of a clique in G
acts as a vertex cover in G.
(⇒) Let C be a clique in G such that |C| = k. Let e = (u, v) be an edge in G. We need
to show that this edge is covered either by u or by v. Observe:
e = (u, v) ∈ E ⇒ (u, v) 6∈ E
⇒ u 6∈ C ∨ v 6∈ C or both
⇒ u∈V \C ∨v ∈V \C
⇒ (u, v) is covered by V \ C
(⇐) Let C ⊆ V be a vertex cover in G of size n − k. Observe that:
∀u, v ∈ V (u, v) ∈ E ⇒ u ∈ C ∨ v ∈ C
now consider the contrapositive of this statement:
∀u, v ∈ V [u 6∈ C ∧ v 6∈ C ⇒ (u, v) ∈ E]
which means that V \ C is a clique in G.
An example demonstrating this relationship can be found in Figure 10.8.
10.5 Beyond P and NP
There are many more complexity classes based on various computational resource bounds:
memory (space); randomization, quantum, etc. A large list of complexity classes is
maintained at the Complexity Zoo: http://www.complexityzoo.com/ [2].
In summary, with respect to the complexity classes examined here,
213
10 Computational Models
b d f b d f
a a
c e g c e g
(a) Graph G with a clique of size 4 highlighted (b) The complement graph G. The corresponding
in purple. vertex cover is highlighted in yellow.
Figure 10.8: Graph and Complement: Clique and Vertex Cover. Graph G with a clique
of size 4 and its complement G with a vertex cover of size 3.
• coRE is the complement class of RE, that is it consists of all decision problems for
which no instances can be verified by a Turing Machine in a finite amount of time.
yes instances are not guaranteed to halt.
• coNP is the complement class of NP. Rather than producing certificates, acceptance
is defined as being able to produce a disqualifier (i.e. if some computation path
produces a no answer, we accept. This is still a nondeterministic class. A good
example: Tautology.
• NP ∩ coNP is the intersection of NP and coNP, P is contained in this class as is
• NPI: NP intermediate, problems in NP, but that are neither in P nor NP-complete.
If one were able to prove a language L ∈ NPI, then it would show that P 6= NP. Thus,
no problems have been shown to be in NPI; however there are many candidates:
Graph Isomorphism, Group Isomorphism, Integer Factorization, Discrete Logarithm,
etc. We do have a partial result: Ladner’s Theorem states that if P 6= NP, then
there are languages in NPI [11].
The major containments are visualized in Figure 10.9.
10.6 Misc
Definition 22. Let G = (V, E) be an undirected graph. A matching is a set, M ⊆ E of
pair-wise non-adjacent edges such that ∀v ∈ V , v is the end point of at most one edge
e ∈ M . A matching is perfect if M covers every vertex in G.
Terminology
• An alternating path: given a partial matching, an alternating path is a path p in G
such that edges in p alternate, e ∈ M, e 6∈ M , etc.
214
10.6 Misc
RE coRE
R
NPC coNPC
NP coNP
P
Figure 10.9: Complexity Hierarchy
TODO
Figure 10.10: Alternating and Augmenting Paths
• An augmenting path: An alternating path is augmenting if it begins and ends in
unmatched edges (e 6∈ M )
• Examples can be found in Figure 10.10
Algorithm for matchings in bipartite graphs (Hopcroft & Karp): find an augmenting
path for each x ∈ L to R (via BFS), then switch out all the edges. Each iteration adds
one more edge to the matching.
Variations & Applications:
• Maximum weighted matching for weighted graphs
• Assignment problem (bipartite matching)
• Efficiently solve (or approximate) special cases of NP-complete problems (Konig’s
Theorem): vertex cover in bipartite graphs
Theorem 10 (Konig’s Theorem). Let G = (V, E) be a bipartite graph. Then any
maximal matching M is also a minimum vertex cover V .
Other Algorithms:
• Edmonds (for general graphs): paths, trees and flowers (graph decomposition)
• Hungarian algorithm (for the assignment problem–equivalent to maximal bipartite
matching)
Related: counting the number of perfect matchings in a bipartite graph is #P-complete
215
10 Computational Models
(a much higher complexity class). This is equivalent to computing the permanent of a 0-1
matrix. Here we have a difficult counting problem whose corresponding search problem
(finding a matching) is easy!
10.7 Exercises
Exercise 10.1. Design a finite-state automaton to accept the language consisting of any
string in which contain no contiguous 0s.
Exercise 10.2. Can you show that polynomial time reductions, ≤P define an equivalence
relation on all languages in NP? What would the implications of such a result be?
Exercise 10.3. Let G = (V, E) be an undirected graph. Given two vertices, u, v ∈ V
it is easy to determine the length of the shortest path between them using Dijkstra’s
algorithm. However, determining the longest path between u, v is NP-complete. Prove
this as follows: assume that a “free” algorithm A exists for the longest path problem. A
takes as input, G, u, v and outputs the length k of the longest path between u and v)
and use it to design a polynomial time algorithm to solve the Hamiltonian Path problem,
a known NP-complete problem.
Exercise 10.4. Show that the Hamiltonian Cycle problem is NP-complete by showing a
reduction from Hamiltonian Path (hint: try adding a vertex and connecting it somehow).
Exercise 10.5. Say that we had an oracle (a subroutine; a “free” function) to answer yes
or no whether a given graph G has a Hamiltonian Path. Design an algorithm to actually
compute a Hamiltonian Path that uses this as a subroutine that runs in polynomial time
(assuming that the oracle subroutine is free).
Exercise 10.6. Consider the following definition and problem:
Definition 23. An independent set of an undirected graph G = (V, E) is a subset of
vertices V 0 ⊆ V such that for any two vertices v, v 0 ∈ V 0 , (v, v 0 ) 6∈ E
Problem 19 (IndependentSet). Given an undirected graph G = (V, E)
Question: Does there exist an independent set of size k?
Prove that IndependentSet is NP-complete by showing a reduction from the clique problem.
Exercise 10.7. Show that the Clique problem is NP-complete by showing a reduction
from the Independent set problem.
Exercise 10.8. Recall the Traveling Salesperson Problem: prove that this problem is
NP-complete by showing a reduction from the Hamiltonian Cycle problem.
Exercise 10.9. Answer the following questions regarding P and NP. Here, ≤P refers to
a polynomial time reduction and P1 , P2 , P3 are problems. Provide a brief justification for
your answers.
216
10.7 Exercises
1. Say that P1 ≤P P2 and that P2 is NP-complete. Can we conclude that P1 is
NP-complete? Why or why not?
2. Say that P1 ≤P P2 and that P2 is in NP. Can we conclude that P2 is NP-complete?
Why or why not?
3. Say that P1 ∈ P and P2 is NP-complete. Can we conclude that P1 ≤P P2 ?
4. Say that P1 ≤P P2 . Can we conclude that P2 is a harder problem than P1 ?
5. Say that P1 ≤P P2 and that P2 is NP-Complete. Can we conclude that P1 ∈ NP?
6. Let P1 be a problem, is it true that P1 ≤P P1 ?
217
11 More Algorithms
11.1 A∗ Search
11.2 Jump Point Search
http://www.aaai.org/ocs/index.php/SOCS/SOCS12/paper/viewFile/5396/5212
https://en.wikipedia.org/wiki/Jump_point_search
11.3 Minimax
219
Glossary
algorithm a process or method that consists of a specified step-by-step set of operations.
221
Acronyms
BFS Breadth First Search. 128
DAG Directed Acyclic Graph. 147
DFS Depth First Search. 121
223
Index
2-3 Trees, 126 Gaussian Elimination, 89
Graphs, 147
Algorithms
Dijkstra’s, 166 Hamiltonian Cycle, 65
Floyd-Warshall, 168 Hamiltonian Path, 65
Kruskal’s Algorithm, 161
Prim’s Algorithm, 163 Kruskal’s Algorithm, 161
Assignment Problem, 99 Language, 198
AVL Trees, 120 Linear Programming, 96
B-Trees, 126 LU Decomposition, 93
Balanced Binary Search Tree Matrix Chain Multiplication, 190
2-3 Trees, 126 Matrix Inverse, 93
AVL Trees, 120 Minimum Spanning Tree, 160, 161
B-Trees, 126
Balanced Binary Search Trees, 118 Neighborhood, 65
Red-Black Trees, 128 NP, 211
Binary Search Trees
Balanced, 118 Optimal Binary Search Trees, 180
Optimal, 180
P, 211
Bipartite Testing, 159
Prim’s Algorithm, 163
Breadth First Search, 154
Problems
Church-Turing Thesis, 205 Assignment Problem, 99
Clique Problem, 216 Clique, 216
Complexity Classes Graph Problems
NP, 211 Bipartite Testing, 159
P, 211 Cycle Detection, 158
Cycle Detection, 158 Minimum Spanning Tree, 160
Shortest Path, 157
Depth First Search, 148 Hamiltonian Path, 65
Determinant, 95 Satisfiability, 64
Dijkstra’s Algorithm, 166 Vertex Cover, 217
Finite State Automata, 201 Red-Black Trees, 128
Floyd-Warshall Algorithm, 168 Reductions, 212
225
Index
Regular Languages, 199
Satisfiability, 64, 214
Shortest Path, 157
Topological Sorting, 157
Turing Machine, 203
Vertex Cover Problem, 217
226
Bibliography
[1] An annotated list of selected NP-complete problems. http://cgi.csc.liv.ac.uk/
~ped/teachadmin/COMP202/annotated_np.html, 2014. Accessed: 2014-08-12.
[2] Complexity zoo. https://complexityzoo.uwaterloo.ca/Complexity_Zoo, 2014.
Accessed: 2014-08-05.
[3] Manindra Agrawal, Neeraj Kayal, and Nitin Saxena. PRIMES is in P. Ann. of
Math, 2:781–793, 2002.
[4] Paul Bachmann. Die Analytische Zahlentheorie. 1894.
[5] R. Bayer and E. McCreight. Organization and maintenance of large ordered indices.
In Proceedings of the 1970 ACM SIGFIDET (Now SIGMOD) Workshop on Data
Description, Access and Control, SIGFIDET ’70, pages 107–141, New York, NY,
USA, 1970. ACM.
[6] Stephen A. Cook. The complexity of theorem-proving procedures. In Proceedings
of the Third Annual ACM Symposium on Theory of Computing, STOC ’71, pages
151–158, New York, NY, USA, 1971. ACM.
[7] Michael R. Garey and David S. Johnson. Computers and Intractability: A Guide to
the Theory of NP-Completeness. W. H. Freeman & Co., New York, NY, USA, 1979.
[8] G. Georgy Adelson-Velsky and E. M. Landis. An algorithm for the organization of
information. In Proceedings of the USSR Academy of Sciences (in Russian), number
146, pages 263–266, 1962.
[9] D.E. Knuth. Optimum binary search trees. Acta Informatica, 1(1):14–25, 1971.
[10] Donald E. Knuth. Big omicron and big omega and big theta. SIGACT News,
8(2):18–24, April 1976.
[11] Richard E. Ladner. On the structure of polynomial time reducibility. J. ACM,
22(1):155–171, January 1975.
[12] Leonid A Levin. Universal sequential search problems. Problemy Peredachi Infor-
matsii, 9(3):115–116, 1973.
[13] Anany V. Levitin. Introduction to the Design and Analysis of Algorithms. Addison-
Wesley Longman Publishing Co., Inc., Boston, MA, USA, 2002.
[14] Michael Sipser. Introduction to the Theory of Computation. International Thomson
227
Bibliography
Publishing, 1st edition, 1996.
[15] Virginia Vassilevska Williams. Multiplying matrices faster than coppersmith-
winograd. In Proceedings of the Forty-fourth Annual ACM Symposium on Theory of
Computing, STOC ’12, pages 887–898, New York, NY, USA, 2012. ACM.
228