Authors Michiel Smid
License CC-BY-SA-4.0
Discrete Structures for Computer Science:
Counting, Recursion, and Probability
Michiel Smid
School of Computer Science
Carleton University
Ottawa, Ontario
Canada
michiel@scs.carleton.ca
July 22, 2019
Contents
Preface vii
1 Introduction 1
1.1 Ramsey Theory . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Sperner’s Theorem . . . . . . . . . . . . . . . . . . . . . . . . 4
1.3 The Quick-Sort Algorithm . . . . . . . . . . . . . . . . . . . . 5
2 Mathematical Preliminaries 9
2.1 Basic Concepts . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.2 Proof Techniques . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.2.1 Direct proofs . . . . . . . . . . . . . . . . . . . . . . . 13
2.2.2 Constructive proofs . . . . . . . . . . . . . . . . . . . . 14
2.2.3 Nonconstructive proofs . . . . . . . . . . . . . . . . . . 14
2.2.4 Proofs by contradiction . . . . . . . . . . . . . . . . . . 15
2.2.5 Proofs by induction . . . . . . . . . . . . . . . . . . . . 16
2.2.6 More examples of proofs . . . . . . . . . . . . . . . . . 18
2.3 Asymptotic Notation . . . . . . . . . . . . . . . . . . . . . . . 20
2.4 Logarithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
2.5 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
3 Counting 25
3.1 The Product Rule . . . . . . . . . . . . . . . . . . . . . . . . . 25
3.1.1 Counting Bitstrings of Length n . . . . . . . . . . . . . 26
3.1.2 Counting Functions . . . . . . . . . . . . . . . . . . . . 26
3.1.3 Placing Books on Shelves . . . . . . . . . . . . . . . . . 29
3.2 The Bijection Rule . . . . . . . . . . . . . . . . . . . . . . . . 31
3.3 The Complement Rule . . . . . . . . . . . . . . . . . . . . . . 33
3.4 The Sum Rule . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
iv Contents
3.5 The Principle of Inclusion and Exclusion . . . . . . . . . . . . 35
3.6 Permutations and Binomial Coefficients . . . . . . . . . . . . . 37
3.6.1 Some Examples . . . . . . . . . . . . . . . . . . . . . . 39
3.6.2 Newton’s Binomial Theorem . . . . . . . . . . . . . . . 40
3.7 Combinatorial Proofs . . . . . . . . . . . . . . . . . . . . . . . 43
3.8 Pascal’s Triangle . . . . . . . . . . . . . . . . . . . . . . . . . 46
3.9 More Counting Problems . . . . . . . . . . . . . . . . . . . . . 50
3.9.1 Reordering the Letters of a Word . . . . . . . . . . . . 50
3.9.2 Counting Solutions of Linear Equations . . . . . . . . . 51
3.10 The Pigeonhole Principle . . . . . . . . . . . . . . . . . . . . . 55
3.10.1 India Pale Ale . . . . . . . . . . . . . . . . . . . . . . . 55
3.10.2 Sequences Containing Divisible Numbers . . . . . . . . 56
3.10.3 Long Monotone Subsequences . . . . . . . . . . . . . . 57
3.10.4 There are Infinitely Many Primes . . . . . . . . . . . . 58
3.11 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
4 Recursion 83
4.1 Recursive Functions . . . . . . . . . . . . . . . . . . . . . . . . 83
4.2 Fibonacci Numbers . . . . . . . . . . . . . . . . . . . . . . . . 85
4.2.1 Counting 00-Free Bitstrings . . . . . . . . . . . . . . . 87
4.3 A Recursively Defined Set . . . . . . . . . . . . . . . . . . . . 88
4.4 A Gossip Problem . . . . . . . . . . . . . . . . . . . . . . . . . 91
4.5 Euclid’s Algorithm . . . . . . . . . . . . . . . . . . . . . . . . 94
4.5.1 The Modulo Operation . . . . . . . . . . . . . . . . . . 95
4.5.2 The Algorithm . . . . . . . . . . . . . . . . . . . . . . 95
4.5.3 The Running Time . . . . . . . . . . . . . . . . . . . . 97
4.6 The Merge-Sort Algorithm . . . . . . . . . . . . . . . . . . . . 99
4.6.1 Correctness of Algorithm MergeSort . . . . . . . . . 100
4.6.2 Running Time of Algorithm MergeSort . . . . . . . 101
4.7 Computing the Closest Pair . . . . . . . . . . . . . . . . . . . 104
4.7.1 The Basic Approach . . . . . . . . . . . . . . . . . . . 105
4.7.2 The Recursive Algorithm . . . . . . . . . . . . . . . . . 111
4.8 Counting Regions when Cutting a Circle . . . . . . . . . . . . 115
4.8.1 A Polynomial Upper Bound on Rn . . . . . . . . . . . 115
4.8.2 A Recurrence Relation for Rn . . . . . . . . . . . . . . 118
4.8.3 Simplifying the Recurrence Relation . . . . . . . . . . . 123
4.8.4 Solving the Recurrence Relation . . . . . . . . . . . . . 124
4.9 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
Contents v
5 Discrete Probability 165
5.1 Anonymous Broadcasting . . . . . . . . . . . . . . . . . . . . 165
5.2 Probability Spaces . . . . . . . . . . . . . . . . . . . . . . . . 170
5.2.1 Examples . . . . . . . . . . . . . . . . . . . . . . . . . 171
5.3 Basic Rules of Probability . . . . . . . . . . . . . . . . . . . . 174
5.4 Uniform Probability Spaces . . . . . . . . . . . . . . . . . . . 179
5.4.1 The Probability of Getting a Full House . . . . . . . . 180
5.5 The Birthday Paradox . . . . . . . . . . . . . . . . . . . . . . 181
5.5.1 Throwing Balls into Boxes . . . . . . . . . . . . . . . . 184
5.6 The Big Box Problem . . . . . . . . . . . . . . . . . . . . . . . 185
5.6.1 The Probability of Finding the Big Box . . . . . . . . . 187
5.7 The Monty Hall Problem . . . . . . . . . . . . . . . . . . . . . 189
5.8 Conditional Probability . . . . . . . . . . . . . . . . . . . . . . 190
5.8.1 Anil’s Children . . . . . . . . . . . . . . . . . . . . . . 191
5.8.2 Rolling a Die . . . . . . . . . . . . . . . . . . . . . . . 192
5.8.3 Flip and Flip or Roll . . . . . . . . . . . . . . . . . . . 195
5.9 The Law of Total Probability . . . . . . . . . . . . . . . . . . 198
5.9.1 Flipping a Coin and Rolling Dice . . . . . . . . . . . . 200
5.10 Please Take a Seat . . . . . . . . . . . . . . . . . . . . . . . . 202
5.10.1 Determining pn,k Using a Recurrence Relation . . . . . 203
5.10.2 Determining pn,k by Modifying the Algorithm . . . . . 206
5.11 Independent Events . . . . . . . . . . . . . . . . . . . . . . . . 209
5.11.1 Rolling Two Dice . . . . . . . . . . . . . . . . . . . . . 209
5.11.2 A Basic Property of Independent Events . . . . . . . . 211
5.11.3 Pairwise and Mutually Independent Events . . . . . . . 212
5.12 Describing Events by Logical Propositions . . . . . . . . . . . 213
5.12.1 Flipping a Coin and Rolling a Die . . . . . . . . . . . . 214
5.12.2 Flipping Coins . . . . . . . . . . . . . . . . . . . . . . 215
5.12.3 The Probability of a Circuit Failing . . . . . . . . . . . 215
5.13 Choosing a Random Element in a Linked List . . . . . . . . . 217
5.14 Long Runs in Random Bitstrings . . . . . . . . . . . . . . . . 219
5.15 Infinite Probability Spaces . . . . . . . . . . . . . . . . . . . . 224
5.15.1 Infinite Series . . . . . . . . . . . . . . . . . . . . . . . 225
5.15.2 Who Flips the First Heads . . . . . . . . . . . . . . . . 227
5.15.3 Who Flips the Second Heads . . . . . . . . . . . . . . . 229
5.16 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231
vi Contents
6 Random Variables and Expectation 279
6.1 Random Variables . . . . . . . . . . . . . . . . . . . . . . . . . 279
6.1.1 Flipping Three Coins . . . . . . . . . . . . . . . . . . . 280
6.1.2 Random Variables and Events . . . . . . . . . . . . . . 281
6.2 Independent Random Variables . . . . . . . . . . . . . . . . . 283
6.3 Distribution Functions . . . . . . . . . . . . . . . . . . . . . . 286
6.4 Expected Values . . . . . . . . . . . . . . . . . . . . . . . . . 287
6.4.1 Some Examples . . . . . . . . . . . . . . . . . . . . . . 288
6.4.2 Comparing the Expected Values of Comparable Ran-
dom Variables . . . . . . . . . . . . . . . . . . . . . . . 290
6.4.3 An Alternative Expression for the Expected Value . . . 291
6.5 Linearity of Expectation . . . . . . . . . . . . . . . . . . . . . 293
6.6 The Geometric Distribution . . . . . . . . . . . . . . . . . . . 296
6.6.1 Determining the Expected Value . . . . . . . . . . . . 297
6.7 The Binomial Distribution . . . . . . . . . . . . . . . . . . . . 299
6.7.1 Determining the Expected Value . . . . . . . . . . . . 299
6.7.2 Using the Linearity of Expectation . . . . . . . . . . . 302
6.8 Indicator Random Variables . . . . . . . . . . . . . . . . . . . 303
6.8.1 Runs in Random Bitstrings . . . . . . . . . . . . . . . 304
6.8.2 Largest Elements in Prefixes of Random Permutations 306
6.8.3 Estimating the Harmonic Number . . . . . . . . . . . . 309
6.9 The Insertion-Sort Algorithm . . . . . . . . . . . . . . . . . . 311
6.10 The Quick-Sort Algorithm . . . . . . . . . . . . . . . . . . . . 313
6.11 Skip Lists . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 316
6.11.1 Algorithm Search . . . . . . . . . . . . . . . . . . . . 318
6.11.2 Algorithms Insert and Delete . . . . . . . . . . . . 319
6.11.3 Analysis of Skip Lists . . . . . . . . . . . . . . . . . . . 321
6.12 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 329
7 The Probabilistic Method 369
7.1 Large Bipartite Subgraphs . . . . . . . . . . . . . . . . . . . . 369
7.2 Ramsey Theory . . . . . . . . . . . . . . . . . . . . . . . . . . 371
7.3 Sperner’s Theorem . . . . . . . . . . . . . . . . . . . . . . . . 374
7.4 The Jaccard Distance between Finite Sets . . . . . . . . . . . 377
7.4.1 The First Proof . . . . . . . . . . . . . . . . . . . . . . 378
7.4.2 The Second Proof . . . . . . . . . . . . . . . . . . . . . 380
7.5 Planar Graphs and the Crossing Lemma . . . . . . . . . . . . 381
7.5.1 Planar Graphs . . . . . . . . . . . . . . . . . . . . . . 382
Contents vii
7.5.2 The Crossing Number of a Graph . . . . . . . . . . . . 386
7.6 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 391
viii Contents
Preface
This is a free textbook for an undergraduate course on Discrete Structures
for Computer Science students, which I have been teaching at Carleton Uni-
versity since the fall term of 2013. The material is offered as the second-year
course COMP 2804 (Discrete Structures II). Students are assumed to have
taken COMP 1805 (Discrete Structures I), which covers mathematical rea-
soning, basic proof techniques, sets, functions, relations, basic graph theory,
asymptotic notation, and countability.
During a 12-week term with three hours of classes per week, I cover most
of the material in this book, except for Chapter 2, which has been included
so that students can review the material from COMP 1805.
Chapter 2 is largely taken from the free textbook Introduction to Theory
of Computation by Anil Maheshwari and Michiel Smid, which is available at
http://cg.scs.carleton.ca/~michiel/TheoryOfComputation/
Please let me know if you find errors, typos, simpler proofs, comments,
omissions, or if you think that some parts of the book “need improvement”.
x
Chapter 1
Introduction
In this chapter, we introduce some problems that will be solved later in this
book. Along the way, we recall some notions from discrete mathematics that
you are assumed to be familiar with. These notions are reviewed in more
detail in Chapter 2.
1.1 Ramsey Theory
Ramsey Theory studies problems of the following form: How many elements
of a given type must there be so that we can guarantee that some property
holds? In this section, we consider the case when the elements are people
and the property is “there is a large group of friends or there is a large group
of strangers”.
Theorem 1.1.1 In any group of six people, there are
• three friends, i.e., three people who know each other,
• or three strangers, i.e., three people, none of which knows the other two.
In order to prove this theorem, we denote the six people by P1 , P2 , . . . , P6 .
Any two people Pi and Pj are either friends or strangers. We define the
complete graph G = (V, E) with vertex set
V = {Pi : 1 ≤ i ≤ 6}
and edge set
E = {Pi Pj : 1 ≤ i < j ≤ 6}.
2 Chapter 1. Introduction
Observe that |V | = 6 and |E| = 15. We draw each edge Pi Pj as a straight-line
segment according to the following rule:
• If Pi and Pj are friends, then the edge Pi Pj is drawn as a solid segment.
• If Pi and Pj are strangers, then the edge Pi Pj is drawn as a dashed
segment.
In the example below, P3 and P5 are friends, whereas P1 and P3 are strangers.
P2 P3
P1 P4
P6 P5
Observe that a group of three friends corresponds to a solid triangle in
the graph G, whereas a group of three strangers corresponds to a dashed
triangle. In the example above, there is no solid triangle and, thus, there
is no group of three friends. Since the triangle P2 P4 P5 is dashed, there is a
group of three strangers.
Using this terminology, Theorem 1.1.1 is equivalent to the following:
Theorem 1.1.2 Consider a complete graph on six vertices, in which each
edge is either solid or dashed. Then there is a solid triangle or a dashed
triangle.
Proof. As before, we denote the vertices by P1 , . . . , P6 . Consider the five
edges that are incident on P1 . Using a proof by contradiction, it can easily
be shown that one of the following two claims must hold:
• At least three of these five edges are solid.
• At least three of these five edges are dashed.
1.1. Ramsey Theory 3
We may assume, without loss of generality, that the first claim holds. (Do
you see why?) Consider three edges incident on P1 that are solid and denote
them by P1 A, P1 B, and P1 C.
If at least one of the edges AB, AC, and BC is solid, then there is a solid
triangle. In the left figure below, AB is solid and we obtain the solid triangle
P1 AB.
A A
P1 B P1 B
C C
Otherwise, all edges AB, AC, and BC are dashed, in which case we
obtain the dashed triangle ABC; see the right figure above.
You should convince yourself that Theorem 1.1.2 also holds for complete
graphs with more than six vertices. The example below shows an example of
a complete graph with five vertices without any solid triangle and without
any dashed triangle. Thus, Theorem 1.1.2 does not hold for complete graphs
with five vertices. Equivalently, Theorem 1.1.1 does not hold for groups of
five people.
P2 P3
P1
P5 P4
What about larger groups of friends/strangers? Let k ≥ 3 be an integer.
The following theorem states that even if we take b2k/2 c people, we are not
guaranteed that there is a group of k friends or a group of k strangers.
A k-clique in a graph is a set of k vertices, any two of which are connected
by an edge. For example, a 3-clique is a triangle.
4 Chapter 1. Introduction
Theorem 1.1.3 Let k ≥ 3 and n ≥ 3 be integers with n ≤ b2k/2 c. There
exists a complete graph with n vertices, in which each edge is either solid or
dashed, such that this graph does not contain a solid k-clique and does not
contain a dashed k-clique.
We will prove this theorem in Section 7.2, using elementary counting
techniques and probability theory. This probably sounds surprising to you,
because Theorem 1.1.3 does not have anything to do with probability. In
fact, in Section 7.2, we will prove the following claim: Take k = 20 and
n = 1024. If you go to the ByWard Market in downtown Ottawa and take a
random group of n people, then with very high probability, this group does
not contain a subgroup of k friends and does not contain a subgroup of k
strangers. In other words, with very high probability, every subgroup of k
people contains two friends and two strangers.
1.2 Sperner’s Theorem
Consider a set S with five elements, say, S = {1, 2, 3, 4, 5}. Let S1 , S2 , . . . , Sm
be a sequence of m subsets of S, such that for all i and j with i 6= j,
Si 6⊆ Sj and Sj 6⊆ Si ,
i.e., Si is not a subset of Sj and Sj is not a subset of Si . How large can m
be? The following example shows that m can be as large as 10:
S1 = {1, 2}, S2 = {1, 3}, S3 = {1, 4}, S4 = {1, 5}, S5 = {2, 3},
S6 = {2, 4}, S7 = {2, 5}, S8 = {3, 4}, S9 = {3, 5}, S10 = {4, 5}.
Observe that these are all subsets of S having size two. Can there be such
a sequence of more than 10 subsets? The following theorem states that the
answer is “no”.
Theorem 1.2.1 (Sperner) Let n ≥ 1 be an integer and let S be a set with
n elements. Let S1 , S2 , . . . , Sm be a sequence of m subsets of S, such that for
all i and j with i 6= j,
Si 6⊆ Sj and Sj 6⊆ Si .
Then
n
m≤ .
bn/2c
1.3. The Quick-Sort Algorithm 5
The right-hand side of the last line is a binomial coefficient, which we will
define in Section 3.6. Its value is equal to the number of subsets of S having
size bn/2c. Observe that these subsets satisfy the property in Theorem 1.2.1.
We will prove Theorem 1.2.1 in Section 7.3, using elementary counting
techniques and probability theory. Again, this probably sounds surprising to
you, because Theorem 1.2.1 does not have anything to do with probability.
1.3 The Quick-Sort Algorithm
You are probably familiar with the QuickSort algorithm. This algorithm
sorts any sequence S of n ≥ 0 pairwise distinct numbers in the following way:
• If n = 0 or n = 1, then there is nothing to do.
• If n ≥ 2, then the algorithm picks one of the numbers in S, let us
call it p (which stands for pivot), scans the sequence S, and splits it
into three subsequences: One subsequence S1 contains all elements in
S that are less than p, one subsequence only consists of the element p,
and the third subsequence S2 contains all elements in S that are larger
than p; see the figure below.
<p p >p
S1 S2
The algorithm then recursively runs QuickSort on the subsequence S1 .
After this recursive call has terminated, the algorithm runs, again re-
cursively, QuickSort on the subsequence S2 .
Running QuickSort recursively on the subsequence S1 means that we first
check if S1 has size at most one; if this is the case, nothing needs to be done,
because S1 is sorted already. If S1 has size at least two, then we choose a
pivot p1 in S1 , use p1 to split S1 into three subsequences, recursively run
QuickSort on the subsequence of S1 consisting of all elements that are
less than p1 , and, finally, recursively run QuickSort on the subsequence of
S1 consisting of all elements that are larger than p1 . (We will see recursive
algorithms in more detail in Chapter 4.)
Algorithm QuickSort correctly sorts any sequence of numbers, no mat-
ter how we choose the pivot element. It turns out, however, that the running
6 Chapter 1. Introduction
time of the algorithm heavily depends on the pivots that are chosen in the
recursive calls.
For example, assume that in each (recursive) call to the algorithm, the
pivot happens to be the largest element in the sequence. Then, in each call,
the subsequence of elements that are larger than the pivot is empty. Let us
see what happens in this case:
• We start with a sequence S of size n. The first pivot p is the largest
element in S. Thus, using the notation given above, the subsequence
S1 contains n − 1 elements (all elements of S except for p), whereas the
subsequence S2 is empty. Computing these subsequences can be done
in n “steps”, after which we are in the following situation:
<p p
n − 1 elements
• We now run QuickSort on a sequence of n − 1 elements. Again, the
pivot p1 is the largest element. In n−1 “steps”, we obtain a subsequence
of n − 2 elements that are less than p1 , and an empty subsequence of
elements that are larger than p1 ; see the figure below.
< p1 p1
n − 2 elements
• Next we run QuickSort on a sequence of n − 2 elements. As before,
the pivot p2 is the largest element. In n − 2 “steps”, we obtain a
subsequence of n − 3 elements that are less than p2 , and an empty
subsequence of elements that are larger than p2 ; see the figure below.
< p2 p2
n − 3 elements
You probably see the pattern. The total running time of the algorithm, i.e.,
the total number of “steps”, is proportional to
n + (n − 1) + (n − 2) + (n − 3) + · · · + 3 + 2 + 1,
1.3. The Quick-Sort Algorithm 7
which, by Theorem 2.2.10, is equal to
1 1 1
n(n + 1) = n2 + n,
2 2 2
which, using the Big-Theta notation (see Section 2.3) is Θ(n2 ), i.e., quadratic
in n. It can be shown that this is, in fact, the worst-case running time of the
QuickSort algorithm.
What would be a good choice for the pivot elements? Intuitively, a pivot
is good if the sequences S1 and S2 have (roughly) the same size. Thus, after
the first call, we are in the following situation:
<p p >p
(n − 1)/2 (n − 1)/2
In Section 4.6, we will prove that, if this happens in each recursive call,
the running time of the QuickSort algorithm is only O(n log n). Obviously,
it is not clear at all how we can guarantee that we always choose a good pivot.
It turns out that there is a simple strategy: In each call, choose the pivot
randomly! That is, among all elements involved in the recursive call, pick one
uniformly at random; thus, each element has the same probability of being
chosen. In Section 6.10, we will prove that this leads to an expected running
time of O(n log n).
8 Chapter 1. Introduction
Chapter 2
Mathematical Preliminaries
2.1 Basic Concepts
Throughout this book, we will assume that you know the following mathe-
matical concepts:
1. A set is a collection of well-defined objects. Examples are (i) the set of
all Dutch Olympic Gold Medallists, (ii) the set of all pubs in Ottawa,
and (iii) the set of all even natural numbers.
2. The set of natural numbers is N = {0, 1, 2, 3, . . .}.
3. The set of integers is Z = {. . . , −3, −2, −1, 0, 1, 2, 3, . . .}.
4. The set of rational numbers is Q = {m/n : m ∈ Z, n ∈ Z, n 6= 0}.
5. The set of real numbers is denoted by R.
6. The empty set is the set that does not contain any element. This set
is denoted by ∅.
7. If A is a finite set, then the size (or cardinality) of A, denoted by |A|,
is the number of elements in A. Observe that |∅| = 0.
8. If A and B are sets, then A is a subset of B, written as A ⊆ B, if every
element of A is also an element of B. For example, the set of even
natural numbers is a subset of the set of all natural numbers. Every
set A is a subset of itself, i.e., A ⊆ A. The empty set is a subset of
10 Chapter 2. Mathematical Preliminaries
every set A, i.e., ∅ ⊆ A. We say that A is a proper subset of B, written
as A ⊂ B, if A ⊆ B and A 6= B.
9. If B is a set, then the power set P(B) of B is defined to be the set of
all subsets of B:
P(B) = {A : A ⊆ B}.
Observe that ∅ ∈ P(B) and B ∈ P(B).
10. If A and B are two sets, then
(a) their union is defined as
A ∪ B = {x : x ∈ A or x ∈ B},
(b) their intersection is defined as
A ∩ B = {x : x ∈ A and x ∈ B},
(c) their difference is defined as
A \ B = {x : x ∈ A and x 6∈ B},
(d) the Cartesian product of A and B is defined as
A × B = {(x, y) : x ∈ A and y ∈ B},
(e) the complement of A is defined as
A = {x : x 6∈ A}.
11. A binary relation on two sets A and B is a subset of A × B.
12. A function f from A to B, denoted by f : A → B, is a binary relation R,
having the property that for each element a in A, there is exactly one
ordered pair in R, whose first component is a. If this unique pair is
(a, b), then we will say that f (a) = b, or f maps a to b, or the image
of a under f is b. The set A is called the domain of f , and the set
{b ∈ B : there is an a ∈ A with f (a) = b}
is called the range of f .
2.2. Proof Techniques 11
13. A function f : A → B is one-to-one (or injective), if for any two distinct
elements a and a0 in A, we have f (a) 6= f (a0 ). The function f is onto
(or surjective), if for each element b in B, there exists an element a
in A, such that f (a) = b; in other words, the range of f is equal to the
set B. A function f is a bijection, if f is both injective and surjective.
14. A set A is countable, if A is finite or there is a bijection f : N → A.
The sets N, Z, and Q are countable, whereas R is not.
15. A graph G = (V, E) is a pair consisting of a set V , whose elements
are called vertices, and a set E, where each element of E is a pair of
distinct vertices. The elements of E are called edges.
16. The Boolean values are 1 and 0, that represent true and false, respec-
tively. The basic Boolean operations include
(a) negation (or NOT ), represented by ¬,
(b) conjunction (or AND), represented by ∧,
(c) disjunction (or OR), represented by ∨,
(d) exclusive-or (or XOR), represented by ⊕,
(e) equivalence, represented by ↔ or ⇔,
(f) implication, represented by → or ⇒.
The following table explains the meanings of these operations.
NOT AND OR XOR equivalence implication
¬0 = 1 0∧0=0 0∨0=0 0⊕0=0 0↔0=1 0→0=1
¬1 = 0 0∧1=0 0∨1=1 0⊕1=1 0↔1=0 0→1=1
1∧0=0 1∨0=1 1⊕0=1 1↔0=0 1→0=0
1∧1=1 1∨1=1 1⊕1=0 1↔1=1 1→1=1
2.2 Proof Techniques
A proof is a proof. What kind of a proof? It’s a proof. A proof
is a proof. And when you have a good proof, it’s because it’s
proven.
— Jean Chrétien, Prime Minister of Canada (1993–2003)
12 Chapter 2. Mathematical Preliminaries
In mathematics, a theorem is a statement that is true. A proof is a se-
quence of mathematical statements that form an argument to show that a
theorem is true. The statements in the proof of a theorem include axioms
(assumptions about the underlying mathematical structures), hypotheses of
the theorem to be proved, and previously proved theorems. The main ques-
tion is “How do we go about proving theorems?” This question is similar
to the question of how to solve a given problem. Of course, the answer is
that finding proofs, or solving problems, is not easy; otherwise life would be
dull! There is no specified way of coming up with a proof, but there are some
generic strategies that could be of help. In this section, we review some of
these strategies. The best way to get a feeling of how to come up with a
proof is by solving a large number of problems. Here are some useful tips.
(You may take a look at the book How to Solve It, by George Pólya).
1. Read and completely understand the statement of the theorem to be
proved. Most often this is the hardest part.
2. Sometimes, theorems contain theorems inside them. For example,
“Property A if and only if property B”, requires showing two state-
ments:
(a) If property A is true, then property B is true (A ⇒ B).
(b) If property B is true, then property A is true (B ⇒ A).
Another example is the theorem “Set A equals set B.” To prove this,
we need to prove that A ⊆ B and B ⊆ A. That is, we need to show
that each element of set A is in set B, and each element of set B is in
set A.
3. Try to work out a few simple cases of the theorem just to get a grip on
it (i.e., crack a few simple cases first).
4. Try to write down the proof once you think you have it. This is to
ensure the correctness of your proof. Often, mistakes are found at the
time of writing.
5. Finding proofs takes time, we do not come prewired to produce proofs.
Be patient, think, express and write clearly, and try to be precise as
much as possible.
In the next sections, we will go through some of the proof strategies.
2.2. Proof Techniques 13
2.2.1 Direct proofs
As the name suggests, in a direct proof of a theorem, we just approach the
theorem directly.
Theorem 2.2.1 If n is an odd positive integer, then n2 is odd as well.
Proof. An odd positive integer n can be written as n = 2k + 1, for some
integer k ≥ 0. Then
n2 = (2k + 1)2 = 4k 2 + 4k + 1 = 2(2k 2 + 2k) + 1.
Since 2(2k 2 + 2k) is even, and “even plus one is odd”, we can conclude that
n2 is odd.
For a graph G = (V, E), the degree of a vertex v, denoted by deg(v), is
defined to be the number of edges that are incident on v.
Theorem 2.2.2 Let G = (V, E) be a graph. Then the sum of the degrees of
all vertices is an even integer, i.e.,
X
deg(v)
v∈V
is even.
Proof. If you do not see the meaning of this statement, then first try it out
for a few graphs. The reason why the statement holds is very simple: Each
edge contributes 2 to the summation (because an edge is incident on exactly
two distinct vertices).
Actually, the proof above proves the following theorem.
Theorem 2.2.3 Let G = (V, E) be a graph. Then the sum of the degrees of
all vertices is equal to twice the number of edges, i.e.,
X
deg(v) = 2|E|.
v∈V
14 Chapter 2. Mathematical Preliminaries
2.2.2 Constructive proofs
This technique not only shows the existence of a certain object, it actually
gives a method of creating it:
Theorem 2.2.4 There exists an object with property P.
Proof. Here is the object: [. . .]
And here is the proof that the object satisfies property P: [. . .]
A graph is called 3-regular, if each vertex has degree three. We prove the
following theorem using a constructive proof.
Theorem 2.2.5 For every even integer n ≥ 4, there exists a 3-regular graph
with n vertices.
Proof. Let
V = {0, 1, 2, . . . , n − 1},
and
E = {{i, i+1} : 0 ≤ i ≤ n−2}∪{{n−1, 0}}∪{{i, i+n/2} : 0 ≤ i ≤ n/2−1}.
Then the graph G = (V, E) is 3-regular.
Convince yourself that this graph is indeed 3-regular. It may help to draw
the graph for, say, n = 8.
2.2.3 Nonconstructive proofs
In a nonconstructive proof, we show that a certain object exists, without
actually creating it. Here is an example of such a proof:
Theorem 2.2.6 There exist irrational numbers x and y such that xy is ra-
tional.
Proof. There are two possible cases.
√ √2
Case 1: 2 ∈ Q. √
In√this case, we take x = y = 2. In Theorem 2.2.9 below, we will prove
that 2 is irrational.
2.2. Proof Techniques 15
√
√ 2
Case 2: 2 6∈ Q. √
√ 2 √
In this case, we take x = 2 and y = 2. Since
√ √2
√ √
2 2
xy = 2 = 2 = 2,
the claim in the theorem follows.
Observe that this proof indeed proves the theorem, but it does not give
an example of a pair of irrational numbers x and y such that xy is rational.
2.2.4 Proofs by contradiction
This is how a proof by contradiction looks like:
Theorem 2.2.7 Statement S is true.
Proof. Assume that statement S is false. Then, derive a contradiction (such
as 1 + 1 = 3).
In other words, we show that the statement “¬S ⇒ false” is true. This
is sufficient, because the contrapositive of the statement “¬S ⇒ false” is the
statement “true ⇒ S”. The latter logical formula is equivalent to S, and
that is what we wanted to show.
Below, we give two examples of proofs by contradiction.
Theorem 2.2.8 Let n be a positive integer. If n2 is even, then n is even.
Proof. We will prove the theorem by contradiction. Thus, we assume that
n2 is even, but n is odd. Since n is odd, we know from Theorem 2.2.1 that
n2 is odd. This is a contradiction, because we assumed that n2 is even.
√ √
Theorem 2.2.9 2 is irrational, i.e., 2 cannot be written as a fraction of
two integers.
Proof.
√ We will prove√the theorem by contradiction. Thus, we assume that
2 is rational. Then
√ 2 can be written as a fraction of two integers m ≥ 1
and n ≥ 1, i.e., 2 = m/n. We may assume that m and n do not share
16 Chapter 2. Mathematical Preliminaries
any common factors, i.e., the greatest common divisor of m and n is equal
to one; if √this is not the case, then we can get rid of the common factors. By
squaring 2 = m/n, we get 2n2 = m2 . This implies that m2 is even. Then,
by Theorem 2.2.8, m is even, which means that we can write m as m = 2k,
for some positive integer k. It follows that 2n2 = m2 = 4k 2 , which implies
that n2 = 2k 2 . Hence, n2 is even. Again by Theorem 2.2.8, it follows that n
is even.
We have shown that m and n are both even. But we know that m and
n
√ are not both even. Hence, we have a contradiction.√Our assumption that
2 is rational is wrong. Thus, we can conclude that 2 is irrational.
There is a nice discussion of this proof in the book My Brain is Open:
The Mathematical Journeys of Paul Erdős by Bruce Schechter.
2.2.5 Proofs by induction
This is a very powerful and important technique for proving theorems.
For each positive integer n, let P (n) be a mathematical statement that
depends on n. Assume we wish to prove that P (n) is true for all positive
integers n. A proof by induction of such a statement is carried out as follows:
Base Case: Prove that P (1) is true.
Induction Step: Prove that for all n ≥ 1, the following holds: If P (n) is
true, then P (n + 1) is also true.
In the induction step, we choose an arbitrary integer n ≥ 1 and assume
that P (n) is true; this is called the induction hypothesis. We then prove that
P (n + 1) is also true.
Theorem 2.2.10 For all positive integers n, we have
n(n + 1)
1 + 2 + 3 + ··· + n = .
2
Proof. We start with the base case of the induction. If n = 1, then both the
left-hand side and the right-hand side are equal to 1. Therefore, the theorem
is true for n = 1.
2.2. Proof Techniques 17
For the induction step, let n ≥ 1 and assume that the theorem is true
for n, i.e., assume that
n(n + 1)
1 + 2 + 3 + ··· + n = .
2
We have to prove that the theorem is true for n + 1, i.e., we have to prove
that
(n + 1)(n + 2)
1 + 2 + 3 + · · · + (n + 1) = .
2
Here is the proof:
1 + 2 + 3 + · · · + (n + 1) = |1 + 2 + 3{z+ · · · + n} +(n + 1)
n(n+1)
= 2
n(n + 1)
= + (n + 1)
2
(n + 1)(n + 2)
= .
2
By the way, here is an alternative proof of the theorem above: Let S =
1 + 2 + 3 + · · · + n. Then,
S = 1 + 2 + 3 + ··· + (n − 2) + (n − 1) + n
S = n + (n − 1) + (n − 2) + ··· + 3 + 2 + 1
2S = (n + 1) + (n + 1) + (n + 1) + ··· + (n + 1) + (n + 1) + (n + 1)
Since there are n terms on the right-hand side, we have 2S = n(n + 1). This
implies that S = n(n + 1)/2.
Theorem 2.2.11 For every positive integer n, a − b is a factor of an − bn .
Proof. A direct proof can be given by providing a factorization of an − bn :
an − bn = (a − b)(an−1 + an−2 b + an−3 b2 + · · · + abn−2 + bn−1 ).
We now prove the theorem by induction. For the base case, let n = 1. The
claim in the theorem is “a − b is a factor of a − b”, which is obviously true.
Let n ≥ 1 and assume that a − b is a factor of an − bn . We have to prove
that a − b is a factor of an+1 − bn+1 . We have
an+1 − bn+1 = an+1 − an b + an b − bn+1 = an (a − b) + (an − bn )b.
18 Chapter 2. Mathematical Preliminaries
The first term on the right-hand side is divisible by a − b. By the induction
hypothesis, the second term on the right-hand side is divisible by a − b as
well. Therefore, the entire right-hand side is divisible by a − b. Since the
right-hand side is equal to an+1 − bn+1 , it follows that a − b is a factor of
an+1 − bn+1 .
We now give an alternative proof of Theorem 2.2.3:
Theorem 2.2.12 Let G = (V, E) be a graph with m edges. Then the sum
of the degrees of all vertices is equal to twice the number of edges, i.e.,
X
deg(v) = 2m.
v∈V
Proof. The proof is by induction on the number m of edges. For the base
case of the induction, assume thatPm = 0. Then the graph G does not
contain any edges and, therefore, v∈V deg(v) = 0. Thus, the theorem is
true if m = 0.
Let m ≥ 0 and assume that the theorem is true for every graph with m
edges.
P Let G be an arbitrary graph with m + 1 edges. We have to prove that
v∈V deg(v) = 2(m + 1).
Let {a, b} be an arbitrary edge in G, and let G0 be the graph obtained
from G by removing the edge {a, b}. Since G0 has m edges, we know from
the induction hypothesis that the sum of the degrees of all vertices in G0 is
equal to 2m. Using this, we obtain
X X
deg(v) = deg(v) + 2 = 2m + 2 = 2(m + 1).
v∈G v∈G0
2.2.6 More examples of proofs
Recall Theorem 2.2.5, which states that for every even integer n ≥ 4, there
exists a 3-regular graph with n vertices. The following theorem explains why
we stated this theorem for even values of n.
Theorem 2.2.13 Let n ≥ 5 be an odd integer. There is no 3-regular graph
with n vertices.
2.2. Proof Techniques 19
Proof. The proof is by contradiction. Thus, we assume that there exists a
graph G = (V, E) with n vertices that is 3-regular. Let m be the number of
edges in G. Since deg(v) = 3 for every vertex v, we have
X
deg(v) = 3n.
v∈V
On the other hand, by Theorem 2.2.3, we have
X
deg(v) = 2m.
v∈V
It follows that
3n = 2m.
Since n is an odd integer, the left-hand side in this equation is an odd integer
as well. The right-hand side, however, is an even integer. This is a contra-
diction.
Let Kn be the complete graph on n vertices. This graph has a vertex set
of size n, and every pair of distinct vertices is joined by an edge.
If G = (V, E) is a graph with n vertices, then the complement G of G is
the graph with vertex set V that consists of those edges of Kn that are not
present in G.
Theorem 2.2.14 Let n ≥ 2 and let G be a graph on n vertices. Then G is
connected or G is connected.
Proof. We prove the theorem by induction on the number n of vertices. For
the base case, assume that n = 2. There are two possibilities for the graph
G:
1. G contains one edge. In this case, G is connected.
2. G does not contain an edge. In this case, the complement G contains
one edge and, therefore, G is connected.
Thus, for n = 2, the theorem is true.
Let n ≥ 2 and assume that the theorem is true for every graph with n
vertices. Let G be graph with n + 1 vertices. We have to prove that G is
connected or G is connected. We consider three cases.
20 Chapter 2. Mathematical Preliminaries
Case 1: There is a vertex v whose degree in G is equal to n.
Since G has n+1 vertices, v is connected by an edge to every other vertex
of G. Therefore, G is connected.
Case 2: There is a vertex v whose degree in G is equal to 0.
In this case, the degree of v in the graph G is equal to n. Since G has n+1
vertices, v is connected by an edge to every other vertex of G. Therefore, G
is connected.
Case 3: For every vertex v, the degree of v in G is in {1, 2, . . . , n − 1}.
Let v be an arbitrary vertex of G. Let G0 be the graph obtained by
deleting from G the vertex v, together with all edges that are incident on v.
Since G0 has n vertices, we know from the induction hypothesis that G0 is
connected or G0 is connected.
Let us first assume that G0 is connected. Then the graph G is connected
as well, because there is at least one edge in G between v and some vertex
of G0 .
If G0 is not connected, then G0 must be connected. Since we are in Case 3,
we know that the degree of v in G is in the set {1, 2, . . . , n − 1}. It follows
that the degree of v in the graph G is in this set as well. Hence, there is at
least one edge in G between v and some vertex in G0 . This implies that G is
connected.
The previous theorem can be rephrased as follows:
Theorem 2.2.15 Let n ≥ 2 and consider the complete graph Kn on n ver-
tices. Color each edge of this graph as either red or blue. Let R be the graph
consisting of all the red edges, and let B be the graph consisting of all the
blue edges. Then R is connected or B is connected.
If you like surprising proofs of various mathematical results, you should
read the book Proofs from THE BOOK by Martin Aigner and Günter Ziegler.
2.3 Asymptotic Notation
Let f : N → R and g : N → R be functions such that f (n) > 0 and g(n) > 0
for all n ∈ N.
2.4. Logarithms 21
• We say that f (n) = O(g(n)) if the following is true: There exist con-
stants c > 0 and k > 0 such that for all n ≥ k,
f (n) ≤ c · g(n).
• We say that f (n) = Ω(g(n)) if the following is true: There exist con-
stants c > 0 and k > 0 such that for all n ≥ k,
f (n) ≥ c · g(n).
• We say that f (n) = Θ(g(n)) if f (n) = O(g(n)) and f (n) = Ω(g(n)).
Thus, there exist constants c > 0, c0 > 0, and k > 0 such that for all
n ≥ k,
c · g(n) ≤ f (n) ≤ c0 · g(n).
For example, for all n ≥ 1, we have
13 + 7n − 5n2 + 8n3 ≤ 13 + 7n + 8n3
≤ 13n3 + 7n3 + 8n3
= 28n3 .
Thus, by taking c = 28 and k = 1, it follows that
13 + 7n − 5n2 + 8n3 = O(n3 ). (2.1)
We also have
13 + 7n − 5n2 + 8n3 ≥ −5n2 + 8n3 .
Since n3 ≥ 5n2 for all n ≥ 5, it follows that, again for all n ≥ 5,
13 + 7n − 5n2 + 8n3 ≥ −5n2 + 8n3
≥ −n3 + 8n3
= 7n3 .
Hence, by taking c = 7 and k = 5, we have shown that
13 + 7n − 5n2 + 8n3 = Ω(n3 ). (2.2)
It follows from (2.1) and (2.2) that
13 + 7n − 5n2 + 8n3 = Θ(n3 ).
22 Chapter 2. Mathematical Preliminaries
2.4 Logarithms
If b and x are real numbers with b > 1 and x > 0, then logb x denotes the
logarithm of x with base b. Note that
logb x = y if and only if by = x.
If b = 2, then we write log x instead of log2 x. We write ln x to refer to the
natural logarithm of x with base e.
Lemma 2.4.1 If b > 1 and x > 0, then
blogb x = x.
Proof. We have seen above that y = logb x if and only if by = x. Thus, if
we write y = logb x, then blogb x = by = x.
For example, if x > 0, then
2log x = x.
Lemma 2.4.2 If b > 1, x > 0, and a is a real number, then
logb (xa ) = a logb x.
Proof. Let y = logb x. Then
a logb x = ay.
Since y = logb x, we have by = x and, thus,
xa = (by )a = bay .
Taking logarithms (with base b) on both sides gives
logb (xa ) = logb (bay ) = ay = a logb x.
For example, for x > 1, we get
2 log log x = log log2 x
and
22 log log x = 2log(log x) = log2 x.
2
2.5. Exercises 23
Lemma 2.4.3 If b > 1, c > 1, and x > 0, then
logc x
logb x = .
logc b
Proof. Let y = logb x. Then by = x, and we get
logc x = logc (by ) = y logc b = logb x logc b.
For example, if x > 0, then
ln x
log x = .
ln 2
2.5 Exercises
Proofs that use a big hammer:
√
Theorem: For any integer n ≥ 3, n 2 is irrational.
√
Proof: Assume n 2√ is rational. Then there exist positive integers
a and b such that n 2 = a/b. Thus, we have 2 = (a/b)n , which is
equivalent to 2 · bn = an , which is equivalent to
b n + b n = an .
This contradicts Fermat’s Last Theorem.
√
2.1 Prove that p is irrational for every prime number p.
√
2.2 Let n be a positive integer that is not a perfect square. Prove that n
is irrational.
2.3 Use induction to prove that every integer n ≥ 2 can be written as a
product of prime numbers.
2.4 Prove by induction that n4 − 4n2 is divisible by 3, for all integers n ≥ 1.
24 Chapter 2. Mathematical Preliminaries
2.5 Prove that n
X 1
2
< 2 − 1/n,
i=1
i
for all integers n ≥ 2.
2.6 Prove that 9 divides n3 + (n + 1)3 + (n + 2)3 , for all integers n ≥ 0.
n
2.7 The Fermat numbers F0 , F1 , F2 , . . . are defined by Fn = 22 +1 for n ≥ 0.
• Prove by induction that
F0 F1 F2 · · · Fn−1 = Fn − 2
for all integers n ≥ 1.
• Prove that for any two distinct integers n ≥ 0 and m ≥ 0, the greatest
common divisor of Fn and Fm is equal to 1.
• Conclude that there are infinitely many prime numbers.
2.8 Prove by induction that n! > 21+n/2 for all integers n ≥ 3.
Chapter 3
Counting
There are three types of people, those who can count and those
who cannot count.
Given a set of 23 elements, how many subsets of size 17 are there? How
many solutions are there to the equation
x1 + x2 + · · · + x12 = 873,
where x1 ≥ 0, x2 ≥ 0, . . . , x12 ≥ 0 are integers? In this chapter, we will
introduce some general techniques that can be used to answer questions of
these types.
3.1 The Product Rule
How many strings of two characters are there, if the first character must be
an uppercase letter and the second character must be a digit? Examples
of such strings are A0, K7, and Z9. The answer is obviously 26 · 10 = 260,
because there are 26 choices for the first character and, no matter which letter
we choose for being the first character, there are 10 choices for the second
character. We can look at this in the following way: Consider the “procedure”
of writing a string of two characters, the first one being an uppercase letter,
and the second one being a digit. Then our original question becomes “how
many ways are there to perform this procedure?” Observe that the procedure
consists of two “tasks”, the first one being writing the first character, and the
26 Chapter 3. Counting
second one being writing the second character. Obviously, there are 26 ways
to do the first task. Next, observe that, regardless of how we do the first
task, there are 10 ways to do the second task. The Product Rule states that
the total number of ways to perform the entire procedure is 26 · 10 = 260.
Product Rule: Assume a procedure consists of performing a se-
quence of m tasks in order. Furthermore, assume that for each
i = 1, 2, . . . , m, there are Ni ways to do the i-th task, regardless
of how the first i − 1 tasks were done. Then, there are N1 N2 · · · Nm
ways to do the entire procedure.
In the example above, we have m = 2, N1 = 26, and N2 = 10.
3.1.1 Counting Bitstrings of Length n
Let n ≥ 1 be an integer. A bitstring of length n is a sequence of 0’s and 1’s.
How many bitstrings of length n are there? To apply the Product Rule, we
have to specify the “procedure” and the “tasks”:
• The procedure is “write a bitstring of length n”.
• For i = 1, 2, . . . , n, the i-th task is “write one bit”.
There are two ways to do the i-th task, regardless of how we did the first
i − 1 tasks. Therefore, we can apply the Product Rule with Ni = 2 for
i = 1, 2, . . . , n, and conclude that there are N1 N2 · · · Nn = 2n ways to do the
entire procedure. As a result, the number of bitstrings of length n is equal
to 2n .
Theorem 3.1.1 For any integer n ≥ 1, the number of bitstrings of length n
is equal to 2n .
3.1.2 Counting Functions
Let m ≥ 1 and n ≥ 1 be integers, let A be a set of size m, and let B be a set
of size n. How many functions f : A → B are there?
Write the set A as A = {a1 , a2 , . . . , am }. Any function f : A → B is
completely specified by the values f (a1 ), f (a2 ), . . . , f (am ), where each such
value can be any element of B. Again, we are going to apply the Product
Rule. Thus, we have to specify the “procedure” and the “tasks”:
3.1. The Product Rule 27
• The procedure is “specify the values f (a1 ), f (a2 ), . . . , f (am )”.
• For i = 1, 2, . . . , m, the i-th task is “specify the value f (ai )”.
ai f (ai)
A B
For each i, f (ai ) can be any of the n elements of B. As a result, there are
Ni = n ways to do the i-th task, regardless of how we did the first i − 1 tasks.
By the Product Rule, there are N1 N2 · · · Nm = nm ways to do the entire
procedure and, hence, this many functions f : A → B. We have proved the
following result:
Theorem 3.1.2 Let m ≥ 1 and n ≥ 1 be integers, let A be a set of size m,
and let B be a set of size n. The number of functions f : A → B is equal
to nm .
Recall that a function f : A → B is one-to-one if for any i and j with
i 6= j, we have f (ai ) 6= f (aj ). How many one-to-one functions f : A → B
are there?
If m > n, then there is no such function. (Do you see why?) Assume
that m ≤ n. To determine the number of one-to-one functions, we use the
same procedure and tasks as before.
• Since f (a1 ) can be any of the n elements of B, there are N1 = n ways
to do the first task.
• In the second task, we have to specify the value f (a2 ). Since the func-
tion f is one-to-one and since we have already specified f (a1 ), we can
choose f (a2 ) to be any of the n − 1 elements in the set B \ {f (a1 )}. As
a result, there are N2 = n − 1 ways to do the second task. Note that
this is true, regardless of how we did the first task.
28 Chapter 3. Counting
• In general, in the i-th task, we have to specify the value f (ai ). Since
we have already specified f (a1 ), f (a2 ), . . . , f (ai−1 ), we can choose f (ai )
to be any of the n − i + 1 elements in the set
B \ {f (a1 ), f (a2 ), . . . , f (ai−1 )}.
As a result, there are Ni = n − i + 1 ways to do the i-th task. Note
that this is true, regardless of how we did the first i − 1 tasks.
By the Product Rule, there are
N1 N2 · · · Nm = n(n − 1)(n − 2) · · · (n − m + 1)
ways to do the entire procedure, which is also the number of one-to-one
functions f : A → B.
Recall the factorial function
1 if k = 0,
k! =
1 · 2 · 3 · · · k if k ≥ 1.
We can simplify the product
n(n − 1)(n − 2) · · · (n − m + 1)
by observing that it is “almost” a factorial:
n(n − 1)(n − 2) · · · (n − m + 1)
(n − m)(n − m − 1) · · · 1
= n(n − 1)(n − 2) · · · (n − m + 1) ·
(n − m)(n − m − 1) · · · 1
n(n − 1)(n − 2) · · · 1
=
(n − m)(n − m − 1) · · · 1
n!
= .
(n − m)!
We have proved the following result:
Theorem 3.1.3 Let m ≥ 1 and n ≥ 1 be integers, let A be a set of size m,
and let B be a set of size n.
1. If m > n, then there is no one-to-one function f : A → B.
2. If m ≤ n, then the number of one-to-one functions f : A → B is equal
to
n!
.
(n − m)!
3.1. The Product Rule 29
3.1.3 Placing Books on Shelves
Let m ≥ 1 and n ≥ 1 be integers, and consider m books B1 , B2 , . . . , Bm and
n bookshelves S1 , S2 , . . . , Sn . How many ways are there to place the books
on the shelves? Placing the books on the shelves means that
• we specify for each book the shelf at which this book is placed, and
• we specify for each shelf the left-to-right order of the books that are
placed on that shelf.
Some bookshelves may be empty. We assume that each shelf is large enough
to fit all books. In the figure below, you see two different placements.
S1 B4 B3 S1
S2 B1 S2 B3 B1 B4
S3 B2 B5 S3 B5 B2
We are again going to use the Product Rule to determine the number of
placements.
• The procedure is “place the m books on the n shelves”.
• For i = 1, 2, . . . , m, the i-th task is “place book Bi on the shelves”.
When placing book Bi , we can place it on the far left or far right
of any shelf or between any two of the books B1 , . . . , Bi−1 that have
already been placed.
Let us see how many ways there are to do each task.
• Just before we place book B1 , all shelves are empty. Therefore, there
are N1 = n ways to do the first task.
• In the second task, we have to place book B2 . Since B1 has already
been placed, we have the following possibilities for placing B2 :
– We place B2 on the far left of any of the n shelves.
30 Chapter 3. Counting
– We place B2 immediately to the right of B1 .
As a result, there are N2 = n + 1 ways to do the second task. Note
that this is true, regardless of how we did the first task.
• In general, in the i-th task, we have to place book Bi . Since the books
B1 , B2 , . . . , Bi−1 have already been placed, we have the following pos-
sibilities for placing Bi :
– We place Bi on the far left of any of the n shelves.
– We place Bi immediately to the right of one of B1 , B2 , . . . , Bi−1 .
As a result, there are Ni = n + i − 1 ways to do the i-th task. Note
that this is true, regardless of how we did the first i − 1 tasks.
Thus, by the Product Rule, there are
N1 N2 · · · Nm = n(n + 1)(n + 2) · · · (n + m − 1)
ways to do the entire procedure, which is also the number of placements of
the m books on the n shelves. As before, we use factorials to simplify this
product:
n(n + 1)(n + 2) · · · (n + m − 1)
1 · 2 · 3 · · · (n − 1)
= · n(n + 1)(n + 2) · · · (n + m − 1)
1 · 2 · 3 · · · (n − 1)
(n + m − 1)!
= .
(n − 1)!
We have proved the following result:
Theorem 3.1.4 Let m ≥ 1 and n ≥ 1 be integers. The number of ways to
place m books on n bookshelves is equal to
(n + m − 1)!
.
(n − 1)!
3.2. The Bijection Rule 31
3.2 The Bijection Rule
Let n ≥ 0 be an integer and let S be a set with n elements. How many
subsets does S have? If n = 0, then S = ∅ and there is only one subset of S,
namely S itself. Assume from now on that n ≥ 1. As we will see below,
asking for the number of subsets of S is exactly the same as asking for the
number of bitstrings of length n.
Let A and B be finite sets. Recall that a function f : A → B is a bijection
if
• f is one-to-one, i.e., if a 6= a0 then f (a) 6= f (a0 ), and
• f is onto, i.e., for each b in B, there is an a in A such that f (a) = b.
This means that
• each element of A corresponds to a unique element of B and
• each element of B corresponds to a unique element of A.
It should be clear that this means that A and B contain the same number
of elements.
Bijection Rule: Let A and B be finite sets. If there exists a bijection
f : A → B, then |A| = |B|, i.e., A and B have the same size.
Let us see how we can apply this rule to the subset problem. We define
the following two sets A and B:
• A = P(S), i.e., the power set of S, which is the set of all subsets of S:
P(S) = {T : T ⊆ S}.
• B is the set of all bitstrings of length n.
We have seen in Theorem 3.1.1 that the set B has size 2n . Therefore, if we
can show that there exists a bijection f : A → B, then, according to the
Bijection Rule, we have |A| = |B| and, thus, the number of subsets of S is
equal to 2n .
Write the set S as S = {s1 , s2 , . . . , sn }. We define the function f : A → B
in the following way:
32 Chapter 3. Counting
• For any T ∈ A (i.e., T ⊆ S), f (T ) is the bitstring b1 b2 . . . bn , where
1 if si ∈ T ,
bi =
0 if si 6∈ T .
For example, assume that n = 5.
• If T = {s1 , s3 , s4 }, then f (T ) = 10110.
• If T = S = {s1 , s2 , s3 , s4 , s5 }, then f (T ) = 11111.
• If T = ∅, then f (T ) = 00000.
It is not difficult to see that each subset of S corresponds to a unique
bitstring of length n, and each bitstring of length n corresponds to a unique
subset of S. Therefore, this function f is a bijection between A and B.
Thus, we have shown that there exists a bijection f : A → B. This,
together with Theorem 3.1.1 and the Bijection Rule, implies the following
result:
Theorem 3.2.1 For any integer n ≥ 0, the number of subsets of a set of
size n is equal to 2n .
You will probably have noticed that we could have proved this result
directly using the Product Rule: The procedure “specify a subset of S =
{s1 , s2 , . . . , sn }” can be carried out by specifying, for i = 1, 2, . . . , n, whether
or not si is contained in the subset. For each i, there are two choices. As a
result, there are 2n ways to do the procedure.
To conclude this section, we remark that we have already been using the
Bijection Rule in Section 3.1!
3.3. The Complement Rule 33
The Product Rule and the Bijection Rule: In order to
apply the Product Rule to a counting problem, we need the
following:
1. Phrase the counting problem in terms of doing a proce-
dure, consisting of a number of tasks.
2. There must be a one-to-one correspondence between the
different ways to do the procedure and the objects we
are counting. In other words:
(a) Each way to do the procedure must correspond to
a unique object we are counting.
(b) Conversely, each object we are counting must cor-
respond to a unique way to do the procedure.
3. Once we have this one-to-one correspondence, the Bi-
jection Rule implies that the number of objects is equal
to the number of ways to do the procedure.
3.3 The Complement Rule
Consider strings consisting of 8 characters, each character being a lowercase
letter or a digit. Such a string is called a valid password, if it contains at
least one digit. How many valid passwords are there? One way to answer
this question is to first count the valid passwords with exactly one digit,
then count the valid passwords with exactly two digits, then count the valid
passwords with exactly three digits, etc. As we will see below, it is easier to
first count the strings that do not contain any digit.
Recall that the difference U \ A of the two sets U and A is defined as
U \ A = {x : x ∈ U and x 6∈ A}.
Complement Rule: Let U be a finite set and let A be a subset of U .
Then
|A| = |U | − |U \ A|.
34 Chapter 3. Counting
This rule follows easily from the fact that |U | = |A| + |U \ A|, which holds
because each element in U is either in A or in U \ A.
To apply the Complement Rule to the password problem, let U be the
set of all strings consisting of 8 characters, each character being a lowercase
letter or a digit, and let A be the set of all valid passwords, i.e., all strings
in U that contain at least one digit. Note that U \ A is the set of all strings
of 8 characters, each character being a lowercase letter or a digit, that do
not contain any digit. In other words, U \ A is the set of all strings of 8
characters, where each character is a lowercase letter.
By the Product Rule, the set U has size 368 , because each string in U
has 8 characters, and there are 26 + 10 = 36 choices for each character.
Similarly, the set U \ A has size 268 , because there are 26 choices for each
of the 8 characters. Then, by the Complement Rule, the number of valid
passwords is equal to
|A| = |U | − |U \ A| = 368 − 268 = 2, 612, 282, 842, 880.
3.4 The Sum Rule
If A and B are two finite sets that are disjoint, i.e., A ∩ B = ∅, then it is
obvious that the size of A ∪ B is equal to the sum of the sizes of A and B.
Sum Rule: Let A1 , A2 , . . . , Am be a sequence of finite and pairwise
disjoint sets. Then
|A1 ∪ A2 ∪ · · · ∪ Am | = |A1 | + |A2 | + · · · + |Am |.
Note that we already used this rule in Section 3.3 when we argued why
the Complement Rule is correct!
To give an example, consider strings consisting of 6, 7, or 8 characters,
each character being a lowercase letter or a digit. Such a string is called a
valid password, if it contains at least one digit. Let A be the set of all valid
passwords. What is the size of A?
For i = 6, 7, 8, let Ai be the set of all valid passwords of length i. It is
obvious that A = A6 ∪ A7 ∪ A8 . Since the three sets A6 , A7 , and A8 are
pairwise disjoint, we have, by the Sum Rule,
|A| = |A6 | + |A7 | + |A8 |.
3.5. Inclusion-Exclusion 35
We have seen in Section 3.3 that |A8 | = 368 − 268 . By the same arguments,
we have |A6 | = 366 − 266 and |A7 | = 367 − 267 . Thus, the number of valid
passwords is equal to
|A| = 366 − 266 + 367 − 267 + 368 − 268 = 2, 684, 483, 063, 360.
3.5 The Principle of Inclusion and Exclusion
The Sum Rule holds only for sets that are pairwise disjoint. Consider two
finite sets A and B that are not necessarily disjoint. How can we determine
the size of the union A ∪ B? We can start with the sum |A| + |B|, i.e., we
include both A and B. In the Venn diagram below,
• x is in A but not in B; it is counted exactly once in |A| + |B|,
• z is in B but not in A; it is counted exactly once in |A| + |B|,
• y is in A and in B; it is counted exactly twice in |A| + |B|.
Based on this, if we subtract the size of the intersection A∩B, i.e., we exclude
A ∩ B, then we have counted every element of A ∪ B exactly once.
x y z
A B
Inclusion-Exclusion: Let A and B be finite sets. Then
|A ∪ B| = |A| + |B| − |A ∩ B|.
To give an example, let us count the bitstrings of length 17 that start
with 010 or end with 11. Let S be the set of all such bitstrings. Define A to
be the set of all bitstrings of length 17 that start with 010, and define B to
be the set of all bitstrings of length 17 that end with 11. Then S = A ∪ B
and, thus, we have to determine the size of A ∪ B.
36 Chapter 3. Counting
• The size of A is equal to the number of bitstrings of length 14, because
the first three bits of every string in A are fixed. Therefore, by the
Product Rule, we have |A| = 214 .
• The size of B is equal to the number of bitstrings of length 15, because
the last two bits of every string in B are fixed. Therefore, by the
Product Rule, we have |B| = 215 .
• Each string in A ∩ B starts with 010 and ends with 11. Thus, five bits
are fixed for every string in A ∩ B. It follows that the size of A ∩ B
is equal to the number of bitstrings of length 12. Therefore, by the
Product Rule, we have |A ∩ B| = 212 .
By applying the Inclusion-Exclusion formula, it follows that
|S| = |A ∪ B| = |A| + |B| − |A ∩ B| = 214 + 215 − 212 = 45, 056.
The Inclusion-Exclusion formula can be generalized to more than two
sets. You are encouraged to verify, using the Venn diagram below, that the
following formula is the correct one for three sets.
Inclusion-Exclusion: Let A, B, and C be finite sets. Then
|A∪B ∪C| = |A|+|B|+|C|−|A∩B|−|A∩C|−|B ∩C|+|A∩B ∩C|.
B
A
C
To give an example, how many bitstrings of length 17 are there that start
with 010, or end with 11, or have 10 at positions1 7 and 8? Let S be the set
1
The positions are numbered 1, 2, . . . , 17.
3.6. Permutations and Binomial Coefficients 37
of all such bitstrings. Define A to be the set of all bitstrings of length 17 that
start with 010, define B to be the set of all bitstrings of length 17 that end
with 11, and define C to be the set of all bitstrings of length 17 that have 10
at positions 7 and 8. Then S = A ∪ B ∪ C and, thus, we have to determine
the size of A ∪ B ∪ C.
• We have seen before that |A| = 214 , |B| = 215 , and |A ∩ B| = 212 .
• We have |C| = 215 , because the bits at positions 7 and 8 are fixed for
every string in C.
• We have |A∩C| = 212 , because 5 bits are fixed for every string in A∩C.
• We have |B ∩ C| = 213 , because 4 bits are fixed for every string in
B ∩ C.
• We have |A ∩ B ∩ C| = 210 , because 7 bits are fixed for every string in
A ∩ B ∩ C.
By applying the Inclusion-Exclusion formula, it follows that
|S| = |A ∪ B ∪ C|
= |A| + |B| + |C| − |A ∩ B| − |A ∩ C| − |B ∩ C| + |A ∩ B ∩ C|
= 214 + 215 + 215 − 212 − 212 − 213 + 210
= 66, 560.
3.6 Permutations and Binomial Coefficients
A permutation of a finite set S is an ordered sequence of the elements of S,
in which each element occurs exactly once. For example, the set S = {a, b, c}
has six permutations:
abc, acb, bac, bca, cab, cba
Theorem 3.6.1 Let n ≥ 0 be an integer and let S be a set with n elements.
There are exactly n! many permutations of S.
Proof. If n = 0, then S = ∅ and the only permutation of S is the empty
sequence. Since 0! = 1, the claim holds for n = 0. Assume that n ≥ 1 and
38 Chapter 3. Counting
denote the elements of S by s1 , s2 , . . . , sn . Consider the procedure “write a
permutation of S” and, for i = 1, 2, . . . , n, the task “write the i-th element in
the permutation”. When we do the i-th task, we have already written i − 1
elements of the permutation; we cannot take any of these elements for the
i-th task. Therefore, there are n − (i − 1) = n − i + 1 ways to do the i-th
task. By the Product Rule, there are
n · (n − 1) · (n − 2) · · · 2 · 1 = n!
ways to do the procedure. This number is equal to the number of permuta-
tions of S.
Note that we could also have used Theorem 3.1.3 to prove Theorem 3.6.1:
A permutation of S can be regarded to be a one-to-one function f : S → S.
Therefore, by applying Theorem 3.1.3 with A = S, B = S and, thus, m = n,
we obtain Theorem 3.6.1.
Consider the set S = {a, b, c, d, e}. How many 3-element subsets does S
have? Recall that in a set, the order of the elements does not matter. Here
is a list of all 10 subsets of S having size 3:
{a, b, c}, {a, b, d}, {a, b, e}, {a, c, d}, {a, c, e},
{a, d, e}, {b, c, d}, {b, c, e}, {b, d, e}, {c, d, e}
Definition 3.6.2 Let n ≥ 0 and k ≥ 0 be integers. The binomial coefficient
n
k
denotes the number of k-element subsets of an n-element set.
The symbol nk is pronounced as “n choose k”.
The example above shows that 53 = 10. Since the empty set has exactly
0
one subset of size zero (the empty set itself), we have 0 =n1. Note that
n
k
= 0 if k > n. Below, we derive a formula for the value of k if 0 ≤ k ≤ n.
Let S be a set with n elements and let A be the set of all ordered sequences
consisting of exactly k pairwise distinct elements of S. We are going to count
the elements of A in two different ways.
The first way is by using the Product Rule. This gives
n!
|A| = n(n − 1)(n − 2) · · · (n − k + 1) = . (3.1)
(n − k)!
Observe that (3.1) also follows from Theorem 3.1.3. (Do you see why?)
In the second way, we do the following:
3.6. Permutations and Binomial Coefficients 39
n
• Write all k
subsets of S having size k.
• For each of these subsets, write a list of all k! permutations of this
subset.
If we put all these lists together, then we obtain a big list in which each
ordered sequence of k pairwise distinct elements of S appears exactly once.
In other words, the big list contains each element of A exactly once. Since
the big list has size nk k!, it follows that
n
|A| = k!. (3.2)
k
Since the right-hand sides of (3.1) and (3.2) are equal (because they are both
equal to |A|), we obtain the following result:
Theorem 3.6.3 Let n and k be integers with 0 ≤ k ≤ n. Then
n n!
= .
k k!(n − k)!
For example,
1·2·3·4·5
5 5! 5!
= = = = 10
3 3!(5 − 3)! 3!2! 1·2·3·1·2
and
0 0! 1
= = = 1;
0 0!0! 1·1
recall that we defined 0! to be equal to 1.
3.6.1 Some Examples
First Example: Consider a standard deck of 52 cards. How many hands
of 5 cards are there? Any such hand is a 5-element subset of the set of 52
cards and, therefore, the number of hands of 5 cards is equal to
52 · 51 · 50 · 49 · 48
52 52!
= = = 2, 598, 960.
5 5!47! 5·4·3·2·1
40 Chapter 3. Counting
Second Example: Let n and k be integers with 0 ≤ k ≤ n. How many
bitstrings of length n have exactly k many 1s? We can answer this question
using the Product Rule:
• The procedure is “write a bitstring of length n having exactly k many
1s”.
• Task 1: Consider the set {1, 2, . . . , n} of positions for the bits of the
string. Choose a k-element subset of this set.
• Task 2: Write a 1 in each of the k positions of the chosen subset.
• Task 3: Write a 0 in each of the n − k remaining positions.
There are nk ways to do the first task, there is one way to do the second task,
and there is one way to do the third task. Thus, by the Product Rule, the
number of ways to do the procedure and, therefore, the number of bitstrings
of length n having exactly k many 1s, is equal to
n n
·1·1= .
k k
We can also use the Bijection Rule, by observing, in the same way as we did
in Section 3.2, that there is a bijection between
• the set of all bitstrings of length n having exactly k many 1s, and
• the set of all k-element subsets of an n-element set.
Since the latter set has size nk , the former set has size nk as well.
Theorem 3.6.4 Let n and k be integers with 0 ≤ k ≤ n. The number of
bitstrings of length n having exactly k many 1s is equal to nk .
3.6.2 Newton’s Binomial Theorem
You have learned in high school that
(x + y)2 = x2 + 2xy + y 2 .
You have probably also seen that
(x + y)3 = x3 + 3x2 y + 3xy 2 + y 3 .
3.6. Permutations and Binomial Coefficients 41
What is the expansion of (x + y)5 ? Observe that
(x + y)5 = (x + y)(x + y)(x + y)(x + y)(x + y).
If we expand the expression on the right-hand side, we get terms
x5 , x4 y, x3 y 2 , x2 y 3 , xy 4 , y 5 ,
each with some coefficient. What is the coefficient of x2 y 3 ? We obtain a term
x2 y 3 , by
• choosing 3 of the 5 terms x + y,
• taking y in each of the 3 chosen terms x + y, and
• taking x in each of the other 2 terms x + y.
Since there are 53 ways to do this, the coefficient of x2 y 3 is equal to 5
3
= 10.
Theorem 3.6.5 (Newton’s Binomial Theorem) For any integer n ≥ 0,
we have n
n
X n n−k k
(x + y) = x y .
k=0
k
Proof. The expression (x+y)n is the product of n terms x+y. By expanding
this product, we get a term xn−k y k for each k = 0, 1, . . . , n, each with some
coefficient. We get a term xn−k y k by
• choosing k of the n terms x + y,
• taking y in each of the k chosen terms x + y, and
• taking x in each of the other n − k terms x + y.
Since there are nk ways to do this, the coefficient of xn−k y k is equal to n
k
.
For example, we have
3 3 3 3 2 3 2 3 3
(x + y) = x + x y+ xy + y
0 1 2 3
= x3 + 3x2 y + 3xy 2 + y 3 .
42 Chapter 3. Counting
To determine the coefficient of x12 y 13 in (x + y) 25
, we take n = 25 and
25
k = 13 in Newton’s Binomial Theorem, and get 13 .
What is the coefficient of x12 y 13 in (2x − 5y)25 ? Observe that
(2x − 5y)25 = ((2x) + (−5y))25 .
By replacing x by 2x, and y by −5y in Newton’s Binomial Theorem, we get
25
25
X 25
(2x − 5y) = (2x)25−k (−5y)k .
k=0
k
By taking k = 13, we obtain the coefficient of x12 y 13 :
25 25−13 13 25
·2 · (−5) = − · 212 · 513 .
13 13
Newton’s Binomial Theorem leads to identities for summations involving
binomial coefficients:
Theorem 3.6.6 For any integer n ≥ 0, we have
n
X n
= 2n .
k=0
k
Proof. Take x = y = 1 in Newton’s Binomial Theorem.
In Section 3.7, we will see a proof of Theorem 3.6.6 that does not use
Newton’s Binomial Theorem.
Theorem 3.6.7 For any integer n ≥ 1, we have
n
X n k
(−1) = 0.
k=0
k
Proof. Take x = 1 and y = −1 in Newton’s Binomial Theorem.
3.7. Combinatorial Proofs 43
3.7 Combinatorial Proofs
In a combinatorial proof, we show the validity of an identity, such as the one
in Theorem 3.6.6, by interpreting it as the answer to a counting problem.
The identity is proved by solving this counting problem in two different ways.
This gives two answers to the same counting problem. Obviously, these two
answers must be equal. Observe that we have already used this approach
n
in Section 3.6: When we determined the formula for k , we counted, in
two different ways, the number of ordered sequences of k pairwise distinct
elements from an n-element set. In this section, we will give several other
examples of combinatorial proofs.
Theorem 3.7.1 For any integers n and k with 0 ≤ k ≤ n, we have
n n
= .
k n−k
Proof. The claim can be proved using Theorem 3.6.3. To obtain a combi-
natorial proof, let S be a set with n elements. Recall that
• nk is the number of ways to choose k elements from the set S,
which is the same as
• the number of ways to not choose n − k elements from the set S.
n
The latter number is equal to n−k .
We can also prove the claim using Theorem 3.6.4:
• The number of bitstrings of length n with exactly k many 1s is equal
to nk .
• The number of bitstrings of length n with exactly n − k many 0s is
n
equal to n−k .
Since these two quantities are equal, the theorem follows.
Theorem 3.7.2 (Pascal’s Identity) For any integers n and k with 1 ≤
k ≤ n, we have
n+1 n n
= + .
k k k−1
44 Chapter 3. Counting
Proof. As in the previous theorem, the claim can be proved using Theo-
rem 3.6.3. To obtain a combinatorial proof, let S be a set with n+1 elements.
We are going to count the k-element subsets of S in two different ways.
First, by definition, the number of k-element subsets of S is equal to
n+1
. (3.3)
k
For the second way, we choose an element x in S and consider the set
T = S \ {x}, i.e., the set obtained by removing x from S. Any k-element
subset of S is of exactly one of the following two types:
• The k-element subset of S does not contain x.
– Any such subset is a k-element subset of T . Since T has size n,
there are nk many k-element subsets of S that do not contain x.
• The k-element subset of S contains x.
– If A is any such subset, then B = A \ {x} is a (k − 1)-element
subset of T .
– Conversely, for any (k − 1)-element subset B of T , the set A =
B ∪ {x} is a k-element subset of S that contains x.
– It follows that the number of k-element subsets of S containing x
(k − 1)-element subsets of T . The latter
is equal to the number of
n
number is equal to k−1 .
Thus, the second way of counting shows that the number of k-element subsets
of S is equal to
n n
+ . (3.4)
k k−1
Since the expressions in (3.3) and (3.4) count the same objects, they must
be equal. Therefore, the proof is complete.
Theorem 3.7.3 For any integer n ≥ 0, we have
n
X n
= 2n .
k=0
k
3.7. Combinatorial Proofs 45
Proof. We have seen in Theorem 3.6.6 that this identity follows from New-
ton’s Binomial Theorem. Below, we give a combinatorial proof.
Consider a set S with n elements. According to Theorem 3.2.1, this set
has 2n many subsets. A different way to count the subsets of S is by dividing
them into (pairwise disjoint) groups according to their sizes. For each k with
0 ≤ k ≤ n, consider all k-element subsets of S. The number of such subsets
n
is equal to k . If we take the sum of all these binomial coefficients, then we
have counted each subset of S exactly once. Thus,
n
X n
k=0
k
is equal to the total number of subsets of S.
Theorem 3.7.4 (Vandermonde’s Identity) For any integers m ≥ 0, n ≥
0, and r ≥ 0 with r ≤ m and r ≤ n, we have
r
X m n m+n
= .
k=0
k r − k r
Proof. Consider a set S with m + n elements. We are going to count the
r-element subsets of S in two different ways.
First, by using the definition of binomial coefficients, the number of r-
m+n
element subsets of S is equal to r .
For the second way, we partition the set S into two subsets A and B,
where A has size m and B has size n. Observe that any r-element subset of
S contains
• some elements of A, say k many, and
• r − k elements of B.
The value of k can be any integer in the set {0, 1, 2, . . . , r}.
Let k be any integer with 0 ≤ k ≤ r, and let Nk be the number of r-
element subsets of S that
Pcontain exactly k elements of A (and, thus, r − k
r
elements of B). Then, k=0 Nk is equal to the total number of r-element
subsets of S and, thus,
r
X m+n
Nk = .
k=0
r
46 Chapter 3. Counting
To determine Nk , we use the Product Rule: We obtain any subset that is
counted in Nk , by
• choosing k elements in A (there are m
k
ways to do this) and
n
• choosing r − k elements in B (there are r−k ways to do this).
It follows that
m n
Nk = .
k r−k
Corollary 3.7.5 For any integer n ≥ 0, we have
n 2
X n 2n
= .
k=0
k n
Proof. By taking m = n = r in Vandermonde’s Identity, we get
n
X n n 2n
= .
k=0
k n−k n
Using Theorem 3.7.1, we get
2
n n n n n
= = .
k n−k k k k
3.8 Pascal’s Triangle
The computational method at the heart of Pascal’s work was
actually discovered by a Chinese mathematician named Jia Xian
around 1050, published by another Chinese mathematician, Zhu
Shijie, in 1303, discussed in a work by Cardano in 1570, and
plugged into the greater whole of probability theory by Pascal,
who ended up getting most of the credit.
— Leonard Mlodinow, The Drunkard’s Walk, 2008
3.8. Pascal’s Triangle 47
We have seen that
n
• 0
= 1 for all integers n ≥ 0,
n
• n
= 1 for all integers n ≥ 0,
n
= n−1 n−1
• k k−1
+ k
for all integers n ≥ 2 and k with 1 ≤ k ≤ n − 1;
see Theorem 3.7.2.
These relations lead to an algorithm for generating binomial coefficients:
Algorithm GenerateBinomCoeff:
BCoeff (0, 0) = 1;
for n = 1, 2, 3, . . .
do BCoeff (n, 0) = 1;
for k = 1 to n − 1
do BCoeff (n, k) = BCoeff (n − 1, k − 1) + BCoeff (n − 1, k)
endfor;
BCoeff (n, n) = 1
endfor
The values BCoeff (n, k) that are computed by this (non-terminating)
algorithm satisfy
n
BCoeff (n, k) = for 0 ≤ k ≤ n.
k
The triangle obtained by arranging
n thesenbinomial coefficients, with the n-th
n
row containing all values 0 , 1 , . . . , n , is called Pascal’s Triangle. The
figure below shows rows 0, 1, . . . , 6:
48 Chapter 3. Counting
0
0
1 1
0 1
2 2 2
0 1 2
3 3 3 3
0 1 2 3
4 4 4 4 4
0 1 2 3 4
5 5 5 5 5 5
0 1 2 3 4 5
6 6 6 6 6 6 6
0 1 2 3 4 5 6
We obtain the values for the binomial coefficients by using the following
rules:
• Each value along the boundary is equal to 1.
• Each value in the interior is equal to the sum of the two values above
it.
In Figure 3.1, you see rows 0, 1, . . . , 12.
Below, we state some of our earlier results using Pascal’s Triangle.
• The values in the n-th row are equal to the coefficients in Newton’s
Binomial Theorem (i.e., Theorem 3.6.5). For example, the coefficients
in the expansion of (x + y)5 are given in the 5-th row:
(x + y)5 = x5 + 5x4 y + 10x3 y 2 + 10x2 y 3 + 5xy 4 + y 5 .
• Theorem 3.6.6 states that the sum of all values in the n-th row is equal
to 2n .
• Theorem 3.7.1 states that reading the n-th row from left to right gives
the same sequence as reading this row from right to left.
• Corollary 3.7.5 states that the sum of the squares of all values in the
n-th row is equal to the middle element in the 2n-th row.
3.9. More Counting Problems 49
1
1 1
1 2 1
1 3 3 1
1 4 6 4 1
1 5 10 10 5 1
1 6 15 20 15 6 1
1 7 21 35 35 21 7 1
1 8 28 56 70 56 28 8 1
1 9 36 84 126 126 84 36 9 1
1 10 45 120 210 252 210 120 45 10 1
1 11 55 165 330 462 462 330 165 55 11 1
1 12 66 220 495 792 924 792 495 220 66 12 1
Figure 3.1: Rows 0, 1, . . . , 12 of Pascal’s Triangle.
50 Chapter 3. Counting
3.9 More Counting Problems
3.9.1 Reordering the Letters of a Word
How many different strings can be made by reordering the letters of the
7-letter word
SUCCESS.
It should be clear that the answer is not 7!: If we swap, for example, the two
occurrences of C, then we obtain the same string.
The correct answer can be obtained by applying the Product Rule. We
start by counting the frequencies of each letter:
• The letter S occurs 3 times.
• The letter C occurs 2 times.
• The letter U occurs 1 time.
• The letter E occurs 1 time.
To apply the Product Rule, we have to specify the procedure and the tasks:
• The procedure is “write the letters occurring in the word SUCCESS”.
• The first task is “choose 3 positions out of 7, and write the letter S in
each chosen position”.
• The second task is “choose 2 positions out of the remaining 4, and write
the letter C in each chosen position”.
• The third task is “choose 1 position out of the remaining 2, and write
the letter U in the chosen position”.
• The fourth task is “choose 1 position out of the remaining 1, and write
the letter E in the chosen position”.
7 4
Since there are 3
ways to do the first task, 2
ways to do the second task,
2 1
1
ways to do the third task, and 1 way to do the fourth task, it follows
that the total number of different strings that can be made by reordering the
letters of the word SUCCESS is equal to
7 4 2 1
= 420.
3 2 1 1
3.9. More Counting Problems 51
In the four tasks above, we first chose the positions for the letter S, then
the positions for the letter C, then the position for the letter U, and finally
the position for the letter E. If we change the order, then we obtain the same
answer. For example, if we choose the positions for the letters in the order
C, E, U, S, then we obtain
7 5 4 3
,
2 1 1 3
which is indeed equal to 420.
3.9.2 Counting Solutions of Linear Equations
Consider the equation
x1 + x2 + x3 = 11.
We are interested in the number of solutions (x1 , x2 , x3 ), where x1 ≥ 0,
x2 ≥ 0, x3 ≥ 0 are integers. Examples of solutions are
(2, 3, 6), (3, 2, 6), (0, 11, 0), (2, 0, 9).
Observe that we consider (2, 3, 6) and (3, 2, 6) to be different solutions.
We are going to use the Bijection Rule to determine the number of solu-
tions. For this, we define A to be the set of all solutions, i.e.,
A = {(x1 , x2 , x3 ) : x1 ≥ 0, x2 ≥ 0, x3 ≥ 0 are integers, x1 + x2 + x3 = 11}.
To apply the Bijection Rule, we need a set B and a bijection f : A → B,
such that it is easy to determine the size of B. This set B should be chosen
such that its elements “encode” the elements of A in a unique way. Consider
the following set B:
• B is the set of all bitstrings of length 13 that contain exactly 2 many
1s (and, thus, exactly 11 many 0s).
The function f : A → B is defined as follows: If (x1 , x2 , x3 ) is an element of
the set A, then f (x1 , x2 , x3 ) is the bitstring
• that starts with x1 many 0s,
• is followed by one 1,
52 Chapter 3. Counting
• is followed by x2 many 0s,
• is followed by one 1,
• and ends with x3 many 0s.
For example, we have
f (2, 3, 6) = 0010001000000,
f (3, 2, 6) = 0001001000000,
f (0, 11, 0) = 1000000000001,
and
f (2, 0, 9) = 0011000000000.
To show that this function f maps elements of A to elements of B, we
have to verify that the string f (x1 , x2 , x3 ) belongs to the set B. This follows
from the following observations:
• The string f (x1 , x2 , x3 ) contains exactly 2 many 1s.
• The number of 0s in the string f (x1 , x2 , x3 ) is equal to x1 + x2 + x3 ,
which is equal to 11, because (x1 , x2 , x3 ) belongs to the set A.
• Thus, f (x1 , x2 , x3 ) is a bitstring of length 13 that contains exactly 2
many 1s.
It should be clear that this function f is one-to-one: If we take two
different elements (x1 , x2 , x3 ) in A, then f gives us two different bitstrings
f (x1 , x2 , x3 ).
To prove that f is onto, we have to show that for every bitstring b in the
set B, there is an element (x1 , x2 , x3 ) in A such that f (x1 , x2 , x3 ) = b. This
element of A is obtained by taking
• x1 to be the number of 0s to the left of the first 1,
• x2 to be the number of 0s between the two 1s, and
• x3 to be the number of 0s to the right of the second 1.
3.9. More Counting Problems 53
For example, if b = 0000110000000, then x1 = 4, x2 = 0, and x3 = 7.
Note that, since b has length 13 and contains exactly 2 many 1s, we have
x1 + x2 + x3 = 11 and, therefore, (x1 , x2 , x3 ) ∈ A.
Thus, we have shown that f : A → B is indeed a bijection. We know from
Theorem 3.6.4 that B has size 13 2
. Therefore, it follows from the Bijection
Rule that
13
|A| = |B| = = 78.
2
The following theorem states this result for general linear equations. You
are encouraged to come up with the proof.
Theorem 3.9.1 Let k ≥ 1 and n ≥ 0 be integers. The number of solutions
to the equation
x1 + x2 + · · · + xk = n,
where x1 ≥ 0, x2 ≥ 0, . . . , xk ≥ 0 are integers, is equal to
n+k−1
.
k−1
Let us now consider inequalities instead of equations. For example, con-
sider the inequality
x1 + x2 + x3 ≤ 11.
Again, we are interested in the number of solutions (x1 , x2 , x3 ), where x1 ≥ 0,
x2 ≥ 0, x3 ≥ 0 are integers. This inequality contains the same solutions as
before, but it has additional solutions such as
(2, 3, 5), (3, 2, 5), (0, 1, 0), (0, 0, 0).
As before, we are going to apply the Bijection Rule. We define
A = {(x1 , x2 , x3 ) : x1 ≥ 0, x2 ≥ 0, x3 ≥ 0 are integers, x1 + x2 + x3 ≤ 11}
and B to be the set of all bitstrings of length 14 that contain exactly 3 many
1s (and, thus, exactly 11 many 0s).
The function f : A → B is defined as follows: If (x1 , x2 , x3 ) is an element
of A, then f (x1 , x2 , x3 ) is the bitstring
• that starts with x1 many 0s,
54 Chapter 3. Counting
• is followed by one 1,
• is followed by x2 many 0s,
• is followed by one 1,
• is followed by x3 many 0s,
• is followed by one 1,
• and ends with 14 − (x1 + x2 + x3 + 3) many 0s.
For example, we have
f (2, 3, 6) = 00100010000001,
f (2, 3, 5) = 00100010000010,
f (0, 1, 0) = 10110000000000,
and
f (0, 0, 0) = 11100000000000.
As before, it can be verified that the string f (x1 , x2 , x3 ) belongs to the set
B and the function f is a bijection. It then follows from the Bijection Rule
that
14
|A| = |B| = = 364.
3
The next theorem gives the answer for the general case. As before, you are
encouraged to give a proof.
Theorem 3.9.2 Let k ≥ 1 and n ≥ 0 be integers. The number of solutions
to the inequality
x1 + x2 + · · · + xk ≤ n,
where x1 ≥ 0, x2 ≥ 0, . . . , xk ≥ 0 are integers, is equal to
n+k
.
k
3.10. The Pigeonhole Principle 55
3.10 The Pigeonhole Principle
In any group of 366 people, there must be two people having the same birth-
day: Since there are 365 days in a year (ignoring leap years), it is not possible
that the birthdays of 366 people are all distinct.
Pigeonhole Principle: Let k ≥ 1 be an integer. If k + 1 or more ob-
jects are placed into k boxes, then there is at least one box containing
two or more objects.
Equivalently, if A and B are two finite sets with |A| > |B|, then there
is no one-to-one function from A to B.
3.10.1 India Pale Ale
President of the
Favorite Drink
Carleton Computer Science Society
Simon Pratt (2013–2014) India Pale Ale
Lindsay Bangs (2014–2015) Wheat Beer
Connor Hillen (2015–2016) Black IPA
Elisa Kazan (2016–2019) Cider
William So (2019–2020) Amber Lager
Simon Pratt loves to drink India Pale Ale (IPA). During each day of the
month of April (which has 30 days), Simon drinks at least one bottle of IPA.
During this entire month, he drinks exactly 45 bottles of IPA. The claim is
that there must be a sequence of consecutive days in April, during which
Simon drinks exactly 14 bottles of IPA.
To prove this, let bi be the number of bottles that Simon drinks on April i,
for i = 1, 2, . . . , 30. We are given that each bi is a positive integer (i.e., bi ≥ 1)
and
b1 + b2 + · · · + b30 = 45.
Define, for i = 1, 2, . . . , 30,
ai = b 1 + b 2 + · · · + b i ,
i.e., ai is the total number of bottles of IPA that Simon drinks during the
first i days of April. Consider the sequence of 60 numbers
a1 , a2 , . . . , a30 , a1 + 14, a2 + 14, . . . , a30 + 14.
56 Chapter 3. Counting
Each number in this sequence is an integer that belongs to the set
{1, 2, . . . , 59}.
Therefore, by the Pigeonhole Principle, these 60 numbers cannot all be dis-
tinct. Observe that there are no duplicates in the sequence a1 , a2 , . . . , a30 ,
because all bi are at least one. Similarly, there are no duplicates in the se-
quence a1 + 14, a2 + 14, . . . , a30 + 14. It follows that there are two indices i
and j such that
ai = aj + 14.
Observe that j must be less than i and
14 = ai − aj = bj+1 + bj+2 + · · · + bi .
Thus, in the period from April j + 1 until April i, Simon drinks exactly 14
bottles of IPA.
3.10.2 Sequences Containing Divisible Numbers
Let A = {1, 2, . . . , 2n} and consider the sequence n + 1, n + 2, . . . , 2n of ele-
ments in A. This sequence has the property that none of its elements divides
any other element in the sequence. Note that the sequence has length n. The
following theorem states that such a sequence of length n + 1 does not exist.
Theorem 3.10.1 Let n ≥ 1 and consider a sequence a1 , a2 , . . . , an+1 of n+1
elements from the set {1, 2, . . . , 2n}. Then there are two distinct indices i and
j such that ai divides aj or aj divides ai .
Proof. For each i with 1 ≤ i ≤ n + 1, write
ai = 2ki · qi ,
where ki ≥ 0 is an integer and qi is an odd integer. For example,
• if ai = 48, then ki = 4 and qi = 3, because 48 = 24 · 3,
• if ai = 1, then ki = 0 and qi = 1, because 1 = 20 · 1,
• if ai = 7, then ki = 0 and qi = 7, because 7 = 20 · 7.
3.10. The Pigeonhole Principle 57
Consider the sequence q1 , q2 , . . . , qn+1 of n+1 integers. Each of these numbers
is an odd integer that belongs to the set
{1, 3, 5, . . . , 2n − 1}.
Since this set has size n, the Pigeonhole Principle implies that there must be
two numbers in the sequence q1 , q2 , . . . , qn+1 that are equal. In other words,
there are two distinct indices i and j such that qi = qj . It follows that
ai 2ki · qi
= kj = 2ki −kj .
aj 2 · qj
Thus, if ki ≥ kj , then aj divides ai . Otherwise, ki < kj , and ai divides aj .
3.10.3 Long Monotone Subsequences
Let n = 3, and consider the sequence 20, 10, 9, 7, 11, 2, 21, 1, 20, 31 of 10 =
n2 + 1 numbers. This sequence contains an increasing subsequence of length
4 = n + 1, namely 10, 11, 21, 31. The following theorem states this result for
arbitrary values of n.
Theorem 3.10.2 Let n ≥ 1 be an integer. Every sequence of n2 + 1 distinct
real numbers contains a subsequence of length n + 1 that is either increasing
or decreasing.
Proof. Let a1 , a2 , . . . , an2 +1 be an arbitrary sequence of n2 + 1 distinct real
numbers. For each i with 1 ≤ i ≤ n2 + 1, let inc i denote the length of
the longest increasing subsequence that starts at ai , and let dec i denote the
length of the longest decreasing subsequence that starts at ai .
Using this notation, the claim in the theorem can be formulated as follows:
There is an index i such that inc i ≥ n + 1 or dec i ≥ n + 1.
We will prove the claim by contradiction. Thus, we assume that inc i ≤ n
and dec i ≤ n for all i with 1 ≤ i ≤ n2 + 1.
Consider the set
B = {(b, c) : 1 ≤ b ≤ n, 1 ≤ c ≤ n},
and think of the elements of B as being boxes. For each i with 1 ≤ i ≤ n2 +1,
the pair (inc i , dec i ) is an element of B. Thus, we have n2 + 1 elements
58 Chapter 3. Counting
(inc i , dec i ), which are placed in the n2 boxes of B. By the Pigeonhole Prin-
ciple, there must be a box that contains two (or more) elements. In other
words, there exist two integers i and j such that i < j and
(inc i , dec i ) = (inc j , dec j ).
Recall that the elements in the sequence are distinct. Hence, ai 6= aj . We
consider two cases.
First assume that ai < aj . Then the length of the longest increasing
subsequence starting at ai must be at least 1+inc j , because we can append ai
to the longest increasing subsequence starting at aj . Therefore, inc i 6= inc j ,
which is a contradiction.
The second case is when ai > aj . Then the length of the longest decreasing
subsequence starting at ai must be at least 1+dec j , because we can append ai
to the longest decreasing subsequence starting at aj . Therefore, dec i 6= dec j ,
which is again a contradiction.
3.10.4 There are Infinitely Many Primes
As a final application of the Pigeonhole Principle, we prove the following
result:
Theorem 3.10.3 There are infinitely many prime numbers.
Proof. The proof is by contradiction. Thus, we assume that there are, say,
k prime numbers, and denote them by
2 = p1 < p2 < · · · < pk .
Note that k is a fixed integer. Since
2n
lim = ∞,
n→∞ (n + 1)k
we can choose an integer n such that
2n > (n + 1)k .
Define the function
f : {1, 2, . . . , 2n } → Nk
3.11. Exercises 59
as follows: For any integer x with 1 ≤ x ≤ 2n , consider its prime factorization
mk
x = pm m2
1 · p2 · · · pk .
1
We define
f (x) = (m1 , m2 , . . . , mk ).
Since
mi ≤ m1 + m2 + · · · + mk
≤ m1 log p1 + m2 log p2 + · · · + mk log pk
mk
= log (pm m2
1 · p2 · · · pk )
1
= log x
≤ n,
it follows that
f (x) ∈ {0, 1, 2, . . . , n}k .
Thus, f is a function
f : {1, 2, . . . , 2n } → {0, 1, 2, . . . , n}k .
It is easy to see that this function is one-to-one. The set on the left-hand
side has size 2n , whereas the set on the right-hand side has size (n + 1)k . It
then follows from the Pigeonhole Principle that
(n + 1)k ≥ 2n ,
which contradicts our choice for n.
3.11 Exercises
3.1 A licence plate number consists of a sequence of four uppercase letters
followed by three digits. How many licence plate numbers are there?
3.2 A multiple-choice exam consists of 100 questions. Each question has
four possible answers a, b, c, and d. How many ways are there to answer the
100 questions (assuming that each question is answered)?
60 Chapter 3. Counting
3.3 For each of the following seven cases, determine how many strings of
eight uppercase letters there are.
• Letters can be repeated.
• No letter can be repeated.
• The strings start with PQ (in this order) and letters can be repeated.
• The strings start with PQ (in this order) and no letter can be repeated.
• The strings start and end with PQ (in this order) and letters can be
repeated.
• The strings start with XYZ (in this order), end with QP (in this order),
and letters can be repeated.
• The strings start with XYZ (in this order) or end with QP (in this
order), and letters can be repeated.
3.4 If n and d are positive integers, then d is a divisor of n, if n/d is an
integer.
Determine the number of divisors of the integer
1, 170, 725, 783, 076, 864 = 217 · 312 · 75 .
3.5 Let k ≥ 1 and n ≥ 1 be integers. Consider k distinct beer bottles and
n distinct students. How many ways are there to hand out the beer bottles
to the students, if there is no restriction on how many bottles a student may
get?
3.6 The Carleton Computer Science Society has a Board of Directors con-
sisting of one president, one vice-president, one secretary, one treasurer, and
a three-person party committee (whose main responsibility is to buy beer
for the other four board members). The entire board consists of seven dis-
tinct students. If there are n ≥ 7 students in Carleton’s Computer Science
program, how many ways are there to choose a Board of Directors?
3.7 The Carleton Computer Science Society has an Academic Events Com-
mittee (AEC) consisting of five students and a Beer Committee (BC) con-
sisting of six students (whose responsibility is to buy beer for the AEC).
3.11. Exercises 61
• Assume there are n ≥ 6 students in Carleton’s Computer Science pro-
gram. Also, assume that a student can be both on the AEC and on the
BC. What is the total number of ways in which these two committees
can be chosen?
• Assume there are n ≥ 11 students in Carleton’s Computer Science
program. Also, assume that a student cannot be both on the AEC
and on the BC. What is the total number of ways in which these two
committees can be chosen?
3.8 Let f ≥ 2, m ≥ 2, and k ≥ 2 be integers such that k ≤ f and k ≤ m.
The Carleton Computer Science program has f female students and m male
students. The Carleton Computer Science Society has a Board of Directors
consisting of k students. At least one of the board members is female and at
least one of the board members is male. Determine the number of ways in
which a Board of Directors can be chosen.
3.9 Let f ≥ 4 and m ≥ 4 be integers. The Carleton Computer Science
program has f female students and m male students that are eligible to be
a TA for COMP 2804. Determine the number of ways to choose eight TAs
out of these f + m students, such that the number of female TAs is equal to
the number of male TAs
3.10 Let m and n be integers with 0 ≤ m ≤ n. There are n + 1 students
in Carleton’s Computer Science program. The Carleton Computer Science
Society has a Board of Directors, consisting of one president and m vice-
presidents. The president cannot be vice-president. Prove that
n n+1
(n + 1) = (n + 1 − m) ,
m m
by determining, in two different ways, the number of ways to choose a Board
of Directors.
3.11 In Tic-Tac-Toe, we are given a 3 × 3 grid, consisting of unmarked cells.
Two players, Xavier and Olivia, take turns marking the cells of this grid.
When it is Xavier’s turn, he chooses an unmarked cell and marks it with the
letter X. Similarly, when it is Olivia’s turn, she chooses an unmarked cell
and marks it with the letter O. The first turn is by Xavier. The players
continue making turns until all cells have been marked. Below, you see an
example of a completely marked grid.
62 Chapter 3. Counting
O O X
X X O
X X O
• What is the number of completely marked grids?
• What is the number of different ways (i.e., ordered sequences) in which
the grid can be completely marked, when following the rules given
above?
3.12 In how many ways can you paint 200 chairs, if 33 of them must be
painted red, 66 of them must be painted blue, and 101 of them must be
painted green?
3.13 Let A be the set of all integers x > 6543 such that the decimal repre-
sentation of x has distinct digits, none of which is equal to 7, 8, or 9. (The
decimal representation does not have leading zeros.) Determine the size of
the set A.
3.14 Let A be the set of all integers x ∈ {1, 2, . . . , 100} such that the decimal
representation of x does not contain the digit 4. (The decimal representation
does not have leading zeros.)
• Determine the size of the set A without using the Complement Rule.
• Use the Complement Rule to determine the size of the set A.
3.15 Let A be a set of size m, let B be a set of size n, and assume that
n ≥ m ≥ 1. How many functions f : A → B are there that are not one-to-
one?
3.16 Consider permutations of the set {a, b, c, d, e, f, g} that do not contain
bge (in this order) and do not contain eaf (in this order). Prove that the
number of such permutations is equal to 4806.
3.17 How many bitstrings of length 8 are there that contain at least 4 con-
secutive 0s or at least 4 consecutive 1s?
3.18 How many bitstrings of length 77 are there that start with 010 (i.e.,
have 010 at positions 1, 2, and 3), or have 101 at positions 2, 3, and 4, or
have 010 at positions 3, 4, and 5?
3.11. Exercises 63
3.19 Let n ≥ 12 be an integer and let {B1 , B2 , . . . , Bn } be a set of n beer
bottles. Consider permutations of these bottles such that there are exactly
10 bottles between B1 and Bn . (B1 can be to the left or right of Bn .) Prove
that the number of such permutations is equal to
n−2
· 10! · 2 · (n − 11)!.
10
3.20 Let n ≥ 3 be an integer. The Gn (or Group of n) is an international
forum where the n leaders of the world meet to drink beer together. Two
of these leaders are Donald Trump and Justin Trudeau. At the end of their
meeting, the n leaders stand on a line and a group photo is taken.
• Determine the number of ways in which the n leaders can be arranged
on a line, if Donald Trump and Justin Trudeau are standing next to
each other.
• Determine the number of ways in which the n leaders can be arranged
on a line, if Donald Trump and Justin Trudeau are not standing next
to each other.
• Determine the number of ways in which the n leaders can be arranged
on a line, if Donald Trump is to the left of Justin Trudeau. (Donald
does not necessarily stand immediately to the left of Justin.)
3.21 A string of letters is called a palindrome, if reading the string from left
to right gives the same result as reading the string from right to left. For
example, madam and racecar are palindromes. Recall that there are five
vowels in the English alphabet: a, e, i, o, and u.
In this exercise, we consider strings consisting of 28 characters, with each
character being a lowercase letter. Determine the number of such strings that
start and end with the same letter, or are palindromes, or contain vowels only.
3.22 A flip in a bitstring is a pair of adjacent bits that are not equal. For
example, the bitstring 010011 has three flips: The first two bits form a flip,
the second and third bits form a flip, and the fourth and fifth bits form a
flip.
• Determine the number of bitstrings of length 7 that have exactly 3 flips
at the following positions: The second and third bits form a flip, the
third and fourth bits form a flip, and the fifth and sixth bits form a
flip.
64 Chapter 3. Counting
• Let n ≥ 2 and k be integers with 0 ≤ k ≤ n − 1. Determine the number
of bitstrings of length n that have exactly k flips.
3.23 Let m and n be integers with m ≥ n ≥ 1. How many ways are there
to place m books on n shelves, if there must be at least one book on each
shelf? As in Section 3.1.3, the order on each shelf matters.
3.24 You are given m distinct books B1 , B2 , . . . , Bm and n identical blocks
of wood. How many ways are there to arrange these books and blocks in a
straight line?
For example, if m = 5 and n = 3, then three possible arrangements are
(W stands for a block of wood)
W B3 B1 W B5 B4 W B2 ,
W B1 B3 W B5 B4 W B2 ,
and
B5 W B3 B1 W W B2 B4 .
3.25 Let n ≥ 1 be an integer and consider n boys and n girls. For each
of the following three cases, determine how many ways there are to arrange
these 2n people on a straight line (the order on the line matters):
• All boys stand next to each other and all girls stand next to each other.
• All girls stand next to each other.
• Boys and girls alternate.
3.26 Elisa Kazan has a set {C1 , C2 , . . . , C50 } consisting of 50 cider bottles.
She divides these bottles among 5 friends, so that each friend receives a
subset consisting of 10 bottles. Determine the number of ways in which Elisa
can divide the bottles.
3.27 Let n ≥ 1 be an integer. Consider a tennis tournament with 2n par-
ticipants. In the first round of this tournament, n games will be played and,
thus, the 2n people have to be divided into n pairs. What is the number of
ways in which this can be done?
3.11. Exercises 65
3.28 The Ottawa Senators and the Toronto Maple Leafs play a best-of-7
series: These two hockey teams play games against each other, and the first
team to win 4 games wins the series. Each game has a winner (thus, no game
ends in a tie).
A sequence of games can be described by a string consisting of the char-
acters S (indicating that the Senators win the game) and L (indicating that
the Leafs win the game). Two possible ways for the Senators to win the
series are (L, S, S, S, S) and (S, L, S, L, S, S).
Determine the number of ways in which the Senators can win the series.
3.29 The Beer Committee of the Carleton Computer Science Society has
bought large quantities of 10 different types of beer. In order to test which
beer students prefer, the committee does the following experiment:
• Out of the n ≥ 10 students in Carleton’s Computer Science program,
10 students are chosen.
• Each of the 10 students chosen drinks one of the 10 beers; no two
students drink the same beer.
What is the number of ways in which this experiment can be done?
3.30 Let m, n, k, and ` be integers such that m ≥ 1, n ≥ 1, 1 ≤ ` ≤ k ≤ `+m
and ` ≤ n.
After a week of hard work, Elisa Kazan goes to her neighborhood pub.
This pub has m different types of beer and n different types of cider on tap.
Elisa decides to order k pints: At most one pint of each type, and exactly `
pints of cider. Determine the number of ways in which Elisa can order these
k pints. The order in which Elisa orders matters.
3.31 Let m ≥ 2 and n ≥ 2 be even integers. You are given m beer bottles
B1 , B2 , . . . , Bm and n cider bottles C1 , C2 , . . . , Cn . Assume you arrange these
m + n bottles on a horizontal line such that
• the leftmost m/2 bottles are all beer bottles, and
• the rightmost n/2 bottles are all cider bottles.
How many such arrangements are there?
66 Chapter 3. Counting
3.32 Consider 10 male students M1 , M2 , . . . , M10 and 7 female students F1 ,
F2 , . . . , F7 . Assume these 17 students are arranged on a horizontal line such
that no two female students are standing next to each other. We are inter-
ested in the number of such arrangements, where the order of the students
matters.
• Explain what is wrong with the following argument:
We are going to use the Product Rule:
– Task 1: Arrange the 7 females on a line. There are 7! ways
to do this.
– Task 2: Choose 6 males. There are 10
6
ways to do this.
– Task 3: Place the 6 males chosen in Task 2 in the 6 “gaps”
between the females. There are 6! ways to do this.
– Task 4: At this moment, we have arranged 13 students on
a line. We are left with 4 males that have to be placed.
∗ Task 4.1: Place one male. There are 14 ways to do
this.
∗ Task 4.2: Place one male. There are 15 ways to do
this.
∗ Task 4.3: Place one male. There are 16 ways to do
this.
∗ Task 4.4: Place one male. There are 17 ways to do
this.
By the Product Rule, the total number of ways to arrange the
17 students is equal to
10
7! · · 6! · 14 · 15 · 16 · 17 = 43, 528, 181, 760, 000.
6
• Determine the number of ways to arrange the 17 students.
Hint: Use the Product Rule. What is easier to count: Placing the
3.11. Exercises 67
female students first and then the male students, or placing the male
students first and then the female students?
3.33 Let n ≥ 1 be an integer. A function f : {1, 2, . . . , n} → {1, 2, . . . , n}
is called awesome, if there is at least one integer i in {1, 2, . . . , n} for which
f (i) = i. Determine the number of awesome functions.
3.34 Let n ≥ 2 be an integer. Consider strings consisting of n digits.
• Determine the number of such strings, in which no two consecutive
digits are equal.
• Determine the number of such strings, in which there is at least one
pair of consecutive digits that are equal.
3.35 Consider strings consisting of 12 characters, each character being a, b,
or c. Such a string is called valid, if at least one of the characters is missing.
For example, abababababab is a valid string, whereas abababacabab is not a
valid string. How many valid strings are there?
3.36 Consider strings consisting of 40 characters, where each character is an
element of {a, b, c}. Such a string is called cool, if it contains exactly 8 many
a’s or exactly 7 many b’s. Determine the number of cool strings.
3.37 A password consists of 100 characters, each character being a digit or
a lowercase letter. A password must contain at least two digits. How many
passwords are there?
3.38 A password is a string of ten characters, where each character is a
lowercase letter, a digit, or one of the eight special characters !, @, #, $, %,
&, (, and ).
A password is called awesome, if it contains at least one digit or at least
one special character. Determine the number of awesome passwords.
3.39 A password consists of 100 characters, each character being a digit, a
lowercase letter, or an uppercase letter. A password must contain at least
one digit, at least one lowercase letter, and at least one uppercase letter.
How many passwords are there?
Hint: Recall De Morgan’s Law
A ∩ B ∩ C = A ∪ B ∪ C.
68 Chapter 3. Counting
3.40 A password is a string of 100 characters, where each character is a digit
or a lowercase letter. A password is called valid, if
• it does not start with abc, and
• it does not end with xyz, and
• it does not start with 3456.
Determine the number of valid passwords.
3.41 A password is a string of 8 characters, where each character is a low-
ercase letter or a digit. A password is called valid, if it contains at least one
digit. In Section 3.3, we have seen that the number of valid passwords is
equal to
368 − 268 = 2, 612, 282, 842, 880.
Explain what is wrong with the following method to count the valid pass-
words.
We are going to use the Product Rule.
• The procedure is “write a valid password”.
• Since a valid password contains at least one digit, we choose,
in the first task, a position for the digit.
• The second task is to write a digit at the chosen position.
• The third task is to write a character (lowercase letter or digit)
at each of the remaining 7 positions.
There are 8 ways to do the first task, 10 ways to do the second task,
and 367 ways to do the third task. Therefore, by the Product Rule,
the number of valid passwords is equal to
8 · 10 · 367 = 6, 269, 133, 127, 680.
3.42 Consider permutations of the 26 lowercase letters a, b, c, . . . , z.
• How many such permutations contain the string wine?
3.11. Exercises 69
• How many such permutations do not contain any of the strings wine,
vodka, or coke?
3.43 Determine the number of integers in the set {1, 2, . . . , 1000} that are
not divisible by any of 5, 7, and 11.
3.44 Let n ≥ 4 be an integer. Determine the number of permutations of
{1, 2, . . . , n}, in which
• 1 and 2 are next to each other, with 1 to the left of 2, or
• 4 and 3 are next to each other, with 4 to the left of 3.
3.45 Determine the number of functions
f : {1, 2, 3, 4} → {a, b, c, . . . , z},
such that f (1) = f (2), or f (3) = f (4), or f (1) 6= f (3).
3.46 Let n ≥ 3 be an integer. Determine the number of permutations of
{1, 2, . . . , n}, in which
• 1 and 2 are next to each other, with 1 to the left of 2, or
• 2 and 3 are next to each other, with 2 to the left of 3.
Compare your answer with the answer to Exercise 3.44.
3.47 Let n and k be integers with 2 ≤ k ≤ n, and consider the set S =
{1, 2, 3, . . . , 2n}. An ordered sequence of k elements of S is called valid, if
• this sequence is strictly increasing, or
• this sequence is strictly decreasing, or
• this sequence contains only even numbers (and duplicate elements are
allowed).
Determine the number of valid sequences.
3.48 Let n ≥ 2 be an integer.
70 Chapter 3. Counting
• Determine the number of strings consisting of n characters, where each
character is an element of the set {a, b, 0}.
• Let S be a set consisting of n elements. Determine the number of
ordered pairs (A, B), where A ⊆ S, B ⊆ S, and A ∩ B = ∅.
• Let S be a set consisting of n elements. Consider ordered pairs (A, B),
where A ⊆ S, B ⊆ S, and |A ∩ B| = 1. Prove that the number of such
pairs is equal to n · 3n−1 .
3.49 In a group of 20 people,
• 6 are blond,
• 7 have green eyes,
• 11 are not blond and do not have green eyes.
How many people in this group are blond and have green eyes?
3.50 Let n ≥ 1 be an integer.
• Assume that n is odd. Determine the number of bitstrings of length n
that contain more 0’s than 1’s. Justify your answer in plain English.
• Assume that n is even.
– Determine the number of bitstrings of length n in which the num-
ber of 0’s is equal to the number of 1’s.
– Determine the number of bitstrings of length n that contain strictly
more 0’s than 1’s.
– Argue that the binomial coefficient
n
n/2
is an even integer.
3.51 Use Pascal’s Identity (Theorem 3.7.2) to prove Newton’s Binomial The-
orem (i.e., Theorem 3.6.5) by induction.
3.11. Exercises 71
3.52 Determine the coefficient of x111 y 444 in the expansion of
(−17x + 71y)555 .
3.53 Nick is not only your friendly TA2 , he also has a part-time job in a
grocery store. This store sells n different types of India Pale Ale (IPA) and
n different types of wheat beer, where n ≥ 2 is an integer. Prove that
2n n
=2 + n2 ,
2 2
by counting, in two different ways, the number of ways to choose two different
types of beer.
3.54 You have won the first prize in the Louis van Gaal Impersonation Con-
test 3 . When you arrive at Louis’ home to collect your prize, you see n beer
bottles B1 , B2 , . . . , Bn , n cider bottles C1 , C2 , . . . , Cn , and n wine bottles
W1 , W2 , . . . , Wn . Here, n is an integer with n ≥ 2. Louis tells you that
your prize consists of one beer bottle of your choice, one cider bottle of your
choice, and one wine bottle of your choice.
Prove that
n3 = (n − 1)3 + 3(n − 1)2 + 3(n − 1) + 1,
by counting, in two different ways, the number of ways in which you can
choose your prize.
3.55 Let n ≥ 4 be an integer and consider the set S = {1, 2, . . . , n}. Let k
be an integer with 2 ≤ k ≤ n − 2. In this exercise, we consider subsets A
of S for which |A| = k and {1, 2} 6⊆ A. Let N denote the number of such
subsets.
• Use the Sum Rule to determine N .
• Use the Complement Rule to determine N .
• Use the above two results to prove that
n−2 n−2 n−2
n
= +2 + .
k k k−1 k−2
2
Winter term 2017
3
Louis van Gaal has been coach of AZ, Ajax, Barcelona, Bayern München, Manchester
United, and the Netherlands.
72 Chapter 3. Counting
3.56 Let k ≥ 1 be an integer and consider a sequence n1 , n2 , . . . , nk of posi-
tive integers. Use a combinatorial proof to show that
n1 + n2 + · · · + nk
n1 n2 nk
+ + ··· + ≤ .
2 2 2 2
Hint: For each i with 1 ≤ i ≤ k, consider the complete graph on ni vertices.
How many edges does this graph have?
3.57 Let n ≥ 1 be an integer. Prove that
n
X n n 2n
= ,
k=1
k k − 1 n + 1
by determining, in two different ways, the number of ways to choose n + 1
people from a group consisting of n men and n women.
3.58 Let n ≥ 1 be an integer. Use Newton’s Binomial Theorem (i.e., Theo-
rem 3.6.5) to prove that
n
X n
10k · 26n−k = 36n − 26n . (3.5)
k=1
k
In the rest of this exercise, you will give a combinatorial proof of this identity.
Consider passwords consisting of n characters, each character being a
digit or a lowercase letter. A password must contain at least one digit.
• Use the Complement Rule of Section 3.3 to show that the number of
passwords is equal to 36n − 26n .
• Let k be an integer with 1 ≤ k ≤ n. Prove that the number of pass-
words with exactly k digits is equal to nk 10k · 26n−k .
• Explain why the above two parts imply the identity in (3.5).
3.59 Use Newton’s Binomial Theorem (i.e., Theorem 3.6.5) to prove that
for every integer n ≥ 1,
n
X n k
2 = 3n . (3.6)
k=0
k
In the rest of this exercise, you will give a combinatorial proof of this identity.
Let A = {1, 2, 3, . . . , n} and B = {a, b, c}. According to Theorem 3.1.2,
the number of functions f : A → B is equal to 3n .
3.11. Exercises 73
• Consider a fixed integer k with 0 ≤ k ≤ n and a fixed subset S of A
having size k. Determine the number of functions f : A → B having
the property that
– for all x ∈ S, f (x) ∈ {a, b}, and
– for all x ∈ A \ S, f (x) = c.
• Explain why this implies the identity in (3.6).
3.60 Use Newton’s Binomial Theorem (i.e., Theorem 3.6.5) to prove that
for every integer n ≥ 2,
n
X n
(n − 1)n−k = nn . (3.7)
k=0
k
In the rest of this exercise, you will give a combinatorial proof of this identity.
Consider the set A = {1, 2, . . . , n}. According to Theorem 3.1.2, the
number of functions f : A → A is equal to nn .
• Consider a fixed integer k with 0 ≤ k ≤ n and a fixed subset S of A
having size k. Determine the number of functions f : A → A having
the property that
– for all x ∈ S, f (x) = x, and
– for all x ∈ A \ S, f (x) 6= x.
• Explain why this implies the identity in (3.7).
3.61 Let n ≥ 66 be an integer and consider the set S = {1, 2, . . . , n}.
• Let k be an integer with 66 ≤ k ≤ n. How many 66-element subsets of
S are there whose largest element is equal to k?
• Use the result in the first part to prove that
n
k−1
X n
= .
k=66
65 66
3.62 Let a ≥ 0, b ≥ 0, and n ≥ 0 be integers, and consider the set S =
{1, 2, 3, . . . , a + b + n + 1}.
74 Chapter 3. Counting
• How many subsets of size a + b + 1 does S have?
• Let k be an integer with 0 ≤ k ≤ n. Consider subsets T of S such that
|T | = a + b + 1 and the (a + 1)-st smallest element in T is equal to
a + k + 1. How many such subsets T are there?
• Use the above results to prove that
n
b+n−k
X a+k a+b+n+1
= .
k=0
k n − k n
3.63 Let n ≥ 0 and k ≥ 0 be integers.
• How many bitstrings of length n + 1 have exactly k + 1 many 1s?
• Let i be an integer with k ≤ i ≤ n. What is the number of bitstrings of
length n+1 that have exactly k +1 many 1s and in which the rightmost
1 is at position i + 1?
• Use the above two results to prove that
n
X i n+1
= .
i=k
k k+1
3.64 Let k, m, and n be integers with 0 ≤ k ≤ m ≤ n, and let S be a set of
size n. Prove that
n−k
n n m
= ,
k m−k m k
by determining, in two different ways, the number of ordered pairs (A, B)
with A ⊆ S, B ⊆ S, A ⊆ B, |A| = k, and |B| = m.
3.65 Let m and n be integers with 0 ≤ m ≤ n, and let S be a set of size n.
Prove that n
X n k n−m n
=2 ,
k=m
k m m
by determining, in two different ways, the number of ordered pairs (A, B)
with A ⊆ S, |A| = m, B ⊆ S, and A ∩ B = ∅.
Hint: The size of B can be any of the values n − m, n − (m + 1), n − (m +
2), . . . , n−n. What is the number of pairs (A, B) having the properties above
and for which |B| = n − k?
3.11. Exercises 75
3.66 Let m and n be integers with 0 ≤ m ≤ n.
• How many bitstrings of length n + 1 have exactly m many 1s?
• Let k be an integer with 0 ≤ k ≤ m. What is the number of bitstrings
of length n+1 that have exactly m many 1s and that start with 1| ·{z
· · 1} 0?
k
• Use the above two results to prove that
m
n−k
X n+1
= .
k=0
m − k m
3.67 Let m and n be integers with 0 ≤ m ≤ n. Use Exercises 3.10, 3.64,
and 3.66 to prove that
m m
X
k n+1
n = .
k=0 k
n+1−m
3.68 Let n ≥ 1 be an integer. Prove that
n 2
2n − 1
X n
k =n ,
k=1
k n − 1
by determining, in two different ways, the number of ways a committee can
be chosen from a group of n men and n women. Such a committee has a
woman as the chair and n − 1 other members.
3.69 Let n ≥ 2 be an integer and consider the set S = {1, 2, . . . , n}. An
ordered triple (A, x, y) is called awesome, if (i) A ⊆ S, (ii) x ∈ A, and (iii)
y ∈ A.
• Let k be an integer with 1 ≤ k ≤ n. Determine the number of awesome
triples (A, x, y) with |A| = k.
• Prove that the number of awesome triples (A, x, y) with x = y is equal
to
n · 2n−1 .
• Determine the number of awesome triples (A, x, y) with x 6= y.
76 Chapter 3. Counting
• Use the above results to prove that
n
X n
2
k = n(n − 1) · 2n−2 + n · 2n−1 .
k=1
k
3.70 Let n ≥ 1 be an integer, and let X and Y be two disjoint sets, each
consisting of n elements. An ordered triple (A, B, C) of sets is called cool, if
A ⊆ X, B ⊆ Y, C ⊆ B, and |A| + |B| = n.
• Let k be an integer with 0 ≤ k ≤ n. Determine the number of cool
triples (A, B, C) for which |A| = k.
• Let k be an integer with 0 ≤ k ≤ n. Determine the number of cool
triples (A, B, C) for which |C| = k.
• Use the above two results to prove that
n 2 n
n 2n − k
X n n−k
X
·2 = .
k=0
k k=0
k n
3.71 Let m ≥ 1 and n ≥ 1 be integers. Consider a rectangle whose horizontal
side has length m and whose vertical side has length n. A path from the
bottom-left corner to the top-right corner is called valid, if in each step, it
either goes one unit to the right or one unit upwards. In the example below,
you see a valid path for the case when m = 5 and n = 3.
3
2
1
0 1 2 3 4 5
How many valid paths are there?
3.11. Exercises 77
3.72 Let n ≥ 1 be an integer. Prove that
n
X n
k = n · 2n−1 .
k=1
k
Hint: Take the derivative of (1 + x)n .
3.73 A string consisting of characters is called cool, if exactly one character
in the string is equal to the letter x and each other character is a digit. Let
n ≥ 1 be an integer.
• Determine the number of cool strings of length n.
• Let k be an integer with 1 ≤ k ≤ n. Determine the number of cool
strings of length n that contain exactly n − k many 0’s.
• Use the above two results to prove that
n
X n k−1
k 9 = n · 10n−1 .
k=1
k
3.74 Let n ≥ 1 be an integer. We consider binary 2 × n matrices, i.e.,
matrices with 2 rows and n columns, in which each entry is 0 or 1. Any
column in such a matrix is of one of four types, based on the bits that occur in
this column. We will refer to these types as 00-columns, 01-columns, 10-columns,
and 11-columns. For example, in the 2 × 7 matrix below, the first, second, and
fifth columns are 01-columns, the third and seventh columns are 11-columns,
the fourth column is a 00-column, and the sixth column is a 10-column.
0 0 1 0 0 1 1
1 1 1 0 1 0 1
For the rest of this exercise, let k be an integer with 0 ≤ k ≤ 2n. A
binary 2 × n matrix is called awesome, if it contains exactly k many 0’s.
• How many 1’s are there in an awesome 2 × n matrix?
• How many awesome 2 × n matrices are there?
• Let i be an integer and consider an arbitrary awesome 2 × n matrix M
with exactly n − i many 11-columns.
78 Chapter 3. Counting
– Prove that dk/2e ≤ i ≤ k.
– Determine the number of 01-columns plus the number of 10-columns
in M .
• Let i be an integer. Prove that the number of awesome 2 × n matrices
with exactly n − i many 11-columns is equal to
2i−k n i
2 .
n − i 2i − k
• Use the above results to prove that
k
X
2i n i k 2n
2 =2 .
i k−i k
i=dk/2e
3.75 How many different strings can be obtained by reordering the letters of
the word MississippiMills. (This is a town close to Ottawa. James Naismith,
the inventor of basketball, was born there.)
3.76 In this exercise, we consider strings that can be obtained by reordering
the letters of the word ENGINE.
• Determine the number of strings that can be obtained.
• Determine the number of strings in which the two letters E are next to
each other.
• Determine the number of strings in which the two letters E are not next
to each other and the two letters N are not next to each other.
3.77 Determine the number of elements x in the set {1, 2, 3, . . . , 99999} for
which the sum of the digits in the decimal representation of x is equal to 8.
An example of such an element x is 3041.
3.78 In Theorems 3.9.1 and 3.9.2, we have seen how many solutions (in
non-negative integers) there are for equations of the type
x1 + x2 + · · · + xk = n
3.11. Exercises 79
and inequalities of the type
x1 + x2 + · · · + xk ≤ n.
Use this to prove the following identity:
n
i+k−1
X n+k
= .
i=0
k−1 k
3.79 Let n and k be integers with n ≥ k ≥ 1. How many solutions are there
to the equation
x1 + x2 + · · · + xk = n,
where x1 ≥ 1, x2 ≥ 1, . . . , xk ≥ 1 are integers?
Hint: In Theorem 3.9.1, we have seen the answer if x1 ≥ 0, x2 ≥ 0, . . . ,
xk ≥ 0.
3.80 In this exercise, we consider sequences consisting of five digits.
• Determine the number of 5-digit sequences d1 d2 d3 d4 d5 , whose digits are
decreasing, i.e., d1 > d2 > d3 > d4 > d5 .
• Determine the number of 5-digit sequences d1 d2 d3 d4 d5 , whose digits are
non-increasing, i.e., d1 ≥ d2 ≥ d3 ≥ d4 ≥ d5 .
Hint: Consider the numbers x1 = d1 −d2 , x2 = d2 −d3 , x3 = d3 −d4 , x4 =
d4 − d5 , x5 = d5 . What do you know about x1 + x2 + x3 + x4 + x5 ?
3.81 The square in the left figure below is divided into nine cells. In each
cell, we write one of the numbers −1, 0, and 1.
0 1 0
1 1 −1
−1 0 −1
Use the Pigeonhole Principle to prove that, among the rows, columns,
and main diagonals, there exist two that have the same sum. For example,
in the right figure above, both main diagonals have sum 0. (Also, the two
topmost rows both have sum 1, whereas the bottom row and the right column
both have sum −2.)
80 Chapter 3. Counting
3.82 Let S be a set consisting of 19 two-digit integers. Thus, each element
of S belongs to the set {10, 11, . . . , 99}.
Use the Pigeonhole Principle to prove that this set S contains two distinct
elements x and y, such that the sum of the two digits of x is equal to the
sum of the two digits of y.
3.83 Let S be a set consisting of 9 people. Every person x in S has an age
age(x), which is an integer with 1 ≤ age(x) ≤ 60.
• Assume that there are two people in S having the same age. Prove that
there exist two distinct subsets A and B of
PS such that (i)
Pboth A and
B are non-empty, (ii) A∩B = ∅, and (iii) x∈A age(x) = x∈B age(x).
• Assume that all people in S having different ages. Use the Pigeonhole
Principle to prove that there exist two distinct subsetsPA and B of S
such
P that (i) both A and B are non-empty, and (ii) x∈A age(x) =
x∈B age(x).
• Assume that all people in S having different ages. Prove that there
exist two distinct subsets A and B ofP
S such that (i) P
both A and B are
non-empty, (ii) A ∩ B = ∅, and (iii) x∈A age(x) = x∈B age(x).
3.84 Let n ≥ 1 be an integer. Use the Pigeonhole Principle to prove that in
any set of n + 1 integers from {1, 2, . . . , 2n}, there are two elements that are
consecutive (i.e., differ by one).
3.85 Let n ≥ 1 be an integer. Use the Pigeonhole Principle to prove that in
any set of n + 1 integers from {1, 2, . . . , 2n}, there are two elements whose
sum is equal to 2n + 1.
3.86 Let S1 , S2 , . . . , S50 be a sequence consisting of 50 subsets of the set
{1, 2, . . . , 55}. Assume that each of these 50 subsets consists of at least seven
elements.
Use the Pigeonhole Principle to prove that there exist two distinct indices
i and j, such that the largest element in Si is equal to the largest element
in Sj .
3.87 Consider five points in a square with sides of length one. Use the Pi-
geonhole Principle
√ to prove that there are two of these points having distance
at most 1/ 2.
3.11. Exercises 81
3.88 Let S1 , S2 , . . . , S26 be a sequence consisting of 26 subsets of the set
{1, 2, . . . , 9}. Assume that each of these 26 subsets consists of at most three
elements. Use the Pigeonhole Principle to prove that there exist two distinct
indices i and j, such that X X
x= x,
x∈Si x∈Sj
i.e., the sum of the elements in Si is equal
P to the sum of the elements in Sj .
Hint: What are the possible values for x∈Si x?
3.89 Let S be a set of 90 positive integers, each one having at most 25 digits
in decimal notation. Use the Pigeonhole Principle to prove that there are
two distinct subsets A and B of S that have the same sum, i.e.,
X X
x= x.
x∈A x∈B
3.90 Let n ≥ 2 be an integer.
• Let S be a set of n + 1 integers. Prove that S contains two elements
whose difference is divisible by n.
Hint: Use the Pigeonhole Principle.
• Prove that there is an integer that is divisible by n and whose decimal
representation only contains the digits 0 and 5.
Hint: Consider the integers 5, 55, 555, 5555, . . .
3.91 In this exercise, we consider the sequence
30 , 31 , 32 , . . . , 31000
of integers.
• Prove that this sequence contains two distinct elements whose difference
is divisible by 1000. That is, prove that there exist two integers m and
n with 0 ≤ m < n ≤ 1000, such that 3n − 3m is divisible by 1000.
Hint: Consider each element in the sequence modulo 1000 and use the
Pigeonhole Principle.
82 Chapter 3. Counting
• Use the first part to prove that the sequence
31 , 32 , . . . , 31000
contains an element whose decimal representation ends with 001. In
other words, the last three digits in the decimal representation are 001.
3.92 Let n ≥ 2 be an integer and let G = (V, E) be a graph whose vertex set
V has size n and whose edge set E is non-empty. The degree of any vertex u
is defined to be the number of edges in E that contain u as a vertex. Prove
that there exist at least two vertices in G that have the same degree.
Hint: Consider the cases when G is connected and G is not connected sepa-
rately. In each case, apply the Pigeonhole Principle. Alternatively, consider
a vertex of maximum degree together with its adjacent vertices and, again,
apply the Pigeonhole Principle.
3.93 Let d ≥ 1 be an integer. A point p in Rd is represented by its d
real coordinates as p = (p1 , p2 , . . . , pd ). The midpoint of two points p =
(p1 , p2 , . . . , pd ) and q = (q1 , q2 , . . . , qd ) is the point
p1 + q1 p2 + q2 pd + qd
, ,..., .
2 2 2
Let P be a set of 2d + 1 points in Rd , all of which have integer coordinates.
Use the Pigeonhole Principle to prove that this set P contains two distinct
elements whose midpoint has integer coordinates.
Hint: The sum of two even integers is even, and the sum of two odd integers
is even.
Chapter 4
Recursion
In order to understand recursion, you must first understand
recursion.
Recursion is the concept where an object (such as a function, a set, or an
algorithm) is defined in the following way:
• There are one or more base cases.
• There are one or more rules that define an object in terms of “smaller”
objects that have already been defined.
In this chapter, we will see several examples of such recursive definitions and
how to use them to solve counting problems.
4.1 Recursive Functions
Recall that N = {0, 1, 2, . . .} denotes the set of natural numbers. Consider
the following recursive definition of a function f : N → N:
f (0) = 3,
f (n) = 2 · f (n − 1) + 3, if n ≥ 1.
These two rules indeed define a function, because f (0) is uniquely defined
and for any integer n ≥ 1, if f (n − 1) is uniquely defined, then f (n) is also
uniquely defined, because it is equal to 2·f (n−1)+3. Therefore, by induction,
for any natural number n, the function value f (n) is uniquely defined. We
can obtain the values f (n) in the following way:
84 Chapter 4. Recursion
• We are given that f (0) = 3.
• If we apply the recursive rule with n = 1, then we get
f (1) = 2 · f (0) + 3 = 2 · 3 + 3 = 9.
• If we apply the recursive rule with n = 2, then we get
f (2) = 2 · f (1) + 3 = 2 · 9 + 3 = 21.
• If we apply the recursive rule with n = 3, then we get
f (3) = 2 · f (2) + 3 = 2 · 21 + 3 = 45.
• If we apply the recursive rule with n = 4, then we get
f (4) = 2 · f (3) + 3 = 2 · 45 + 3 = 93.
Can we “solve” this recurrence relation? That is, can we express f (n) in
terms of n only? By looking at these values, you may see a pattern, i.e., you
may guess that for each n ≥ 0,
f (n) = 3 · 2n+1 − 3. (4.1)
We prove by induction that this is correct: If n = 0, then f (n) = f (0) = 3
and 3 · 2n+1 − 3 = 3 · 20+1 − 3 = 3. Thus, (4.1) is true for n = 0. Let n ≥ 1
and assume that (4.1) is true for n − 1, i.e., assume that
f (n − 1) = 3 · 2n − 3.
Then
f (n) = 2 · f (n − 1) + 3
= 2 (3 · 2n − 3) + 3
= 3 · 2n+1 − 3.
Thus, we have proved by induction that (4.1) holds for all integers n ≥ 0.
4.2. Fibonacci Numbers 85
A recursive definition of factorials: Consider the following recursive
definition of a function g : N → N:
g(0) = 1,
g(n) = n · g(n − 1), if n ≥ 1.
As in the previous example, a simple induction proof shows that these rules
uniquely define the value g(n) for each n ≥ 0. We leave it to the reader to
verify that g is the factorial function, i.e., g(n) = n! for each n ≥ 0.
A recursive definition of binomial coefficients: Consider the following
recursive definition of a function B : N × N → N with two variables:
B(n, 0) = 1, if n ≥ 0,
B(n, n) = 1, if n ≥ 0,
B(n, k) = B(n − 1, k − 1) + B(n − 1, k), if n ≥ 2 and 1 ≤ k ≤ n − 1.
The recursive rule has the same form as Pascal’sIdentity in Theorem 3.7.2.
n
The first base case shows that B(n,
0) = 1 = 0 , whereas the second base
n
case shows that B(n,
n) = 1 = n . From this, it can be shown by induction
that B(n, k) = nk for all n and k with 0 ≤ k ≤ n.
4.2 Fibonacci Numbers
I’ll have an order of the Fibonachos.
The Fibonacci numbers are defined using the following rules:
f0 = 0,
f1 = 1,
fn = fn−1 + fn−2 , if n ≥ 2.
In words, there are two base cases (i.e., 0 and 1) and each next element in the
sequence is the sum of the previous two elements. This gives the sequence
0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, . . .
The following theorem states that we can “solve” this recurrence relation.
That is, we can express the n-th Fibonacci number fn in a non-recursive
way, i.e., without using any other Fibonacci numbers.
86 Chapter 4. Recursion
√ √
Theorem 4.2.1 Let ϕ = 1+2 5 and ψ = 1−2 5 be the two solutions of the
quadratic equation x2 = x + 1. Then, for all n ≥ 0, we have
ϕn − ψ n
fn = √ .
5
Proof. We prove the claim by induction on n. There are two base cases1 :
ϕ0√
−ψ 0
• Both f0 and 5
are equal to 0.
ϕ1√
−ψ 1
• Both f1 and 5
are equal to 1.
Let n ≥ 2 and assume that the claim is true for n − 2 and n − 1. In other
words, assume that
ϕn−2 − ψ n−2
fn−2 = √
5
and
ϕn−1 − ψ n−1
fn−1 = √ .
5
We have to prove that the claim is true for n as well. Using the definition
of fn , the two assumptions, and the identities ϕ2 = ϕ + 1 and ψ 2 = ψ + 1,
we get
fn = fn−1 + fn−2
ϕn−1 − ψ n−1 ϕn−2 − ψ n−2
= √ + √
5 5
n−2 n−2
ϕ (ϕ + 1) ψ (ψ + 1)
= √ − √
5 5
n−2 2 n−2 2
ϕ ·ϕ ψ ·ψ
= √ − √
5 5
n n
ϕ −ψ
= √ .
5
1
Do you see why there are two base cases?
4.2. Fibonacci Numbers 87
4.2.1 Counting 00-Free Bitstrings
A bitstring is called 00-free, if it does not contain two 0’s next to each other.
Examples of 00-free bitstrings are 10, 010, 0101010101, and 1111111. On the
other hand, neither of the two bitstrings 101001 and 0100011 is 00-free.
For any integer n ≥ 1, what is the number of 00-free bitstrings having
length n? Since we do not know the answer yet, we introduce a variable Bn ,
one for each n ≥ 1, for the number of such strings. Thus,
• Bn denotes the number of 00-free bitstrings of length n.
Let us start by determining Bn for some small values of n. There are two
bitstrings of length 1:
0, 1.
Since neither of them contains 00, we have B1 = 2. There are four bitstrings
of length 2:
00, 10, 01, 11.
Since three of them do not contain 00, we have B2 = 3. Similarly, there are
eight bitstrings of length 3:
000, 001, 010, 100, 011, 101, 110, 111.
Since five of them do not contain 00, we have B3 = 5.
Let n ≥ 3. We are going to express Bn in terms of the previous two values
Bn−1 and Bn−2 . This, together with the two base cases B1 = 2 and B2 = 3,
will give a recurrence relation for the entire sequence.
Consider a matrix that contains all 00-free bitstrings of length n, one
string per row. Since the number of such strings is equal to Bn , the matrix
has Bn rows. Also, the matrix has n columns, because the strings have
length n.
We rearrange the rows of the matrix such that all strings in the top part
start with 1 and all strings in the bottom part start with 0.
• How many rows are there in the top part? Any string in the top part
starts with 1 and is followed by a bitstring of length n − 1 that does not
contain 00. Thus, if we take the rows in the top part and delete the first
bit from each row, then we obtain all 00-free bitstrings of length n − 1.
Since the number of 00-free bitstrings of length n − 1 is equal to Bn−1 ,
it follows that the top part of the matrix consists of Bn−1 rows.
88 Chapter 4. Recursion
• How many rows are there in the bottom part? Any string in the bottom
part starts with 0. Since the string does not contain 00, the second bit
must be 1. After these first two bits, we have a bitstring of length n − 2
that does not contain 00. Thus, if we take the rows in the bottom part
and delete the first two bits from each row, then we obtain all 00-free
bitstrings of length n − 2. Since the number of 00-free bitstrings of
length n − 2 is equal to Bn−2 , it follows that the bottom part of the
matrix consists of Bn−2 rows.
Thus, on the one hand, the matrix has Bn rows. On the other hand, this
matrix has Bn−1 + Bn−2 rows. Therefore, we have Bn = Bn−1 + Bn−2 .
To summarize, we have proved that the values Bn , for n ≥ 1, satisfy the
following recurrence relation:
B1 = 2,
B2 = 3,
Bn = Bn−1 + Bn−2 , if n ≥ 3.
This recurrence relation is the same as the one for the Fibonacci numbers,
except that the two base cases are different. The sequence Bn , n ≥ 1, consists
of the integers
2, 3, 5, 8, 13, 21, 34, 55, 89, 144, . . .
We obtain this sequence by removing the first three elements (i.e., f0 , f1 , and
f2 ) from the Fibonacci sequence. We leave it to the reader to verify (using
induction) that for all n ≥ 1,
Bn = fn+2 .
4.3 A Recursively Defined Set
Consider the set S that is defined by the following two rules:
• 5 is an element of the set S.
• If x and y are elements of the set S, then x − y is also an element of
the set S.
Thus, if we already know that x and y belong to the set S, then the second
rule gives us a new element, i.e., x − y, that also belongs to S.
4.3. A Recursively Defined Set 89
Can we give a simple description of the set S? We are going to use the
rules to obtain some elements of S. From these examples, we then hope to
see a pattern from which we guess the simple description of S. The final step
consists of proving that our guess is correct.
• We are given that 5 is an element of S.
• Applying the rule with x = 5 and y = 5 implies that x − y = 0 is also
an element of S.
• Applying the rule with x = 0 and y = 5 implies that x − y = −5 is also
an element of S.
• Applying the rule with x = 5 and y = −5 implies that x − y = 10 is
also an element of S.
• Applying the rule with x = 0 and y = 10 implies that x − y = −10 is
also an element of S.
• Applying the rule with x = 5 and y = −10 implies that x − y = 15 is
also an element of S.
• Applying the rule with x = 0 and y = 15 implies that x − y = −15 is
also an element of S.
Thus, we have obtained the following elements of S:
−15, −10, −5, 0, 5, 10, 15
Since there is clearly a pattern, it is natural to guess that
S = {5n : n ∈ Z}, (4.2)
where Z is the set of all (positive and negative) integers, including 0. To
prove that this is correct, we will first prove that the set on the left-hand
side is a subset of the set on the right-hand side. Then we prove that the set
on the right-hand side is a subset of the set on the left-hand side.
We start by proving that
S ⊆ {5n : n ∈ Z},
90 Chapter 4. Recursion
which is equivalent to proving that
every element of S is a multiple of 5. (4.3)
How do we prove this? The set S is defined using a base case and a recursive
rule. The only way to obtain an element of S is by starting with the base
case and then applying the recursive rule a finite number of times. Therefore,
the following will prove that (4.3) holds:
• The element in the base case, i.e., 5, is a multiple of 5.
• Let x and y be two elements of S and assume that they are both
multiples of 5. Then x − y (which is the “next” element of S) is also a
multiple of 5.
Next we prove that
{5n : n ∈ Z} ⊆ S.
We will do this by proving that for all n ≥ 0,
5n ∈ S and − 5n ∈ S. (4.4)
The proof is by induction on n. For the base case, i.e., when n = 0, we
observe that, from the definition of S, x = 5 and y = 5 are in S and,
therefore, x − y = 0 is also in S. Therefore, (4.4) is true for n = 0.
Let n ≥ 0 and assume that (4.4) is true for n, i.e., assume that
5n ∈ S and − 5n ∈ S.
We have to show that (4.4) is also true for n + 1, i.e.,
5(n + 1) ∈ S and − 5(n + 1) ∈ S.
• It follows from the definition of S and our assumption that both x = 5
and y = −5n are in S. Therefore, x − y = 5(n + 1) is also in S.
• It follows from the definition of S and our assumption that both x =
−5n and y = 5 are in S. Therefore, x − y = −5(n + 1) is also in S.
Thus, we have shown by induction that (4.4) holds for all n ≥ 0.
Since we have shown that both (4.3) and (4.4) hold, we conclude that (4.2)
holds as well. In other words, we have indeed obtained a simple description
of the set S: It is the set of all multiples of 5.
4.4. A Gossip Problem 91
4.4 A Gossip Problem
Let n ≥ 4 be an integer and consider a group P1 , P2 , . . . , Pn of n people.
Assume that each person Pi knows some scandal Si that nobody else knows.
For any i and j, if person Pi makes a phone call with person Pj , they exchange
the scandals they know at that moment, i.e., Pi tells all scandals she knows
to Pj , and Pj tells all scandals he knows to Pi . How many phone calls are
needed until each of the n people knows all n scandals?
An obvious solution is that each pair of people in the group makes one
phone call. At the end, each person knows all scandals. The number of phone
calls is
n(n − 1)
n
= ,
2 2
which is quadratic in the number n of people. We will see below that only a
linear number of phone calls are needed.
Let us first consider the case when n = 4. At the start, each person Pi
only knows the scandal Si , which we visualize in the following table:
P1 P2 P3 P4
S1 S2 S3 S4
Consider the following sequence of phone calls:
1. P1 calls P2 . After this phone call, the table looks as follows:
P1 P2 P3 P4
S1 S2 S1 S2 S3 S4
2. P3 calls P4 . After this phone call, the table looks as follows:
P1 P2 P3 P4
S1 S2 S1 S2 S3 S4 S3 S4
3. P1 calls P3 . After this phone call, the table looks as follows:
P1 P2 P3 P4
S1 S2 S3 S4 S1 S2 S1 S2 S3 S4 S3 S4
4. P2 calls P4 . After this phone call, the table looks as follows:
92 Chapter 4. Recursion
P1 P2 P3 P4
S1 S2 S3 S4 S1 S2 S3 S4 S1 S2 S3 S4 S1 S2 S3 S4
We see that after four phone calls, each person knows all four scandals.
Observe that the number of phone calls is 42 = 6 if we would have used the
obvious solution mentioned above.
We now have an algorithm that schedules the phone calls for groups of
four people. Below, we will extend this “base case” to a recursive algorithm
that schedules the phone calls for any group of n ≥ 4 people. The approach
is as follows:
• We assume that we know how to schedule the phone calls for groups of
n − 1 people.
• We use this assumption to schedule the phone calls for groups of n
people.
Let us see how this is done.
• At the start, P1 knows S1 , P2 knows S2 , . . . , Pn knows Sn .
• Pn−1 calls Pn . After this phone call, P1 knows S1 , P2 knows S2 , . . . ,
Pn−2 knows Sn−2 , and both Pn−1 and Pn know Sn−1 and Sn .
0
• Consider Sn−1 and Sn to be one scandal Sn−1 .
• Schedule the phone calls for the group P1 , P2 , . . . , Pn−1 of n − 1 people,
0
using the scandals S1 , S2 , . . . , Sn−2 , Sn−1 . (We have assumed that we
know how to do this!) At the end, each of P1 , P2 , . . . , Pn−1 knows all
scandals S1 , S2 , . . . , Sn .
• At this moment, Pn only knows Sn−1 and Sn . Therefore, Pn−1 again
calls Pn and tells her all scandals S1 , S2 , . . . , Sn ; the first n − 2 of these
are new to Pn .
Below, you see this recursive algorithm in pseudocode.
4.4. A Gossip Problem 93
Algorithm gossip(n):
// n ≥ 4, this algorithm schedules phone calls for P1 , P2 , . . . , Pn
if n = 4
then P1 calls P2 ;
P3 calls P4 ;
P1 calls P3 ;
P2 calls P4
else Pn−1 calls Pn ;
gossip(n − 1);
Pn−1 calls Pn
endif
We are now going to determine the number of phone calls made when
running algorithm gossip(n). Since we do not know the answer yet, we
introduce a variable C(n) to denote this number. It follows from the pseu-
docode that
C(4) = 4.
Let n ≥ 5. Algorithm gossip(n) starts and ends with the same phone call:
Pn−1 calls Pn . In between, it runs algorithm gossip(n − 1), during which,
by definition, C(n − 1) phone calls are made. It follows that
C(n) = 2 + C(n − 1) for n ≥ 5.
Thus, we have obtained a recurrence relation for the numbers C(n). The
first few numbers in the sequence are
C(4) = 4,
C(5) = 2 + C(4) = 2 + 4 = 6,
C(6) = 2 + C(5) = 2 + 6 = 8,
C(7) = 2 + C(6) = 2 + 8 = 10.
From this, we guess that
C(n) = 2n − 4 for n ≥ 4.
We can easily prove by induction that our guess is correct. Indeed, since
both C(4) and 2 · 4 − 4 are equal to 4, the claim is true for n = 4. If n ≥ 5
and C(n − 1) = 2(n − 1) − 4, then
C(n) = 2 + C(n − 1) = 2 + (2(n − 1) − 4) = 2n − 4.
94 Chapter 4. Recursion
This shows that C(n) = 2n − 4 for all n ≥ 4.
It can be shown that algorithm gossip is optimal: Any algorithm that
schedules phone calls for n ≥ 4 people must make at least 2n − 4 phone calls.
You may wonder why the base case for algorithm gossip(n) is when
n = 4. You will find the reason in Exercise 4.52.
4.5 Euclid’s Algorithm
We might call Euclid’s method the granddaddy of all algorithms,
because it is the oldest nontrivial algorithm that has survived to
the present day.
— Donald E. Knuth, The Art of Computer Programming,
Vol. 2, 1997
The greatest common divisor of two integers a ≥ 1 and b ≥ 1 is the
largest integer that divides both a and b. We denote this largest integer by
gcd (a, b). For example, the common divisors of 75 and 45 are 1, 3, 5, and 15.
Since 15 is the largest among them, gcd (75, 45) = 15. Observe that for any
integer a ≥ 1, gcd (a, a) = a.
Assume we are given two large integers a and b, say a = 371, 435, 805 and
b = 137, 916, 675. How can we compute their greatest common divisor? One
approach is to determine the prime factorizations of a and b:
a = 371, 435, 805 = 32 · 5 · 134 · 172
and
b = 137, 916, 675 = 34 · 52 · 133 · 31.
From this, we see that
gcd (a, b) = 32 · 5 · 133 = 98, 865.
Unfortunately, it is not known how to obtain, by an efficient algorithm, the
prime factorization of a very large integer. As a result, this approach to
compute the greatest common divisor of two large integers is not good.
Around 300 BC, Euclid published an algorithm that is both very simple
and efficient. This algorithm is based on the modulo operation, which we
introduce first.
4.5. Euclid’s Algorithm 95
4.5.1 The Modulo Operation
Let a ≥ 1 and b ≥ 1 be integers. If we divide a by b, then we obtain a
quotient q and a remainder r, which are the unique integers that satisfy
a = qb + r, q ≥ 0, and 0 ≤ r ≤ b − 1.
The modulo operation, denoted by a mod b, is the function that maps the
pair (a, b) to the remainder r. Thus, we will write
a mod b = r.
For example,
• 17 mod 5 = 2, because 17 = 3 · 5 + 2,
• 17 mod 17 = 0, because 17 = 1 · 17 + 0,
• 17 mod 1 = 0, because 17 = 17 · 1 + 0,
• 17 mod 19 = 17, because 17 = 0 · 19 + 17.
4.5.2 The Algorithm
Euclid’s algorithm takes as input two positive integers a and b, where a ≥ b,
and returns gcd (a, b).
The algorithm starts by computing a mod b and stores the result in a
variable r. If r = 0, then the algorithm returns the value b. Otherwise, we
have r ≥ 1, in which case the algorithm recursively computes the greatest
common divisor of b and r. The algorithm is presented in pseudocode below.
Algorithm Euclid(a, b):
// a and b are integers with a ≥ b ≥ 1
r = a mod b;
if r = 0
then return b
else Euclid(b, r)
// observe that b > r ≥ 1
endif
96 Chapter 4. Recursion
Let us look at an example. If we run Euclid(75, 45), then the algo-
rithm computes 75 mod 45, which is 30. Then, it runs Euclid(45, 30), dur-
ing which the algorithm computes 45 mod 30, which is 15. Next, it runs
Euclid(30, 15), during which the algorithm computes 30 mod 15, which is 0.
At this moment, the algorithm returns 15, which is indeed the greatest com-
mon divisor of the input values 75 and 45.
The following lemma is the basis for a proof that algorithm Euclid(a, b)
correctly returns gcd (a, b) for any input values a ≥ b ≥ 1.
Lemma 4.5.1 Let a and b be integers with a ≥ b ≥ 1, and let r = a mod b.
1. If r = 0, then gcd (a, b) = b.
2. If r ≥ 1, then gcd (a, b) = gcd (b, r).
Proof. Let q and r be the integers that satisfy a = qb + r, q ≥ 1, and
0 ≤ r ≤ b − 1. (Observe that q cannot be equal to 0, because a ≥ b.)
If r = 0, then a = qb. In this case, it is clear that gcd (a, b) = b. Assume
that r ≥ 1. We claim that the common divisors of a and b are the same as
the common divisors of b and r:
• Let d ≥ 1 be an integer that divides both a and b. Since r = a − qb, it
follows that d divides r. Thus, d divides both b and r.
• Let d ≥ 1 be an integer that divides both b and r. Since a = qb + r, it
follows that d divides a. Thus, d divides both a and b.
Since the two pairs a, b and b, r have the same common divisors, their greatest
common divisors are equal as well.
Theorem 4.5.2 For any two integers a and b with a ≥ b ≥ 1, algorithm
Euclid(a, b) returns gcd (a, b).
Proof. If algorithm Euclid(a, b) generates the recursive call Euclid(b, r),
then r < b. Thus, in each recursive call to Euclid, the second argument
decreases. Since this second argument is a positive integer, the algorithm
terminates.
We leave it to the reader to use Lemma 4.5.1 to prove that the output of
algorithm Euclid(a, b) is gcd (a, b).
4.5. Euclid’s Algorithm 97
4.5.3 The Running Time
In the beginning of Section 4.5, we mentioned that Euclid’s algorithm is
efficient. In this section, we will formalize this.
We are going to bound the total number of modulo operations that are
performed when running algorithm Euclid(a, b). This number will be de-
noted by M (a, b).
For example, when running Euclid(75, 45), the modulo operation is per-
formed three times: The algorithm computes 75 mod 45, 45 mod 30, and
30 mod 15. Therefore, M (75, 45) = 3.
Our goal is to prove an upper bound on M (a, b) in terms of a and b. In
fact, as we will see, we will obtain an upper bound in terms of b only, i.e., the
upper bound only depends on the smaller of the two input values a and b.
As a first upper bound, we have seen in the proof of Theorem 4.5.2 that
in each recursive call to algorithm Euclid, the second argument decreases.
Since in the initial call Euclid(a, b), this second argument is equal to b, the
number of modulo operations cannot be larger than b. It follows that, for all
integers a and b with a ≥ b ≥ 1,
M (a, b) ≤ b.
This gives an upper bound that is linear in b. Below, we will prove a much
better upper bound: The value of M (a, b) is at most logarithmic in b. We
will use the Fibonacci numbers of Section 4.2 to obtain this result. Recall
that these numbers are defined by
f0 = 0,
f1 = 1,
fn = fn−1 + fn−2 , if n ≥ 2.
As mentioned above, we are going to prove an upper bound on M (a, b)
in terms of the logarithm of b. Usually, when analyzing the running time of
an algorithm, we consider a given input and derive an upper bound on the
running time in terms of the input. For algorithm Euclid(a, b), we use the
opposite approach: We fix a value m for the running time M (a, b), and then
prove a lower bound on both a and b in terms of m. The following lemma
makes this precise.
Lemma 4.5.3 Let a and b be integers with a > b ≥ 1, and let m = M (a, b).
Then a ≥ fm+2 and b ≥ fm+1 .
98 Chapter 4. Recursion
Proof. The proof is by induction on m. The base case is when m = 1. Since
a ≥ b + 1 ≥ 2 = f3 and b ≥ 1 = f2 , the claim in the lemma holds.
For the induction step, assume that m ≥ 2. Consider the integers q
and r that satisfy a = qb + r, q ≥ 1, and 0 ≤ r ≤ b − 1. Observe that
algorithm Euclid(a, b) computes the value a mod b, which is equal to r.
Since m ≥ 2, we have r ≥ 1 and the total number of modulo operations
performed during the recursive call Euclid(b, r) is equal to m − 1. In other
words, M (b, r) = m − 1. Thus, by induction, we have b ≥ fm+1 and r ≥ fm .
We observe that
a = qb + r ≥ b + r ≥ fm+1 + fm = fm+2 .
This completes the induction step.
In Theorem 4.2.1, we have seen that√ the Fibonacci√ numbers can be ex-
1+ 5 1− 5
pressed in terms of the numbers ϕ = 2 and ψ = 2 . You are encouraged
to prove, by induction and using the fact that ϕ2 = ϕ+1, that for any integer
n ≥ 2,
fn ≥ ϕn−2 . (4.5)
Theorem 4.5.4 Let a and b be integers with a ≥ b ≥ 1. Then
M (a, b) ≤ 1 + logϕ b,
i.e., the total number of modulo operations performed by algorithm Euclid(a, b)
is O(log b).
Proof. If a = b, then M (a, b) = 1 and the claim obviously holds. Assume
that a > b. Let m = M (a, b). By Lemma 4.5.3 and (4.5), we have
b ≥ fm+1 ≥ ϕm−1 .
By taking logarithms with base ϕ, we conclude that
m − 1 ≤ logϕ b,
i.e.,
M (a, b) = m ≤ 1 + logϕ b = O(log b).
4.6. The Merge-Sort Algorithm 99
4.6 The Merge-Sort Algorithm
MergeSort is a recursive sorting algorithm that works as follows. To sort
the sequence a1 , a2 , . . . , an of numbers,
• it recursively sorts the sequence a1 , a2 , . . . , am , where m = bn/2c, and
stores the sorted sequence in a list L1 ,
• it recursively sorts the sequence am+1 , am+2 , . . . , an and stores the sorted
sequence in a list L2 ,
• it merges the two sorted lists L1 and L2 into one sorted list.
Below, you see this recursive algorithm in pseudocode.
Algorithm MergeSort(L, n):
// L is a list of n ≥ 0 numbers
if n ≥ 2
then m = bn/2c;
L1 = list consisting of the first m elements of L;
L2 = list consisting of the last n − m elements of L;
L1 = MergeSort(L1 , m);
L2 = MergeSort(L2 , n − m);
L = Merge(L1 , L2 )
endif;
return L
We still have to specify algorithm Merge(L1 , L2 ). Of course, this algo-
rithm uses the fact that both L1 and L2 are sorted lists. The task is to merge
them into one sorted list. This is done in the following way. Initialize an
empty list L. (At the end, this list will contain the final sorted sequence.)
• Let x be the first element of L1 and let y be the first element of L2 .
• If x ≤ y, then remove x from L1 and append it to L (i.e., add x at the
end of L).
• Otherwise (i.e., if x > y), remove y from L2 and append it to L.
100 Chapter 4. Recursion
Repeat these steps until one of L1 and L2 is empty. If L1 is empty, then
append L2 to L. Otherwise, append L1 to L. Here is the algorithm in
pseudocode:
Algorithm Merge(L1 , L2 ):
// L1 and L2 are sorted lists
L = empty list;
while L1 is not empty and L2 is not empty
do x = first element of L1 ;
y = first element of L2 ;
if x ≤ y
then remove x from L1 ;
append x to L
else remove y from L2 ;
append y to L
endif
endwhile;
if L1 is empty
then append L2 to L
else append L1 to L
endif;
return L
4.6.1 Correctness of Algorithm MergeSort
I hope you are convinced that the output L of algorithm Merge(L1 , L2 ) is
a sorted list that contains all elements of L1 and L2 (and no other elements).
How do we prove that algorithm MergeSort(L, n) is correct, i.e., correctly
sorts the elements in any list L of n numbers? Since the algorithm is recursive,
we prove this by induction.
The two base cases are when n = 0 or n = 1. It follows from the
pseudocode for MergeSort(L, n) that it simply returns the input list L,
which is obviously sorted.
Let n ≥ 2 and assume that for any integer k with 0 ≤ k < n and for any
list L0 of k numbers, algorithm MergeSort(L0 , k) returns a list containing
the elements of L0 in sorted order. Let L be a list of n numbers. By going
4.6. The Merge-Sort Algorithm 101
through the pseudocode for MergeSort(L, n), we observe the following:
• The recursive call MergeSort(L1 , m) is on a list with less than n
numbers. Therefore, by the induction hypothesis, its output, which is
the list L1 , is sorted.
• The recursive call MergeSort(L2 , n − m) is on a list with less than n
numbers. Again by the induction hypothesis, its output, which is the
list L2 , is sorted.
• Algorithm Merge(L1 , L2 , ) gets as input the two sorted lists L1 and
L2 , and returns a list L. Since algorithm Merge is correct, it then
follows that L is a sorted list.
It follows that the final list L, which is returned by algorithm MergeSort,
is sorted.
This proves the correctness of algorithm MergeSort(L, n) for any inte-
ger n ≥ 0 and any list L of n numbers.
4.6.2 Running Time of Algorithm MergeSort
We now analyze the running time of algorithm MergeSort. It follows from
the pseudocode that, when running this algorithm together with its recursive
calls, several calls are made to algorithm Merge. We are going to count the
total number of comparisons that are made. That is, we will determine
the total number of times that the line “if x ≤ y” in algorithm Merge is
executed when running algorithm MergeSort(L, n).
We first observe that the number of comparisons made by algorithm
Merge(L1 , L2 ) is at most |L1 | + |L2 |.
Let n be an integer and assume for simplicity that n is a power of two, i.e.,
n = 2k for some integer k ≥ 0. We define T (n) to be the maximum number of
comparisons made when running algorithm MergeSort(L, n) on any input
list L of n numbers. Note that we include in T (n) all comparisons that are
made during all calls to Merge that are part of all recursive calls that are
generated when running MergeSort(L, n).
Consider a list L of n numbers, where n is a power of two. For n = 1, it
follows from the pseudocode for MergeSort(L, n) that
T (1) = 0.
102 Chapter 4. Recursion
Assume that n ≥ 2 and consider again the pseudocode for MergeSort(L, n).
Which parts of the algorithm make comparisons between input elements?
• The call MergeSort(L1 , m) is a recursive call on a list of m = n/2
numbers. By definition, the total number of comparisons made in this
call (together with all its recursive subcalls) is at most T (n/2).
• The call MergeSort(L2 , n − m) is a recursive call on a list of n − m =
n/2 numbers. By definition, the total number of comparisons made in
this call (together with all its recursive subcalls) is at most T (n/2).
• Finally, algorithm MergeSort(L, n) calls the non-recursive algorithm
Merge(L1 , L2 ). We have seen above that the number of comparisons
made in this call is at most |L1 | + |L2 | = n.
By adding the number of comparisons, we get
T (n) ≤ T (n/2) + T (n/2) + n = 2 · T (n/2) + n.
Thus, we obtain the following recurrence relation:
T (1) = 0,
T (n) ≤ 2 · T (n/2) + n, if n ≥ 2 and n is a power of 2. (4.6)
Our goal was to determine T (n), but at this moment, we only have a recur-
rence relation for this function. We will solve this recurrence relation using
a technique called unfolding:
Recall that we assume that n = 2k for some integer k ≥ 0. We further-
more assume that n is a large integer. We know from (4.6) that
T (n) ≤ 2 · T (n/2) + n.
If we replace n by n/2 in (4.6), which is a valid thing to do, we get
T (n/2) ≤ 2 · T (n/22 ) + n/2.
By combining these two inequalities, we get
T (n) ≤ 2 · T (n/2) + n
≤ 2 2 · T (n/22 ) + n/2 + n
= 22 · T (n/22 ) + 2n.
4.6. The Merge-Sort Algorithm 103
Let us repeat this: Replacing n by n/22 in (4.6) gives
T (n/22 ) ≤ 2 · T (n/23 ) + n/22 .
By substituting this into the inequality for T (n), we get
T (n) ≤ 22 · T (n/22 ) + 2n
≤ 22 2 · T (n/23 ) + n/22 + 2n
= 23 · T (n/23 ) + 3n.
In the next step, we replace n by n/23 in (4.6), which gives
T (n/23 ) ≤ 2 · T (n/24 ) + n/23 .
By substituting this into the inequality for T (n), we get
T (n) ≤ 23 · T (n/23 ) + 3n
≤ 23 2 · T (n/24 ) + n/23 + 3n
= 24 · T (n/24 ) + 4n.
At this moment, you will see the pattern and, at the end, we get the inequality
T (n) ≤ 2k · T (n/2k ) + kn.
Since n = 2k , we have T (n/2k ) = T (1), which is 0 from the base case of the
recurrence relation. Also, n = 2k implies that k = log n. We conclude that
T (n) ≤ n · T (1) + n log n = n log n.
We thus have solved the recurrence relation. In case you have doubts about
the validity of the unfolding method, we verify by induction that indeed
T (n) ≤ n log n, for any integer n that is a power of 2.
The base case is when n = 1. In this case, we have T (1) = 0 and 1 log 1 =
1 · 0 = 0. Let n ≥ 2 be a power of 2 and assume that
T (n/2) ≤ (n/2) log(n/2).
From the recurrence relation, we get
T (n) ≤ 2 · T (n/2) + n.
104 Chapter 4. Recursion
By substituting the induction hypothesis into this inequality, we get
T (n) ≤ 2 · (n/2) log(n/2) + n
= n log(n/2) + n
= n (log n − log 2) + n
= n (log n − 1) + n
= n log n.
Thus, by induction, T (n) ≤ n log n for any integer n that is a power of 2.
Until now, we have only counted the number of comparisons made by
algorithm MergeSort. It follows from the pseudocode that the total run-
ning time, i.e., the total number of “elementary” steps, is within a constant
factor of the total number of comparisons. Therefore, if n is a power of 2,
the running time of algorithm MergeSort(L, n) is O(n log n).
For general values of n, the recurrence relation for the number of com-
parisons becomes the following:
T (n) = 0, if n = 0 or n = 1,
T (n) ≤ T (bn/2c) + T (dn/2e) + n, if n ≥ 2.
It can be shown by induction that this recurrence relation solves to T (n) =
O(n log n). We have proved the following result:
Theorem 4.6.1 For any list L of n numbers, the running time of algorithm
MergeSort(L, n) is O(n log n).
4.7 Computing the Closest Pair
For a long time researchers felt that there might be a quadratic
lower bound on the complexity of the closest-pair problem.
— Jon Louis Bentley,
— Communications of the ACM, volume 23, page 226, 1980
If p = (p1 , p2 ) and q = (q1 , q2 ) are two points in R2 , then their distance
d(p, q) is given by
p
d(p, q) = (p1 − q1 )2 + (p2 − q2 )2 .
This follows by applying Pythagoras’ Theorem to the right triangle in the
following figure.
4.7. Computing the Closest Pair 105
q
d(p, q)
|p2 − q2|
p
|p1 − q1|
Let S be a set of n points in R2 , where n ≥ 2 is an integer. The closest-
pair distance in S, denoted by δ(S), is the minimum distance between any
two distinct points of S, i.e.,
δ(S) = min{d(p, q) : p ∈ S, q ∈ S, p 6= q}.
δ(S)
In this section, we consider the problem of designing an efficient algorithm
that, when given an arbitrary set S of n points in R2 , with n ≥ 2, returns
the closest-pair distance δ(S).
A trivial algorithm considers all 2-element subsets of S. For each such
subset {p, q}, the algorithm computes the distance d(p, q). After all these
subsets have been considered, the algorithm returns the smallest distance
found. Obviously, the running time of this algorithm is proportional to the
number of 2-element subsets of S, which is
n(n − 1)
n
= Θ n2 .
=
2 2
In this section, we will show that the closest pair problem can be solved,
by a recursive algorithm, in O(n log n) time. In Section 4.7.1, we start by
presenting a high-level overview of the basic approach. Then, in Section 4.7.2,
we present the details of the recursive algorithm.
4.7.1 The Basic Approach
We are given a set S of n points in R2 , where n ≥ 2. We assume that
106 Chapter 4. Recursion
• n is a power of two,
• no two points of S have the same x-coordinate,
• no two points of S have the same y-coordinate.
We remark that neither of these assumptions is necessary. We only make
them to simplify the presentation.
As mentioned above, our algorithm will be recursive. The base case is
when n = 2, i.e., the set S consists of exactly two points, say p and q. In
this case, the algorithm simply returns the distance d(p, q).
From now on, we assume that n ≥ 4. The algorithm performs the follow-
ing four steps:
Step 1: Let ` be a vertical line that splits the set S into two subsets of equal
size. The algorithm computes the set S1 consisting of all points of S that are
to the left of `, and the set S2 consisting of all points of S that are to the
right of `. Observe that |S1 | = |S2 | = n/2.
Step 2: The algorithm recursively computes the closest-pair distance δ1 in
the set S1 .
Step 3: The algorithm recursively computes the closest-pair distance δ2 in
the set S2 .
δ1
δ2
`
S1 S2
Step 4: Let δ = min(δ1 , δ2 ). Consider the set
A = {{p, q} : p ∈ S1 , q ∈ S2 , d(p, q) < δ}.
4.7. Computing the Closest Pair 107
• If A = ∅, then the algorithm returns the value of δ.
• Assume that A 6= ∅. The algorithm considers all pairs {p, q} ∈ A and
computes the distances d(p, q). Let δ1,2 be the smallest distance found
after all these pairs have been considered. Then, the algorithm returns
the value of δ1,2 .
It should be clear that this algorithm correctly returns the closest-pair
distance δ(S) in the point set S. What is not clear, however, is how to
efficiently perform the last step that involves the set A. For this, we have to
answer two questions: First, how do we efficiently obtain all pairs {p, q} that
belong to the set A? Second, is there a “small” upper bound on the size of
this set A?
Let `1 be the vertical line that is at distance δ to the left of `, and let S10
be the set of all points in S1 that are between `1 and `. Similarly, let `2 be
the vertical line that is at distance δ to the right of `, and let S20 be the set
of all points in S2 that are between ` and `2 . Refer to the figure below for
an illustration.
δ δ
δ1
δ2
`1 ` `2
S10 S20
Any point that is on or to the left of `1 has distance at least δ to any
point that is on or to the right of `. Similarly, any point that is on or to the
left of ` has distance at least δ to any point that is on or to the right of `2 .
This implies that the set A in Step 4 of the algorithm satisfies
A = {{p, q} : p ∈ S10 , q ∈ S20 , d(p, q) < δ}. (4.7)
108 Chapter 4. Recursion
Unfortunately, even using this alternative characterization of the set A, it
is not clear how to obtain all elements of this set in an efficient way. Below,
we will define a superset of A, i.e., a set C of ordered pairs (r, s), with
r ∈ S10 ∪ S20 and s ∈ S10 ∪ S20 , that contains2 all elements of A. As we will
see, the size of this new set C is O(n) and its elements can be obtained in
an efficient way. As a result, the algorithm will use this new set C in Step 4,
0
instead of A. If C 6= ∅, let δ1,2 be the smallest distance of any pair (r, s) in
C. The algorithm will return the value
0
min δ, δ1,2 .
Note that, since A ⊆ C, the algorithm, with the revised Step 4, still correctly
returns the closest-pair distance in the set S.
Before we define the new set C, we introduce a preliminary set B that is
a superset of A, i.e., A ⊆ B. We will use this set B to define the set C that
we are looking for. This set C will satisfy B ⊆ C and, thus, A ⊆ C.
We introduce the following notation, which is illustrated in the figure
below. Let r be any point that is between the two lines `1 and `2 . We denote
by Rr the rectangle that has r on its bottom side, whose left side is on `1 ,
whose right side is on `2 , and whose height is equal to δ. Observe that the
width of Rr is equal to 2δ.
δ δ δ δ
Rp p Rq
δ q δ
p q
`1 ` `2 `1 ` `2
Consider the set
B = B1 ∪ B2 ,
2
Even though A consists of unordered pairs and C consists of ordered pairs, we will
cheat a bit and say that C contains A.
4.7. Computing the Closest Pair 109
where
B1 = {(p, q) : p ∈ S10 , q ∈ S20 , q ∈ Rp }
and
B2 = {(q, p) : p ∈ S10 , q ∈ S20 , p ∈ Rq }.
Lemma 4.7.1 The set B is a superset of the set A, i.e., A ⊆ B.
Proof. We have to show that every element of the set A belongs (as an
ordered pair) to the set B. To prove this, consider an arbitrary element
{p, q} of A. We will show that one of the ordered pairs (p, q) and (q, p) is an
element of the set B.
It follows from (4.7) that p ∈ S10 and q ∈ S20 . Thus, to prove that one of
(p, q) and (q, p) is an element of B, it remains to be shown that
q ∈ Rp or p ∈ Rq . (4.8)
Since {p, q} ∈ A, we have d(p, q) < δ. This implies that the vertical
distance between p and q is less than δ. That is, if p = (p1 , p2 ) and q = (q1 , q2 ),
then |p2 − q2 | < δ.
If p2 < q2 , then the point q is contained in the rectangle Rp and, therefore,
(4.8) holds. Otherwise, p2 > q2 , in which case the point p is contained in the
rectangle Rq and, thus, (4.8) also holds.
Is there a non-trivial upper bound on the size of the set B? Since each
of the two sets S10 and S20 can have n/2 elements, it is clear that |B| ≤
n/2 · n/2 = n2 /4. In words, the size of B is at most quadratic in n. The
following lemma states that the size of B is, in fact, at most linear in n:
Lemma 4.7.2 The size of the set B is at most 4n.
Proof. Let p be an arbitrary point in S10 . We claim that there are at most
four points q such that (p, q) ∈ B1 . We will prove this claim by contradiction.
Thus, assume that there are at least five such points q. Observe that for any
such point q, we have q ∈ S20 and q ∈ Rp . Therefore, all these points q are
contained in the part of Rp that is to the right of the line `. This part is a
square with sides of length δ. By Exercise 3.87, there are two of these points
that have distance at most √
δ/ 2 < δ.
110 Chapter 4. Recursion
Thus, the set S20 contains two points having distance less than δ. That is,
the closest-pair distance in the set S2 is less than δ.
On the other hand, recall that δ = min(δ1 , δ2 ) and δ2 is the closest-pair
distance of the set S2 . It follows that all distances in the set S2 are at least
equal to δ. This is a contradiction.
Thus, we have shown that, for this fixed point p in S10 , there are at most
four points q such that (p, q) ∈ B1 . Therefore,
|B1 | ≤ 4|S10 | ≤ 4|S1 | = 4 · n/2 = 2n.
By a symmetric argument, for any fixed point q in S20 , there are at most
four points p such that (q, p) ∈ B2 . This implies that the set B2 contains at
most 2n elements. We conclude that
|B| = |B1 | + |B2 | ≤ 2n + 2n = 4n.
We are now ready to define the set C that we are looking for. Let
0
S1,2 = S10 ∪ S20 .
0
Imagine that we have the points of this set S1,2 in increasing order of their
0
y-coordinates. Consider an arbitrary point r of S1,2 . The seven y-successors
0
of r are the seven points of S1,2 that immediately follow r in this increasing
order. In the figure below, these are the points a, b, . . . , g.
δ δ
g
f
e
d
c
b
a
r
`1 ` `2
4.7. Computing the Closest Pair 111
Observe that the number of points that follow r may be less than seven.
In this case, we abuse our terminology a bit and still talk about the seven
y-successors of r, even though there are fewer of them.
Our final set C is defined as follows:
C = {(r, s) : r, s ∈ S10 ∪ S20 , s is one of the seven y-successors of r}. (4.9)
Lemma 4.7.3 The set C is a superset of the set A, i.e., A ⊆ C.
Proof. We will prove that B ⊆ C. It will then follow from Lemma 4.7.1
that A ⊆ C.
Let (p, q) be an arbitrary element in the set B1 . It follows from the
definition of B1 that p ∈ S10 , q ∈ S20 , and q ∈ Rp . To prove that (p, q) is an
element of the set C, we have to argue that q is one of the seven y-successors
of p.
As in the proof of Lemma 4.7.2, (i) the part of Rp that is to the left of
the line ` contains at most four points of S10 and (ii) the part of Rp that is
to the right of ` contains at most four points of S20 . Thus, the rectangle Rp
contains at most eight points of S10 ∪ S20 . Since p is one of them and p is on
the bottom side of Rp , the point q must be one of the seven y-successors of p.
Thus, we have shown that B1 ⊆ C. By a symmetric argument, B2 ⊆ C.
Consider the elements (r, s) of the set C. There are at most n choices
for the point r. For each choice of r, there are at most seven choices for the
point s. This proves the following lemma:
Lemma 4.7.4 The size of the set C is at most 7n.
4.7.2 The Recursive Algorithm
Consider a set of n points in R2 . We make the same assumptions as in
Section 4.7.1. Thus, n ≥ 2, n is a power of two, no two points have the same
x-coordinate, and no two points have the same y-coordinate. Our goal is to
compute the closest-pair distance in this point set. The base case, i.e., when
n = 2, is easy. Assume that n ≥ 4. In Section 4.7.1, we have seen that the
algorithm will make the following steps:
112 Chapter 4. Recursion
Step 1: Determine a vertical line ` that splits the point set into two subsets,
each having size n/2. This step is easy to perform if we have the points in
sorted order of their x-coordinates.
Steps 2 and 3: Run the algorithm recursively, once for all points to the left
of `, and once for all points to the right of `.
Step 4: Compute and traverse the set C that is defined in (4.9). This step is
easy to perform if we have the points in sorted order of their y-coordinates.
We assume that the set of input points is stored in a list L. The entire
algorithm, which we denote by ClosestPair(L, n), is given in Figure 4.1. In
the pseudocode, Merge(·, ·, y) refers to the merge algorithm of Section 4.6
that merges two lists, based on the y-coordinates of the points.
• The input to the call ClosestPair(L, n) is a list L that stores n points
in R2 , where n ≥ 2 and n is a power of two. This list stores the points
in increasing order of their x-coordinates.
• The call ClosestPair(L, n) returns the closest-pair distance between
any two distinct points that are stored in L.
• At termination, the list L stores the same points, but in sorted order
of their y-coordinates.
The algorithm starts by checking if it is in the base case. Clearly, this
base case is easy to handle. Assume that the algorithm is not in the base
case, i.e., n ≥ 4.
• Since L stores the input points in sorted order of their x-coordinates,
the algorithm obtains the lists L1 and L2 by a simple traversal of L.
Observe that, at this moment, the points in both lists L1 and L2 are
sorted by their x-coordinates. The value z that is chosen by the algo-
rithm is the x-coordinate of the vertical line `.
• In the first recursive call ClosestPair(L1 , n/2), the algorithm recur-
sively computes the closest-pair distance δ1 in L1 , whereas in the second
recursive call ClosestPair(L2 , n/2), it computes the closest-pair dis-
tance δ2 in L2 . After these two recursive calls have terminated, the
points in both lists L1 and L2 are sorted by their y-coordinates.
4.7. Computing the Closest Pair 113
Algorithm ClosestPair(L, n):
if n = 2
then δ = the distance between the two points in L;
sort the points in L by their y-coordinates;
return δ
else L1 = list consisting of the first n/2 points in L;
L2 = list consisting of the last n/2 points in L;
z = any value between the x-coordinates of the last point
of L1 and the first point of L2 ;
// both L1 and L2 are sorted by x-coordinate
δ1 = ClosestPair(L1 , n/2);
δ2 = ClosestPair(L2 , n/2);
// both L1 and L2 are sorted by y-coordinate
δ = min(δ1 , δ2 );
L01 = list consisting of all points p of L1 with p1 > z − δ;
L02 = list consisting of all points q of L2 with q1 < z + δ;
// both L01 and L02 are sorted by y-coordinate
L01,2 = Merge (L01 , L02 , y);
L = Merge (L1 , L2 , y);
// both L01,2 and L are sorted by y-coordinate
if L01,2 is empty
then return δ
0
else δ1,2 = min{d(r, s) : r, s ∈ L01,2 , s is one of the seven
y-successors of r};
0
return min(δ, δ1,2 )
endif
endif;
Figure 4.1: The recursive closest pair algorithm.
114 Chapter 4. Recursion
• By simple traversals of the lists L1 and L2 , the algorithm computes
the lists L01 and L02 . Observe that L01 stores all points of L1 that are
to the right of the vertical line `1 that is at distance δ to the left of `.
Similarly, L02 stores all points of L2 that are to the left of the vertical
line `2 that is at distance δ to the right of `.
Since both lists L01 and L02 are in sorted y-order, the algorithm can
use algorithm Merge (L01 , L02 , y) to merge these two lists into one list
L01,2 that is also in sorted y-order. Similarly, the algorithm can run
Merge (L1 , L2 , y) to merge the two lists L1 and L2 into one list L that
is in sorted y-order.
In the final step, if the list L01,2 is non-empty, the algorithm computes
0
the value of δ1,2 using a nested for-loop: The outer-loop iterates over
0
all points r of L1,2 . For each such point r, the inner-loop iterates over
the seven successors of r in the list L01,2 .
Note that the input list L must contain the points in sorted order of
their x-coordinates. Therefore, before the first call to ClosestPair, we run
algorithm MergeSort(L, n) of Section 4.6 to sort the input points by their
x-coordinates. By Theorem 4.6.1, this takes O(n log n) time.
We now analyze the running time of algorithm ClosestPair(L, n). Let
T (n) denote the worst-case running time of this algorithm, when given as
input a list of size n whose points are in sorted x-order. If n = 2, then
the running time is bounded by some constant, say c. If n ≥ 2, then the
algorithm spends 2 · T (n/2) time for the two recursive calls, whereas the rest
of the algorithm takes at most c0 n time, where c0 is some constant. Thus,
the function T (n) satisfies the following recurrence:
T (1) ≤ c,
T (n) ≤ 2 · T (n/2) + c0 n, if n ≥ 2 and n is a power of 2.
As in Section 4.6.2, this recurrence solves to T (n) = O(n log n). Thus, we
have proved the following result:
Theorem 4.7.5 For any list L of n points in R2 , algorithm ClosestPair(L, n)
computes their closest-pair distance in O(n log n) time.
4.8. Counting Regions when Cutting a Circle 115
4.8 Counting Regions when Cutting a Circle
Take a circle, place n points on it, and connect each pair of points by a
straight-line segment. The points must be placed in such a way that no
three segments pass through one point. These segments divide the circle into
regions. Define Rn to be the number of such regions. Can we determine Rn ?
By looking at the figure above, we see that
R1 = 1, R2 = 2, R3 = 4, R4 = 8, R5 = 16.
There seems to be a clear pattern and it is natural to guess that Rn is equal
to 2n−1 for all n ≥ 1. To prove this, we have to argue that the number of
regions doubles if we increase n by 1. If you try to do this, however, then
you will fail! The reason is that Rn is not equal to 2n−1 for all n ≥ 1; our
guess was correct only for 1 ≤ n ≤ 5.
We will prove below that Rn grows only polynomially in n. This will
imply that Rn cannot be equal to 2n−1 for all n, because the latter function
grows exponentially.
4.8.1 A Polynomial Upper Bound on Rn
Let n be a (large) integer,
consider a placement of n points on a circle, and
n
connect each of the 2 pairs of points by a straight-line segment. Recall
that we assume that no three segments pass through one point. We define
the following graph:
• Each of the n points on the circle is a vertex.
• Each intersection point between two segments is a vertex.
116 Chapter 4. Recursion
• These vertices divide the segments into subsegments and the circle into
arcs in a natural way. Each such subsegment and arc is an edge of the
graph.
The figure below illustrates this for the case when n = 5. The graph on
the right has 10 = 5 + 5 vertices: Each of the 5 points on the circle leads to
one vertex and each of the 5 intersection points leads to one vertex. These
5
10 vertices divide the 2 = 10 segments into 20 straight-line edges and the
circle into 5 circular edges. Therefore, the graph has 20 + 5 = 25 edges.
Note that, strictly speaking, this process does not define a proper graph,
because any two consecutive vertices on the circle are connected by two edges
(one straight-line edge and one circular edge), whereas in a proper graph,
there can be only one edge between any pair of vertices. For simplicity,
however, we will refer to the resulting structure as a graph.
Let Vn and En be the number of vertices and edges of the graph, respec-
tively. We claim that
n
Vn ≤ n + 2 . (4.10)
2
This claim follows from the following observations:
• There are exactly n vertices on the circle.
• The n points on the circle are connected by n2 segments, and any
two such segments intersect at most once. Therefore, the number of
vertices inside the circle is at most the number of pairs of segments.
The latter quantity is equal to
n
2 .
2
We next claim that
Vn
En ≤ n + . (4.11)
2
This claim follows from the following observations:
4.8. Counting Regions when Cutting a Circle 117
• There are exactly n edges on the circle.
• Any straight-line edge joins two vertices. Therefore, the number of
straight-line edges is at most the number of pairs of vertices, which is
Vn
2
.
The final claim is that
Rn ≤ En . (4.12)
To prove this claim, we do the following. For each region r, choose a point
pr inside r, such that the y-coordinate of pr is not equal to the y-coordinate
of any vertex. Let f (r) be the first edge that is reached when walking from
pr horizontally to the right.
r
pr
f (r)
This defines a one-to-one function f from the set of regions to the set of
edges. Therefore, the number of regions, which is Rn , is at most the number
of edges, which is En .
By combining (4.10), (4.11), and (4.12), we get
Rn ≤ En
Vn
≤ n+
2
n + (22 )
n
≤ n+ .
2
In order to estimate the last quantity, we are going to use asymptotic nota-
tion; see Section 2.3. First observe that
n(n − 1)
n
= = O(n2 ).
2 2
118 Chapter 4. Recursion
This implies that n
O(n2 )
2 = = O(n4 ),
2 2
which implies that
n
n+ 2 = n + O(n4 ) = O(n4 ),
2
which implies that
n + (22 )
n
O(n4 )
= = O(n8 ),
2 2
which implies that
n + (22 )
n
Rn ≤ n + = n + O(n8 ) = O(n8 ).
2
Thus, we have proved our claim that Rn grows polynomially in n and, there-
fore, for large values of n, Rn is not equal to 2n−1 . (Using results on pla-
nar graphs that we will see in Section 7.5.1, it can be shown that, in fact,
Rn = O(n4 ).)
We remark that there is a shorter way to prove that Rn is not equal to
2n−1 for all n ≥ 1: You can verify by hand that R6 = 31. Still, this single
example does not rule out the possibility that Rn grows exponentially. The
analysis that we gave above does rule this out.
We have proved above that Rn = O(n8 ). We also mentioned that this
upper bound can be improved to O(n4 ). In the following subsections, we
will prove that the latter upper bound cannot be improved. That is, we will
prove that Rn = Θ(n4 ). In fact, we will determine an exact formula, in terms
of n, for the value of Rn .
4.8.2 A Recurrence Relation for Rn
Let n ≥ 2 be an integer and consider a placement of n points on a circle. We
denote these points by p1 , p2 , . . . , pn and assume that they are
numbered in
n
counterclockwise order. As before, we connect each of the 2 pairs of points
by a straight-line segment. We assume that no three segments pass through
one point. We are going to derive a recurrence relation for the number Rn
of regions in the following way:
4.8. Counting Regions when Cutting a Circle 119
• Remove all segments that have pn as an endpoint. At this moment, the
number of regions is, by definition, equal to Rn−1 .
• Add the n − 1 line segments p1 pn , p2 pn , . . . , pn−1 pn one by one. For
each segment pk pn added, determine the increase Ik in the number of
regions.
• Take the sum of Rn−1 and all increases Ik , i.e.,
n−1
X
Rn−1 + Ik .
k=1
This sum is equal to Rn , because in the entire process, we have counted
each of the regions for n points exactly once.
• Thus, together with the base case R1 = 1, we obtain a recurrence
relation for the values Rn .
We start by illustrating this process for the case when n = 6. The figure
below shows the situation after we have removed all segments that have p6
as an endpoint. The number of regions is equal to R5 = 16.
p2 p1
p6
p3
p4 p5
We are going to add, one by one, the five segments that have p6 as an
endpoint. When we add p1 p6 , one region gets cut into two. Thus, the number
of regions increases by one. Using the notation introduced above, we have
I1 = 1.
p2 p1
p6
p3
p4 p5
120 Chapter 4. Recursion
When we add p2 p6 , four regions get cut into two. Thus, the number of
regions increases by four, and we have I2 = 4.
p2 p1
p6
p3
p4 p5
When we add p3 p6 , five regions get cut into two. Thus, the number of
regions increases by five, and we have I3 = 5.
p2 p1
p6
p3
p4 p5
When we add p4 p6 , four regions get cut into two. Thus, the number of
regions increases by four, and we have I4 = 4.
p2 p1
p6
p3
p4 p5
Finally, when we add p5 p6 , one region gets cut into two. Thus, the number
of regions increases by one, and we have I5 = 1.
p2 p1
p6
p3
p4 p5
4.8. Counting Regions when Cutting a Circle 121
After having added the five segments with endpoint p6 , we have accounted
for all regions determined by the six points. In other words, the number of
regions we have at the end is equal to R6 . Since the number of regions at
the end is also equal to the sum of (i) the number of regions we started with,
which is R5 , and (ii) the total increase, we have
R6 = R5 + I1 + I2 + I3 + I4 + I5 = 31.
Let us look at this more carefully. We have seen that I3 = 5. That is,
when adding the segment p3 p6 , the number of regions increases by 5. Where
does this number 5 come from? The segment p3 p6 intersects 4 segments,
namely p1 p4 , p1 p5 , p2 p4 , and p2 p5 . The increase in the number of regions is
one more than the number of intersections. Thus, when adding a segment,
if we determine the number X of intersections between this new segment
and existing segments, then the increase in the number of regions is equal to
1 + X.
When we add p3 p6 , we have X = 4. Where does this number 4 come
from? We make the following observations:
• Any segment that intersects p3 p6 has one endpoint above p3 p6 and one
endpoint below p3 p6 .
• Any pair (a, b) of points on the circle, with a above p3 p6 and b below
p3 p6 , defines a segment ab that intersects p3 p6 .
• Thus, the value of X is equal to the number of pairs (a, b) of points in
{p1 , p2 , p4 , p5 }, where a is above p3 p6 and b is below p3 p6 . Since there
are 2 choices for a (viz., p1 and p2 ) and 2 choices for b (viz., p4 and p5 ),
it follows from the Product Rule that X = 2 · 2 = 4.
Now that we have seen the basic approach, we are going to derive the
recurrence relation for Rn for an arbitrary integer n ≥ 2. After having
removed all segments that have pn as an endpoint, we have Rn−1 regions.
For each integer k with 1 ≤ k ≤ n − 1, we add the segment pk pn . What is
the number of existing segments that are intersected by this new segment?
122 Chapter 4. Recursion
pi
pk−1
p1
pk pn
pk+1
pn−1
pj
We observe that for i < j,
• pi pj intersects pk pn if and only if 1 ≤ i ≤ k − 1 and k + 1 ≤ j ≤ n − 1.
Since there are k − 1 choices for i and n − k − 1 choices for j, the Product
Rule implies that the number of intersections due to pk pn is equal to (k −
1)(n − k − 1). Thus, the segment pk pn goes through 1 + (k − 1)(n − k − 1)
regions, and each of them is cut into two. It follows that, when adding pk pn ,
the increase Ik in the number of regions is equal to
Ik = 1 + (k − 1)(n − k − 1).
We conclude that
n−1
X
Rn = Rn−1 + Ik
k=1
n−1
X
= Rn−1 + (1 + (k − 1)(n − k − 1)) .
k=1
In the summation on the right-hand side
• the term 1 occurs exactly n − 1 times, and
• the term (k − 1)(n − k − 1) is non-zero only if 2 ≤ k ≤ n − 2.
It follows that, for n ≥ 2,
n−2
X
Rn = Rn−1 + (n − 1) + (k − 1)(n − k − 1). (4.13)
k=2
Thus, together with the base case
R1 = 1, (4.14)
we have determined the recurrence relation we were looking for.
4.8. Counting Regions when Cutting a Circle 123
4.8.3 Simplifying the Recurrence Relation
In this subsection, we will use a combinatorial proof (see Section 3.7) to show
that the summation on the right-hand side of (4.13) satisfies
n−2
n−1
X
(k − 1)(n − k − 1) = , (4.15)
k=2
3
for any integer n ≥ 2. (In fact, (4.15) is a special case of the result in
Exercise 3.62.)
If n ∈ {2, 3}, then both sides of (4.15) are equal to zero. Assume that
n ≥ 4 and consider the set S = {1, 2, . . . , n−1}. We know that the number of
3-element subsets of S is equal to n−13
. As we will see below, the summation
on the left-hand side of (4.15) counts exactly the same subsets.
We divide the 3-element subsets of S into groups based on their mid-
dle element. Observe that the middle element can be any of the values
2, 3, . . . , n − 2. Thus, for any k with 2 ≤ k ≤ n − 2, the k-th group Gk
consists of all 3-element subsets of S whose middle element is equal to k.
Since the groups are pairwise disjoint, we have
n−2
n−1
X
= |Gk |.
3 k=2
What is the size of the k-th group Gk ? Any 3-element subset in Gk consists
of
• one element from {1, 2, . . . , k − 1},
• the element k, and
• one element from {k + 1, k + 2, . . . , n − 1}.
It then follows from the Product Rule that
|Gk | = (k − 1) · 1 · (n − k − 1) = (k − 1)(n − k − 1).
Thus, we have proved the identity in (4.15), and the recurrence relation in
(4.13) and (4.14) becomes
R1 = 1,
n−1
(4.16)
Rn = Rn−1 + (n − 1) + 3
, if n ≥ 2.
124 Chapter 4. Recursion
4.8.4 Solving the Recurrence Relation
Now that we have a recurrence relation that looks reasonable, we are going
to apply the unfolding technique of Section 4.6 to solve it. Let n ≥ 2 be an
integer. By repeatedly applying the recurrence relation in (4.16), we get
n−1
Rn = (n − 1) + + Rn−1
3
n−1 n−2
= (n − 1) + (n − 2) + + + Rn−2
3 3
n−1 n−2 n−3
= (n − 1) + (n − 2) + (n − 3) + + +
3 3 3
+ Rn−3 .
By continuing, we get
Rn = (n − 1) + (n − 2) + (n − 3) + · · · + 3 + 2 + 1
n−1 n−2 n−3
3 2 1
+ + + + ··· + + +
3 3 3 3 3 3
+ R1 .
Since 23 = 13 = 0 and R1 = 1, we get
Rn = (n − 1) + (n − 2) + (n − 3) + · · · + 3 + 2 + 1
n−1 n−2 n−3
3
+ + + + ··· +
3 3 3 3
+ 1.
Since, by Theorem 2.2.10, the first summation is equal to
n
1 + 2 + 3 + · · · + (n − 1) = n(n − 1)/2 = ,
2
we get
Xn−1
n k
Rn = 1 + + .
2 k=3
3
The final step is to simplify the summation on the right-hand side. We
will use a combinatorial proof to show that
n−1
X k n
= , (4.17)
k=3
3 4
4.9. Exercises 125
for any integer n ≥ 2. (As was the case for (4.15), the identity in (4.17) is a
special case of the result in Exercise 3.62.)
If n ∈ {2, 3}, then both sides of (4.17) are equal to zero. Assume that
n ≥ 4 and consider all 4-element subsets of the set S = {1, 2, . . . , n}. We
know that there are n4 many such subsets. We divide these subsets into
groups based on their largest element. For any k with 3 ≤ k ≤ n − 1, the
k-th group Gk consists of all 4-element subsets of S whose largest element is
equal to k + 1. It should be clear that
X n−1
n
= |Gk |.
4 k=3
To determine the size of the group Gk , we observe that any 4-element subset
in Gk consists of
• three elements from {1, 2, . . . , k} and
• the element k + 1.
It then follows from the Product Rule that
k k
|Gk | = ·1= ,
3 3
completing the proof of (4.17).
After (finally!) having solved and simplified our recurrence relation, we
conclude that for any integer n ≥ 1,
n n
Rn = 1 + + .
2 4
In Exercise 4.76, you will see a shorter way to determine the exact value
of Rn . We went for the long derivation, because it allowed us to illustrate,
along the way, several techniques from previous sections.
4.9 Exercises
4.1 The function f : N → Z is recursively defined as follows:
f (0) = 7,
f (n) = f (n − 1) + 6n − 3 if n ≥ 1.
Prove that f (n) = 3n2 + 7 for all integers n ≥ 0.
126 Chapter 4. Recursion
4.2 The function f : N → Z is recursively defined as follows:
f (0) = −18,
f (n) = 9(n − 2)(n − 3) + f (n − 1) if n ≥ 1.
Prove that
f (n) = 3(n − 1)(n − 2)(n − 3)
for all integers n ≥ 0.
4.3 The function f : N → Z is recursively defined as follows:
f (0) = 3,
f (n) = 2 · f (n − 1) − (f (n − 1))2 if n ≥ 1.
n n
Prove that f (n) = 1 − 22 for all integers n ≥ 1. (Note that 22 denotes 2 to
the power of 2n .)
4.4 The function f : N → N is defined by
f (0) = 1,
f (n) = 21 · 4n · f (n − 1) if n ≥ 1.
Prove that for every integer n ≥ 0,
2
f (n) = 2n ;
this reads as 2 to the power n2 .
4.5 The function f : N → N is defined by
f (0) = 0,
f (1) = 0,
f (n) = f (n − 2) + 2n−1 if n ≥ 2.
• Prove that for every even integer n ≥ 0,
2n+1 − 2
f (n) = .
3
• Prove that for every odd integer n ≥ 1,
2n+1 − 4
f (n) = .
3
4.9. Exercises 127
4.6 The function f : {1, 2, 3, . . .} → R is defined by
f (1) = 2,
f (n) = 12 f (n − 1) + 1
f (n−1)
if n ≥ 2.
• Prove that for every integer n ≥ 1,
n−1
32 + 1
f (n) = 2n−1 .
3 −1
n−1
Note that 32 denotes 3 to the power of 2n−1 .
4.7 You are asked to come up with an exam question about recursive func-
tions. You write down some recurrence, which you then solve. Afterwards,
you give the recurrence to the students, together with the solution. The
students must then prove that the given solution is indeed correct.
This is a painful process, because you must solve the recurrence yourself.
Since you are lazy, you start with the following:
Exam Question:
The function f : N → N is defined by
f (0) = XXX,
f (n) = f (n − 1) + Y Y Y if n ≥ 1.
Prove that for every integer n ≥ 0,
f (n) = 7n2 − 2n + 9.
• Complete the question, i.e., fill in XXX and Y Y Y , so that you obtain
a complete recurrence that has the given solution.
4.8 The function f : N → Z is defined by
f (n) = 2n(n − 6)
for each integer n ≥ 0. Derive a recursive form of this function.
128 Chapter 4. Recursion
4.9 The function f : N2 → N is defined by
f (0, n) = 2n if n ≥ 0,
f (m, 0) = 0 if m ≥ 1,
f (m, 1) = 2 if m ≥ 1,
f (m, n) = f (m − 1, f (m, n − 1)) if m ≥ 1 and n ≥ 2.
• Determine f (2, 2).
• Determine f (1, n) for n ≥ 1.
• Determine f (3, 3).
4.10 The function f : N3 → N is defined as follows:
f (k, n, 0) = k+n if k ≥ 0 and n ≥ 0,
f (k, 0, 1) = 0 if k ≥ 0,
f (k, 0, 2) = 1 if k ≥ 0,
f (k, 0, i) = k if k ≥ 0 and i ≥ 3,
f (k, n, i) = f (k, f (k, n − 1, i), i − 1) if k ≥ 0, i ≥ 1, and n ≥ 1.
Determine f (2, 3, 2).
4.11 The functions f : N → N and g : N2 → N are recursively defined as
follows:
f (0) = 1,
f (n) = g(n, f (n − 1)) if n ≥ 1,
g(m, 0) = 0 if m ≥ 0,
g(m, n) = m + g(m, n − 1) if m ≥ 0 and n ≥ 1.
Solve these recurrence relations for f , i.e., express f (n) in terms of n.
4.12 The functions f : N → N and g : N2 → N are recursively defined as
follows:
f (0) = 1,
f (1) = 2,
f (n) = g(f (n − 2), f (n − 1)) if n ≥ 2,
g(m, 0) = 2m if m ≥ 0,
g(m, n) = g(m, n − 1) + 1 if m ≥ 0 and n ≥ 1.
Solve these recurrence relations for f , i.e., express f (n) in terms of n.
4.9. Exercises 129
4.13 The functions f : N → N and g : N2 → N are recursively defined as
follows:
f (0) = 1,
f (n) = g(f (n − 1), 2n) if n ≥ 1,
g(0, n) = 0 if n ≥ 0,
g(m, n) = g(m − 1, n) + n if m ≥ 1 and n ≥ 0.
Solve these recurrence relations for f , i.e., express f (n) in terms of n.
4.14 The functions f : N → N, g : N2 → N, and h : N → N are recursively
defined as follows:
f (n) = g(n, h(n)) if n ≥ 0,
g(m, 0) = 0 if m ≥ 0,
g(m, n) = g(m, n − 1) + m if m ≥ 0 and n ≥ 1,
h(0) = 1,
h(n) = 2 · h(n − 1) if n ≥ 1.
Solve these recurrences for f , i.e., express f (n) in terms of n.
4.15 The sequence an of numbers, for n ≥ 0, is recursively defined as follows:
a0 = 5,
a1 = 3,
an = 6 · an−1 − 9 · an−2 if n ≥ 2.
• Determine an for n = 0, 1, 2, 3, 4, 5.
• Prove that for every integer n ≥ 0,
an = (5 − 4n) · 3n .
√ √
4.16 Let ϕ = 1+2 5 and ψ = 1− 5
2
, and let n ≥ 0 be an integer. We have
seen in Theorem 4.2.1 that
ϕn − ψ n
√ (4.18)
5
is equal to the n-th Fibonacci number fn . Since the Fibonacci numbers are
obviously integers, the number in (4.18) is an integer as well.
Prove that the number in (4.18) is a rational number using only Newton’s
Binomial Theorem (i.e., Theorem 3.6.5).
130 Chapter 4. Recursion
4.17 In Section 4.2, we have defined the Fibonacci numbers f0 , f1 , f2 , . . . In
this exercise, you will prove that there exists a Fibonacci number whose 2018
rightmost digits (when written in decimal notation) are all zero.
In the rest of this exercise, N denotes the number 104036 . For any integer
n ≥ 0, define
gn = fn mod 102018 .
• Consider the ordered pairs (gn , gn+1 ), for n = 0, 1, 2, . . . , N . Use the
Pigeonhole Principle to prove that these ordered pairs cannot all be
distinct. That is, prove that there exist integers m ≥ 0, p ≥ 1, such
that m + p ≤ N and
(gm , gm+1 ) = (gm+p , gm+p+1 ).
• Prove that (gm−1 , gm ) = (gm+p−1 , gm+p ).
• Prove that (g0 , g1 ) = (gp , gp+1 ).
• Consider the decimal representation of fp . Prove that the 2018 right-
most digits of fp are all zero.
• Let b ≥ 2 and k ≥ 1 be integers. Prove that there exists a Fibonacci
number whose k rightmost digits (when written in base-b notation) are
all zero.
4.18 In Section 4.2, we have defined the Fibonacci numbers f0 , f1 , f2 , . . .
Prove that for each integer n ≥ 1,
n
X
f2i = f2n+1 − 1
i=1
and
f12 + f22 + f32 + · · · + fn2 = fn fn+1 .
4.19 In Section 4.2, we have defined the Fibonacci numbers f0 , f1 , f2 , . . .
Prove that for each integer n ≥ 0,
• f3n is even,
• f3n+1 is odd,
4.9. Exercises 131
• f3n+2 is odd,
• f4n is a multiple of 3.
4.20 In Section 4.2, we have defined the Fibonacci numbers f0 , f1 , f2 , . . . In
Section 4.2.1, we have seen that for any integer m ≥ 1, the number of 00-free
bitstrings of length m is equal to fm+2 .
Let n ≥ 2 be an integer.
• How many 00-free bitstrings of length n do not contain any 0?
• How many 00-free bitstrings of length n have the following property:
The rightmost 0 is at position 1.
• How many 00-free bitstrings of length n have the following property:
The rightmost 0 is at position 2.
• Let k be an integer with 3 ≤ k ≤ n. How many 00-free bitstrings of
length n have the following property: The rightmost 0 is at position k.
• Use the previous results to prove that
n
X
fn+2 = 1 + fk .
k=1
4.21 In Section 4.2, we have defined the Fibonacci numbers f0 , f1 , f2 , . . . In
Section 4.2.1, we have seen that for any integer m ≥ 1, the number of 00-free
bitstrings of length m is equal to fm+2 .
• Let n ≥ 2 be an integer. What is the number of 00-free bitstrings of
length 2n − 1 for which the bit in the middle position is equal to 1?
• Let n ≥ 3 be an integer. What is the number of 00-free bitstrings of
length 2n − 1 for which the bit in the middle position is equal to 0?
• Use the previous results to prove that for any integer n ≥ 3,
f2n+1 = fn2 + fn+1
2
.
132 Chapter 4. Recursion
4.22 In Section 4.2, we have defined the Fibonacci numbers f0 , f1 , f2 , . . . In
Section 4.2.1, we have seen that for any integer m ≥ 1, the number of 00-free
bitstrings of length m is equal to fm+2 .
Let n ≥ 1 be an integer.
• How many 00-free bitstrings of length n + 2 do not contain any 0?
• How many 00-free bitstrings of length n + 2 contain exactly one 0?
• How many 00-free bitstrings of length n+2 have the following property:
The bitstring contains at least two 0’s, and the second rightmost 0 is
at position 1.
• How many 00-free bitstrings of length n+2 have the following property:
The bitstring contains at least two 0’s, and the second rightmost 0 is
at position 2.
• Let k be an integer with 3 ≤ k ≤ n. How many 00-free bitstrings of
length n+2 have the following property: The bitstring contains at least
two 0’s, and the second rightmost 0 is at position k.
• Let k be an element of {n + 1, n + 2}. How many 00-free bitstrings
of length n + 2 have the following property: The bitstring contains at
least two 0’s, and the second rightmost 0 is at position k.
• Use the previous results to prove that
n
X
(n − k + 1) · fk = fn+4 − n − 3,
k=1
i.e.,
n · f1 + (n − 1) · f2 + (n − 2) · f3 + · · · + 2 · fn−1 + 1 · fn = fn+4 − n − 3.
4.23 Use basic algebra to prove that
2
x2 + y 2 + (x + y)2 = 2 x4 + y 4 + (x + y)4 .
In Section 4.2, we have defined the Fibonacci numbers f0 , f1 , f2 , . . . Prove
that for each integer n ≥ 0,
2
fn2 + fn+1
2 2
= 2 fn4 + fn+1
4 4
+ fn+2 + fn+2 .
4.9. Exercises 133
4.24 Let n ≥ 1 be an integer and consider a 2 × n board Bn consisting of 2n
square cells. The top part of the figure below shows B13 .
A brick is a horizontal or vertical board consisting of 2 square cells; see
the bottom part of the figure above. A tiling of the board Bn is a placement
of bricks on the board such that
• the bricks exactly cover Bn and
• no two bricks overlap.
The figure below shows a tiling of B13 .
For n ≥ 1, let Tn be the number of different tilings of the board Bn .
Determine the value of Tn , i.e., express Tn in terms of numbers that we have
seen in this chapter.
4.25 Let n be a positive integer and consider a 5 × n board Bn consisting
of 5n cells, each one having sides of length one. The top part of the figure
below shows B12 .
134 Chapter 4. Recursion
A brick is a horizontal or vertical board consisting of 2 × 3 = 6 cells; see
the bottom part of the figure above. A tiling of the board Bn is a placement
of bricks on the board such that
• the bricks exactly cover Bn and
• no two bricks overlap.
The figure below shows a tiling of B12 .
Let Tn be the number of different tilings of the board Bn .
• Let n ≥ 6 be a multiple of 6. Determine the value of Tn .
• Let n be a positive integer that is not a multiple of 6. Prove that
Tn = 0.
4.26 Let n be a positive integer and consider a 1 × n board Bn consisting of
n cells, each one having sides of length one. The top part of the figure below
shows B9 .
R B W Y G
We have an unlimited supply of bricks, which are of the following types
(see the bottom part of the figure above):
• There are red (R) and blue (B) bricks, both of which are 1 × 1 cells.
• There are white (W ), yellow (Y ), and green (G) bricks, all of which
are 1 × 2 cells.
A tiling of the board Bn is a placement of bricks on the board such that
4.9. Exercises 135
• the bricks exactly cover Bn and
• no two bricks overlap.
In a tiling, a color can be used more than once and some colors may not be
used at all. The figure below shows a tiling of B9 , in which each color is used
and the color red is used twice.
B W R G R Y
Let Tn be the number of different tilings of the board Bn .
• Determine T1 and T2 .
• Let n ≥ 3 be an integer. Prove that
Tn = 2 · Tn−1 + 3 · Tn−2 .
• Prove that for any integer n,
2(−1)n−1 + 3(−1)n−2 = (−1)n .
• Prove that for any integer n ≥ 1,
3n+1 + (−1)n
Tn = .
4
4.27 The sequence of numbers an , for n ≥ 0, is recursively defined as follows:
a0 = 0,
a1 = 1,
an = 2 · an−1 + an−2 if n ≥ 2.
• Determine an for n = 0, 1, 2, 3, 4, 5.
• Prove that √ n √ n
1+ 2 − 1− 2
an = √ (4.19)
2 2
for all integers n ≥ 0.
Hint: What are the solutions of the equation x2 = 2x + 1?
136 Chapter 4. Recursion
• Since the numbers an , for n ≥ 0, are obviously integers, the fraction on
the right-hand side of (4.19) is an integer as well. Prove that the frac-
tion on the right-hand side of (4.19) is an integer using only Newton’s
Binomial Theorem (i.e., Theorem 3.6.5).
4.28 Let n be a positive integer and consider a 1 × n board Bn consisting of
n cells, each one having sides of length one. The top part of the figure below
shows B9 .
R B G
You have an unlimited supply of bricks, which are of the following types
(see the bottom part of the figure above):
• There are red (R) and blue (B) bricks, both of which are 1 × 1 cells.
We refer to these bricks as squares.
• There are green (G) bricks, which are 1 × 2 cells. We refer to these as
dominoes.
A tiling of the board Bn is a placement of bricks on the board such that
• the bricks exactly cover Bn and
• no two bricks overlap.
In a tiling, a color can be used more than once and some colors may not be
used at all. The figure below shows an example of a tiling of B9 .
G B B R B G R
Let Tn be the number of different tilings of the board Bn .
• Determine T1 , T2 , and T3 .
• For any integer n ≥ 1, express Tn in terms of the numbers that appear
in Exercise 4.27 .
4.9. Exercises 137
4.29 In this exercise, we use the notation of Exercise 4.28. Let n ≥ 1 be an
integer and consider the 1 × n board Bn .
• Consider strings consisting of characters, where each character is S or
D. Let k be an integer with 0 ≤ k ≤ bn/2c. Determine the number of
such strings of length n − k, that contain exactly k many D’s.
• Let k be an integer with 0 ≤ k ≤ bn/2c. Determine the number of
tilings of the board Bn that use exactly k dominoes.
Hint: How many bricks are used for such a tiling? In the first part,
imagine that S stands for “square” and D stands for “domino”.
• Use the results of the previous part to prove that
bn/2c
n−k
X
Tn = · 2n−2k .
k=0
k
4.30 In this exercise, we consider strings of characters, where each character
is an element of {a, b, c}. For any integer n ≥ 1, let En be the number of
such strings of length n that have an even number of c’s, and let On be the
number of such strings of length n that have an odd number of c’s. (Recall
that 0 is even.)
• Determine E1 , O1 , E2 , and O2 .
• Explain, in plain English, why
En + On = 3n .
• Prove that for every integer n ≥ 2,
En = 2 · En−1 + On−1 .
• Prove that for every integer n ≥ 1,
1 + 3n
En = .
2
138 Chapter 4. Recursion
4.31 Consider strings of n characters, where each character is an element of
{a, b, c, d}, that contain an even number of as. (Recall that 0 is even.) Let
En be the number of such strings. Prove that for any integer n ≥ 1,
En+1 = 2 · En + 4n .
4.32 Let An be the number of bitstrings of length n that contain 000. Prove
that for n ≥ 4,
An = An−1 + An−2 + An−3 + 2n−3 .
4.33 Let n ≥ 1 be an integer and define An to be the number of bitstrings
of length n that do not contain 101.
• Determine A1 , A2 , A3 , and A4 .
• Prove that for each integer n ≥ 4,
An = 3 + A1 + A2 + A3 + · · · + An−4 + An−3 + An−1
n−3
X
= 3+ Ak + An−1 .
k=1
Hint: Divide the strings into groups depending on the number of leading 1s.
4.34 Let n ≥ 1 be an integer and consider n people P1 , P2 , . . . , Pn . Let An
be the number of ways these n people can be divided into groups, such that
each group consists of either one or two people.
• Determine A1 , A2 , A3 , and A4 .
• Prove that for each integer n ≥ 3,
An = An−1 + (n − 1) · An−2 .
4.35 In this exercise, we consider strings of characters, where each character
is an element of {a, b, c}. Such a string is called aa-free, if it does not contain
two consecutive a’s. For any integer n ≥ 1, let Fn be the number of aa-free
strings of length n.
• Determine F1 , F2 , and F3 .
4.9. Exercises 139
• Let n ≥ 3 be an integer. Express Fn in terms of Fn−1 and Fn−2 .
• Prove that for every integer n ≥ 1,
√ n √ n
1 1 1 1
Fn = +√ 1+ 3 + −√ 1− 3 .
2 3 2 3
Hint: What are the solutions of the equation x2 = 2x + 2? Using these
solutions will simplify the proof.
4.36 In this exercise, we consider strings of characters, where each character
is an element of {a, b, c}. Such a string is called awesome, if it does not
contain the substring ab and does not contain the substring ba. For any
integer n ≥ 1, let
1. Sn denote the number of awesome strings of length n,
2. An denote the number of awesome strings of length n that start with a,
3. Bn denote the number of awesome strings of length n that start with b,
4. Cn denote the number of awesome strings of length n that start with c.
• Determine S1 and S2 .
• Let n ≥ 1 be an integer. Express Sn in terms of An , Bn , and Cn .
• Let n ≥ 2 be an integer. Express Cn in terms of Sn−1 .
• Let n ≥ 2 be an integer. Prove that
Sn = (Sn−1 − Bn−1 ) + (Sn−1 − An−1 ) + Sn−1 .
• Let n ≥ 3 be an integer. Prove that
Sn = 2 · Sn−1 + Sn−2 .
• Prove that for every integer n ≥ 1,
1 √ n+1 1 √ n+1
Sn = 1+ 2 + 1− 2 .
2 2
Hint: What are the solutions of the equation x2 = 2x + 1? Using these
solutions will simplify the proof.
140 Chapter 4. Recursion
4.37 A block in a bitstring is a maximal consecutive substring of 1’s. For
example, the bitstring 1100011110100111 has four blocks: 11, 1111, 1, and
111. These blocks have lengths 2, 4, 1, and 3, respectively.
Let n ≥ 1 be an integer and let Bn be the number of bitstrings of length
n that do not contain any block of odd length; in other words, every block
in these bitstrings has an even length.
• Determine B1 , B2 , B3 , and B4 .
• Determine the value of Bn , i.e., express Bn in terms of numbers that
we have seen in this chapter.
4.38 Let n ≥ 1 be an integer and let Sn be the number of ways in which n
can be written as a sum of 1s and 2s; the order in which the 1s and 2s occur
in the sum matters. For example, S3 = 3, because
3 = 1 + 1 + 1 = 1 + 2 = 2 + 1.
• Determine S1 , S2 , and S4 .
• Determine the value of Sn , i.e., express Sn in terms of numbers that we
have seen in this chapter.
4.39 Ever since he was a child, Nick has been dreaming to be like Spiderman.
As you all know, Spiderman can climb up the outside of a building; if he is
at a particular floor, then, in one step, he can move up several floors. Nick
is not that advanced yet. In one step, Nick can move up either one floor or
two floors.
Let n ≥ 1 be an integer and consider a building with n floors, numbered
1, 2, . . . , n. (The first floor has number 1; this is not the ground floor.) Nick
is standing in front of this building, at the ground level. There are different
ways in which Nick can climb to the n-th floor. For example, here are three
different ways for the case when n = 5:
1. move up 2 floors, move up 1 floor, move up 2 floors.
2. move up 1 floor, move up 2 floors, move up 2 floors.
3. move up 1 floor, move up 2 floors, move up 1 floor, move up 1 floor.
Let Sn be the number of different ways, in which Nick can climb to the
n-th floor.
4.9. Exercises 141
• Determine, S1 , S2 , S3 , and S4 .
• Determine the value of Sn , i.e., express Sn in terms of numbers that we
have seen in this chapter.
4.40 Let n ≥ 1 be an integer and consider the set Sn = {1, 2, . . . , n}. A non-
neighbor subset of Sn is any subset T of S having the following property: If
k is any element of T , then k + 1 is not an element of T . (Observe that the
empty set is a non-neighbor subset of Sn .)
For example, if n = 3, then {1, 3} is a non-neighbor subset, whereas {2, 3}
is not a non-neighbor subset.
Let Nn denote the number of non-neighbor subsets of the set Sn .
• Determine N1 , N2 , and N3 .
• Determine the value of Nn , i.e., express Nn in terms of numbers that
we have seen in this chapter.
4.41 Let n ≥ 1 be an integer and consider the set S = {1, 2, . . . , n}.
• Assume we arrange the elements of S in sorted order on a horizontal
line. Let Bn be the number of subsets of S that do not contain any
two elements that are neighbors on this line. For example, if n = 4,
then both subsets {1, 3} and {1, 4} are counted in B4 , but neither of
the subsets {2, 3} and {2, 3, 4} is counted.
For each integer n ≥ 1, express Bn in terms of numbers that we have
seen in this chapter.
• Assume we arrange the elements of S in sorted order along a circle. Let
Cn be the number of subsets of S that do not contain any two elements
that are neighbors on this circle. For example, if n = 4, then the subset
{1, 3} is counted in C4 , but neither of the subsets {2, 3} and {1, 4} is
counted.
For each integer n ≥ 4, express Cn in terms of numbers that we have
seen in this chapter.
4.42 For any integer n ≥ 1, a permutation a1 , a2 , . . . , an of the set {1, 2, . . . , n}
is called awesome, if the following condition holds:
142 Chapter 4. Recursion
• For every i with 1 ≤ i ≤ n, the element ai in the permutation belongs
to the set {i − 1, i, i + 1}.
For example, for n = 5, the permutation 2, 1, 3, 5, 4 is awesome, whereas
2, 1, 5, 3, 4 is not an awesome permutation.
Let Pn denote the number of awesome permutations of the set {1, 2, . . . , n}.
• Determine P1 , P2 , and P3 .
• Determine the value of Pn , i.e., express Pn in terms of numbers that
we have seen in this chapter.
Hint: Derive a recurrence relation. What are the possible values for
the last element an in an awesome permutation?
4.43 A block in a bitstring is a maximal consecutive substring of 1’s. For
example, the bitstring 1100011110100111 has four blocks: 11, 1111, 1, and
111.
For a given integer n ≥ 1, consider all 2n bitstrings of length n. Let Bn
be the total number of blocks in all these bitstrings.
For example, the left part of the table below contains all 8 bitstrings of
length 3. Each entry in the rightmost column shows the number of blocks in
the corresponding bitstring. Thus,
B3 = 0 + 1 + 1 + 1 + 1 + 2 + 1 + 1 = 8.
0 0 0 0
0 0 1 1
0 1 0 1
1 0 0 1
0 1 1 1
1 0 1 2
1 1 0 1
1 1 1 1
• Determine B1 and B2 .
• Let n ≥ 3 be an integer.
– Consider all bitstrings of length n that start with 0. What is the
total number of blocks in these bitstrings?
4.9. Exercises 143
– Determine the number of blocks in the bitstring
1| ·{z
· · 1} .
n
– Determine the number of blocks in the bitstring
1| ·{z
· · 1} 0.
n−1
– Let k be an integer with 2 ≤ k ≤ n − 1. Consider all bitstrings of
length n that start with
1| ·{z
· · 1} 0.
k−1
Prove that the total number of blocks in these bitstrings is equal
to
2n−k + Bn−k .
– Prove that
n−1
X
2n−k + Bn−k .
Bn = 2 + Bn−1 +
k=2
– Use 1 + 2 + 22 + 23 + · · · + 2n−2 = 2n−1 − 1, to prove that
Bn = 2n−1 + B1 + B2 + · · · + Bn−1 . (4.20)
• Prove that (4.20) also holds for n = 2.
• Let n ≥ 3. Prove that
Bn = 2n−2 + 2 · Bn−1 . (4.21)
Hint: Write (4.20) on one line. Below this line, write (4.20) with n
replaced by n − 1.
• Prove that for every n ≥ 1,
n+1 n
Bn = ·2 .
4
144 Chapter 4. Recursion
• The derivation of the recurrence in (4.21) was quite involved. Prove
this recurrence in a direct way.
4.44 Let n ≥ 1 be an integer and consider a set S consisting of n elements.
A function f : S → S is called cool, if for all elements x of S,
f (f (f (x))) = x.
Let An be the number of cool functions f : S → S.
• Let f : S → S be a cool function, and let x be an element of S. Prove
that the set
{x, f (x), f (f (x))}
has size 1 or 3.
• Let f : S → S be a cool function, and let x and y be two distinct
elements of S. Assume that f (y) = y. Prove that f (x) 6= y.
• Prove that for any integer n ≥ 4,
An = An−1 + (n − 1)(n − 2) · An−3 .
Hint: Let y be a fixed element in S. Some cool functions f have the
property that f (y) = y, whereas some other cool functions f have the
property that f (y) 6= y.
4.45 Let S be the set of ordered pairs of integers that is recursively defined
in the following way:
• (0, 0) ∈ S.
• If (a, b) ∈ S then (a + 2, b + 3) ∈ S.
• If (a, b) ∈ S then (a + 3, b + 2) ∈ S.
Prove that for every element (a, b) in S, a + b is divisible by 5.
4.46 Let S be the set of integers that is recursively defined in the following
way:
• 4 is an element of S.
4.9. Exercises 145
• If x and y are elements of S, then x + y 2 is an element of S.
Prove that every element of S is divisible by 4.
4.47 Let S be the set of ordered triples of integers that is recursively defined
in the following way:
• (66, 55, 1331) ∈ S.
• If (a, b, c) ∈ S then (a + 7, b + 5, 14a − 10b + c + 24) ∈ S.
Prove that for every element (a, b, c) in S,
a2 − b2 = c.
4.48 Let S be the set of integers that is recursively defined in the following
way:
• 1 is an element of S.
√
• If x is an element of S, then x + 2 x + 1 is also an element of S.
Give a simple description of the set S and prove that your answer is correct.
4.49 The set S of bitstrings is recursively defined in the following way:
• The string 00 is an element of the set S.
• The string 01 is an element of the set S.
• The string 10 is an element of the set S.
• If the string s is an element of the set S, then the string 0s (i.e., the
string obtained by adding the bit 0 at the front of s) is also an element
of the set S.
• If the string s is an element of the set S, then the string 10s (i.e.,
the string obtained by adding the bits 10 at the front of s) is also an
element of the set S.
Let s be an arbitrary string in the set S. Prove that s does not contain the
substring 11.
146 Chapter 4. Recursion
4.50 A binary tree is
• either one single node
• or a node whose left subtree is a binary tree and whose right subtree is
a binary tree.
single node
or binary binary
tree tree
Prove that any binary tree with n leaves has exactly 2n − 1 nodes.
4.51 In this exercise, we will denote Boolean variables by lowercase letters,
such as p and q. A proposition is any Boolean formula that can be obtained
by applying the following recursive rules:
1. For every Boolean variable p, p is a proposition.
2. If f is a proposition, then ¬f is also a proposition.
3. If f and g are propositions, then (f ∨ g) is also a proposition.
4. If f and g are propositions, then (f ∧ g) is also a proposition.
• Let p and q be Boolean variables. Prove that
¬ ((p ∧ ¬q) ∨ (¬p ∨ q))
is a proposition.
• Let ↑ denote the not-and operator. In other words, if f and g are
Boolean formulas, then (f ↑ g) is the Boolean formula that has the
following truth table (0 stands for false, and 1 stands for true):
f g (f ↑ g)
0 0 1
0 1 1
1 0 1
1 1 0
4.9. Exercises 147
– Let p be a Boolean variable. Use a truth table to prove that the
Boolean formulas (p ↑ p) and ¬p are equivalent.
– Let p and q be Boolean variables. Use a truth table to prove that
the Boolean formulas ((p ↑ p) ↑ (q ↑ q)) and p ∨ q are equivalent.
– Let p and q be Boolean variables. Express the Boolean formula
(p ∧ q) as an equivalent Boolean formula that only uses the ↑-
operator. Use a truth table to justify your answer.
• Prove that any proposition can be written as an equivalent Boolean
formula that only uses the ↑-operator.
4.52 In Section 4.4, we have seen the recursive algorithm gossip(n), which
computes a sequence of phone calls for the persons P1 , P2 , . . . , Pn . The base
case for this algorithm was when n = 4. Assume we change the base case to
n = 2: In this new base case, there are only two people P1 and P2 , and only
one phone call is needed. The rest of the algorithm remains unchanged.
Prove that the modified algorithm gossip(n) results in a sequence of
2n − 3 phone calls for any integer n ≥ 2. (Thus, for n ≥ 4, it makes one
more phone call than the algorithm in Section 4.4.)
4.53 In Section 4.4, we have seen the recursive algorithm gossip(n), which
computes a sequence of phone calls for the persons P1 , P2 , . . . , Pn , for any
integer n ≥ 4.
Give an iterative (i.e., non-recursive) version of this algorithm in pseu-
docode. Your algorithm must produce exactly the same sequence of phone
calls as algorithm gossip(n).
4.54 In Section 4.5, we have seen algorithm Euclid(a, b), which takes as
input two integers a and b with a ≥ b ≥ 1, and returns their greatest common
divisior.
Assume we run algorithm Euclid(a, b) with two input integers a and b
that satisfy b > a ≥ 1. What is the output of this algorithm?
4.55 The following recursive algorithm Fib takes as input an integer n ≥ 0
and returns the n-th Fibonacci number fn :
148 Chapter 4. Recursion
Algorithm Fib(n):
if n = 0 or n = 1
then f = n
else f = Fib(n − 1) + Fib(n − 2)
endif;
return f
Let an be the number of additions made by algorithm Fib(n), i.e., the
total number of times the +-function in the else-case is called. Prove that
for all n ≥ 0,
an = fn+1 − 1.
4.56 Consider the following recursive algorithm Beer(n), which takes as
input an integer n ≥ 1:
Algorithm Beer(n):
if n = 1
then eat some peanuts
else choose an arbitrary integer m with 1 ≤ m ≤ n − 1;
Beer(m);
drink one pint of beer;
Beer(n − m)
endif
• Explain why, for any integer n ≥ 1, algorithm Beer(n) terminates.
• Let B(n) be the number of pints of beer you drink when running algo-
rithm Beer(n). Determine the value of B(n).
4.57 Consider the following recursive algorithm Silly, which takes as input
an integer n ≥ 1 which is a power of 2:
4.9. Exercises 149
Algorithm Silly(n):
if n = 1
then drink one pint of beer
else if n = 2
then fart once
else fart once;
Silly(n/2);
fart once
endif
endif
For n a power of 2, let F (n) be the number of times you fart when running
algorithm Silly(n). Determine the value of F (n).
4.58 In the fall term of 2015, Nick took the course COMP 2804 at Carleton
University. Nick was always sitting in the back of the classroom and spent
most of his time eating bananas. Nick uses the following scheme to buy
bananas:
• At the start of week 0, there are 2 bananas in Nick’s fridge.
• For any integer n ≥ 0, Nick does the following during week n:
– At the start of week n, Nick determines the number of bananas in
his fridge and stores this number in a variable x.
– Nick goes to Jim’s Banana Empire, buys x bananas, and puts
them in his fridge.
– Nick takes n + 1 bananas out of his fridge and eats them during
week n.
For any integer n ≥ 0, let B(n) be the number of bananas in Nick’s fridge at
the start of week n. Determine the value of B(n).
4.59 Jennifer loves to drink India Pale Ale (IPA). After a week of hard work,
Jennifer goes to the pub and runs the following recursive algorithm, which
takes as input an integer n ≥ 1, which is a power of 4:
150 Chapter 4. Recursion
Algorithm JenniferDrinksIPA(n):
if n = 1
then place one order of chicken wings
else for k = 1 to 4
do JenniferDrinksIPA(n/4);
drink n pints of IPA
endfor
endif
For n a power of 4, let
• P (n) be the number of pints of IPA that Jennifer drinks when running
algorithm JenniferDrinksIPA(n),
• C(n) be the number of orders of chicken wings that Jennifer places
when running algorithm JenniferDrinksIPA(n).
Determine the values of P (n) and C(n).
4.60 Elisa Kazan loves to drink cider. During the weekend, Elisa goes to
the pub and runs the following recursive algorithm, which takes as input an
integer n ≥ 0:
Algorithm ElisaDrinksCider(n):
if n = 0
then order Fibonachos
else if n is even
then ElisaDrinksCider(n/2);
drink n2 /2 pints of cider;
ElisaDrinksCider(n/2)
else for i = 1 to 4
do ElisaDrinksCider((n − 1)/2);
drink (n − 1)/2 pints of cider
endfor;
drink 1 pint of cider
endif
endif
4.9. Exercises 151
For n ≥ 0, let C(n) be the number of pints of cider that Elisa drinks when
running algorithm ElisaDrinksCider(n). Determine the value of C(n).
4.61 Elisa Kazan loves to drink cider. After a week of bossing the Vice-
Presidents around, Elisa goes to the pub and runs the following recursive
algorithm, which takes as input an integer n ≥ 0:
Algorithm ElisaGoesToThePub(n):
if n = 0
then drink one bottle of cider
else for k = 0 to n − 1
do ElisaGoesToThePub(k);
drink one bottle of cider
endfor
endif
For n ≥ 0, let C(n) be the number of bottles of cider that Elisa drinks
when running algorithm ElisaGoesToThePub(n).
Prove that for every integer n ≥ 1,
C(n) = 3 · 2n−1 − 1.
Hint: 1 + 2 + 22 + 23 + · · · + 2n−2 = 2n−1 − 1.
4.62 Elisa Kazan loves to drink cider. On Saturday night, Elisa goes to her
neighborhood pub and runs the following recursive algorithm, which takes
as input an integer n ≥ 1:
152 Chapter 4. Recursion
Algorithm ElisaDrinksCider(n):
if n = 1
then drink one pint of cider
else if n is even
then ElisaDrinksCider(n/2);
drink one pint of cider;
ElisaDrinksCider(n/2)
else drink one pint of cider;
ElisaDrinksCider(n − 1);
drink one pint of cider
endif
endif
For any integer n ≥ 1, let P (n) be the number of pints of cider that
Elisa drinks when running algorithm ElisaDrinksCider(n). Determine
the value of P (n).
4.63 Let n ≥ 2 be an integer and consider a sequence s1 , s2 , . . . , sn of n
pairwise distinct numbers. The following algorithm computes the smallest
and largest elements in this sequence:
Algorithm MinMax(s1 , s2 , . . . , sn ):
min = s1 ;
max = s1 ;
for i = 2 to n
do if si < min (1)
then min = si
endif;
if si > max (2)
then max = si
endif
endwhile;
return (min, max )
This algorithm makes comparisons between input elements in lines (1)
and (2). Determine the total number of comparisons as a function of n.
4.9. Exercises 153
4.64 Let n ≥ 2 be a power of 2 and consider a sequence S of n pairwise
distinct numbers. The following algorithm computes the smallest and largest
elements in this sequence:
Algorithm FastMinMax(S, n):
if n = 2
then let x and y be the two elements in S;
if x < y (1)
then min = x;
max = y
else min = y;
max = x
endif
else divide S into two subsequences S1 and S2 , both of size n/2;
(min 1 , max 1 ) = FastMinMax(S1 , n/2);
(min 2 , max 2 ) = FastMinMax(S2 , n/2);
if min 1 < min 2 (2)
then min = min 1
else min = min 2
endif;
if max 1 < max 2 (3)
then max = max 2
else max = max 1
endif
endif;
return (min, max )
This algorithm makes comparisons between input elements in lines (1),
(2), and (3). Let C(n) be the total number of comparisons made by algorithm
FastMinMax on an input sequence of length n.
• Derive a recurrence relation for C(n).
• Use this recurrence relation to prove that C(n) = 23 n−2 for each n ≥ 2
that is a power of 2.
4.65 Consider the following recursive algorithm, which takes as input a se-
quence (a1 , a2 , . . . , an ) of length n, where n ≥ 1:
154 Chapter 4. Recursion
Algorithm Mystery(a1 , a2 , . . . , an ):
if n = 1
then return the sequence (a1 )
else (b1 , b2 , . . . , bn−1 ) = Mystery(a1 , a2 , . . . , an−1 );
return the sequence (an , b1 , b2 , . . . , bn−1 )
endif
• Express the output of algorithm Mystery(a1 , a2 , . . . , an ) in terms of
the input sequence (a1 , a2 , . . . , an ).
4.66 Consider the following recursive algorithm, which takes as input a se-
quence (a1 , a2 , . . . , an ) of n numbers, where n is a power of two, i.e., n = 2k
for some integer k ≥ 0:
Algorithm Mystery(a1 , a2 , . . . , an ):
if n = 1
then return a1
else for i = 1 to n/2
do bi = min(a2i−1 , a2i ) (∗)
endfor;
Mystery(b1 , b2 , . . . , bn/2 )
endif
• Express the output of algorithm Mystery(a1 , a2 , . . . , an ) in terms of
the input sequence (a1 , a2 , . . . , an ).
• For any integer n ≥ 1 that is a power of two, let T (n) be the to-
tal number of times that line (∗) is executed when running algorithm
Mystery(a1 , a2 , . . . , an ). Derive a recurrence for T (n) and use it to
prove that for any integer n ≥ 1 that is a power of two,
T (n) = n − 1.
4.67 Let k be a positive integer and let n = 2k . You are given an n×n board
Bn , all of whose (square) cells are white, except for one, which is black. (The
left part of the figure below gives an example where k = 3 and n = 8.)
4.9. Exercises 155
A tromino is an L-shaped object consisting of three 1 × 1 cells. Each
tromino can appear in four different orientations; see the right part of the
figure below.
A tiling of the board Bn is a placement of trominoes on the board such
that
• the trominoes cover exactly all white cells (thus, the black cell is not
covered by any tromino) and
• no two trominoes overlap.
Here is a tiling of the board given above:
Describe a recursive algorithm that
• takes as input a board Bn having exactly one black cell (which can be
anywhere on the board) and
• returns a tiling of this board.
Hint: Look at the following figure:
156 Chapter 4. Recursion
4.68 Let n ≥ 1 be an integer and consider a set S consisting of n points
in R2 . Each point p of S is given by its x- and y-coordinates px and py ,
respectively. We assume that no two points of S have the same x-coordinate
and no two points of S have the same y-coordinate.
A point p of S is called maximal in S if there is no point in S that is to
the north-east of p, i.e.,
{q ∈ S : qx > px and qy > py } = ∅.
The figure below shows an example, in which the •-points are maximal and
the ×-points are not maximal. Observe that, in general, there is more than
one maximal element in S.
•
•
×
•
× •
× ×
× •
Describe a recursive algorithm MaxElem that has the same structure as
algorithm MergeSort in Section 4.6 and has the following specification:
Algorithm MaxElem(S, n):
Input: A set S of n points in R2 , in sorted order of their x-
coordinates.
Output: All maximal elements of S, in sorted order of their x-
coordinates.
The running time of your algorithm must be O(n log n).
4.69 The Hadamard matrices H0 , H1 , H2 , . . . are recursively defined as fol-
lows:
H0 = (1)
and for k ≥ 1,
Hk−1 Hk−1
Hk = .
Hk−1 −Hk−1
4.9. Exercises 157
Thus, H0 is a 1 × 1 matrix whose only entry is 1,
1 1
H1 = ,
1 −1
and
1 1 1 1
1 −1 1 −1
H2 =
1 1 −1 −1 .
1 −1 −1 1
Observe that Hk has 2k rows and 2k columns.
If x is a column vector of length 2k , then Hk x is the column vector of
length 2k obtained by multiplying the matrix Hk with the vector x.
Describe a recursive algorithm Mult that has the following specification:
Algorithm Mult(k, x):
Input: An integer k ≥ 0 and a column vector x of length n = 2k .
Output: The column vector Hk x (having length n).
The running time T (n) of your algorithm must be O(n log n).
Hint: The input only consists of k and x. The matrix Hk , which has n2 en-
tries, is not given as part of the input. Since you are aiming for an O(n log n)–
time algorithm, you cannot compute all entries of the matrix Hk .
4.70 Let m ≥ 1 and n ≥ 1 be integers and consider an m × n matrix A. The
rows of this matrix are numbered 1, 2, . . . , m, and its columns are numbered
1, 2, . . . , n. Each entry of A stores one number and, for each row, all numbers
in this row are pairwise distinct. For each i = 1, 2, . . . , m, define
g(i) = the position (i.e., column number) of the smallest number in row i.
We say that the matrix A is awesome, if
g(1) ≤ g(2) ≤ g(3) ≤ . . . ≤ g(m).
In the matrix below, the smallest number in each row is in boldface. For this
example, we have m = 4, n = 10, g(1) = 3, g(2) = 3, g(3) = 5, and g(4) = 8.
Thus, this matrix is awesome.
13 12 5 8 6 9 15 20 19 7
3 4 1 17 6 13 7 10 2 5
A= 19 5 12 7 2 4 11 13 6 3 .
7 4 17 10 5 14 12 3 20 6
158 Chapter 4. Recursion
From now on, we assume that the m × n matrix A is awesome.
• Let i be an integer with 1 ≤ i ≤ m. Describe an algorithm that
computes g(i) in O(n) time.
• Describe an algorithm that computes all values g(1), g(2), . . . , g(m) in
O(mn) total time.
In the rest of this exercise, you will show that all values g(1), g(2), . . . , g(m)
can be computed in O(m + n log m) total time.
• Assume that m is even and assume that you are given the values
g(2), g(4), g(6), g(8), . . . , g(m).
Describe an algorithm that computes the values
g(1), g(3), g(5), g(7), . . . , g(m − 1)
in O(m + n) total time.
• Assume that m = 2k , i.e., m is a power of two. Describe a recursive
algorithm FindRowMinima that has the following specification:
Algorithm FindRowMinima(A, i):
Input: An m × n awesome matrix A and an integer i with
0 ≤ i ≤ k.
Output: The values g (j · m/2i ) for j = 1, 2, 3, . . . , 2i .
For each i with 0 ≤ i ≤ k, let T (i) denote the running time of algorithm
FindRowMinima(A, i). The running time of your algorithm must
satisfy the recurrence
T (0) = O(n),
T (i) = T (i − 1) + O 2i + n , if 1 ≤ i ≤ k.
• Assume again that m = 2k . Prove that all values g(1), g(2), . . . , g(m)
can be computed in O(m + n log m) total time.
Hint: 1 + 2 + 22 + 23 + · · · + 2k ≤ 2m.
4.9. Exercises 159
4.71 Prove, for example by induction, that for n ≥ 1,
n(n + 1)
1 + 2 + 3 + ··· + n = ,
2
and
n(n + 1)(2n + 1)
12 + 22 + 32 + · · · + n2 = .
6
4.72 Assume you remember that
12 + 22 + 32 + · · · + n2
is equal to a polynomial of degree three, i.e.,
12 + 22 + 32 + · · · + n2 = An3 + Bn2 + Cn + D,
but you have forgotten the values of A, B, C, and D. How can you determine
these four values?
4.73 In Section 4.8.3, we have shown that
n−2
n−1
X
(k − 1)(n − k − 1) = .
k=2
3
Use Exercise 4.71 to give an alternative proof.
4.74 In Section 4.8.4, we have used the fact that
n−1
X n
k= ,
k=1
2
which follows from Theorem 2.2.10. Give an alternative proof that uses the
approach that we used to prove the identity in (4.17).
4.75 In Section 4.8.4, we have shown that
n−1
X k n
= .
k=3
3 4
Use induction and Pascal’s Identity (see Theorem 3.7.2) to give an alternative
proof.
160 Chapter 4. Recursion
4.76 Consider the numbers Rn that we defined in Section 4.8. The n points
n
on the circle define 2 line segments, one segment for each
pair of points.
Let X be the total number of intersections among these n2 line segments.
• Prove that
n
Rn = 1 + + X.
2
n
Hint: Start with only the circle and the n points. Then add the 2
line segments one by one.
• Prove that
n
X= .
4
4.77 For an integer n ≥ 1, draw n straight lines, such that no two of them
are parallel and no three of them intersect in one single point. These lines
divide the plane into regions (some of which are bounded and some of which
are unbounded). Denote the number of these regions by Cn .
Derive a recurrence relation for the numbers Cn and use it to prove that
for n ≥ 1,
n(n + 1)
Cn = 1 + .
2
4.78 Let n ≥ 1 be an integer. Consider 2n straight lines `1 , `01 , . . . , `n , `0n
such that
• for each i with 1 ≤ i ≤ n, `i and `0i are parallel,
• no two of the lines `1 , . . . , `n are parallel,
• no two of the lines `01 , . . . , `0n are parallel,
• no three of the 2n lines intersect in one single point.
These lines divide the plane into regions (some of which are bounded and
some of which are unbounded). Denote the number of these regions by Rn .
From the figure below, you can see that R1 = 3, R2 = 9, and R3 = 19.
4.9. Exercises 161
`01 `01 `2
`1
`1 `02
R1 = 3
R2 = 9
`01 `2
`1 `02
`03
`3
R3 = 19
• Derive a recurrence relation for the numbers Rn and use it to prove
that Rn = 2n2 + 1 for n ≥ 1.
4.79 Let m ≥ 1 and n ≥ 1 be integers. Consider m horizontal lines and n
non-horizontal lines such that
• no two of the non-horizontal lines are parallel,
• no three of the m + n lines intersect in one single point.
These lines divide the plane into regions (some of which are bounded and
some of which are unbounded). Denote the number of these regions by Rm,n .
From the figure below, you can see that R4,3 = 23.
162 Chapter 4. Recursion
• Derive a recurrence relation for the numbers Rm,n and use it to prove
that
n+1
Rm,n = 1 + m(n + 1) + .
2
4.80 A line is called slanted if it is neither horizontal nor vertical. Let k ≥ 1,
m ≥ 1, and n ≥ 0 be integers. Consider k horizontal lines, m vertical lines,
and n slanted lines, such that
• no two of the slanted lines are parallel,
• no three of the k + m + n lines intersect in one single point.
These lines divide the plane into regions (some of which are bounded and
some of which are unbounded). Denote the number of these regions by Rk,m,n .
From the figure below, you can see that R4,2,2 = 30.
• Prove that
Rk,m,0 = (k + 1)(m + 1).
• Derive a recurrence relation for the numbers Rk,m,n and use it to prove
that
n+1
Rk,m,n = (k + 1)(m + 1) + (k + m)n + .
2
4.81 The sequence SF 0 , SF 1 , SF 2 , . . . of snowflakes is recursively defined in
the following way:
• The snowflake SF 0 is an equilateral triangle with edges of length 1.
• For any integer n ≥ 1, the snowflake SF n is obtained by taking the
snowflake SF n−1 and doing the following for each of its edges:
4.9. Exercises 163
– Divide this edge into three edges of equal length.
– Draw an equilateral triangle that has the middle edge from the
previous step as its base, and that is outside of SF n−1 .
– Remove the edge that is the base of the equilateral triangle from
the previous step.
In the figure below, you see the snowflakes SF 0 up to SF 5 .
• For any integer n ≥ 0, let Nn be the total number of edges of SF n .
Determine the value of Nn , by deriving a recurrence relation and solving
it.
• For any integer n ≥ 0, let `n be the length of one single edge of SF n .
Determine the value of `n , by deriving a recurrence relation and solving
it.
• For any integer n ≥ 0, let Ln be the total length of all edges of SF n .
Prove that n
4
Ln = 3 · .
3
• Let a0 be the area of the triangle SF 0 . For any integer n ≥ 1, let
an be the area of one single triangle that is added when constructing
164 Chapter 4. Recursion
SF n from SF n−1 . Determine the value of an , by deriving a recurrence
relation and solving it.
• For any integer n ≥ 1, let An be the total area of all triangles that are
added when constructing SF n from SF n−1 . Prove that
n
3 4
An = · · a0 .
4 9
• Let n ≥ 0 be an integer. Prove that the total area of SF n is equal to
n
a0 4
· 8−3· .
5 9
Hint: For any real number x 6= 1,
n
X 1 − xn
xk = x · .
k=1
1−x
Chapter 5
Discrete Probability
We all have some intuitive understanding of the notions of “chance” and
“probability”. When buying a lottery ticket, we know that there is a chance
of winning the jackpot, but we also know that this chance is very small.
Before leaving home in the morning, we check the weather forecast and see
that, with probability 80%, we get 15 centimetres of snow in Ottawa. In
this chapter, we will give a formal definition of this notion of “probability”.
We start by presenting a surprising application of probability and random
numbers.
5.1 Anonymous Broadcasting
Consider a group of n people P1 , P2 , . . . , Pn , for some integer n ≥ 3. One
person in this group, say Pk , would like to broadcast, anonymously, a message
to all other people in the group. That is, Pk wants to broadcast a message
such that
• everybody in the group receives the message,
• nobody knows that the message was broadcast by Pk .
In other words, when Pi (with i 6= k) receives the message, she only knows
that it was broadcast by one of P1 , . . . , Pi−1 , Pi+1 , . . . , Pn ; she cannot deter-
mine who broadcast the message.
At first sight, it seems to be impossible to do this. In 1988, however,
David Chaum published, in the Journal of Cryptology, a surprisingly simple
166 Chapter 5. Discrete Probability
protocol that does achieve this. Chaum referred to the problem as the Dining
Cryptographers Problem.
We will present and analyze the protocol for the case when n = 3. Thus,
there are three people P1 , P2 , and P3 . We assume that exactly one of them
broadcasts a message and refer to this person as the broadcaster. We also
assume that the message is a bitstring. The broadcaster will announce the
message one bit at a time.
The three people P1 , P2 , and P3 sit at a table, in clockwise order of their
indices. Let b be the current bit that the broadcaster wants to announce.
The protocol for broadcasting this bit is as follows:
Step 1: Each person Pi generates a random bit bi , for example, by flipping
a coin. Thus, with 50% probability, bi = 0 and with 50% probability, bi = 1.
Step 2: Each person Pi shows the bit bi to her clockwise neighbor.
b1
P2
b2
P1
P3
b3
At the end of this second step,
• P1 knows b1 and b3 , but not b2 ,
• P2 knows b1 and b2 , but not b3 ,
• P3 knows b2 and b3 , but not b1 .
Step 3: Each person Pi computes the sum si (modulo 2) of the bits that she
knows. Thus,
• P1 computes s1 = (b1 + b3 ) mod 2,
• P2 computes s2 = (b1 + b2 ) mod 2,
• P3 computes s3 = (b2 + b3 ) mod 2.
Step 4: Each person Pi does the following:
5.1. Anonymous Broadcasting 167
• If Pi is not the broadcaster, she sets ti = si .
• If Pi is the broadcaster, she sets ti = (si + b) mod 2. Recall that b is the
current bit that the broadcaster wants to announce. (Thus, if b = 1,
then Pi “secretly” flips the bit si and stores the result in ti .)
Step 5: Each person Pi shows her bit ti to the other two people.
Step 6: Each person Pi computes the sum (modulo 2) of the three bits t1 ,
t2 , and t3 , i.e., the value (t1 + t2 + t3 ) mod 2.
This concludes the description of the protocol for broadcasting one bit b.
Observe that for any bit x, we have (x + x) mod 2 = 0. Therefore, the bit
computed in the last step is equal to
t1 + t2 + t3 = s1 + s2 + s3 + b
= (b1 + b3 ) + (b1 + b2 ) + (b2 + b3 ) + b
= (b1 + b1 ) + (b2 + b2 ) + (b3 + b3 ) + b
= b,
where all arithmetic is done modulo 2. In other words, the bit computed in
the last step is equal to the bit that the broadcaster wants to announce. This
shows that each person in the group receives this bit.
It remains to show that a non-broadcaster cannot determine who broad-
cast the bit. In the analysis below, we assume that
• b = 1, i.e., the broadcaster announces the bit 1,
• P2 is not the broadcaster.
We have to show that P2 cannot determine whether P1 or P3 is the broad-
caster. Note that P2 knows the values
b1 , b2 , t1 , t2 , t3 ,
but does not know the bit b3 . We consider the cases when b1 = b2 and b1 6= b2
separately.
Case 1: b1 = b2 . This case has two subcases depending on the value of b3 .
Case 1.1: b3 = b1 ; thus, all three b-bits are equal.
168 Chapter 5. Discrete Probability
• If P1 is the broadcaster, then (all arithmetic is done modulo 2)
t1 = s1 + 1 = b1 + b3 + 1 = 1
and
t3 = s3 = b2 + b3 = 0.
• If P3 is the broadcaster, then
t1 = s1 = b1 + b3 = 0
and
t3 = s3 + 1 = b2 + b3 + 1 = 1.
Thus, the broadcaster is the person whose t-bit is equal to 1.
Case 1.2: b3 6= b1 and, thus, b3 6= b2 .
• If P1 is the broadcaster, then
t1 = s1 + 1 = b1 + b3 + 1 = 0
and
t3 = s3 = b2 + b3 = 1.
• If P3 is the broadcaster, then
t1 = s1 = b1 + b3 = 1
and
t3 = s3 + 1 = b2 + b3 + 1 = 0.
Thus, the broadcaster is the person whose t-bit is equal to 0.
Since P2 knows b1 and b2 , she knows when Case 1 occurs. Since P2 does
not know b3 , however, she cannot determine whether Case 1.1 or 1.2 occurs.
As a result, P2 cannot determine whether P1 or P3 is the broadcaster.
Case 2: b1 6= b2 . This case has two subcases depending on the value of b3 .
Case 2.1: b3 = b1 and, thus, b3 6= b2 .
5.1. Anonymous Broadcasting 169
• If P1 is the broadcaster, then
t1 = s1 + 1 = b1 + b3 + 1 = 1
and
t3 = s3 = b2 + b3 = 1.
• If P3 is the broadcaster, then
t1 = s1 = b1 + b3 = 0
and
t3 = s3 + 1 = b2 + b3 + 1 = 0.
Thus, t1 is always equal to t3 , no matter whether P1 or P3 is the broadcaster.
Case 2.2: b3 6= b1 and, thus, b3 = b2 .
• If P1 is the broadcaster, then
t1 = s1 + 1 = b1 + b3 + 1 = 0
and
t3 = s3 = b2 + b3 = 0.
• If P3 is the broadcaster, then
t1 = s1 = b1 + b3 = 1
and
t3 = s3 + 1 = b2 + b3 + 1 = 1.
Thus, t1 is always equal to t3 , no matter whether P1 or P3 is the broadcaster.
Since P2 knows b1 and b2 , she knows when Case 2 occurs. Since P2 does
not know b3 , however, she cannot determine whether Case 2.1 or 2.2 occurs.
As in Case 1, P2 cannot determine whether P1 or P3 is the broadcaster.
We conclude from Cases 1 and 2 that the broadcasting of the bit b = 1 is
indeed anonymous. Now consider the case when the bit b to be announced
is equal to 0. It follows from the protocol that in this case, there is no
“secret bit flipping” done in Step 4 and all three people use the same rules
to compute the s-values and the t-values. In this case, t1 = s1 , t2 = s2 , and
170 Chapter 5. Discrete Probability
t3 = s3 , and P2 can determine the bit b3 . She cannot, however, determine
whether P1 or P3 is the broadcaster.
To conclude this section, we remark that for each bit to be announced,
the entire protocol must be followed. That is, in each round of the protocol,
one bit is broadcast and each person Pi must flip a coin to determine the bit
bi that is used in this round. We also remark that the protocol works only if
exactly one person is the broadcaster.
5.2 Probability Spaces
In this section, we give a formal definition of the notion of “probability” in
terms of sets and functions.
Definition 5.2.1 A sample space S is a non-empty countable set. Each
element of S is called an outcome and each subset of S is called an event.
In daily life, we express probabilities in terms of percentages. For exam-
ple, the weather forecast may tell us that, with 80% probability, we will be
getting a snowstorm today. In probability theory, probabilities are expressed
in terms of numbers in the interval [0, 1]. A probability of 80% becomes a
probability of 0.8.
Definition 5.2.2 Let S be a sample space. A probability function on S is a
function Pr : S → R such that
• for all ω ∈ S, 0 ≤ Pr(ω) ≤ 1, and
P
• ω∈S Pr(ω) = 1.
For any outcome ω in the sample space S, we will refer to Pr(ω) as the
probability that the outcome is equal to ω.
Definition 5.2.3 A probability space is a pair (S, Pr), where S is a sample
space and Pr : S → R is a probability function on S.
A probability function Pr : S → R maps each element of the sample
space S (i.e., each outcome) to a real number in the interval [0, 1]. It turns
5.2. Probability Spaces 171
out to be useful to extend this function so that it maps any event to a real
number in [0, 1]. If A is an event (i.e., A ⊆ S), then we define
X
Pr(A) = Pr(ω). (5.1)
ω∈A
We will refer to Pr(A) as the probability that the event A occurs.
Note that since S ⊆ S, the entire sample space S is an event and
X
Pr(S) = Pr(ω) = 1,
ω∈S
where the last equality follows from the second condition in Definition 5.2.2.
5.2.1 Examples
Flipping a coin: Assume we flip a coin. Since there are two possible
outcomes (the coin comes up either heads (H) or tails (T )), the sample
space is the set S = {H, T }. If the coin is fair, i.e., the probabilities of H
and T are equal, then the probability function Pr : S → R is given by
Pr(H) = 1/2,
Pr(T ) = 1/2.
Observe that this function Pr satisfies the two conditions in Definition 5.2.2.
Since this sample space has two elements, there are four events, one event
for each subset. These events are
∅, {H}, {T }, {H, T },
and it follows from (5.1) that
Pr(∅) = 0,
Pr({H}) = Pr(H) = 1/2,
Pr({T }) = Pr(T ) = 1/2,
Pr({H, T }) = Pr(H) + Pr(T ) = 1/2 + 1/2 = 1.
Flipping a coin twice: If we flip a fair coin twice, then there are four
possible outcomes, and the sample space becomes S = {HH, HT, T H, T T }.
For example, HT indicates that the first flip resulted in heads, whereas the
172 Chapter 5. Discrete Probability
second flip resulted in tails. In this case, the probability function Pr : S → R
is given by
Pr(HH) = Pr(HT ) = Pr(T H) = Pr(T T ) = 1/4.
Observe again that this function Pr satisfies the two conditions in Defini-
tion 5.2.2. Since the sample space consists of 4 elements, the number of
events is equal to 24 = 16. For example, A = {HT, T H} is an event and it
follows from (5.1) that
Pr(A) = Pr(HT ) + Pr(T H) = 1/4 + 1/4 = 1/2.
In words, when flipping a fair coin twice, the probability that we see one
heads and one tails (without specifying the order) is equal to 1/2.
Rolling a die twice: If we roll a fair die, then there are six possible
outcomes (1, 2, 3, 4, 5, and 6), each one occurring with probability 1/6. If
we roll this die twice, we obtain the sample space
S = {(i, j) : 1 ≤ i ≤ 6, 1 ≤ j ≤ 6},
where i is the result of the first roll and j is the result of the second roll. Note
that |S| = 6 × 6 = 36. Since the die is fair, each outcome has the same prob-
ability. Therefore, in order to satisfy the two conditions in Definition 5.2.2,
we must have
Pr(i, j) = 1/36
for each outcome (i, j) in S.
If we are interested in the sum of the results of the two rolls, then we
define the event
Ak = “the sum of the results of the two rolls is equal to k”,
which, using the notation of sets, is the same as
Ak = {(i, j) ∈ S : i + j = k}.
Consider, for example, the case when k = 4. There are three possible out-
comes of two rolls that result in a sum of 4. These outcomes are (1, 3), (2, 2),
and (3, 1). Thus, the event A4 is equal to
A4 = {(1, 3), (2, 2), (3, 1)}. (5.2)
5.2. Probability Spaces 173
In the matrix below, the leftmost column indicates the result of the first
roll, the top row indicates the result of the second roll, and each entry is the
sum of the results of the two corresponding rolls.
1 2 3 4 5 6
1 2 3 4 5 6 7
2 3 4 5 6 7 8
3 4 5 6 7 8 9
4 5 6 7 8 9 10
5 6 7 8 9 10 11
6 7 8 9 10 11 12
As can be seen from this matrix, the event Ak is non-empty only if k ∈
{2, 3, . . . , 12}. For any other k, the event Ak is empty, which means that it
can never occur.
It follows from (5.1) that
X X
Pr (Ak ) = Pr(i, j) = 1/36 = |Ak |/36.
(i,j)∈Ak (i,j)∈Ak
For example, the number 4 occurs three times in the matrix and, therefore,
the event A4 has size three. Observe that we have already seen this in (5.2).
It follows that
Pr (A4 ) = |A4 |/36 = 3/36 = 1/12.
In a similar way, we see that
Pr (A2 ) = 1/36,
Pr (A3 ) = 2/36 = 1/18,
Pr (A4 ) = 3/36 = 1/12,
Pr (A5 ) = 4/36 = 1/9,
Pr (A6 ) = 5/36,
Pr (A7 ) = 6/36 = 1/6,
Pr (A8 ) = 5/36,
Pr (A9 ) = 4/36 = 1/9,
Pr (A10 ) = 3/36 = 1/12,
Pr (A11 ) = 2/36 = 1/18,
Pr (A12 ) = 1/36.
174 Chapter 5. Discrete Probability
A sample space is not necessarily uniquely defined. In the last example,
where we were interested in the sum of the results of two rolls of a die, we
could also have taken the sample space to be the set
S 0 = {2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12}.
The probability function Pr0 corresponding to this sample space S 0 is given
by
Pr0 (k) = Pr (Ak ) ,
because Pr0 (k) is the probability that we get the outcome k in the sample
space S 0 , which is the same as the probability that event Ak occurs in the
sample space S. You should verify that this function Pr0 satisfies the two
conditions in Definition 5.2.2 and, thus, is a valid probability function on S 0 .
5.3 Basic Rules of Probability
In this section, we prove some basic properties of probability functions. As
we will see, all these properties follow from Definition 5.2.2. Throughout this
section, (S, Pr) is a probability space.
Recall that an event is a subset of the sample space S. In particular, the
empty set ∅ is an event. Intuitively, Pr(∅) must be zero, because it is the
probability that there is no outcome, which can never occur. The following
lemma states that this is indeed the case.
Lemma 5.3.1 Pr(∅) = 0.
Proof. By (5.1), we have
X
Pr(∅) = Pr(ω).
ω∈∅
Since there are zero terms in this summation, its value is equal to zero.
We say that two events A and B are disjoint, if A ∩ B = ∅. A sequence
A1 , A2 , . . . , An of events is pairwise disjoint, if any pair in this sequence is
disjoint. The following lemma is similar to the Sum Rule of Section 3.4.
5.3. Basic Rules of Probability 175
Lemma 5.3.2 If A1 , A2 , . . . , An is a sequence of pairwise disjoint events,
then n
X
Pr(A1 ∪ A2 ∪ · · · ∪ An ) = Pr (Ai ) .
i=1
Proof. Let A = A1 ∪ A2 ∪ · · · ∪ An . Using (5.1), we have
X
Pr(A) = Pr(ω)
ω∈A
Xn X
= Pr(ω)
i=1 ω∈Ai
Xn
= Pr (Ai ) .
i=1
To give an example, assume we roll a fair die twice. What is the proba-
bility that the sum of the two results is even? If you look at the matrix in
Section 5.2, then you see that there are 18 entries, out of 36, that are even.
Therefore, the probability of having an even sum is equal to 18/36 = 1/2.
Below we will give a different way to determine this probability.
The sample space is the set
S = {(i, j) : 1 ≤ i ≤ 6, 1 ≤ j ≤ 6},
where i is the result of the first roll and j is the result of the second roll. Each
element of S has the same probability 1/36 of being an outcome of rolling
the die twice.
The event we are interested in is
A = {(i, j) ∈ S : i + j is even}.
Observe that i + j is even if and only if both i and j are even or both i and
j are odd. Therefore, we split the event A into two disjoint events
A1 = {(i, j) ∈ S : both i and j are even}
and
A2 = {(i, j) ∈ S : both i and j are odd}.
176 Chapter 5. Discrete Probability
By Lemma 5.3.2, we have
Pr(A) = Pr (A1 ) + Pr (A2 ) .
The set A1 has 3 · 3 = 9 elements, because there are 3 choices for i and 3
choices for j. Similarly, the set A2 has 9 elements. It follows that
Pr(A) = Pr (A1 ) + Pr (A2 ) = 9/36 + 9/36 = 1/2.
In the next lemma, we relate the probability that an event occurs to the
probability that the event does not occur. If A is an event, then A denotes
its complement, i.e., A = S \ A. Intuitively, the sum of Pr(A) and Pr A
must be equal to one, because the event A either occurs or does not occur.
The following lemma states that this is indeed the case. Observe that this is
similar to the Complement Rule of Section 3.3.
Lemma 5.3.3 For any event A,
Pr(A) = 1 − Pr A .
Proof. Since A and A are disjoint and S = A∪A, it follows from Lemma 5.3.2
that
Pr(S) = Pr A ∪ A = Pr(A) + Pr A .
We have seen in Section 5.2 that Pr(S) = 1.
Consider again the sample space that we saw after Lemma 5.3.2. We
showed that, when rolling a fair die twice, we get an even sum with probabil-
ity 1/2. It follows from Lemma 5.3.3 that we get an odd sum with probability
1 − 1/2 = 1/2.
The next lemma is similar to the Principle of Inclusion and Exclusion
that we have seen in Section 3.5.
Lemma 5.3.4 If A and B are events, then
Pr(A ∪ B) = Pr(A) + Pr(B) − Pr(A ∩ B).
Proof. Since B \ A and A ∩ B are disjoint and B = (B \ A) ∪ (A ∩ B), it
follows from Lemma 5.3.2 that
Pr(B) = Pr(B \ A) + Pr(A ∩ B).
5.3. Basic Rules of Probability 177
Next observe that A and B \ A are disjoint. Since A ∪ B = A ∪ (B \ A), we
again apply Lemma 5.3.2 and obtain
Pr(A ∪ B) = Pr(A) + Pr(B \ A).
By combining these two equations, we obtain the claim in the lemma.
To give an example, assume we choose a number x in the sample space S =
{1, 2, . . . , 1000}, such that each element has the same probability 1/1000 of
being chosen. What is the probability that x is divisible by 2 or 3? Consider
the events
A = {i ∈ S : i is divisible by 2}
and
B = {i ∈ S : i is divisible by 3}.
Then we want to determine Pr(A ∪ B), which, by Lemma 5.3.4, is equal to
Pr(A ∪ B) = Pr(A) + Pr(B) − Pr(A ∩ B).
Since there are b1000/2c = 500 even numbers in S, we have
Pr(A) = 500/1000.
Since there are b1000/3c = 333 elements in S that are divisible by 3, we have
Pr(B) = 333/1000.
Observe that i belongs to A ∩ B if and only if i is divisible by 6, i.e.,
A ∩ B = {i ∈ S : i is divisible by 6}.
Since there are b1000/6c = 166 elements in S that are divisible by 6, we have
Pr(A ∩ B) = 166/1000.
We conclude that
Pr(A ∪ B) = Pr(A) + Pr(B) − Pr(A ∩ B)
= 500/1000 + 333/1000 − 166/1000
= 667/1000.
178 Chapter 5. Discrete Probability
Lemma 5.3.5 (Union Bound) For any integer n ≥ 1, if A1 , A2 , . . . , An is
a sequence of events, then
n
X
Pr (A1 ∪ A2 ∪ · · · ∪ An ) ≤ Pr (Ai ) .
i=1
Proof. The proof is by induction on n. If n = 1, we have equality and, thus,
the claim obviously holds. Let n ≥ 2 and assume the claim is true for n − 1,
i.e., assume that
n−1
X
Pr (A1 ∪ A2 ∪ · · · ∪ An−1 ) ≤ Pr (Ai ) .
i=1
Let B = A1 ∪ A2 ∪ · · · ∪ An−1 . Then it follows from Lemma 5.3.4 that
Pr (B ∪ An ) = Pr(B) + Pr (An ) − Pr (B ∩ An ) ≤ Pr(B) + Pr (An ) ,
because Pr (B ∩ An ) ≥ 0 (this follows from the first condition in Defini-
tion 5.2.2). Since we assumed that
n−1
X
Pr(B) ≤ Pr (Ai ) ,
i=1
it follows that
Pr (A1 ∪ A2 ∪ · · · ∪ An ) = Pr (B ∪ An )
≤ Pr(B) + Pr (An )
n−1
X
≤ Pr (Ai ) + Pr (An )
i=1
n
X
= Pr (Ai ) .
i=1
Lemma 5.3.6 If A and B are events with A ⊆ B, then
Pr(A) ≤ Pr(B).
5.4. Uniform Probability Spaces 179
Proof. Using (5.1) and the fact that Pr(ω) ≥ 0 for each ω in S, we have
X
Pr(A) = Pr(ω)
ω∈A
X
≤ Pr(ω)
ω∈B
= Pr(B).
5.4 Uniform Probability Spaces
In this section, we consider finite sample spaces S in which each outcome has
the same probability. Since, by Definition 5.2.2, all probabilities add up to
one, the probability of each outcome must be equal to 1/|S|.
Definition 5.4.1 A uniform probability space is a pair (S, Pr), where S is a
finite sample space and the probability function Pr : S → R satisfies
1
Pr(ω) = ,
|S|
for each outcome ω in S.
The probability spaces that we have seen in Section 5.2.1 are all uniform,
except the space (S 0 , Pr0 ) that we saw at the end of that section.
To give another example, when playing Lotto 6/49, you choose a 6-
element subset of the set A = {1, 2, . . . , 49}. Twice a week, the Ontario Lot-
tery and Gaming Corporation (OLG) draws the six “winning numbers” uni-
formly at random from A. If your numbers are equal to those drawn by OLG,
then you can withdraw from this course and spend the rest of your life on the
beach. Most people find it silly to choose the subset {1, 2, 3, 4, 5, 6}. They
think that it is better to choose, for example, the subset {2, 5, 16, 36, 41, 43}.
Is this true?
For this example, the sample space is the set S consisting of all 6-element
subsets of A. Since S has size 49
6
and the subset drawn by OLG is uniform,
each outcome (i.e., each 6-element subset of S) has a probability of
1 1
49
= ≈ 0.000000072.
6
13, 983, 816
180 Chapter 5. Discrete Probability
In particular, both {1, 2, 3, 4, 5, 6} and {2, 5, 16, 36, 41, 43} have the same
probability of being the winning numbers. (Still, the latter subset was drawn
by OLG on February 8, 2014.)
The lemma below states that in a uniform probability space (S, Pr), the
probability of an event A is the ratio of the size of A and the size of S.
Lemma 5.4.2 If (S, Pr) is a uniform probability space and A is an event,
then
|A|
Pr(A) = .
|S|
Proof. By using (5.1) and Definition 5.4.1, we get
X X 1 1 X |A|
Pr(A) = Pr(ω) = = 1= .
ω∈A ω∈A
|S| |S| ω∈A
|S|
5.4.1 The Probability of Getting a Full House
In a standard deck of 52 cards, each card has a suit and a rank. There are
four suits (spades ♠, hearts ♥, clubs ♣, and diamonds ♦), and 13 ranks
(Ace, 2, 3, 4, 5, 6, 7, 8, 9, 10, Jack, Queen, and King).
A hand of five cards is called a full house, if three of the cards are of
the same rank and the other two cards are also of the same (but necessarily
different) rank. For example, the hand
7♠, 7♥, 7♦, Q♠, Q♣
is a full house, because it consists of three sevens and two Queens.
Assume we get a uniformly random hand of five cards. What is the
probability that this hand is a full house? To answer this question, first
observe that a hand of five cards is a subset of the set of all 52 cards. Thus,
the sample space is the set S consisting of all 5-element subsets of the set of
52 cards and, therefore,
52
|S| = = 2, 598, 960.
5
5.5. The Birthday Paradox 181
Each hand of five cards has a probability of 1/|S| of being chosen.
Since we are interested in the probability of a random hand being a full
house, we define the event A to be the set of all elements in S that are full
houses. By Lemma 5.4.2, we have
|A|
Pr(A) = .
|S|
Thus, to determine Pr(A), it remains to determine the size of the set A, i.e.,
the total number of full houses. For this, we will use the Product Rule of
Section 3.1:
• The procedure is “choose a full house”.
• First task: Choose the rank of the three cards in the full house. There
are 13 ways to do this.
• Second task: Choose the suits of these three cards. There are 43 ways
to do this.
• Third task: Choose the rank of the other two cards in the full house.
There are 12 ways to do this.
• Fourth task: Choose the suits of these two cards. There are 42 ways
to do this.
Thus, the number of full houses is equal to
4 4
|A| = 13 · · 12 · = 3, 744.
3 2
We conclude that the probability of getting a full house is equal to
|A| 3, 744
Pr(A) = = ≈ 0.00144.
|S| 2, 598, 960
5.5 The Birthday Paradox
Let n ≥ 2 be an integer and consider a group of n people. In this section,
we will determine the probability pn that at least two of them have the same
birthday. We will ignore leap years, so that there are 365 days in one year.
182 Chapter 5. Discrete Probability
Below, we will show that p2 = 1/365. If n ≥ 366, then it follows from the
Pigeonhole Principle (see Section 3.10) that there must be at least two people
with the same birthday and, therefore, pn = 1. Intuitively, if n increases from
2 to 365, the value of pn increases as well. What is the value of n such that
pn is larger than 1/2 for the first time? That is, what is the value of n for
which pn−1 ≤ 1/2 < pn ? In this section, we will see that this question can be
answered using simple counting techniques that we have seen in Chapter 3.
We denote the people by P1 , P2 , . . . , Pn , we denote the number of days in
one year by d, and we number the days in one year as 1, 2, . . . , d. The sample
space is the set
Sn = {(b1 , b2 , . . . , bn ) : bi ∈ {1, 2, . . . , d} for each 1 ≤ i ≤ n},
where bi denotes the birthday of Pi . Note that
|Sn | = dn .
We consider the uniform probability space: For each element (b1 , b2 , . . . , bn )
in Sn , we have
1 1
Pr (b1 , b2 , . . . , bn ) = = n.
|Sn | d
The event we are interested in is
An = “at least two of the numbers in b1 , b2 , . . . , bn are equal”.
Using the notation of sets, this is the same as
An = {(b1 , b2 , . . . , bn ) ∈ Sn : b1 , b2 , . . . , bn contains duplicates}.
The probability pn that we introduced above is equal to
pn = Pr (An ) .
As mentioned above, the Pigeonhole Principle implies that pn = 1 for n > d.
Therefore, we assume from now on that n ≤ d.
Let us start by determining p2 . Since we consider the uniform probability
space, we have, by Lemma 5.4.2,
|A2 |
p2 = Pr (A2 ) = .
|S2 |
5.5. The Birthday Paradox 183
We know already that |S2 | = d2 . The event A2 is equal to
A2 = {(1, 1), (2, 2), . . . , (d, d)}.
Thus, |A2 | = d and we obtain
|A2 | d 1
p2 = = 2 = .
|S2 | d d
To determine pn for larger values of n, it is easier to determine the
probability of the complement An . The latter probability, together with
Lemma 5.3.3, will give us the value of pn . Note that
An = {(b1 , b2 , . . . , bn ) ∈ Sn : b1 , b2 , . . . , bn are pairwise distinct}.
In words, An is the set of all ordered sequences consisting of n pairwise
distinct elements of {1, 2, . . . , d}. In Section 3.6, see (3.1), we have seen that
d!
|An | = .
(d − n)!
We conclude that, for any n with 2 ≤ n ≤ d,
pn = Pr (An )
= 1 − Pr An
|An |
= 1−
|Sn |
d!
= 1− .
(d − n)!dn
By taking d = 365, we get p22 = 0.476 and p23 = 0.507. Thus, in a random
group of 23 people1 , the probability that at least two of them have the same
birthday is more than 50%. Most people are very surprised when they see
this for the first time, because our intuition says that a much larger group is
needed to have a probability of more than 50%. The values pn approach 1
pretty fast. For example, p40 = 0.891 and p100 = 0.9999997.
1
two soccer teams plus the referee
184 Chapter 5. Discrete Probability
5.5.1 Throwing Balls into Boxes
When we derived the expression for pn , we did not use the fact that the
value of d is equal to 365. In other words, the expression is valid for any
value of d. For general values of d, we can interpret the birthday problem
in the following way: Consider d boxes B1 , B2 , . . . , Bd , where d is a large
integer. Assume that we throw n balls into these boxes so that each ball
lands in a uniformly random box. Then pn is the probability that there is at
least one box that contains more than one ball. Since it is not easy to see
how the expression
d!
pn = 1 −
(d − n)!dn
depends on n, we will approximate it by a simpler expression. For this, we
will use the inequality
1 − x ≤ e−x , (5.3)
which is valid for any real number x. If x is close to zero, then the inequality
is very accurate. The easiest way to prove this inequality is by showing that
the minimum of the function f (x) = x + e−x is equal to f (0) = 1, using
techniques from calculus.
If we define qn = 1 − pn , then we have
d!
qn = .
(d − n)!dn
Using (5.3), we get
d d−1 d−2 d−3 d − (n − 1)
qn = · · · ···
d d d d d
d−1 d−2 d−3 d − (n − 1)
= · · ···
d d d d
= (1 − 1/d) · (1 − 2/d) · (1 − 3/d) · · · (1 − (n − 1)/d)
≤ e−1/d · e−2/d · e−3/d · · · e−(n−1)/d
= e−(1+2+3+···+(n−1))/d .
Using the equality
1 + 2 + 3 + · · · + (n − 1) = n(n − 1)/2,
see Theorem 2.2.10, we thus get
qn ≤ e−n(n−1)/(2d) ,
5.6. The Big Box Problem 185
and therefore,
pn = 1 − qn ≥ 1 − e−n(n−1)/(2d) .
If n is large, then n(n − 1)/(2d) is very close to n2 /(2d) and, thus,
2 /(2d)
pn & 1 − e−n .
√
If we take n = 2d, then we get
pn & 1 − e−1 ≈ 0.632.
√
Thus, for large values of d, if we throw 2d balls into d boxes, then with
probability (approximately) at least 1 − 1/e, there is a box that contains
more than one ball.
5.6 The Big Box Problem
Keith chooses two distinct elements x and y, with x < y, from the set
A = {0, 1, 2, . . . , 100}; he does not show these two numbers to us. He takes
two identical boxes, and puts x dollars in one box and y dollars in the other
box. Then Keith closes the two boxes, shuffles them, and puts them on a
table. At this moment, we can see the two boxes, they look identical to us,
and the only information we have is that they contain different amounts of
money, where each amount is an element of the set A.
$x $y or $y $x
We will refer to the box containing x dollars as the small box and to the
box containing y dollars as the big box. Our goal is to find the big box. We
are allowed to do the following:
1. We can choose one of the two boxes, open it, and determine how much
money is inside it.
2. Now we have to make our final decision: Either we keep the box we
just opened or we take the other box.
186 Chapter 5. Discrete Probability
For example, assume that the box we pick in the first step contains $33.
Then we know that the other box contains either less than $33 or more
than $33. It seems that the only reasonable thing to do is to flip a fair coin
when making our final decision. If we do that, then we find the big box with
probability 0.5.
In the rest of this section, we will show the surprising result that we can
find the big box with probability at least 0.505.
The idea is as follows. Assume that we know a number z such that
x < z < y. (Keep in mind that we do not know x and we do not know y.
Thus, we assume that we know a number z that is between the two unknown
numbers x and y.)
• If the box we choose in the first step contains more than z dollars, then
we know that this is the big box and, therefore, we keep it.
• If the box we choose in the first step contains less than z dollars, then
we know that this is the small box and, therefore, we take the other
box.
Thus, if we know this number z with x < z < y, then we are guaranteed to
find the big box.
Of course, it is not realistic to assume that we know this magic number z.
The trick is to choose a random z and hope that it is between x and y. If z
is between x and y, then we find the big box with probability 1; otherwise,
we find the big box with probability 1/2. As we will see later, the overall
probability of finding the big box will be at least 0.505.
In order to avoid the case when z = x or z = y, we will choose z from
the set
B = {1/2, 3/2, 5/2, . . . , 100 − 1/2}.
Note that |B| = 100. Our algorithm that attempts to find the big box does
the following:
5.6. The Big Box Problem 187
Algorithm FindBigBox:
Step 1: Choose one of the two boxes uniformly at random, open it,
and determine the amount of money inside it; let this amount be a.
Step 2: Choose z uniformly at random from the set B.
Step 3: Do the following:
• If a > z, then keep the box chosen in Step 1.
• Otherwise (i.e., if a < z), take the other box.
5.6.1 The Probability of Finding the Big Box
We are now going to determine the probability that this algorithm finds the
big box. First, we have to ask ourselves what the sample space is. There are
two places in the algorithm where a random element is chosen:
• In Step 1, we choose the element a, which is a random element from
the set {x, y}. We know that this value a is equal to one of x and y.
However, at the end of Step 1, we do not know whether a = x or a = y.
• In Step 2, we choose a random element from the set B.
Based on this, the sample space S is the Cartesian product
S = {x, y} × B = {(a, z) : a ∈ {x, y}, z ∈ B}
and Steps 1 and 2 can be replaced by
• choose a uniformly random element (a, z) in S.
Note that |S| = 200.
We say that algorithm FindBigBox is successful if it finds the big box.
Thus, we want to determine Pr(W ), where W is the event
W = “algorithm FindBigBox is successful”.
We are going to write this event as a subset of the sample space S. For this, we
have to determine all elements (a, z) in S for which algorithm FindBigBox
is successful.
First consider the case when a = x. In this case, the box we choose in
Step 1 is the small box. There are two possibilities for z:
188 Chapter 5. Discrete Probability
• If x = a > z, then the algorithm keeps the small box and, thus, is not
successful.
• If x = a < z, then the algorithm takes the other box (which is the big
box) and, thus, is successful.
Thus, the event W contains the set
Wx = {(x, z) : z ∈ {x + 1/2, x + 3/2, . . . , 100 − 1/2}}.
You can verify that
|Wx | = 100 − x.
The second case to consider is when a = y. In this case, the box we
choose in Step 1 is the big box. Again, there are two possibilities for z:
• If y = a > z, then the algorithm keeps the big box and, thus, is
successful.
• If y = a < z, then the algorithm takes the other box (which is the small
box) and, thus, is not successful.
Thus, the event W contains the set
Wy = {(y, z) : z ∈ {1/2, 3/2, . . . , y − 1/2}}.
You can verify that
|Wy | = y.
Since W = Wx ∪ Wy and the events Wx and Wy are disjoint, we have, by
Lemma 5.3.2,
Pr(W ) = Pr (Wx ∪ Wy )
= Pr (Wx ) + Pr (Wy ) .
Since the element (a, z) is chosen uniformly at random from the sample
space S, we can use Lemma 5.4.2 to determine the probability that algorithm
FindBigBox is successful:
Pr(W ) = Pr (Wx ) + Pr (Wy )
|Wx | |Wy |
= +
|S| |S|
100 − x y
= +
200 200
1 y−x
= + .
2 200
5.8. Conditional Probability 189
Since x and y are distinct integers and x < y, we have y − x ≥ 1, and we
conclude that
1 1
Pr(W ) ≥ + = 0.505.
2 200
5.7 The Monty Hall Problem
The Monty Hall Problem is a well-known puzzle in probability theory. It
is named after the host, Monty Hall, of the American television game show
Let’s Make a Deal. The problem became famous in 1990, when (part of) a
reader’s letter was published in Marilyn vos Savant’s column Ask Marilyn in
the magazine Parade:
Suppose you’re on a game show, and you’re given the choice of three
doors: Behind one door is a car; behind the others, goats. You pick
a door, say No. 1, and the host, who knows what’s behind the doors,
opens another door, say No. 3, which has a goat. He then says to you,
“Do you want to pick door No. 2?” Is it to your advantage to switch
your choice?
Note that the host can always open a door that has a goat behind it.
After the host has opened No. 3, we know that the car is either behind No. 1
or No. 2, and it seems that both these doors have the same probability (i.e.,
50%) of having the car behind them. We will prove below, however, that this
is not true: It is indeed to our advantage to switch our choice.
We assume that the car is equally likely to be behind any of the three
doors. Moreover, the host knows what is behind each door.
• We initially choose one of the three doors uniformly at random; this
door remains closed.
• The host opens one of the other two doors that has a goat behind it.
• Our final choice is to switch to the other door that is still closed.
Let A be the event that we win the car and let B be the event that the initial
door has a goat behind it. Then it is not difficult to see that event A occurs
if and only if event B occurs. Therefore, the probability that we win the car
is equal to
Pr(A) = Pr(B) = 2/3.
190 Chapter 5. Discrete Probability
5.8 Conditional Probability
Anil Maheshwari has two children. We are told that one of them is a boy.
What is the probability that the other child is also a boy? Most people will
say that this probability is 1/2. We will show below that this is not the
correct answer.
Since Anil has two children, the sample space is
S = {(b, b), (b, g), (g, b), (g, g)},
where, for example, (b, g) indicates that the youngest child is a boy and the
oldest child is a girl. We assume a uniform probability function, so that each
outcome has a probability of 1/4.
We are given the additional information that one of the two children is
a boy, or, to be more precise, that at least one of the two children is a boy.
This means that the actual sample space is not S, but
{(b, b), (b, g), (g, b)}.
When asking for the probability that the other child is also a boy, we are
really asking for the probability that both children are boys. Since there is
only one possibility (out of three) for both children to be boys, it follows that
this probability is equal to 1/3.
This is an example of a conditional probability: We are asking for the
probability of an event (both children are boys), given that another event (at
least one of the two children is a boy) occurs.
Definition 5.8.1 Let (S, Pr) be a probability space and let A and B be two
events with Pr(B) > 0. The conditional probability Pr(A | B), pronounced
as “the probability of A given B”, is defined as
Pr(A ∩ B)
Pr(A | B) = .
Pr(B)
Let us try to understand where this definition comes from. Initially, the
sample space is equal to S. When we are given the additional information
that event B occurs, the sample space “shrinks” to B, and event A occurs if
and only if event A ∩ B occurs.
5.8. Conditional Probability 191
S
B
A
You may think that Pr(A | B) should therefore be defined to be Pr(A∩B).
However, since the sum of all probabilities must be equal to 1, we have to
normalize, i.e., divide by Pr(B). Equivalently, if A = B, we get Pr(A | A),
which is the probability that event A occurs, given that event A occurs. This
probability should be equal to 1. Indeed, using the definition, we do get
Pr(A ∩ A) Pr(A)
Pr(A | A) = = = 1.
Pr(A) Pr(A)
In Exercise 5.24, you are asked to give a formal proof that our definition
gives a valid probability function on the sample space S.
5.8.1 Anil’s Children
Returning to Anil’s two children, we saw that the sample space is
S = {(b, b), (b, g), (g, b), (g, g)}
and we assumed a uniform probability function. The events we considered
are
A = “both children are boys”
and
B = “at least one of the two children is a boy”,
and we wanted to know Pr(A | B). Writing A and B as subsets of the sample
space S, we get
A = {(b, b)}
and
B = {(b, b), (b, g), (g, b)}.
Using Definition 5.8.1, it follows that
Pr(A ∩ B) Pr(A) |A|/|S| 1/4
Pr(A | B) = = = = = 1/3,
Pr(B) Pr(B) |B|/|S| 3/4
which is the same answer as we got before.
192 Chapter 5. Discrete Probability
5.8.2 Rolling a Die
Assume we roll a fair die, i.e., we choose an element uniformly at random
from the sample space
S = {1, 2, 3, 4, 5, 6}.
Consider the events
A = “the result is 3”
and
B = “the result is an odd integer”.
What is the conditional probability Pr(A | B)? To determine this proba-
bility, we assume that event B occurs, i.e., the roll of the die resulted in one
of 1, 3, and 5. Given that event B occurs, event A occurs in one out of these
three possibilities. Thus, Pr(A | B) should be equal to 1/3. We are going
to verify that this is indeed the answer we get when using Definition 5.8.1:
Since
A = {3}
and
B = {1, 3, 5},
we have
Pr(A ∩ B) Pr(A) |A|/|S| 1/6
Pr(A | B) = = = = = 1/3.
Pr(B) Pr(B) |B|/|S| 3/6
Let us now consider the conditional probability Pr(B | A). Thus, we are
given that event A occurs, i.e., the roll of the die resulted in 3. Since 3 is an
odd integer, event B is guaranteed to occur. Therefore, Pr(B | A) should be
equal to 1. Again, we are going to verify that this is indeed the answer we
get when using Definition 5.8.1:
Pr(B ∩ A) Pr(A)
Pr(B | A) = = = 1.
Pr(A) Pr(A)
This shows that, in general, Pr(A | B) is not equal to Pr(B | A). Observe
that this is not surprising. (Do you see why?)
Consider the event
C = “the result is a prime number”,
5.8. Conditional Probability 193
which, when written as a subset of the sample space, is
C = {2, 3, 5}.
Then Pr(C | B) should be equal to 2/3 and Pr(C | A) should be equal to 1.
Indeed, we have
Pr(C ∩ B) |C ∩ B|/|S| 2/6
Pr(C | B) = = = = 2/3
Pr(B) |B|/|S| 3/6
and
Pr(C ∩ A) Pr(A)
Pr(C | A) = = = 1.
Pr(A) Pr(A)
Recall that B denotes the complement of the event B. Thus, this is the
event
B = “the result is an even integer”,
which, when written as a subset of the sample space, is
B = {2, 4, 6}.
Then Pr C | B should be equal to 1/3. Indeed, we have
Pr C ∩ B |C ∩ B|/|S| 1/6
Pr C | B = = = = 1/3.
Pr B |B|/|S| 3/6
Observe that
Pr(C | B) + Pr C | B = 2/3 + 1/3 = 1.
You may think that this is true for any two events B and C. This is, however,
not the case: Since
A = {1, 2, 4, 5, 6},
we have
Pr C ∩ A |C ∩ A|/|S| 2/6
Pr C | A = = = = 2/5
Pr A |A|/|S| 5/6
and, thus,
Pr(C | A) + Pr C | A = 1 + 2/5 6= 1.
194 Chapter 5. Discrete Probability
It should be an easy exercise to verify that
Pr(A | C) + Pr A | C = 1.
Intuitively, this should be true for any two events A and C: When we are
given that event C occurs, then either A occurs or A does not occur (in
which case A occurs). The following lemma states that this intuition is
indeed correct.
Lemma 5.8.2 Let (S, Pr) be a probability space and let A and B be two
events with Pr(B) > 0. Then
Pr(A | B) + Pr A | B = 1.
Proof. By definition, we have
Pr(A ∩ B) Pr A ∩ B
Pr(A | B) + Pr A | B = +
Pr(B) Pr(B)
Pr(A ∩ B) + Pr A ∩ B
= .
Pr(B)
Since the events A ∩ B and A ∩ B are disjoint, we have, by Lemma 5.3.2,
Pr(A ∩ B) + Pr A ∩ B = Pr (A ∩ B) ∪ A ∩ B .
By drawing a Venn diagram, you will see that
(A ∩ B) ∪ A ∩ B = B,
implying that
Pr(A ∩ B) + Pr A ∩ B = Pr(B).
We conclude that
Pr(B)
Pr(A | B) + Pr A | B = = 1.
Pr(B)
5.8. Conditional Probability 195
5.8.3 Flip and Flip or Roll
We are given a fair red coin, a fair blue coin, and a fair die. First, we flip
the red coin. If the result of this flip is heads, then we flip the blue coin and
return the result of this second flip. Otherwise, the red coin came up tails,
in which case we roll the die and return the result of this roll.
What is the probability that the value 5 is returned? Our intuition says
that this probability is equal to 1/12: The value 5 is returned if and only
if the red coin comes up tails (which happens with probability 1/2) and the
result of rolling the die is 5 (which happens with probability 1/6). We will
prove that this is indeed the correct answer.
We start by modifying the above random process so that it better reflects
the random choices that are being made:
Algorithm FlipAndFlipOrRoll:
fr = the result of flipping the red coin;
if fr = H
then fb = the result of flipping the blue coin;
return the ordered pair (fr , fb )
else d = the result of rolling the die;
return the ordered pair (fr , d)
endif
The possible executions of this algorithm are visualized in the following
tree diagram:
flip red coin
H T
flip blue coin roll die
H T
1 2 3 4 5 6
HH HT T1 T2 T3 T4 T5 T6
The sample space is the set S of all possible values that can be returned
by algorithm FlipAndFlipOrRoll. Thus, we have
S = {(H, H), (H, T ), (T, 1), (T, 2), (T, 3), (T, 4), (T, 5), (T, 6)}.
196 Chapter 5. Discrete Probability
We are interested in the probability that the algorithm returns the value
(T, 5), i.e., the probability of the event
A = {(T, 5)}.
Since the event A obviously depends on the result of flipping the red coin,
we consider the event
R = “the result of flipping the red coin is tails”.
If we write this event as a subset of the sample space, we get
R = {(T, 1), (T, 2), (T, 3), (T, 4), (T, 5), (T, 6)}.
Observe that Pr(R) = 1/2 and A ∩ R = A, which implies that
Pr(A) = Pr(A ∩ R).
If we rewrite the expression for the conditional probability Pr(A | R) in
Definition 5.8.1, we get
Pr(A) = Pr(A ∩ R) = Pr(R) · Pr(A | R).
We have seen already that Pr(R) = 1/2. To determine Pr(A | R), we assume
that event R occurs. Under this assumption, event A occurs if and only if
the result of rolling the die is 5, which happens with probability 1/6. Thus,
Pr(A | R) = 1/6
and we conclude that
Pr(A) = 1/2 · 1/6 = 1/12,
which is the answer we were expecting to see.
You may object to this method of determining Pr(A): When we deter-
mined Pr(A | R), we did not use the definition of conditional probability,
i.e.,
Pr(A ∩ R)
Pr(A | R) = .
Pr(R)
Instead, we used the “informal definition”, by determining the probability
that event A occurs, assuming that event R occurs. Thus, we do not yet
5.8. Conditional Probability 197
have a formal justification as to why Pr(A) is equal to 1/12. In the rest of
this section, we do present a formal justification.
For each integer i with 1 ≤ i ≤ 6, we consider the event
Ai = {(T, i)}
and its probability
pi = Pr (Ai ) .
First observe that
p1 = p2 = p 3 = p4 = p5 = p6 ,
because the die is fair. Let p denote the common value of the pi ’s. Next
observe that
R = A1 ∪ A2 ∪ A3 ∪ A4 ∪ A5 ∪ A6 ,
where the six events on the right-hand side are pairwise disjoint. We have
seen already that Pr(R) = 1/2. It follows that
1/2 = Pr(R)
6
!
[
= Pr Ai
i=1
6
X
= Pr (Ai )
i=1
6
X
= p
i=1
= 6p,
implying that p = 1/12. Since the event A we are interested in is equal to
the event A5 , we conclude that
Pr(A) = Pr (A5 ) = p5 = p = 1/12.
Thus, we have obtained a formal proof of the fact that the probability of the
event A is equal 1/12.
198 Chapter 5. Discrete Probability
Using the definition of conditional probability, we can now also formally
determine Pr(A | R):
Pr(A ∩ R)
Pr(A | R) =
Pr(R)
Pr(A)
=
Pr(R)
1/12
=
1/2
= 1/6.
5.9 The Law of Total Probability
Both Mick and Keith have a random birthday. What is the probability
that they have the same birthday? We have seen in Section 5.5 that this
probability is equal to 1/365. A common way to determine this probability
is as follows: Consider Mick’s birthday, which can be any of the 365 days
of the year. By symmetry, it does not really matter what Mick’s birthday
is, so we just assume that it is July 26. Then Mick and Keith have the
same birthday if and only if Keith’s birthday is also on July 26. Therefore,
since Keith has a random birthday, the probability that Mick and Keith
have the same birthday is equal to 1/365. The following theorem explains
this reasoning.
Theorem 5.9.1 (Law of Total Probability) Let (S, Pr) be a probability
space and let A be an event. Assume that B1 , B2 , . . . , Bn is a sequence of
events such that
1. Pr (Bi ) > 0 for all i with 1 ≤ i ≤ n,
2. the events B1 , B2 , . . . , Bn are pairwise disjoint, and
3. ni=1 Bi = S.
S
Then
n
X
Pr(A) = Pr (A | Bi ) · Pr (Bi ) .
i=1
5.9. The Law of Total Probability 199
Proof. The assumptions imply that
A = A∩S !
n
[
= A∩ Bi
i=1
n
[
= (A ∩ Bi ) .
i=1
Since the events A ∩ B1 , A ∩ B2 , . . . , A ∩ Bn are pairwise disjoint, it follows
from Lemma 5.3.2 that
n
!
[
Pr(A) = Pr (A ∩ Bi )
i=1
n
X
= Pr (A ∩ Bi ) .
i=1
The theorem follows by observing that, from Definition 5.8.1,
Pr (A ∩ Bi ) = Pr (A | Bi ) · Pr (Bi ) .
Let us consider the three conditions in this theorem. The first condition
is that Pr (Bi ) > 0, i.e., there is a positive probability that event Bi occurs.
The second and third conditions, i.e.,
• the events B1 , B2 , . . . , Bn are pairwise disjoint, and
• ni=1 Bi = S,
S
are equivalent to
• exactly one of the events B1 , B2 , . . . , Bn is guaranteed to occur.
In the example in the beginning of this section, we wanted to know Pr(A),
where A is the event
A = “Mick and Keith have the same birthday”.
In order to apply Theorem 5.9.1, we define a sequence B1 , B2 , . . . of events
that satisfy the conditions in this theorem and for which Pr (A | Bi ) is easy
200 Chapter 5. Discrete Probability
to determine. For this example, we define the event Bi , for each i with
1 ≤ i ≤ 365, to be
Bi = “Mick’s birthday is on the i-th day of the year”.
It is clear that (i) Pr (Bi ) = 1/365 > 0 and (ii) exactly one of the events
B1 , B2 , . . . , B365 is guaranteed to occur. It follows that
365
X
Pr(A) = Pr (A | Bi ) · Pr (Bi ) .
i=1
To determine Pr (A | Bi ), we assume that the event Bi occurs, i.e., we fix
Mick’s birthday to be the i-th day of the year. Given this event Bi , event A
occurs if and only if Keith’s birthday is also on the i-th day. Thus, we have
Pr (A | Bi ) = 1/365 and it follows that
365
X
Pr(A) = (1/365) · Pr (Bi )
i=1
365
X
= (1/365) Pr (Bi )
i=1
= (1/365) · 1
= 1/365,
which is the same answer as we got in the beginning of this section.
5.9.1 Flipping a Coin and Rolling Dice
Consider the following experiment:
• We flip a fair coin.
– If the coin comes up heads, then we roll a fair die. Let R denote
the result of this die.
– If the coin comes up tails, then we roll two fair dice. Let R denote
the sum of the results of these dice.
5.10. Please Take a Seat 201
What is the probability that the value of R is equal to 2? That is, if we
define the event A to be
A = “the value of R is equal to 2”,
then we want to know Pr(A). Since the value of R depends on whether the
coin comes up heads or tails, we define the event
B = “the coin comes up heads”.
Since (i) both B and its complement B occur with a positive probability
and (ii) exactly one of B and B is guaranteed to occur, we can apply Theo-
rem 5.9.1 and get
Pr(A) = Pr(A | B) · Pr(B) + Pr A | B · Pr B .
We determine the four terms on the right-hand side:
• It should be clear that
Pr(B) = Pr B = 1/2.
• To determine Pr(A | B), we assume that the event B occurs, i.e., the
coin comes up heads. Because of this assumption, we roll one die, and
the event A occurs if and only if the result of this roll is 2. It follows
that
Pr(A | B) = 1/6.
• To determine Pr A | B , we assume that the event B occurs, i.e., the
coin comes up tails. Because of this assumption, we roll two dice, and
the event A occurs if and only if both rolls result in 1. Since there are
36 possible outcomes when rolling two dice, it follows that
Pr A | B = 1/36.
We conclude that
Pr(A) = Pr(A | B) · Pr(B) + Pr A | B · Pr B
= 1/6 · 1/2 + 1/36 · 1/2
= 7/72.
202 Chapter 5. Discrete Probability
5.10 Please Take a Seat
Let n ≥ 2 and k ≥ 0 be integers. There are n + k chairs C1 , C2 , . . . , Cn+k
inside a room. Outside this room, there are n people P1 , P2 , . . . , Pn . These
people are told to enter the room one by one, in increasing order of their
indices, and each person must sit down in the chair having her index: For
i = 1, 2, . . . , n, person Pi enters the room and sits down in chair Ci .
The first person P1 did not listen to the instructions and, instead of taking
chair C1 , chooses one of the n + k chairs uniformly at random and sits down
in the chosen chair. (Note that chair C1 may be the chosen chair.) From
then on, for i = 2, 3, . . . , n, person Pi checks if chair Ci is available. If this
is the case, then Pi sits down in chair Ci . Otherwise, Pi chooses one of the
available chairs uniformly at random and sits down in the chosen chair.
We want to determine the probability pn,k that, at the end, the last person
Pn sits in chair Cn . Before we analyze this probability, we present this process
in pseudocode:
Algorithm TakeASeat(n, k):
// n ≥ 2 and k ≥ 0;
// the input consists of n people P1 , P2 , . . . , Pn and
// n + k chairs C1 , C2 , . . . , Cn+k
j = uniformly random element in {1, 2, . . . , n + k};
person P1 sits down in chair Cj ;
for i = 2 to n
do if chair Ci is available
then person Pi sits down in chair Ci
else j = index of a uniformly random available chair;
person Pi sits down in chair Cj
endif
endfor
We consider the event
An,k = “after algorithm TakeASeat(n, k) has terminated,
person Pn sits in chair Cn ”.
The probability that was mentioned above is given by
pn,k = Pr (An,k ) .
5.10. Please Take a Seat 203
In the for-loop in algorithm TakeASeat(n, k), the variable i runs from
2 to n. We will label the iterations of this loop by the value of the variable i.
Thus, iteration 3 will refer to the iteration in which i = 3; observe that this
is actually the second time that the algorithm goes through the for-loop.
At this moment, you should convince yourself (for example, by induction
on i) that the following holds for each i = 2, 3, . . . , n:
• At the start of iteration i,
– all chairs C2 , C3 , . . . , Ci−1 have been taken, and
– exactly one of the chairs C1 , Ci , Ci+1 , . . . , Cn+k has been taken.
If we take i = n, then we see that at the start of iteration n
• all chairs C2 , C3 , . . . , Cn−1 have been taken, and
• exactly one of the k + 2 chairs C1 , Cn , Cn+1 , . . . , Cn+k has been taken.
Event An,k occurs if and only if chair Cn is available (i.e., has not been taken)
at the start of iteration n.
Is it true that the chair among C1 , Cn , Cn+1 , . . . , Cn+k that has been taken
at the start of iteration n is a uniformly random chair from these k +2 chairs?
If this is the case, then chair Cn has been taken with probability 1/(k + 2)
and, thus, Cn is available with probability 1 − 1/(k + 2) = (k + 1)/(k + 2).
In other words, if the question above has a positive answer, then
k+1
pn,k = Pr (An,k ) = .
k+2
In the rest of this section, we will present two ways to prove that this is
indeed the correct value of pn,k . In both proofs, we will use the Law of Total
Probability of Section 5.9.
Note that pn,k does not depend on n. In particular, if k = 0, then the
probability that person Pn sits in chair Cn is equal to 1/2.
5.10.1 Determining pn,k Using a Recurrence Relation
Let us start with the case when n = 2. Thus, there are two people P1 and P2 ,
and 2 + k chairs C1 , C2 , . . . , C2+k . Event A2,k occurs if and only if P1 chooses
204 Chapter 5. Discrete Probability
one of the 1 + k chairs C1 , C3 , C4 , . . . , C2+k . Since P1 chooses a uniformly
random chair out of 2 + k chairs, it follows that
k+1
p2,k = Pr (A2,k ) = .
k+2
Assume from now on that n ≥ 3. We are going to derive a recurrence
relation that expresses pn,k in terms of p2,k , p3,k , . . . , pn−1,k .
Consider the (random) index j of the chair that P1 chooses in the first
line of algorithm TakeASeat(n, k). We consider three cases, depending on
the value of j.
• Assume that j ∈ {1, n + 1, n + 2, . . . , n + k}. Then for each i =
2, 3, . . . , n, chair Ci is available at the start of iteration i and person Pi
sits down in chair Ci . In particular, during iteration n, Pn sits down in
chair Cn and event An,k occurs.
• Assume that j = n. Then chair Cn has been taken at the start of
iteration n and event An,k does not occur.
• Assume that j ∈ {2, 3, . . . , n − 1}. Then for each i = 2, 3, . . . , j − 1,
chair Ci is available at the start of iteration i and person Pi sits down in
chair Ci . At the start of iteration j, the chairs C1 , Cj+1 , Cj+2 , . . . , Cn+k
are available and person Pj chooses one of these chairs uniformly at ran-
dom. Thus, iterations j, j +1, . . . , n can be viewed as running algorithm
TakeASeat(n−j+1, k), where the n−j+1 people are Pj , Pj+1 , . . . , Pn
and the n − j + 1 + k chairs are C1 , Cj+1 , Cj+2 , . . . , Cn+k . In this case,
event An,k occurs if and only if, after algorithm TakeASeat(n − j +
1, k) has terminated, person Pn sits in chair Cn , i.e., event An−j+1,k
occurs.
Thus, we can determine the probability that event An,k occurs, if we are given
the value of j; note that this is a conditional probability. Since j is a random
element in the set {1, 2, . . . , n + k}, we are going to use the Law of Total
Probability (Theorem 5.9.1): For each j ∈ {1, 2, . . . , n + k}, we consider the
event
Bn,k,j = “in the second line of algorithm TakeASeat(n, k),
person P1 sits down in chair Cj ”.
5.10. Please Take a Seat 205
Since exactly one of these events is guaranteed to occur, we can apply The-
orem 5.9.1 and obtain
n+k
X
Pr (An,k ) = Pr (An,k | Bn,k,j ) · Pr (Bn,k,j ) .
j=1
It follows from the first line in algorithm TakeASeat(n, k) that, for each j
with 1 ≤ j ≤ n + k,
1
Pr (Bn,k,j ) = .
n+k
• Assume that j ∈ {1, n + 1, n + 2, . . . , n + k}. We have seen above that
event An,k occurs. Thus,
Pr (An,k | Bn,k,j ) = 1.
• Assume that j = n. We have seen above that event An,k does not
occur. Thus,
Pr (An,k | Bn,k,n ) = 0.
• Assume that j ∈ {2, 3, . . . , n − 1}. We have seen above that event An,k
occurs if and only if event An−j+1,k occurs. Thus,
Pr (An,k | Bn,k,j ) = Pr (An−j+1,k ) = pn−j+1,k .
We conclude that
pn,k = Pr (An,k )
n+k
X
= Pr (An,k | Bn,k,j ) · Pr (Bn,k,j )
j=1
n+k
X 1
= Pr (An,k | Bn,k,j ) ·
j=1
n+k
n+k
1 X
= Pr (An,k | Bn,k,j )
n + k j=1
n−1
!
1 X
= (k + 1) + pn−j+1,k .
n+k j=2
206 Chapter 5. Discrete Probability
If we write out the terms in this summation, then we get, for each n ≥ 3,
k+1 1
pn,k = + (p2,k + p3,k + · · · + pn−1,k ) .
n+k n+k
As we have seen above, the base case is given by
k+1
p2,k = .
k+2
It remains to solve the recurrence relation. If you use the recurrence to
determine pn,k for some small values of n, then you will notice that they are
all equal to (k + 1)/(k + 2). This suggests that
k+1
pn,k =
k+2
for all integers n ≥ 2. (Recall that we already suspected this.) Using induc-
tion on n, it can easily be proved that this is indeed the case.
5.10.2 Determining pn,k by Modifying the Algorithm
Our second solution is obtained by modifying algorithm TakeASeat(n, k):
Person P1 again did not listen to the instructions and, instead of taking
chair C1 , chooses one of the n + k chairs uniformly at random and sits down
in the chosen chair. From then on, for i = 2, 3, . . . , n − 1, person Pi checks
if chair Ci is available. If this is the case, then Pi sits down in chair Ci .
Otherwise, P1 is sitting in Ci , in which case (i) Pi kicks P1 out of chair Ci ,
(ii) Pi sits down in chair Ci , and (iii) P1 chooses one of the available chairs
uniformly at random and sits down in the chosen chair. In the last step,
person Pn checks if chair Cn is available. If this is the case, then Pn sits down
in chair Cn . Otherwise, Pn chooses one of the available chairs uniformly at
random and sits down in the chosen chair. In pseudocode, this modified
algorithm looks as follows:
5.10. Please Take a Seat 207
Algorithm TakeASeat0 (n, k):
// n ≥ 2 and k ≥ 0;
// the input consists of n people P1 , P2 , . . . , Pn and
// n + k chairs C1 , C2 , . . . , Cn+k
j = uniformly random element in {1, 2, . . . , n + k};
person P1 sits down in chair Cj ;
for i = 2 to n − 1
do // P2 sits in C2 , P3 sits in C3 , . . . , Pi−1 sits in Ci−1
if chair Ci has been taken
then // P1 sits in Ci
j = uniformly random element
in {1, i + 1, i + 2, . . . , n + k};
person P1 sits down in chair Cj
endif;
person Pi sits down in chair Ci
endfor;
// P2 sits in C2 , P3 sits in C3 , . . . , Pn−1 sits in Cn−1
if chair Cn is available
then person Pn sits down in chair Cn
else j = uniformly random element
in {1, n + 1, n + 2, . . . , n + k};
person Pn sits down in chair Cj
endif
As in the previous subsection, we label the iterations of the for-loop by
the value of the variable i. Moreover, we consider the first two lines of the
algorithm to be iteration 1. Thus, up to the end of the for-loop, algorithm
TakeASeat0 (n, k) makes iterations that are labeled 1, 2, . . . , n − 1.
It follows from algorithm TakeASeat0 (n, k) that, after the for-loop has
terminated,
• for each i = 2, 3, . . . , n − 1, person Pi sits in chair Ci , and
• person P1 sits in one of the chairs C1 , Cn , Cn+1 , . . . , Cn+k .
Recall that An,k is the event that person Pn sits in chair Cn , after the
original algorithm TakeASeat(n, k) has terminated. It follows from the
modified algorithm TakeASeat0 (n, k) that event An,k occurs if and only if,
208 Chapter 5. Discrete Probability
after the for-loop of algorithm TakeASeat0 (n, k) has terminated, person P1
sits in one of the chairs C1 , Cn+1 , Cn+2 , . . . , Cn+k . In other words, pn,k =
Pr (An,k ) is equal to the probability that, after the for-loop has terminated,
P1 sits in one of C1 , Cn+1 , Cn+2 , . . . , Cn+k .
We have seen that, after the for-loop has terminated, P1 sits in one of the
chairs C1 , Cn , Cn+1 , . . . , Cn+k . Thus, there is a value of i with 1 ≤ i ≤ n − 1,
such that P1 sits down in one of these chairs during iteration i. As soon as P1
sits down in one of these chairs, P1 stays there until the end of the algorithm.
This implies that there is exactly one integer i having this property. Based
on this, and since this integer i is random, we are again going to use the
Law of Total Probability (Theorem 5.9.1): For each i ∈ {1, 2, . . . , n − 1}, we
consider the event
Bn,k,i = “during iteration i, person P1 chooses one of
the chairs C1 , Cn , Cn+1 , . . . , Cn+k .”
Since exactly one of these events is guaranteed to occur, Theorem 5.9.1 im-
plies that
n−1
X
Pr (An,k ) = Pr (An,k | Bn,k,i ) · Pr (Bn,k,i ) .
i=1
Consider the event
Bn,k = “a uniformly random element from the set
{1, n, n + 1, . . . , n + k} is not equal to n.”
Then for each i with 1 ≤ i ≤ n − 1, we have
k+1
Pr (An,k | Bn,k,i ) = Pr (Bn,k ) = .
k+2
It follows that
pn,k = Pr (An,k )
n−1
X k+1
= · Pr (Bn,k,i )
i=1
k + 2
n−1
k+1X
= Pr (Bn,k,i )
k + 2 i=1
k+1
= ,
k+2
because the last summation is equal to 1.
5.11. Independent Events 209
5.11 Independent Events
Consider two events A and B in a sample space S. In this section, we will
define the notion of these two events being “independent”. Intuitively, this
should express that (i) the probability that event A occurs does not depend
on whether or not event B occurs, and (ii) the probability that event B
occurs does not depend on whether or not event A occurs. Thus, if we
assume that Pr(A) > 0 and Pr(B) > 0, then (i) Pr(A) should be equal to
the conditional probability Pr(A | B), and (ii) Pr(B) should be equal to
the conditional probability Pr(B | A). As we will show below, the following
definition exactly captures this.
Definition 5.11.1 Let (S, Pr) be a probability space and let A and B be
two events. We say that A and B are independent if
Pr(A ∩ B) = Pr(A) · Pr(B).
In this definition, it is not assumed that Pr(A) > 0 and Pr(B) > 0. If
Pr(B) > 0, then
Pr(A ∩ B)
Pr(A | B) = ,
Pr(B)
and A and B are independent if and only if
Pr(A | B) = Pr(A).
Similarly, if Pr(A) > 0, then A and B are independent if and only if
Pr(B | A) = Pr(B).
5.11.1 Rolling Two Dice
Assume we roll a red die and a blue die; thus, the sample space is
S = {(i, j) : 1 ≤ i ≤ 6, 1 ≤ j ≤ 6},
where i is the result of the red die and j is the result of the blue die. We
assume a uniform probability function. Thus, each outcome has a probability
of 1/36.
210 Chapter 5. Discrete Probability
Let D1 denote the result of the red die and let D2 denote the result of
the blue die. Consider the events
A = “D1 + D2 = 7”
and
B = “D1 = 4”.
Are these events independent?
• Since
A = {(1, 6), (2, 5), (3, 4), (4, 3), (5, 2), (6, 1)},
we have Pr(A) = 6/36 = 1/6.
• Since
B = {(4, 1), (4, 2), (4, 3), (4, 4), (4, 5), (4, 6)},
we have Pr(B) = 6/36 = 1/6.
• Since
A ∩ B = {(4, 3)},
we have Pr(A ∩ B) = 1/36.
• It follows that Pr(A ∩ B) = Pr(A) · Pr(B) and we conclude that A and
B are independent.
As an exercise, you should verify that the events
A0 = “D1 + D2 = 11”
and
B 0 = “D1 = 5”
are not independent.
Now consider the two events
A00 = “D1 + D2 = 4”
and
B 00 = “D1 = 4”.
5.11. Independent Events 211
Since A00 ∩ B 00 = ∅, we have
Pr (A00 ∩ B 00 ) = Pr(∅) = 0.
On the other hand, Pr (A00 ) = 1/12 and Pr (B 00 ) = 1/6. Thus,
Pr (A00 ∩ B 00 ) 6= Pr (A00 ) · Pr (B 00 )
and the events A00 and B 00 are not independent. This is not surprising: If we
know that B 00 occurs, then A00 does not occur, i.e., Pr (A00 | B 00 ) = 0. Thus,
the event B 00 has an effect on the probability that the event A00 occurs.
5.11.2 A Basic Property of Independent Events
Consider two events A and B in a sample space S. If these events are
independent, then the probability that A occurs does not depend on whether
or not B occurs. Since whether or not B occurs is the same as whether the
complement B does not or does occur, it should not be a surprise that the
events A and B are independent as well. The following lemma states that
this is indeed the case.
Lemma 5.11.2 Let (S, Pr) be a probability space and let A and B be two
events. If A and B are independent, then A and B are also independent.
Proof. To prove that A and B are independent, we have to show that
Pr A ∩ B = Pr(A) · Pr B .
Using Lemma 5.3.3, this is equivalent to showing that
Pr A ∩ B = Pr(A) · (1 − Pr(B)) . (5.4)
Since the events A ∩ B and A ∩ B are disjoint and
A = (A ∩ B) ∪ A ∩ B ,
it follows from Lemma 5.3.2 that
Pr(A) = Pr(A ∩ B) + Pr A ∩ B .
Since A and B are independent, we have
Pr(A ∩ B) = Pr(A) · Pr(B).
It follows that
Pr(A) = Pr(A) · Pr(B) + Pr A ∩ B ,
which is equivalent to (5.4).
212 Chapter 5. Discrete Probability
5.11.3 Pairwise and Mutually Independent Events
We have defined the notion of two events being independent. The following
definition generalizes this in two ways to sequences of events:
Definition 5.11.3 Let (S, Pr) be a probability space, let n ≥ 2, and let
A1 , A2 , . . . , An be a sequence of events.
1. We say that this sequence is pairwise independent if for any two distinct
indices i and j, the events Ai and Aj are independent, i.e.,
Pr (Ai ∩ Aj ) = Pr (Ai ) · Pr (Aj ) .
2. We say that this sequence is mutually independent if for all k with
2 ≤ k ≤ n and all indices i1 < i2 < . . . < ik ,
Pr (Ai1 ∩ Ai2 ∩ · · · ∩ Aik ) = Pr (Ai1 ) · Pr (Ai2 ) · · · Pr (Aik ) .
Thus, in order to show that the sequence A1 , A2 , . . . , An is pairwise inde-
n
pendent, we have to verify 2 equalities. On the otherPhand, to show that
this sequence is mutually independent, we have to verify nk=2 nk = 2n −1−n
equalities.
For example, if we want to prove that the sequence A, B, C of three events
is mutually independent, then we have to show that
Pr(A ∩ B) = Pr(A) · Pr(B),
Pr(A ∩ C) = Pr(A) · Pr(C),
Pr(B ∩ C) = Pr(B) · Pr(C),
and
Pr(A ∩ B ∩ C) = Pr(A) · Pr(B) · Pr(C).
To give an example, consider flipping a coin three times and assume that
the result is a uniformly random element from the sample space
S = {HHH, HHT, HT H, T HH, HT T, T HT, T T H, T T T },
where, e.g., HHT indicates that the first two flips result in heads and the
third flip results in tails. For i = 1, 2, 3, let fi denote the result of the i-th
flip, and consider the events
A = “f1 = f2 ”,
5.12. Describing Events by Logical Propositions 213
B = “f2 = f3 ”,
and
C = “f1 = f3 ”.
If we write these events as subsets of the sample space, then we get
A = {HHH, HHT, T T H, T T T },
B = {HHH, T HH, HT T, T T T },
and
C = {HHH, HT H, T HT, T T T }.
It follows that
Pr(A) = |A|/|S| = 4/8 = 1/2,
Pr(B) = |B|/|S| = 4/8 = 1/2,
Pr(C) = |C|/|S| = 4/8 = 1/2,
Pr(A ∩ B) = |A ∩ B|/|S| = 2/8 = 1/4,
Pr(A ∩ C) = |A ∩ C|/|S| = 2/8 = 1/4,
Pr(B ∩ C) = |B ∩ C|/|S| = 2/8 = 1/4.
Thus, the sequence A, B, C is pairwise independent. Since
A ∩ B ∩ C = {HHH, T T T },
we have
Pr(A ∩ B ∩ C) = |A ∩ B ∩ C|/|S| = 2/8 = 1/4.
Thus,
Pr(A ∩ B ∩ C) 6= Pr(A) · Pr(B) · Pr(C)
and, therefore, the sequence A, B, C is not mutually independent. Of course,
this is not surprising: If both events A and B occur, then event C also occurs.
5.12 Describing Events by Logical Proposi-
tions
We have defined an event to be a subset of a sample space. In several
examples, however, we have described events in plain English or as logical
propositions.
214 Chapter 5. Discrete Probability
• Since the intersection (∩) of sets corresponds to the conjunction (∧) of
propositions, we often write A ∧ B for the event “both A and B occur”.
• Similarly, since the union (∪) of sets corresponds to the disjunction (∨)
of propositions, we often write A ∨ B for the event “A or B occurs”.
5.12.1 Flipping a Coin and Rolling a Die
If we flip a coin and roll a die, the sample space is
S = {H1, H2, H3, H4, H5, H6, T 1, T 2, T 3, T 4, T 5, T 6}.
The events
A = “the coin comes up heads”
and
B = “the result of the die is even”
correspond to the subsets
A = {H1, H2, H3, H4, H5, H6}
and
B = {H2, H4, H6, T 2, T 4, T 6}
of the sample space S, respectively. The event that both A and B occur is
written as A ∧ B and corresponds to the subset
A ∩ B = {H2, H4, H6}
of S. The event that A or B occurs is written as A ∨ B and corresponds to
the subset
A ∪ B = {H1, H2, H3, H4, H5, H6, T 2, T 4, T 6}
of S.
Assume that both the coin and the die are fair, and the results of rolling
the die and flipping the coin are independent. The probability that both A
and B occur, i.e., Pr(A ∧ B), is equal to |A ∩ B|/|S| = 3/12 = 1/4. We can
also use independence to determine this probability:
Pr(A ∧ B) = Pr(A) · Pr(B) = 1/2 · 3/6 = 1/4.
5.12. Describing Events by Logical Propositions 215
Observe that when we determine Pr(A), we do not consider the entire sample
space S. Instead, we consider the coin’s sample space, which is {H, T }.
Similarly, when we determine Pr(B), we consider the die’s sample space,
which is {1, 2, 3, 4, 5, 6}.
The probability that A or B occurs, i.e., Pr(A ∨ B), is equal to
Pr(A ∨ B) = |A ∪ B|/|S| = 9/12 = 3/4.
5.12.2 Flipping Coins
Let n ≥ 2 be an integer and assume we flip n fair coins. For each i with
1 ≤ i ≤ n, consider the event
Ai = “the i-th coin comes up heads”.
We assume that the coin flips are independent of each other, by which we
mean that the sequence A1 , A2 , . . . , An of events is mutually independent.
Consider the event
A = A1 ∧ A2 ∧ · · · ∧ An .
What is Pr(A), i.e., the probability that all n coins come up heads? Since
there are 2n many possible outcomes for n coin flips and only one of them
satisfies event A, this probability is equal to 1/2n . Alternatively, we can use
independence to determine Pr(A):
Pr(A) = Pr (A1 ∧ A2 ∧ · · · ∧ An )
= Pr (A1 ) · Pr (A2 ) · · · Pr (An ) .
Since each coin is fair, we have Pr (Ai ) = 1/2 and, thus, we get
Pr(A) = (1/2) · (1/2) · · · (1/2) = (1/2)n = 1/2n .
| {z }
n times
5.12.3 The Probability of a Circuit Failing
Consider a circuit C that consists of n components C1 , C2 , . . . , Cn . Let p
be a real number with 0 < p < 1 and assume that any component fails
with probability p, independently of the other components. For each i with
1 ≤ i ≤ n, consider the event
Ai = “component Ci fails”.
216 Chapter 5. Discrete Probability
Let A be the event
A = “the entire circuit fails”.
• Assume that the entire circuit fails when at least one component fails.
What is Pr(A), i.e., the probability that the circuit fails? By our as-
sumption, we have
A = A1 ∨ A2 ∨ · · · ∨ An
and, thus, using De Morgan’s Law,
A = A1 ∧ A2 ∧ · · · ∧ An .
Using independence and Lemmas 5.3.3 and 5.11.2, we get
Pr(A) = 1 − Pr A
= 1 − Pr A1 ∧ A2 ∧ · · · ∧ An
= 1 − Pr A1 · Pr A2 · · · Pr An
= 1 − (1 − p)(1 − p) · · · (1 − p)
| {z }
n times
= 1 − (1 − p)n .
Since 0 < p < 1, we have limn→∞ Pr(A) = 1. We conclude that for
large values of n, it is very likely that the circuit fails.
• Now assume that the entire circuit fails when all components fail.
Again, we want to know the probability Pr(A) that the circuit fails.
In this case, we have
A = A1 ∧ A2 ∧ · · · ∧ An ,
and we get
Pr(A) = Pr (A1 ∧ A2 ∧ · · · ∧ An )
= Pr (A1 ) · Pr (A2 ) · · · Pr (An )
= p · p···p
| {z }
n times
= pn .
Since 0 < p < 1, we have limn→∞ Pr(A) = 0. Thus, for large values
of n, it is very likely that the circuit does not fail.
5.13. Choosing a Random Element in a Linked List 217
5.13 Choosing a Random Element in a Linked
List
Consider a linked list L. Each node u in L stores a pointer to its successor
node succ(u). If u is the last node in L, then u does not have a successor and
succ(u) = nil . We are also given a pointer to the first node head (L) of L.
head (L) u succ(u) nil
Our task is to choose, uniformly at random, a node in L. Thus, if this list
has n nodes, then each node must have a probability of 1/n of being chosen.
We assume that we are given a function Random: For any integer
i ≥ 1, a call to Random(i) returns a uniformly random element from the
set {1, 2, . . . , i}; the value returned is independent of all other calls to this
function.
To make the problem interesting, we assume that we do not know the
value of n, i.e., at the start, we do not know the number of nodes in the
list L. Also, we are allowed to only make one pass over this list. We will
prove below that the following algorithm solves the problem:
Algorithm ChooseRandomNode(L):
u = head (L);
i = 1;
while u 6= nil
do r = Random(i);
if r = 1
then x = u
endif;
u = succ(u);
i=i+1
endwhile;
return x
In one iteration of the while-loop, the call to Random(i) returns a uni-
formly random element r from the set {1, 2, . . . , i}. If r = 1, which happens
with probability 1/i, the value of x is set to the currently visited node. If
r 6= 1, which happens with probability 1−1/i, the value of x does not change
during this iteration of the while-loop. Thus,
218 Chapter 5. Discrete Probability
• in the first iteration, x is set to the first node of L with probability 1,
• in the second iteration, x is set to the second node of L with probability
1/2, whereas the value of x does not change with probability 1/2,
• in the third iteration, x is set to the third node of L with probability
1/3, whereas the value of x does not change with probability 2/3,
• in the last iteration, x is set to the last node of L with probability 1/|L|,
whereas the value of x does not change with probability (|L| − 1)/|L|.
We now prove that the output x of algorithm ChooseRandomNode(L)
is a uniformly random node of the list L. Let n denote the number of nodes
in L and let v be an arbitrary node in L. We will prove that, after the
algorithm has terminated, x = v with probability 1/n.
Let k be the integer such that v is the k-th node in L; thus, 1 ≤ k ≤ n.
We observe that, after the algorithm has terminated, x = v if and only if
• during the k-th iteration, the value of x is set to v, and
• for all i = k + 1, k + 2, . . . , n, during the i-th iteration, the value of x
does not change.
Consider the event
A = “after the algorithm has terminated, x = v”.
For each i with 1 ≤ i ≤ n, consider the event
Ai = “the value of x changes during the i-th iteration”.
Then
A = Ak ∧ Ak+1 ∧ Ak+2 ∧ Ak+3 ∧ · · · ∧ An .
Recall that we assume that the output of the function Random is inde-
pendent of all other calls to this function. This implies that the events
5.14. Long Runs in Random Bitstrings 219
A1 , A2 , . . . , An are mutually independent. It follows that
Pr(A) = Pr Ak ∧ Ak+1 ∧ Ak+2 ∧ Ak+3 ∧ · · · ∧ An
= Pr (Ak ) · Pr Ak+1 · Pr Ak+2 · Pr Ak+3 · · · Pr An
1 1 1 1 1
= · 1− · 1− · 1− ··· 1 −
k k+1 k+2 k+3 n
1 k k+1 k+2 n−1
= · · · ···
k k+1 k+2 k+3 n
1
= .
n
5.14 Long Runs in Random Bitstrings
Let n be a large integer and assume we flip a fair coin n times, where all flips
are mutually independent. If we write 0 for heads and 1 for tails, then we
obtain a random bitstring
R = r1 r2 . . . rn .
A run of length k is a substring of R, all of whose bits are the same. For
example, the bitstring
00111100101000011000
contains, among others, the following substrings in bold,
00111100101000011000,
which are runs of lengths 4, 2, and 1, respectively.
Would you be surprised to see a “long” run in the random bitstring R,
say a run of length about log n? Most people will answer this question with
“yes”. We will prove below, however, that the correct answer is “no”: The
probability that this happens is about 1−1/n2 ; thus, it converges to 1 when n
goes to infinity. In other words, you should be surprised if a random bitstring
does not contain a run of length about log n.
We choose a positive integer k and consider the event
A = “R contains a run of length at least k”.
220 Chapter 5. Discrete Probability
We are going to prove a lower bound on Pr(A) in terms of n and k. At the
end, we will show that by taking k to be slightly less than log n, we have
Pr(A) ≥ 1 − 1/n2 .
For each i with 1 ≤ i ≤ n − k + 1, we consider the event
Ai = “the substring of length k starting at position i is a run”.
Since a run of length at least k can start at any of the positions 1, 2, . . . , n −
k + 1, we have
A = A1 ∨ A2 ∨ · · · ∨ An−k+1 ,
implying that
Pr(A) = Pr (A1 ∨ A2 ∨ · · · ∨ An−k+1 ) .
Observe that the events A1 , A2 , . . . , An−k+1 are not pairwise disjoint. As a
result, the probability on the right-hand side is difficult to analyze; it requires
the Principle of Inclusion and Exclusion (see Section 3.5). Because of this,
we consider the complement of the event A, i.e., the event
A = “each run in R has length less than k”.
Using De Morgan’s Law, we get
A = A1 ∧ A2 ∧ · · · ∧ An−k+1 ,
where Ai is the complement of Ai , i.e, the event
Ai = “the substring of length k starting at position i is not a run”.
It follows that
Pr A = Pr A1 ∧ A2 ∧ · · · ∧ An−k+1 . (5.5)
We determine Pr Ai , by first determining Pr (Ai ). The event Ai occurs
if and only if
ri = ri+1 = · · · = ri+k−1 = 0
or
ri = ri+1 = · · · = ri+k−1 = 1.
Since the coin flips are mutually independent, it follows that
Pr (Ai ) = 1/2k + 1/2k = 1/2k−1
5.14. Long Runs in Random Bitstrings 221
and, therefore,
Pr Ai = 1 − Pr (Ai ) = 1 − 1/2k−1 .
Is the probability on the right-hand side of (5.5) equal to the product of
the individual probabilities? If the events A1 , A2 , . . . , An−k+1 are mutually
independent, then the answer is “yes”. However, it should be clear that, for
example, the events A1 and A2 are not independent: If we are told that event
A1 occurs, then the first k bits in the bitstring R are equal; let us say they
are all equal to 0. In this case, the probability that event A2 occurs is equal
to the probability that the (k + 1)-st bit in R is 0, which is equal to 1/2 and
not 1/2k−1 (assuming that k ≥ 3). It seems that we are stuck. Fortunately,
there is a way out:
Let us assume that the integer k is chosen such that n/k is an integer.
We divide the bitstring R = r1 r2 . . . rn into n/k blocks, each having length k.
Thus,
• the first block is the substring r1 r2 . . . rk ,
• the second block is the substring rk+1 rk+2 . . . r2k ,
• the third block is the substring r2k+1 r2k+2 . . . r3k ,
• the (n/k)-th block is the substring rn−k+1 rn−k+2 . . . rn .
For each i with 1 ≤ i ≤ n/k, we consider the event
Bi = “the i-th block is a run”.
Thus, the complement of Bi is the event
B i = “the i-th block is not a run”.
Since Bi = A(i−1)k+1 and B i = A(i−1)k+1 , we have
Pr B i = 1 − 1/2k−1 .
Observe that
• the events B 1 , B 2 , . . . , B n/k are mutually independent, because the
blocks do not overlap, and
• if the event A occurs, then the event B 1 ∧ B 2 ∧ · · · ∧ B n/k also occurs
(but, in general, the converse is not true!).
222 Chapter 5. Discrete Probability
Using Lemma 5.3.6, it follows that
Pr A ≤ Pr B 1 ∧ B 2 ∧ · · · ∧ B n/k
= Pr B 1 · Pr B 2 · · · Pr B n/k
= 1 − 1/2k−1 · 1 − 1/2k−1 · · · 1 − 1/2k−1
n/k
= 1 − 1/2k−1 .
Using the inequality 1 − x ≤ e−x , see (5.3), we get
k−1 k
1 − 1/2k−1 ≤ e−1/2 = e−2/2
and, thus,
k n/k
k
Pr A ≤ e−2/2 = e−2n/(k2 ) . (5.6)
Note that until now, k was arbitrary. We choose k to be
k = log n − 2 log log n.
Using basic properties of logarithms, see Section 2.4, we will show below that,
for this choice of k, the right-hand side in (5.6) is a “nice” function of n.
In Section 2.4, we have seen that
2log n = n
and
22 log log n = log2 n.
It follows that
2log n n
2k = 2log n−2 log log n = = .
22 log log n log2 n
Thus,
2n 2 log2 n
=
k2k k
2 log2 n
=
log n − 2 log log n
2 log2 n
≥
log n
= 2 log n
ln n
= 2
ln 2
≥ 2 ln n,
5.14. Long Runs in Random Bitstrings 223
implying that
k
Pr A ≤ e−2n/(k2 )
≤ e−2 ln n
= 1/n2 .
We conclude that, for the value of k chosen above,
Pr(A) = 1 − Pr A ≥ 1 − 1/n2 .
Thus, with probability at least 1 − 1/n2 , a random bitstring of length n
contains a run of length at least log n − 2 log log n.
We remark that we have been cheating, because we assumed that both
m
k and n/k are integers. Assume that n is of the form 22 , for some positive
integer m. Then both log n and log log n are integers and, thus, k is an
integer as well. In a correct derivation, we divide the bitstring R into bn/kc
blocks of size k and, if n/k is not an integer, one block of length less than k.
We then get
bn/kc
1 − 1/2k−1
Pr A ≤
bn/kc
−2/2k
≤ e
k
= e−2bn/kc/2 .
As we have seen before, for k = log n − 2 log log n, we have 2k = n/ log2 n.
Since
bn/kc > n/k − 1,
we get
2bn/kc 2(n/k − 1)
k
>
2 2k
(2 log2 n)(n/k − 1)
=
n
2 log2 n 2 log2 n
= −
k n
2 log2 n
≥ 2 ln n −
n
224 Chapter 5. Discrete Probability
and, thus,
k
≤ e−2bn/kc/2
Pr A
2
≤ e−2 ln n+(2 log n)/n
2
= e−2 ln n · e(2 log n)/n
= (1/n2 ) · 1 + O((log2 n)/n)
= 1/n2 + O (log2 n)/n3 .
This upper bound is larger than the upper bound we had before by only a
small additive factor of O((log2 n)/n3 ).
5.15 Infinite Probability Spaces
In Section 5.2, we defined a sample space to be any non-empty countable set.
All sample spaces that we have seen so far are finite. In some cases, infinite
(but countable) sample spaces arise in a natural way. To give an example,
assume we flip a fair coin repeatedly and independently until it comes up
heads for the first time. The sample space S is the set of all sequences of
coin flips that can occur. If we denote by T n H the sequence consisting of n
tails followed by one heads, then
S = {H, T H, T T H, T T T H, T T T T H, . . .}
= {T n H : n ≥ 0},
which is indeed an infinite set.
Since the coin is fair and the coin flips are mutually independent, the
outcome T n H has a probability of (1/2)n+1 , i.e.,
Pr (T n H) = (1/2)n+1 .
Recall that according to Definition 5.2.2, in order for this to be a valid
probability function, the sum of all probabilities must be equal to 1, i.e.,
the infinite series ∞ ∞
X X
Pr (T n H) = (1/2)n+1
n=0 n=0
must be equal to 1. Since you may have forgotten about infinite series, we
recall the definition in the following subsection.
5.15. Infinite Probability Spaces 225
5.15.1 Infinite Series
The divergent series are the invention of the devil, and it is a
shame to base on them any demonstration whatsoever.
— Niels Henrik Abel, 1828
Definition 5.15.1 Let a0 , a1 , a2 , . . . be an infinite sequence of real numbers.
If
N
X
lim an = lim (a0 + a1 + a2 + · · · + aN )
N →∞ N →∞
n=0
P∞
exists, then we say that the infinite series n=0 an converges. In this case,
the value of this infinite series is equal to
∞
X N
X
an = lim an .
N →∞
n=0 n=0
For example, let x be a real number with x 6= 1, and define an = xn for
n ≥ 0. We claim that
N N
X X
n 2 1 − xN +1 N
an = x = 1 + x + x + ··· + x = ,
n=0 n=0
1−x
which can be proved either by induction on N or by verifying that
(1 − x) 1 + x + x2 + · · · + xN = 1 − xN +1 .
If −1 < x < 1, then limN →∞ xN +1 = 0 and it follows that
∞
X N
X
xn = lim xn
N →∞
n=0 n=0
1 − xN +1
= lim
N →∞ 1−x
1
= .
1−x
We have proved the following result:
226 Chapter 5. Discrete Probability
Lemma 5.15.2 If x is a real number with −1 < x < 1, then
∞
X 1
xn = .
n=0
1−x
Now we can return to the coin flipping example that we saw in the be-
ginning of Section 5.15. If we take x = 1/2 in Lemma 5.15.2, then we get
∞
X ∞
X
n
Pr (T H) = (1/2)n+1
n=0 n=0
∞
X
= (1/2) (1/2)n
n=0
1
= (1/2) ·
1 − 1/2
= 1.
Thus, we indeed have a valid probability function on the infinite sample space
S = {T n H : n ≥ 0}.
The limit does not exist.
— Cady Heron (played by Lindsay Lohan),
— Mean Girls, 2004
We have seen in Lemma 5.15.2 that the infinite series ∞ n
P
n=0 x converges
if −1 < x < 1. It is not difficult to see that for all other values of x, the limit
N
X
lim xn
N →∞
n=0
does not exist. As a result, if x ≥ 1 or x ≤ −1, the infinite series ∞ n
P
n=0 x
does not converge. Another example of an infinite series that does not con-
verge is
∞
X
1/n = 1 + 1/2 + 1/3 + 1/4 + · · ·
n=1
In Section 6.8.3, we will prove that
N
X
1/n = 1 + 1/2 + 1/3 + 1/4 + · · · + 1/N
n=1
5.15. Infinite Probability Spaces 227
is about ln N . It follows that
N
X
lim 1/n
N →∞
n=0
is about
lim ln N,
N →∞
which clearly does not exist.
5.15.2 Who Flips the First Heads
Consider a game in which two players P1 and P2 take turns flipping, inde-
pendently, a fair coin. Thus, first P1 flips the coin, then P2 flips the coin,
then P1 flips the coin, then P2 flips the coin, etc. The player who flips heads
first is the winner of the game.
Who is more likely to win this game? Our intuition says that P1 has
an advantage, because he is the player who starts: If the first flip is heads,
then the game is over and P1 wins. We will prove below that this intuition
is correct: P1 has a probability of 2/3 of winning the game and, thus, the
winning probability of P2 is only 1/3.
The sample space S is the set of all sequences of coin flips that can occur.
Since the game is over as soon as a heads is flipped, we have
S = {T n H : n ≥ 0}.
Since P1 starts, the event
A = “P1 wins the game”
corresponds to the subset
A = {T n H : n ≥ 0 and n is even},
which we rewrite as
A = {T 2m H : m ≥ 0}.
The probability that P1 wins the game is equal to Pr(A). How do we deter-
mine this probability? According to (5.1) in Section 5.2,
X
Pr(A) = Pr(ω).
ω∈A
228 Chapter 5. Discrete Probability
Since each outcome ω in A is of the form T 2m H, we have
∞
X
Pr T 2m H .
Pr(A) =
m=0
Thus, we have
∞
X
Pr T 2m H
Pr(A) =
m=0
X∞
= (1/2)2m+1
m=0
∞
X
= (1/2) (1/2)2m
m=0
X∞
= (1/2) (1/4)m .
m=0
By taking x = 1/4 in Lemma 5.15.2, we get
1
Pr(A) = (1/2) · = 2/3.
1 − 1/4
Let B be the event
B = “P2 wins the game”.
Since either P1 or P2 wins the game, we have
Pr(B) = 1 − Pr(A) = 1 − 2/3 = 1/3.
Let us verify, using an infinite series, that Pr(B) is indeed equal to 1/3. The
event B corresponds to the subset
B = {T n H : n ≥ 0 and n is odd},
which we rewrite as
B = {T 2m+1 H : m ≥ 0}.
5.15. Infinite Probability Spaces 229
The probability that P2 wins the game is thus equal to
∞
X
Pr T 2m+1 H
Pr(B) =
m=0
X∞
= (1/2)2m+2
m=0
∞
X
= (1/4) (1/2)2m
m=0
X∞
= (1/4) (1/4)m
m=0
1
= (1/4) · = 1/3.
1 − 1/4
5.15.3 Who Flips the Second Heads
Let us change the game from the previous subsection: Again, the two players
P1 and P2 take turns flipping, independently, a fair coin, where P1 starts. The
game ends as soon as a second heads comes up. The player who flips the
second heads wins the game.
Before you continue reading: Who do you think has a higher probability
of winning this game?
In this game, a sequence of coin flips can occur if and only if (i) the
sequence contains exactly two heads and (ii) the last element in the sequence
is heads. Thus, the sample space S is given by
S = {T m HT n H : m ≥ 0, n ≥ 0}.
The event
A = “P1 wins the game”
corresponds to the subset
A = {T m HT n H : m ≥ 0, n ≥ 0, m + n is odd}.
Below, we will determine Pr(A), i.e., the probability that P1 wins the game.
We split the event A into two events
A1 = “P1 flips both the first and the second heads”
230 Chapter 5. Discrete Probability
and
A2 = “P2 flips the first heads and P1 flips the second heads”.
If we write these two events as subsets of the sample space S, we get
A1 = {T m HT n H : m ≥ 0, n ≥ 0, m is even and n is odd}
= {T 2k HT 2`+1 H : k ≥ 0, ` ≥ 0}
and
A2 = {T m HT n H : m ≥ 0, n ≥ 0, m is odd and n is even}
= {T 2k+1 HT 2` H : k ≥ 0, ` ≥ 0}.
Observe that A1 ∩ A2 = ∅ and A = A1 ∪ A2 , implying that
Pr(A) = Pr (A1 ) + Pr (A2 ) .
We determine the two probabilities on the right-hand side.
We have
∞ X
X ∞
Pr T 2k HT 2`+1 H
Pr (A1 ) =
k=0 `=0
∞ X
X ∞
= (1/2)2k+2`+3
k=0 `=0
∞
X ∞
X
3 2k
= (1/2 ) (1/2) (1/2)2`
k=0 `=0
∞
X ∞
X
= (1/8) (1/4)k (1/4)`
k=0 `=0
∞
X 1
= (1/8) (1/4)k ·
k=0
1 − 1/4
∞
X
= (1/6) (1/4)k
k=0
1
= (1/6) ·
1 − 1/4
= 2/9
5.16. Exercises 231
and
∞ X
X ∞
Pr T 2k+1 HT 2` H
Pr (A2 ) =
k=0 `=0
∞ X
X ∞
= (1/2)2k+2`+3
k=0 `=0
= 2/9.
Thus, the probability that P1 wins the game is equal to
Pr(A) = Pr (A1 ) + Pr (A2 )
= 2/9 + 2/9
= 4/9.
The probability that P2 wins the game is equal to
1 − Pr(A) = 5/9.
Thus, P2 has a slightly larger probability of winning the game.
You will agree that this was a painful way of determining Pr(A). In Exer-
cise 5.91, you will see an easier way to determine this probability: The game
of this subsection can be seen as two rounds of the game in Section 5.15.2.
This observation, together with the Law of Total Probability (Theorem 5.9.1)
leads to an easier way to prove that Pr(A) = 4/9.
5.16 Exercises
5.1 Consider a coin that has 0 on one side and 1 on the other side. We flip
this coin once and roll a die twice, and are interested in the product of the
three numbers.
• What is the sample space?
• How many possible events are there?
• If both the coin and the die are fair, how would you define the proba-
bility function Pr for this sample space?
232 Chapter 5. Discrete Probability
5.2 Consider the sample space S = {a, b, c, d} and a probability function
Pr : S → R on S. Consider the events A = {a}, B = {a, b}, C = {a, b, c},
and D = {b, d}. You are given that Pr(A) = 1/10, Pr(B) = 1/2, and
Pr(C) = 7/10. Determine Pr(D).
5.3 Let n be a positive integer. We flip a fair coin 2n times and consider the
possible outcomes, which are strings of length 2n with each character being
H (= heads) or T (= tails). Thus, we take the sample space S to be the set
of all such strings. Since our coin is fair, each string of S should have the
same probability. Thus, we define Pr(s) = 1/|S| for each string s in S. In
other words, we have a uniform probability space.
You are asked to determine the probability that in the sequence of 2n
flips, the coin comes up heads exactly n times:
• What is the event A that describes this?
• Determine Pr(A).
5.4 A cup contains two pennies (P), one nickel (N), and one dime (D). You
choose one coin uniformly at random, and then you choose a second coin
from the remaining coins, again uniformly at random.
• Let S be the sample space consisting of all ordered pairs of letters P,
N, and D that represent the possible outcomes. Write out all elements
of S.
• Determine the probability for each element in this sample space.
5.5 You are given a box that contains the 8 lowercase letters a, b, c, d, e, f, g, h
and the 5 uppercase letters V, W, X, Y, Z.
In this exercise, we will consider two ways to choose 4 random letters
from the box. In the first way, we do uniform sampling without replacement,
whereas in the second way, we do uniform sampling with replacement. For
each case, you are asked to determine the probability that the 4-th letter
chosen is an uppercase letter. Before starting this exercise, spend a few
minutes and guess for which case this probability is smaller.
• You choose 4 letters from the box: These letters are chosen in 4 steps,
and in each step, you choose a uniformly random letter from the box;
this letter is removed from the box.
5.16. Exercises 233
– What is the sample space?
– Consider the event
A = “the 4-th letter chosen is an uppercase letter ”.
Determine Pr(A).
• You choose 4 letters from the box: These letters are chosen in 4 steps,
and in each step, you choose a uniformly random letter from the box;
this letter is not removed from the box.
– What is the sample space?
– Consider the event
B = “the 4-th letter chosen is an uppercase letter”.
Determine Pr(B).
5.6 You flip a fair coin, independently, six times.
• What is the sample space?
• Consider the events
A = “the coin comes up heads at least four times”,
B = “the number of heads is equal to the number of tails”,
C = “there are at least four consecutive heads”.
Determine Pr(A), Pr(B), Pr(C), Pr(A | B), and Pr(C | A).
5.7 Let k ≥ 2 be an integer and consider the sample space S consisting
of all sequences of k characters, where each character is one of the digits
0, 1, 2, . . . , 9.
If we choose a sequence s uniformly at random from the sample space S,
what is the probability that none of the digits in s is equal to 5?
5.8 You are given a red coin and a blue coin. Both coins have the number
1 on one side and the number 2 on the other side. You flip both coins once
(independently of each other) and take the sum of the two results. Consider
the events
A = “the sum of the results equals 2”,
B = “the sum of the results equals 3”,
C = “the sum of the results equals 4”.
234 Chapter 5. Discrete Probability
• Assume both coins are fair. Determine Pr(A), Pr(B), and Pr(C).
• Let p and q be real numbers with 0 < p < 1 and 0 < q < 1. Assume
the red coin comes up “1” with probability p and the blue coin comes
up “1” with probability q. Is it possible to choose p and q such that
Pr(A) = Pr(B) = Pr(C)?
5.9 Let p1 , p2 , . . . , p6 , q1 , q2 , . . . , q6 be real numbers such that each pi is strictly
positive, each qi is strictly positive, and p1 +p2 +· · ·+p6 = q1 +q2 +· · ·+q6 = 1.
You are given a red die and a blue die. For any i with 1 ≤ i ≤ 6, if you
roll the red die, then the result is i with probability pi , and if you roll the
blue die, then the result is i with probability qi .
You roll each die once (independently of each other) and take the sum of
the two results. For any s ∈ {2, 3, . . . , 12}, consider the event
As = “the sum of the results equals s”.
• Let x > 0 and y > 0 be real numbers. Prove that
x y
+ ≥ 2.
y x
Hint: Rewrite this inequality until you get an equivalent inequality
which obviously holds.
• Assume that Pr (A2 ) = Pr (A12 ) and denote this common value by a.
Prove that
Pr (A7 ) ≥ 2a.
• Is it possible to choose p1 , p2 , . . . , p6 , q1 , q2 , . . . , q6 such that for any s ∈
{2, 3, . . . , 12}, Pr (As ) = 1/11?
5.10 The Fibonacci numbers are defined as follows: f0 = 0, f1 = 1, and
fn = fn−1 + fn−2 for n ≥ 2.
Let n be a large integer. A Fibonacci die is a die that has fn faces. Such
a die is fair: If we roll it, each face is on top with the same probability 1/fn .
There are three different types of Fibonacci dice:
• D1 : fn−2 of its faces show the number 1 and the other fn−1 faces show
the number 4.
5.16. Exercises 235
• D2 : Each face shows the number 3.
• D3 : fn−2 of its faces show the number 5 and the other fn−1 faces show
the number 2.
Assume we roll each of D1 , D2 , and D3 once, independently of each
other. Let R1 , R2 , and R3 be the numbers on the top faces of D1 , D2 , and
D3 , respectively. Determine
Pr(R1 > R2 )
and
Pr(R2 > R3 ),
and show that
fn−2 fn+1
Pr(R3 > R1 ) = .
fn2
5.11 You are given a fair die. If you roll this die repeatedly, then the results
of the rolls are independent of each other.
• You roll the die 6 times. Consider the event
A = “there is at least one 6 in this sequence of 6 rolls”.
Determine Pr(A).
• You roll the die 12 times. Consider the event
B = “there are at least two 6’s in this sequence of 12 rolls”.
Determine Pr(B).
• You roll the die 18 times. Consider the event
C = “there are at least three 6’s in this sequence of 18 rolls”.
Determine Pr(C).
Before starting this exercise, spend a few minutes and guess which of these
three probabilities is the smallest.
236 Chapter 5. Discrete Probability
5.12 When Tri is a big boy, he wants to have four children. Assuming that
the genders of these children are uniformly random, which of the following
three events has the highest probability?
1. All four kids are of the same gender.
2. Exactly three kids are of the same gender.
3. Two kids are boys and two kids are girls.
5.13 A group of ten people sits down, uniformly at random, around a table.
Lindsay and Simon are part of this group. Determine the probability that
Lindsay and Simon sit next to each other.
5.14 Consider five people, each of which has a uniformly random and inde-
pendent birthday. (We ignore leap years.) Consider the event
A = “at least three people have the same birthday”.
Determine Pr(A).
5.15 Donald Trump wants to hire two secretaries. There are n applicants
a1 , a2 , . . . , an , where n ≥ 2 is an integer. Each of these applicants has a
uniformly random birthday, and all birthdays are mutually independent. (We
ignore leap years.)
Since Donald is too busy making America great again, he does not have
time to interview the applicants. Instead, he uses the following strategy: If
there is an index i such that ai and ai+1 have the same birthday, then he
chooses the smallest such index i and hires ai and ai+1 . In this case, the
hiring process is a tremendous success. If such an index i does not exist,
then nobody is hired and the hiring process is a total disaster.
Determine the probability that the hiring process is a tremendous success.
5.16 Let d and n be integers such that d ≥ 1, n ≥ d, and n + d is even.
You live on Somerset Street and want to go to your local pub, which is also
located on Somerset Street, at distance d to the east from your home.
d
home pub
5.16. Exercises 237
You use the following strategy:
• Initially, you are at your home.
• For each i = 1, 2, . . . , n, you do the following:
– You flip a fair and independent coin.
– If the coin comes up heads, you walk a distance 1 to the east.
– If the coin comes up tails, you walk a distance 1 to the west.
Consider the event
A = “after these n steps, you are at your local pub”.
Prove that
n n
Pr(A) = n+d /2 .
2
5.17 In Section 5.4.1, we have seen the different cards that are part of a
standard deck of cards.
• You choose 2 cards uniformly at random from the 13 spades in a deck
of 52 cards. Determine the probability that you choose an Ace and a
King.
• You choose 2 cards uniformly at random from a deck of 52 cards. De-
termine the probability that you choose an Ace and a King.
• You choose 2 cards uniformly at random from a deck of 52 cards. De-
termine the probability that you choose an Ace and a King of the same
suit.
5.18 In Section 5.4.1, we have seen the different cards that are part of a
standard deck of cards.
A hand of cards is a subset consisting of five cards. A hand of cards is
called a straight, if the ranks of these five cards are consecutive and the cards
are not all of the same suit.
An Ace and a 2 are considered to be consecutive, whereas a King and an
Ace are also considered to be consecutive. For example, each of the three
hands below is a straight:
8♠, 9♥, 10♦, J♠, Q♣
238 Chapter 5. Discrete Probability
A♦, 2♥, 3♠, 4♠, 5♣
10♦, J♥, Q♠, K♠, A♣
• Assume you get a uniformly random hand of cards. Determine the
probability that this hand is a straight.
5.19 Three people P1 , P2 , and P3 are in a dark room. Each person has a bag
containing one red hat and one blue hat. Each person chooses a uniformly
random hat from her bag and puts it on her head. Afterwards, the lights are
turned on.
Each person does not know the color of her hat, but can see the colors of
the other two hats. Each person Pi can do one of the following:
• Person Pi announces “my hat is red”.
• Person Pi announces “my hat is blue”.
• Person Pi says “I pass”.
The game is a success if at least one person announces the correct color of
her hat and no person announces the wrong color of her hat. (If a person
passes, then she does not announce any color.)
• Assume person P1 announces “my hat is red” and both P2 and P3 pass.
Consider the event
A = “the game is a success”.
Determine Pr(A).
• Assume each person Pi does the following:
– If the two hats that Pi sees have different colors, then Pi passes.
– If the two hats that Pi sees are both red, then Pi announces “my
hat is blue”.
– If the two hats that Pi sees are both blue, then Pi announces “my
hat is red”.
Consider the event
B = “the game is a success”.
Determine Pr(B).
5.16. Exercises 239
5.20 Let A be an event in some probability space (S, Pr). You are given
that the events A and A are independent2 . Determine Pr(A).
5.21 You are given three events A, B, and C in some probability space
(S, Pr). Is the following true or false?
Pr A ∩ B ∩ C = Pr(A ∪ B ∪ C) − Pr(B) − Pr(C) + Pr (B ∩ C) .
5.22 Let S be a set consisting of 6 positive integers and 8 negative integers.
Choose a 4-element subset of S uniformly at random, and multiply the ele-
ments in this subset. Denote the product by x. Determine the probability
that x > 0.
5.23 Prove the inequality in (5.3), i.e., prove that
1 − x ≤ e−x
for all real numbers x.
5.24 Let (S, Pr) be a probability space and let B be an event with Pr(B) > 0.
Consider the function Pr0 : S → R by
(
Pr(ω)
if ω ∈ B,
Pr0 (ω) = Pr(B)
0 if ω 6∈ B.
• Prove that Pr0 is a probability function on S according to Defini-
tion 5.2.2.
• Prove that for any event A,
Pr(A ∩ B)
Pr0 (A) = .
Pr(B)
5.25 Consider two events A and B in some probability space (S, Pr).
• Assume that Pr(A) = 1/2 and Pr B | A = 3/5. Determine Pr(A∪B).
• Assume that Pr(A∪B) = 5/6 and Pr A | B = 1/3. Determine Pr(B).
2
This is not a typo.
240 Chapter 5. Discrete Probability
5.26 Give an example of a sample space S and six events A, B, C, D, E,
and F such that
• Pr(A | B) = Pr(A),
• Pr(C | D) < Pr(C),
• Pr(E | F ) > Pr(E).
Hint: The sequence of six events may contain duplicates. Try to make the
sample space S as small as you can.
5.27 You roll a fair die twice. Consider the events
A = “the sum of the two rolls is 7”,
B = “the result of the first roll is 4”.
Determine the conditional probabilities Pr(A | B) and Pr(B | A).
5.28 You flip a fair coin three times. Consider the four events (recall that
zero is even)
A = “the coin comes up heads an odd number of times”,
B = “the coin comes up heads an even number of times”,
C = “the coin comes up tails an odd number of times”,
D = “the coin comes up tails an even number of times”.
• Determine Pr(A), Pr(B), Pr(C), Pr(D), Pr(A | C), and Pr(A | D).
• Are there any two events in the sequence A, B, C, and D that are
independent?
5.29 Consider a box that contains four beer bottles b1 , b2 , b3 , b4 and two cider
bottles c1 , c2 . You choose a uniformly random bottle from the box (and do
not put it back), after which you again choose a uniformly random bottle
from the box.
Consider the events
A = “the first bottle chosen is a beer bottle”,
B = “the second bottle chosen is a beer bottle”.
5.16. Exercises 241
• What is the sample space?
• For each element ω in your sample space, determine Pr(ω).
• Determine Pr(A).
• Determine Pr(B).
• Are the events A and B independent?
5.30 A standard deck of 52 cards contains 13 spades (♠), 13 hearts (♥), 13
clubs (♣), and 13 diamonds (♦). You choose a uniformly random card from
this deck. Consider the events
A = “the chosen card is a clubs or a diamonds card”,
B = “the chosen card is a clubs or a hearts card”,
C = “the chosen card is a clubs or a spades card”.
• Are the events A, B, and C pairwise independent?
• Are the events A, B, and C mutually independent?
5.31 You roll a fair die twice. Consider the events
A = “the sum of the results is at least 9”,
B = “at least one of the two rolls results in 2”,
C = “at least one of the two rolls results in 5”.
• Determine Pr(A), Pr(B), and Pr(C).
• Determine Pr(B | C).
• Are the events A and B independent?
• Are the events A and C independent?
5.32 A hand of 13 cards is chosen uniformly at random from a standard deck
of 52 cards. Consider the events
A = “the hand has at least one Ace”,
B = “the hand has at least two Aces”,
C = “the hand has the Ace of spades”.
Determine the conditional probabilities Pr(A | B), Pr(B | A), and Pr(B | C).
242 Chapter 5. Discrete Probability
5.33 We take a uniformly random permutation of a standard deck of 52
cards, so that each permutation has a probability of 1/52!. Consider the
events
A = “the top card is an Ace”,
B = “the bottom card is the Ace of spades”,
C = “the bottom card is the Queen of spades”.
Determine the conditional probabilities Pr(A | B) and Pr(A | C).
5.34 Consider two dice, each one having one face showing the letter a, two
faces showing the letter b, and the remaining three faces showing the letter c.
You roll each die once, independently of the other die.
• What is the sample space?
• Consider the events
A = “at least one of the two dice shows the letter b on its top face”,
B = “both dice show the same letter on their top faces”.
Determine Pr(A), Pr(B), and Pr(A | B).
5.35 You flip a fair coin, independently, three times. Consider the events
A = “the first flip results in heads”,
B = “the coin comes up heads exactly once”.
Determine the conditional probabilities Pr(A | B) and Pr(B | A).
5.36 You roll a fair die twice. Consider the events
A = “the sum of the results is even”,
B = “the sum of the results is at least 10”.
Determine the conditional probability Pr(A | B).
5.37 You flip a fair coin seven times, independently of each other. Consider
the events
A = “the number of heads is at least six”,
B = “the number of heads is at least five”,
C = “the number of tails is at least two”,
D = “the number of heads is at least four”.
Determine the conditional probabilities Pr(A | B) and Pr(C | D).
5.16. Exercises 243
5.38 Consider the set Y = {1, 2, 3, . . . , 10}. We choose, uniformly at ran-
dom, a 6-element subset X of Y . Consider the events
A = “5 is an element of X”,
B = “6 is an element of X”,
C = “6 is an element of X or 7 is an element of X”.
• Determine Pr(A), Pr(B), and Pr(C).
• Determine Pr(A | B), Pr(A | C), and Pr(B | C).
5.39 Let A and B be two events
in some probability space (S, Pr) such that
Pr(A) = 2/5 and Pr A ∪ B = 3/10.
• Assume that A and B are disjoint. Determine Pr(B).
• Assume that A and B are independent. Determine Pr(B).
5.40 In this exercise, we assume that, when a child is born, its gender is
uniformly random, its day of birth is uniformly random, the gender and day
of birth are independent of each other and independent of other children.
Anil Maheshwari has two children. You are given that at least one of
Anil’s kids is a boy who was born on a Sunday. Determine the probability
that Anil has two boys.
5.41 Elisa and Nick go to Tan Tran’s Darts Bar. When Elisa throws a dart,
she hits the dartboard with probability p. When Nick throws a dart, he
hits the dartboard with probability q. Here, p and q are real numbers with
0 < p < 1 and 0 < q < 1. Elisa and Nick throw one dart each, independently
of each other. Consider the events
E = “Elisa’s dart hits the dartboard”,
N = “Nick’s dart hits the dartboard”.
Determine Pr(E | E ∪ N ) and Pr(E ∩ N | E ∪ N ).
5.42 As everyone knows, Elisa Kazan loves to drink cider. You may not be
aware that Elisa is not a big fan of beer.
244 Chapter 5. Discrete Probability
Consider a round table that has six seats numbered 1, 2, 3, 4, 5, 6. Elisa
is sitting in seat 1. On top of the table, there is a rotating tray3 . On this
tray, there are five bottles of beer (B) and one bottle of cider (C), as in the
figure below. After the tray has been spun, there is always a bottle exactly
in front of Elisa. (In other words, you can only spin the tray by a multiple of
60 degrees.) Moreover, Elisa can only see the bottle that is in front of her.
4
3 B 5
B B
B B
2 C 6
1
Elisa spins the tray uniformly at random in clockwise order. After the
tray has come to a rest, there is a bottle of beer in front of her. Since Elisa is
obviously not happy, she gets a second chance, i.e., Elisa can choose between
one of the following two options:
1. Spin the tray again uniformly at random and independently of the first
spin. After the tray has come to a rest, Elisa must drink the bottle
that is in front of her.
2. Rotate the tray one position (i.e., 60 degrees) in clockwise order, after
which Elisa must drink the bottle that is in front of her.
• Elisa decides to go for the first option. Determine the probability that
she drinks the bottle of cider.
• Elisa decides to go for the second option. Determine the probability
that she drinks the bottle of cider.
5.43 You are given three dice D1 , D2 , and D3 :
3
According to Wikipedia, such a tray is called a Lazy Susan or Lazy Suzy. You may
have seen them in Chinese restaurants.
5.16. Exercises 245
• Die D1 has 0 on two of its faces and 1 on the other four faces.
• Die D2 has 0 on all six faces.
• Die D3 has 1 on all six faces.
You throw these three dice in a box so that they end up at uniformly random
orientations. You pick a uniformly random die in the box and observe that it
has 0 on its top face. Determine the probability that the die that you picked
is D1 .
Hint: You want to determine Pr(A | B), where A is the event that you pick
D1 and B is the event that you see a 0 on the top face of the die that you
picked. There are different ways to define the sample space S. One way is
to take
S = {(D1 , 0), (D1 , 1), (D2 , 0), (D3 , 1)},
where, for example, (D1 , 1) is the outcome in which you observe 1 on top of
die D1 . Note that this is not a uniform probability space.
5.44 According to Statistics Canada, a random person in Canada has
• a probability of 4/5 to live to at least 70 years old and
• a probability of 1/2 to live to at least 80 years old.
John (a random person in Canada) has just celebrated his 70-th birthday.
What is the probability that John will celebrate his 80-th birthday?
5.45 Nick is taking the course SPID 2804 (The Effect of Spiderman on the
Banana Industry). The final exam for this course consists of one true/false
question. To answer this question, Nick uses the following approach:
1. If Nick knows that the answer to the question is “true”, he
answers “true”.
2. If Nick knows that the answer is “false”, he answers “false”.
3. If Nick does not know the answer, he flips a fair coin.
(a) If the coin comes up heads, he answers “true”.
(b) If the coin comes up tails, he answers “false”.
246 Chapter 5. Discrete Probability
You are given that Nick knows the answer to the question with probabil-
ity 0.8. Consider the event
A = “Nick gives the correct answer to the question”.
Determine Pr(A).
5.46 Let A and B be events in some probability space (S, Pr), such that
Pr(A) 6= 0 and Pr(B) 6= 0. Use the definition of conditional probability to
prove Bayes’ Theorem:
Pr(B | A) · Pr(A)
Pr(A | B) = .
Pr(B)
5.47 Medical doctors have developed a test for detecting disease X.
• The test is 98% effective on people who have X: If a person has X,
then with probability 0.98, the test says that the person indeed has X.
• The test gives a false reading for 3% of the population without the
disease: If a person does not have X, then with probability 0.03, the
test says that the person does have X.
• It is known that 0.1% of the population has X.
Assume we choose a person uniformly at random from the population and
test this person for disease X.
• Determine the probability that the test says that the person has X.
• Assume the test says that the person has X. Use Exercise 5.46 to
determine the probability that the person indeed has X.
5.48 In this exercise, we consider a standard deck of 52 cards.
• We choose, uniformly at random, one card from the deck. Consider the
events
A = “the rank of the chosen card is Ace”,
B = “the suit of the chosen card is diamonds”.
Are the events A and B independent?
5.16. Exercises 247
• Assume we remove the Queen of hearts from the deck. We choose,
uniformly at random, one card from the remaining 51 cards. Consider
the events
C = “the rank of the chosen card is Ace”,
D = “the suit of the chosen card is diamonds”.
Are the events C and D independent?
5.49 Let n ≥ 2 and m ≥ 1 be integers and consider two sets A and B, where
A has size n and B has size m. We choose a uniformly random function
f : A → B. For any two integers i and k with 1 ≤ i ≤ n and 1 ≤ k ≤ m,
consider the event
Aik = “f (i) = k”.
• For two integers i and k, determine Pr (Aik ).
• For two distinct integers i and j, and for an integer k, are the two
events Aik and Ajk independent?
5.50 Consider three events A, B, and C in some probability space (S, Pr),
and assume that Pr(B ∩ C) 6= 0 and Pr(C) 6= 0. Prove that
Pr(A ∩ B ∩ C) = Pr(A | B ∩ C) · Pr(B | C) · Pr(C).
5.51 You have a fair die and do the following experiment:
• Roll the die once; let x be the outcome.
• Roll the die x times (independently); let y be the smallest outcome of
these x rolls.
• Roll the die y times (independently); let z be the largest outcome of
these y rolls.
Use Exercise 5.50 to determine
Pr(x = 1 and y = 2 and z = 3).
5.52 A standard deck of 52 cards has four Aces.
248 Chapter 5. Discrete Probability
• You get a uniformly random hand of three cards. Consider the event
A = “the hand consists of three Aces”.
Determine Pr(A).
• You get three cards, which are chosen one after another. Each of these
three cards is chosen uniformly at random from the current deck of
cards. (When a card has been chosen, it is removed from the current
deck.) Consider the events
B = “all three cards are Aces”
and, for i = 1, 2, 3,
Bi = “the i-th card is an Ace.”
Express the event B in terms of B1 , B2 , and B3 , and use this expression,
together with Exercise 5.50, to determine Pr(B).
5.53 Let p be a real number with 0 < p < 1. You are given two coins C1
and C2 . The coin C1 is fair, i.e., if you flip this coin, it comes up heads with
probability 1/2 and tails with probability 1/2. If you flip the coin C2 , it
comes up heads with probability p and tails with probability 1 − p. You pick
one of these two coins uniformly at random, and flip it twice. These two coin
flips are independent of each other. Consider the events
A = “the first coin flip results in heads”,
B = “the second coin flip results in heads”.
• Determine Pr(A).
• Assume that p = 1/4. Are the events A and B independent?
• Determine all values of p for which the events A and B are independent.
5.54 Let n ≥ 2 be an integer. Assume we have n balls and 10 boxes. We
throw the balls independently and uniformly at random in the boxes. Thus,
for each k and i with 1 ≤ k ≤ n and 1 ≤ i ≤ 10,
Pr( the k-th ball falls in the i-th box ) = 1/10.
5.16. Exercises 249
Consider the event
An = “there is a box that contains at least two balls”
and let pn = Pr (An ).
• Determine the smallest value of n for which pn ≥ 1/2.
• Determine the smallest value of n for which pn ≥ 2/3.
5.55 Donald Trump wants to hire a secretary and receives n applications for
this job, where n ≥ 1 is an integer. Since he is too busy in making important
announcements on Twitter, he appoints a three-person hiring committee.
After having interviewed the n applicants, each committee member ranks
the applicants from 1 to n. An applicant is hired for the job if he/she is
ranked first by at least two committee members.
Since the committee members do not have the ability to rank the appli-
cants, each member chooses a uniformly random ranking (i.e., permutation)
of the applicants, independently of each other.
John is one of the applicants. Determine the probability that John is
hired.
5.56 Edward, Francois-Xavier, Omar, and Yaser are sitting at a round table,
as in the figure below.
E
Y FX
O
At 11:59am, they all lower their heads. At noon, each of the boys chooses
a uniformly random element from the set {CW , CCW , O}; these choices are
independent of each other. If a boy chooses CW , then he looks at his clock-
wise neighbor, if he chooses CCW , then he looks at his counter-clockwise
neighbor, and if he chooses O, then he looks at the boy at the other side of
the table. When two boys make eye contact, they both shout Vive le Québec
libre.
250 Chapter 5. Discrete Probability
• Consider the event
A = “both Edward and Francois-Xavier shout Vive le Québec
libre, whereas neither Omar nor Yaser does”.
Determine Pr(A).
• Consider the event
B = “both Francois-Xavier and Yaser shout Vive le Québec
libre, whereas neither Edward nor Omar does”.
Determine Pr(B).
• For any integer i with 0 ≤ i ≤ 4, consider the event
Ci = “exactly i boys shout Vive le Québec libre”.
Determine
4
X
Pr (Ci ) .
i=0
Justify your answer in plain English.
• Determine each of the five probabilities Pr (C0 ), Pr (C1 ), . . . , Pr (C4 ).
5.57 You are given a fair die. For any integer n ≥ 1, you roll this die n times
(the rolls are independent). Consider the events
An = “the sum of the results of the n rolls is even”
and
Bn = “the last roll in the sequence of n rolls results in an even number”,
and their probabilities
pn = Pr (An )
and
qn = Pr (Bn ) .
• Determine p1 .
5.16. Exercises 251
• For any integer n ≥ 1, determine qn .
• For any integer n ≥ 2, express the event An in terms of the events An−1
and Bn .
• Use the previous parts to determine pn for any integer n ≥ 2.
5.58 You are asked to design a random bit generator. You find a coin in
your pocket, but, unfortunately, you are not sure if it is a fair coin. After
some thought, you come up with the following algorithm GenerateBit(n),
which takes as input an integer n ≥ 1:
Algorithm GenerateBit(n):
// all coin flips made are mutually independent
flip the coin n times;
k = the number of heads in the sequence of n coin flips;
if k is odd
then return 0
else return 1
endif
In this exercise, you will show that, when n → ∞, the output of algorithm
GenerateBit(n) is a uniformly random bit.
Let p be the real number with 0 < p < 1, such that, if the coin is flipped
once, it comes up heads with probability p and tails with probability 1 − p.
(Note that algorithm GenerateBit does not need to know the value of p.)
For any integer n ≥ 1, consider the two events
An = “algorithm GenerateBit(n) returns 0”
and
Bn = “the n-th coin flip made by algorithm GenerateBit(n)
results in heads”,
and define
Pn = Pr (An )
and
Qn = Pn − 1/2.
252 Chapter 5. Discrete Probability
• Determine P1 and Q1 .
• For any integer n ≥ 2, prove that
Pn = p + (1 − 2p) · Pn−1 .
Hint: Express the event An in terms of the events An−1 and Bn .
• For any integer n ≥ 2, prove that
Qn = (1 − 2p) · Qn−1 .
• For any integer n ≥ 1, prove that
Qn = (1 − 2p)n−1 · (p − 1/2).
• Prove that
lim Qn = 0
n→∞
and
lim Pn = 1/2.
n→∞
5.59 In this exercise, we will use the product notation. In case you are not
familiar with this notation:
• For k ≤ m, m
Q
i=k xi denotes the product
xk · xk+1 · xk+2 · · · xm .
Qm
• If k > m, then i=k xi is an “empty” product, which we define to be
equal to 1.
Let n ≥ 1 be an integer, and for each i = 1, 2, . . . , n, let pi be a real
number such that 0 < pi < 1. In this exercise, you will prove that
n
X n
Y n
Y
pi (1 − pj ) = 1 − (1 − pi ). (5.7)
i=1 j=i+1 i=1
For example,
5.16. Exercises 253
• for n = 1, (5.7) becomes
p1 = 1 − (1 − p1 ),
• for n = 2, (5.7) becomes
p1 (1 − p2 ) + p2 = 1 − (1 − p1 )(1 − p2 ),
• for n = 3, (5.7) becomes
p1 (1 − p2 )(1 − p3 ) + p2 (1 − p3 ) + p3 = 1 − (1 − p1 )(1 − p2 )(1 − p3 ).
Assume we do an experiment consisting of n tasks T1 , T2 , . . . , Tn . Each
task is either a success or a failure, independently of the other tasks. For
each i = 1, 2, . . . , n, let pi be the probability that Ti is a success. Consider
the event
A = “at least one task is a success”.
• Prove (5.7) by determining Pr(A) in two different ways.
5.60 Let n ≥ 0 be an integer. In this exercise, you will prove that
n
X 1 n+k
k
· = 2n . (5.8)
k=0
2 k
The Ottawa Senators and the Toronto Maple Leafs play a best-of-(2n+1)
series: These two hockey teams play games against each other, and the first
team to win n + 1 games wins the series. Assume that
• each game has a winner (thus, no game ends in a tie),
• in any game, the Sens have a probability of 1/2 of defeating the Leafs,
and
• the results of the games are mutually independent.
Consider the events
A = “the Sens win the series”
and
B = “the Leafs win the series”.
254 Chapter 5. Discrete Probability
• Explain in plain English why Pr(A) = Pr(B) = 1/2.
• For each k with 0 ≤ k ≤ n, consider the event
Ak = “the Sens win the series after winning the (n + k + 1)-st game”.
Express the event A in terms of the events A0 , A1 , . . . , An .
• Consider a fixed value of k with 0 ≤ k ≤ n. Prove that
1 n+k
Pr (Ak ) = n+k+1 · .
2 k
Hint: Assume event Ak occurs. Which team wins the (n + k + 1)-st
game? In the first n + k games, how many games are won by the Leafs?
• Prove that (5.8) holds by combining the results of the previous parts.
5.61 Let n ≥ 0 be an integer. In this exercise, you will prove that
n
X 1 n 1
2n+1 − 1 .
= (5.9)
k=0
k+1 k n+1
There are n + 1 students in Carleton’s Computer Science program. We
denote these students by P1 , P2 , . . . , Pn+1 . We play the following game:
1. We choose a uniformly random subset X of {P1 , P2 , . . . , Pn+1 }.
2. (a) If X 6= ∅, then we choose a uniformly random student in X. The
chosen student wins a six-pack of cider.
(b) If X = ∅, then nobody wins the six-pack.
The random choices made are independent of each other.
• Consider the event
A0 = “nobody wins the six-pack”.
Determine Pr (A0 ).
5.16. Exercises 255
• For each i = 1, 2, . . . , n + 1, consider the event
Ai = “student Pi wins the six-pack”.
Explain in plain English why
Pr (A1 ) = Pr (A2 ) = . . . = Pr (An+1 ) .
• Prove that
1 − 1/2n+1
Pr (A1 ) = .
n+1
• For each k with 0 ≤ k ≤ n, consider the event
Bk = “X has size k + 1 and P1 wins the six-pack”.
Prove that n
k 1
Pr (Bk ) = · .
2n+1 k+1
• Express the event A1 in terms of the events B0 , B1 , . . . , Bn .
• Prove that (5.9) holds by combining the results of the previous parts.
5.62 Let n and k be integers with 1 ≤ n ≤ k ≤ 2n. In this exercise, you will
prove that
n
2n − k
X k 2n
= . (5.10)
i=k−n
i n − i n
Jim is working on his assignment for the course COMP 4999 (Computa-
tional Aspects of Growing Cannabis). There are 2n questions on this assign-
ment and each of them is worth 1 mark. Two minutes before the deadline,
Jim has completed the first k questions. Jim is very smart and all answers
to these k questions are correct. Jim knows that the instructor, Professor
Mary Juana, does not accept late submissions. Because of this, Jim leaves
the last 2n − k questions blank and hands in his assignment.
Tri is a teaching assistant for this course. Since Tri is lazy, he does not
want to mark all questions. Instead, he chooses a uniformly random subset of
n questions out of the 2n questions, and only marks the n chosen questions.
For each correct answer, Tri gives 2 marks, whereas he gives 0 marks for each
wrong (or blank) answer.
For each integer i ≥ 0, consider the event
Ai = “Jim receives exactly 2i marks for his assignment”.
256 Chapter 5. Discrete Probability
P
• Determine the value of the summation i Pr (Ai ). Explain your answer
in plain English.
• Determine all values of i for which the event Ai is non-empty. For each
such value i, determine Pr (Ai ).
• Prove that (5.10) holds by combining the results of the previous parts.
5.63 Let a and z be integers with a > z ≥ 1, and let p be a real number
with 0 < p < 1. Alexa and Zoltan play a game consisting of several rounds.
In one round,
1. Alexa receives a points with probability p and 0 points with probability
1 − p,
2. Zoltan receives z points (with probability 1).
We assume that the results of different rounds are independent.
• Consider the event
A = “in one round, Alexa receives more points than Zoltan”.
We say that Alexa is a better player than Zoltan, if Pr(A) > 1/2.
For which values of p is Alexa a better player than Zoltan?
• Assume that a = 3, z = 2, and
√ p is chosen such that p > 1/2 and
2
p < 1/2. (For example, p = ( 5 − 1)/2.)
– Is Alexa a better player than Zoltan?
– Alexa and Zoltan play a game consisting of two rounds. We con-
sider the total number of points that each player wins during these
two rounds. Consider the event
B = “in two rounds, Alexa receives more points than Zoltan”.
Prove that Pr(B) < 1/2. (This seems to suggest that Zoltan is a
better player than Alexa.)
• Let n be a large integer, and assume that a = n + 1, z = n, and p
is chosen very close to (but less than) 1. (For example, n = 500 and
p = 0.99.)
5.16. Exercises 257
– Is Alexa a better player than Zoltan?
– Alexa and Zoltan play a game consisting of n rounds. We consider
the total number of points that each player wins during these n
rounds. Consider the event
C = “in n rounds, Alexa receives more points than Zoltan”.
Prove that Pr(C) = pn . (If n = 500 and p = 0.99, then pn ≈
0.0066. This seems to suggest that Zoltan is a much better player
than Alexa.)
5.64 Let k ≥ 1 be an integer. √
Assume we live on a planet on which one year
has d = 4k 2 days. Consider d = 2k people P1 , P2 , . . . , P2k living on our
planet. Each person has a uniformly random birthday, and the birthdays of
these 2k people are mutually independent. Consider the event
A = “at least two of P1 , P2 , . . . , P2k have the same birthday”.
This exercise will lead you through a proof of the claim that
0.221 < Pr(A) < 0.5.
√
Thus, if one year has d days, then d people are enough to have a good
chance that not all birthdays are distinct. (This result is similar to the one
we obtained in Section 5.5.1.)
• For each i with 1 ≤ i ≤ 2k, consider the event
Bi = “Pi has the same birthday as at least one of P1 , P2 , . . . , Pi−1 ”.
Prove that
i−1
Pr (Bi ) ≤ .
d
• Express the event A in terms of the events B1 , B2 , . . . , B2k .
• Use the Union Bound (Lemma 5.3.5) to prove that
Pr(A) < 1/2.
258 Chapter 5. Discrete Probability
• Consider the event
B = “at least two of Pk+1 , Pk+2 , . . . , P2k have the same birthday”
and for each i with 1 ≤ i ≤ k, the event
Ci = “Pi has the same birthday as at least one of Pk+1 , Pk+2 , . . . , P2k ”.
Prove that
1
Pr Ci | B = .
4k
• Prove that if the event A occurs, then the event
C1 ∩ B ∩ C2 ∩ B ∩ · · · ∩ Ck ∩ B
also occurs.
• Prove that k
1
Pr A ≤ 1− .
4k
You may use the fact that the events C 1 ∩ B, C 2 ∩ B, . . . , C k ∩ B are
mutually independent.
• Use the inequality 1 − x ≤ e−x to prove that
Pr(A) ≥ 1 − e−1/4 > 0.221.
5.65 Let n be a large power of two (thus, log n is an integer). Consider a
binary string s = s1 s2 . . . sn , where each bit si is 0 with probability 1/2, and
1 with probability 1/2, independently of the other bits.
A run of length k is a substring of length k, all of whose bits are equal. In
Section 5.14, we have seen that it is very likely that the bitstring s contains
a run of length at least log n − 2 log log n. In this exercise, you will prove
that it is very unlikely that s contains a run of length more than 2 log n.
• Let k be an integer with 1 ≤ k ≤ n. Consider the event
A = “the bitstring s contains a run of length at least k”.
5.16. Exercises 259
For each i with 1 ≤ i ≤ n − k + 1, consider the event
Ai = “the substring si si+1 . . . si+k−1 is a run”.
Use the Union Bound (Lemma 5.3.5) to prove that
n−k+1
Pr(A) ≤ .
2k−1
• Let k = 2 log n. Prove that
Pr(A) ≤ 2/n.
5.66 A hand of 5 cards is chosen uniformly at random from a standard deck
of 52 cards. Consider the event
A = “the hand has at least one Ace”.
• Explain what is wrong with the following argument:
We are going to determine Pr(A). Event A states that the hand
has at least one Ace. By symmetry, we may assume that A is
the event that the hand has the Ace of spades. Since there are
52 51
5
hands of five cards and exactly 4 of them contain the Ace
of spades, it follows that
51
4 5
Pr(A) = 52 = .
5
52
• Explain what is wrong with the following argument:
260 Chapter 5. Discrete Probability
We are going to determine Pr(A) using the Law of Total Proba-
bility (Theorem 5.9.1). For each x ∈ {♠, ♥, ♣, ♦}, consider the
event
Bx = “the hand has the Ace of suit x”.
We observe that
51
4 5
Pr (Bx ) = 52
= .
5
52
We next observe that
Pr (A | Bx ) = 1,
because if event Bx occurs, then event A also occurs. Thus,
using the Law of Total Probability, we get
X
Pr(A) = Pr (A | Bx ) · Pr (Bx )
x
X
= 1 · Pr (Bx )
x
X 5
=
x
52
5
= 4·
52
5
= .
13
• Determine the value of Pr(A).
5.67 You are doing two projects P and Q. The probability that project P
is successful is equal to 2/3 and the probability that project Q is successful
is equal to 4/5. Whether or not these two projects are successful are inde-
pendent of each other. What is the probability that both P and Q are not
successful?
5.68 Consider two independent events A and B in some probability space
(S, Pr). Assume that A and B are disjoint, i.e., A ∩ B = ∅. What can you
say about Pr(A) and Pr(B)?
5.16. Exercises 261
5.69 You flip three fair coins independently of each other. Let A be the event
“at least two flips in a row are heads” and let B be the event “the number
of heads is even”. (Note that zero is even.) Are A and B independent?
5.70 You flip three fair coins independently of each other. Consider the
events
A = “there is at most one tails”
and
B = “not all flips are identical”.
Are A and B independent?
5.71 Let n ≥ 2 be an integer and consider two fixed integers a and b with
1 ≤ a < b ≤ n.
• Use the Product Rule to determine the number of permutations of
{1, 2, . . . , n} in which a is to the left of b.
• Consider a uniformly random permutation of the set {1, 2, . . . , n}, and
define the event
A = “in this permutation, a is to the left of b”.
Use your answer to the first part of this exercise to determine Pr(A).
5.72 Let n ≥ 4 be an integer and consider a uniformly random permutation
of the set {1, 2, . . . , n}. Consider the event
A = “in this permutation, both 3 and 4 are to the left of both 1 and 2”.
Determine Pr(A).
5.73 Let n ≥ 3 be an integer, consider a uniformly random permutation of
the set {1, 2, . . . , n}, and define the events
A = “in this permutation, 2 is to the left of 3”
and
B = “in this permutation, 1 is to the left of 2 and 1 is to the left of 3”.
Are these two events independent?
262 Chapter 5. Discrete Probability
5.74 Let n ≥ 4 be an integer. Consider a uniformly random permutation of
{1, 2, . . . , n} and define the events
A = “1 and 2 are next to each other, with 1 to the left of 2, or
4 and 3 are next to each other, with 4 to the left of 3”
and
B = “1 and 2 are next to each other, with 1 to the left of 2, or
2 and 3 are next to each other, with 2 to the left of 3”.
Determine Pr(A) and Pr(B). (Before you determine these probabilities,
spend a few minutes and guess which probability is larger.)
5.75 You flip two fair coins independently of each other. Consider the events
A = “the number of heads is odd”,
B = “the first coin comes up heads”,
C = “the second coin comes up heads”.
• Are the events A and B independent?
• Are the events A and C independent?
• Are the events B and C independent?
• Are the events A, B, and C pairwise independent?
• Are the events A, B, and C mutually independent?
5.76 You roll a fair die once. Consider the events
A = “the result is an element of {1, 3, 4}”,
B = “the result is an element of {3, 4, 5, 6}”.
Are these two events independent?
5.77 You roll a fair die once. Consider the events
A = “the result is even”,
B = “the result is odd”,
C = “the result is at most 4”.
5.16. Exercises 263
• Are the events A and B independent?
• Are the events A and C independent?
• Are the events B and C independent?
5.78 You are given a tetrahedron, which is a die with four faces. Each of
these faces has one of the bitstrings 110, 101, 011, and 000 written on it.
Different faces have different bitstrings.
We roll the tetrahedron so that each face is at the bottom with equal
probability 1/4. For k = 1, 2, 3, consider the event
Ak = “the bitstring written on the bottom face has 0 at position k”.
For example, if the bitstring at the bottom face is 101, then A1 is false, A2
is true, and A3 is false.
• Are the events A1 and A2 independent?
• Are the events A1 and A3 independent?
• Are the events A2 and A3 independent?
• Are the events A1 , A2 , A3 pairwise independent?
• Are the events A1 , A2 , A3 mutually independent?
5.79 In a group of 100 children, 34 are boys and 66 are girls. You are given
the following information about the girls:
• Each girl has green eyes or is blond or is left-handed.
• 20 of the girls have green eyes.
• 40 of the girls are blond.
• 50 of the girls are left-handed.
• 10 of the girls have green eyes and are blond.
• 14 of the girls have green eyes and are left-handed.
• 4 of the girls have green eyes, are blond, and are left-handed.
264 Chapter 5. Discrete Probability
We choose one of these 100 children uniformly at random. Consider the
events
G = “the child chosen is a girl with green eyes”,
B = “the child chosen is a blond girl”,
L = “the child chosen is a left-handed girl”.
• Are the events G and B independent?
• Are the events G and L independent?
• Are the events B and L independent?
• Verify whether or not the following equation holds:
Pr(G ∧ B ∧ L) = Pr(G) · Pr(B) · Pr(L).
5.80 Let S be a sample space consisting of 100 elements. Consider three
events A, B, and C, as indicated in the figure below. For example, the event
A consists of 50 elements, 20 of which are only in A, 20 of which are only in
A ∩ B, 5 of which are only in A ∩ C, and 5 of which are in A ∩ B ∩ C.
S
B
A
20 20 20
5
5 5
10
C
15
Consider the uniform probability function on this sample space.
• Are the events A and B independent?
5.16. Exercises 265
• Determine whether or not
Pr(A ∩ B | C) = Pr(A | C) · Pr(B | C).
5.81 Annie, Boris, and Charlie write an exam that consists of only one
question: What is 26 times 26? Calculators are not allowed during the
exam. Both Annie and Boris are pretty clever and each of them gives the
correct answer with probability 9/10. Charlie has trouble with two-digit
numbers and gives the correct answer with probability 6/10.
• Assume that the three students do not cheat, i.e., each student answers
the question independently of the other two students. Determine the
probability that at least two of them give the correct answer.
• Assume that Annie and Boris do not cheat, but Charlie copies Annie’s
answer. Determine the probability that at least two of them give the
correct answer.
Hint: The answer to the second part is smaller than the answer to the first
part.
5.82 Alexa and Zoltan play the following game:
AZ-game:
Step 1: Alexa chooses a uniformly random element from the set
{1, 2, 3}. Let a denote the element that Alexa chooses.
Step 2: Zoltan chooses a uniformly random element from the set
{1, 2, 3}. Let z denote the element that Zoltan chooses.
Step 3: Using one of the three strategies mentioned below, Alexa
chooses an element from the set {1, 2, 3} \ {a}. Let a0 denote the
element that Alexa chooses.
Step 4: Using one of the three strategies mentioned below, Zoltan
chooses an element from the set {1, 2, 3} \ {z}. Let z 0 denote the
element that Alexa chooses.
The AZ-game is a success if a0 6= z 0 .
• MinMin Strategy: In Step 3, Alexa chooses the smallest element in the
set {1, 2, 3} \ {a}, and Zoltan chooses the smallest element in the set
{1, 2, 3} \ {z}.
266 Chapter 5. Discrete Probability
– Describe the sample space for this strategy.
– For this strategy, determine the probability that the AZ-game is
a success.
• MinMax Strategy: In Step 3, Alexa chooses the smallest element in the
set {1, 2, 3} \ {a}, and Zoltan chooses the largest element in the set
{1, 2, 3} \ {z}.
– Describe the sample space for this strategy.
– For this strategy, determine the probability that the AZ-game is
a success.
• Random Strategy: In Step 3, Alexa chooses a uniformly random element
in the set {1, 2, 3}\{a}, and Zoltan chooses a uniformly random element
in the set {1, 2, 3} \ {z}.
– Describe the sample space for this strategy.
– For this strategy, determine the probability that the AZ-game is
a success.
5.83 You are given a box that contains one red ball and one blue ball.
Consider the following algorithm RandomRedBlue(n) that takes as input an
integer n ≥ 3:
Algorithm RandomRedBlue(n):
// n ≥ 3
// initially, the box contains one red ball and one blue ball
// all random choices are mutually independent
for k = 1 to n − 2
do choose a uniformly random ball in the box;
if the chosen ball is red
then put the chosen ball back in the box;
add one red ball to the box
else put the chosen ball back in the box;
add one blue ball to the box
endif
endfor
5.16. Exercises 267
For any integers n ≥ 3 and i with 1 ≤ i ≤ n − 1, consider the event
Ani = “at the end of algorithm RandomRedBlue(n),
the number of red balls in the box is equal to i”.
In this exercise, you will prove that for any integers n ≥ 3 and i with
1 ≤ i ≤ n − 1,
1
Pr (Ani ) = . (5.11)
n−1
• Let n ≥ 3 and k be integers with 1 ≤ k ≤ n − 2. When running
algorithm RandomRedBlue(n),
– how many balls does the box contain at the start of the k-th
iteration,
– how many balls does the box contain at the end of the k-th iter-
ation?
• Let n ≥ 3 be an integer. After algorithm RandomRedBlue(n) has
terminated, how many balls does the box contain?
• For any integer n ≥ 3, prove that
1
Pr (An1 ) = .
n−1
• For any integer n ≥ 3, prove that
1
Pr Ann−1 =
.
n−1
• Let n = 3. Prove that (5.11) holds for all values of i in the indicated
range.
• Let n ≥ 4. Consider the event
A = “in the (n − 2)-th iteration of algorithm RandomRedBlue(n),
a red ball is chosen”.
For any integer i with 2 ≤ i ≤ n − 2, express the event Ani in terms of
the events An−1 n−1
i−1 , Ai , and A.
268 Chapter 5. Discrete Probability
• Let n ≥ 4. For any integer i with 2 ≤ i ≤ n − 2, prove that
n−1 n−1
Pr (Ani ) = Pr A | Ai−1 + Pr A | Ain−1 · Pr Ain−1 .
· Pr Ai−1
• Let n ≥ 4. Prove that (5.11) holds for all values of i in the indicated
range.
5.84 Prove that for any real number x 6= 1 and any integer N ≥ 0,
N
X 1 − xN +1
xn = .
n=0
1−x
5.85 Use the following argumentation to convince yourself that
∞
X
1/2n = 2.
n=0
Take the interval I = [0, 2) of length 2 on the real line and, for each n ≥ 0, an
interval In of length 1/2n . It is possible to place all intervals In with n ≥ 0
in I such that
• no two intervals In and Im , with m 6= n, overlap and
• all intervals In with n ≥ 0 completely cover the interval I.
5.86 Alexa, Tri, and Zoltan play the OddPlayer game: In one round, each
player flips a fair coin.
1. Assume that not all flips are equal. Then the coin flips of exactly two
players are equal. The player whose coin flip is different is called the
odd player. In this case, the odd player wins the game. For example, if
Alexa flips tails, Tri flips heads, and Zoltan flips tails, then Tri is the
odd player and wins the game.
2. If all three coin flips are equal, then the game is repeated.
Below, this game is presented in pseudocode:
5.16. Exercises 269
Algorithm OddPlayer:
// all coin flips are mutually independent
each player flips a fair coin;
if not all coin flips are equal
then the game terminates and the odd player wins
else OddPlayer
endif
• What is the sample space?
• Consider the event
A = “Alexa wins the game”.
Express this event as a subset of the sample space.
• Use your expression from the previous part to determine Pr(A).
• Use symmetry to determine Pr(A). Explain your answer in plain En-
glish and a few sentences.
Hint: What is the probability that Tri wins the game? What is the
probability that Zoltan wins the game?
5.87 Two players P1 and P2 take turns rolling two fair and independent dice,
where P1 starts the game. The first player who gets a sum of seven wins the
game. Determine the probability that player P1 wins the game.
5.88 By flipping a fair coin repeatedly and independently, we obtain a se-
quence of H’s and T ’s. We stop flipping the coin as soon as the sequence
contains either HH or T H.
Two players P1 and P2 play a game, in which P1 wins if the last two
symbols in the sequence are HH. Otherwise, the last two symbols in the
sequence are T H, in which case P2 wins. Determine the probability that
player P1 wins the game.
5.89 Two players P1 and P2 play a game in which they take turns flipping,
independently, a fair coin: First P1 flips the coin, then P2 flips the coin, then
P1 flips the coin, then P2 flips the coin, etc. The game ends as soon as the
270 Chapter 5. Discrete Probability
sequence of coin flips contains either HH or T T . The player who flips the
coin for the last time is the winner of the game. For example, if the sequence
of coin flips is HT HT HH, then P2 wins the game.
Determine the probability that player P1 wins the game.
5.90 We flip a fair coin repeatedly and independently, and stop as soon as
we see one of the two sequences HT T and HHT . Let A be the event that
the process stops because HT T is seen.
• Prove that the event A is given by the set
{T m (HT )n HT T : m ≥ 0, n ≥ 0}.
In other words, event A holds if and only if the sequence of coin flips
is equal to T m (HT )n HT T for some m ≥ 0 and n ≥ 0.
• Prove that Pr(A) = 1/3.
5.91 For i ∈ {1, 2}, consider the game Gi , in which two players P1 and P2
take turns flipping, independently, a fair coin, where Pi starts. The game
ends as soon as heads comes up. The player who flips heads first is the
winner of the game Gi . For j ∈ {1, 2}, consider the event
Bij = “Pj wins the game Gi ”.
In Section 5.15.2, we have seen that
Pr (B11 ) = Pr (B22 ) = 2/3 (5.12)
and
Pr (B12 ) = Pr (B21 ) = 1/3. (5.13)
Consider the game G, in which P1 and P2 take turns flipping, indepen-
dently, a fair coin, where P1 starts. The game ends as soon as a second heads
comes up. The player who flips the second heads wins the game. Consider
the event
A = “P1 wins the game G”.
In Section 5.15.3, we used an infinite series to show that
Pr(A) = 4/9. (5.14)
Use the Law of Total Probability (Theorem 5.9.1) to give an alternative proof
of (5.14). You are allowed to use (5.12) and (5.13).
5.16. Exercises 271
5.92 Consider two players P1 and P2 :
• P1 has one fair coin.
• P2 has two coins. One of them is fair, whereas the other one is 2-headed
(Her Majesty is on both sides of this coin).
The two players P1 and P2 play a game in which they alternate making turns:
P1 starts, after which it is P2 ’s turn, after which it is P1 ’s turn, after which
it is P2 ’s turn, etc.
• When it is P1 ’s turn, she flips her coin once.
• When it is P2 ’s turn, he does the following:
– P2 chooses one of his two coins uniformly at random. Then he
flips the chosen coin once.
– If the first flip did not results in heads, then P2 repeats this process
one more time: P2 again chooses one of his two coins uniformly
at random and flips the chosen coin once.
The player who flips heads first is the winner of the game.
• Determine the probability that P2 wins this game, assuming that all
random choices and coin flips made are mutually independent.
5.93 Jennifer loves to drink India Pale Ale (IPA), whereas Connor Hillen
prefers Black IPA. Jennifer and Connor decide to go to their favorite pub
Chez Lindsay et Simon. The beer menu shows that this pub has ten beers
on tap:
• Phillips Cabin Fever Imperial Black IPA,
• Big Rig Black IPA,
• Leo’s Early Breakfast IPA,
• Goose Island IPA,
• Caboose IPA,
• and five other beers, neither of which is an IPA.
272 Chapter 5. Discrete Probability
Each of the first five beers is an IPA, whereas each of the first two beers is a
Black IPA.
Jennifer and Connor play a game, in which they alternate ordering beer:
Connor starts, after which it is Jennifer’s turn, after which it is Connor’s
turn, after which it is Jennifer’s turn, etc.
• When it is Connor’s turn, he orders two beers; each of these is chosen
uniformly at random from the ten beers (thus, these two beers may be
equal).
• When it is Jennifer’s turn, she orders one of the ten beers, uniformly
at random.
The game ends as soon as (i) Connor has ordered at least one Black IPA, in
which case he pays the bill, or (ii) Jennifer has ordered at least one IPA, in
which case she pays the bill.
• Determine the probability that Connor pays the bill, assuming that all
random choices made are mutually independent.
5.94 You would like to generate a uniformly random bit, i.e., with proba-
bility 1/2, this bit is 0, and with probability 1/2, it is 1. You find a coin in
your pocket, but you are not sure if it is a fair coin: It comes up heads (H)
with probability p and tails (T ) with probability 1 − p, for some real number
p that is unknown to you. In particular, you do not know if p = 1/2. In this
exercise, you will show that this coin can be used to generate a uniformly
random bit.
Consider the following recursive algorithm GetRandomBit, which does
not take any input:
Algorithm GetRandomBit:
// all coin flips made are mutually independent
flip the coin twice;
if the result is HT
then return 0
else if the result is T H
then return 1
else GetRandomBit
endif
endif
5.16. Exercises 273
• The sample space S is the set of all sequences of coin flips that can oc-
cur when running algorithm GetRandomBit. Determine this sample
space S.
• Prove that algorithm GetRandomBit returns a uniformly random
bit.
5.95 You would like to generate a biased random bit: With probability 2/3,
this bit is 0, and with probability 1/3, it is 1. You find a fair coin in your
pocket: This coin comes up heads (H) with probability 1/2 and tails (T )
with probability 1/2. In this exercise, you will show that this coin can be
used to generate a biased random bit.
Consider the following recursive algorithm GetBiasedBit, which does
not take any input:
Algorithm GetBiasedBit:
// all coin flips made are mutually independent
flip the coin;
if the result is H
then return 0
else b = GetBiasedBit;
return 1 − b
endif
• The sample space S is the set of all sequences of coin flips that can occur
when running algorithm GetBiasedBit. Determine this sample space
S.
• Prove that algorithm GetBiasedBit returns 0 with probability 2/3.
5.96 Both Alexa and Shelly have an infinite bitstring. Alexa’s bitstring is
denoted by a1 a2 a3 . . ., whereas Shelly’s bitstring is denoted by s1 s2 s3 . . ..
Alexa can see her bitstring, but she cannot see Shelly’s bitstring. Similarly,
Shelly can see her bitstring, but she cannot see Alexa’s bitstring. The bits
in both bitstrings are uniformly random and independent.
The ladies play the following game: Alexa chooses a positive integer k
and Shelly chooses a positive integer `. The game is a success if sk = 1 and
a` = 1. In words, the game is a success if Alexa chooses a position in Shelly’s
274 Chapter 5. Discrete Probability
bitstring that contains a 1, and Shelly chooses a position in Alexa’s bitstring
that contains a 1.
• Assume Alexa chooses k = 4 and Shelly chooses ` = 7. Determine the
probability that the game is a success.
• Assume Alexa chooses the position, say k, of the leftmost 1 in her
bitstring, and Shelly chooses the position, say `, of the leftmost 1 in
her bitstring.
– If k 6= `, is the game a success?
– Determine the probability that the game is a success.
5.97 Alexa and Shelly take turns flipping, independently, a coin, where Alexa
starts. The game ends as soon as heads comes up. The lady who flips heads
first is the winner of the game.
Alexa proposes that they both use a fair coin. Of course, Shelly does
not agree, because she knows from Section 5.15.2 that this gives Alexa a
probability of 2/3 of winning the game.
The ladies agree on the following: Let p and q be real numbers with
0 < p < 1 and 0 ≤ q ≤ 1. Alexa uses a coin that comes up heads with
probability p, and Shelly uses a coin that comes up heads with probability q.
• Assume that p = 1/2. Determine the value of q for which Alexa and
Shelly have the same probability of winning the game.
• From now on, assume that 0 < p < 1 and 0 < q < 1.
– Determine the probability that Alexa wins the game.
– Assume that p > 1/2. Prove that for any q with 0 < q < 1, the
probability that Alexa wins the game is strictly larger than 1/2.
– Assume that p < 1/2. Determine the value of q for which Alexa
and Shelly have the same probability of winning the game.
5.98 Let n ≥ 2 be an integer and consider a uniformly random permutation
(a1 , a2 , . . . , an ) of the set {1, 2, . . . , n}. For each k with 1 ≤ k ≤ n, consider
the event
Ak = “ak is the largest element among the first k elements
in the permutation”.
5.16. Exercises 275
• Let k and ` be two integers with 1 ≤ k < ` ≤ n. Prove that the events
Ak and A` are independent.
Hint: Use the Product Rule to determine the number of permutations
that define Ak , A` , and Ak ∩ A` , respectively.
• Prove that the sequence A1 , A2 , . . . , An of events is mutually indepen-
dent.
5.99 Let n ≥ 2 be an integer. We generate a random bitstring R =
r1 r2 · · · rn , by setting, for each i = 1, 2, . . . , n, ri = 1 with probability 1/i
and, thus, ri = 0 with probability 1 − 1/i. All random choices made when
setting these bits are mutually independent.
For each i with 1 ≤ i ≤ n, consider the events
Bi = “ri = 1”
and
Ri = “the rightmost 1 in the bitstring R is at position i”.
• Determine Pr (Ri ).
The following algorithm TryToFindRightmostOne(R, n, m) takes as
input the bitstring R = r1 r2 · · · rn of length n and an integer m with 1 ≤
m ≤ n. As the name suggests, this algorithm tries to find the position of the
rightmost 1 in the string R.
276 Chapter 5. Discrete Probability
Algorithm TryToFindRightmostOne(R, n, m):
for i = 1 to m
do if ri = 1
then k = i
endif
endfor;
// k is the position of the rightmost 1 in the substring
// r1 r2 · · · rm .
// the next while-loop finds the position of the leftmost 1
// in the substring rm+1 rm+2 · · · rn , if this position exists.
` = m + 1;
while ` ≤ n and r` = 0
do ` = ` + 1
endwhile;
// if ` ≤ n, then ` is the position of the leftmost 1 in the
// substring rm+1 rm+2 · · · rn .
if ` ≤ n
then return `
else return k
endif
Consider the event
Em = “there is exactly one 1 in the substring rm+1 rm+2 · · · rn ”.
• Prove that
m 1 1 1
Pr (Em ) = + + ··· + .
n m m+1 n−1
Consider the event
A = “TryToFindRightmostOne(R, n, m) returns the position of the
rightmost 1 in the string R”.
• Prove that
m 1 1 1
Pr (A) = 1+ + + ··· + .
n m m+1 n−1
5.16. Exercises 277
5.100 You realize that it is time to buy a pair of shoes. You look up all n
shoe stores in Ottawa and visit them in random order. While shopping, you
create a bitstring r1 r2 · · · rn of length n: For each i with 1 ≤ i ≤ n, you set
ri to 1 if and only if the i-th store has the best pair of shoes, among the first
i stores that you have visited.
• Use Exercise 5.98 to prove that this bitstring satisfies the condition in
Exercise 5.99.
After you have visited the first m shoe stores, you are bored of shopping.
You keep on visiting shoe stores, but as soon as you visit a store that has
a pair of shoes that you like more than the previously best pair you have
found, you buy the former pair of shoes.
• Use Exercise 5.99 to determine the probability that you buy the best
pair of shoes that is available in Ottawa.
278 Chapter 5. Discrete Probability
Chapter 6
Random Variables and
Expectation
A natural question: What is the definition of random variable?
Classically, and in many of today’s textbooks, you see definitions
such as, a random variable is the observed value of a random
quantity. What on earth does that mean? How can any sort of
theory be built on such vagueness?
— Persi Diaconis and Brian Skyrms, Ten Great Ideas About
Chance, 2018
6.1 Random Variables
We have already seen random variables in Chapter 5, even though we did not
use that term there. For example, in Section 5.2.1, we rolled a die twice and
were interested in the sum of the results of these two rolls. In other words,
we did an “experiment” (rolling a die twice) and asked for a function of the
outcome (the sum of the results of the two rolls).
Definition 6.1.1 Let S be a sample space. A random variable on the sample
space S is a function X : S → R.
In the example given above, the sample space is
S = {(i, j) : 1 ≤ i ≤ 6, 1 ≤ j ≤ 6}
280 Chapter 6. Random Variables and Expectation
and the random variable is the function X : S → R defined by
X(i, j) = i + j
for all (i, j) in S.
Note that the term “random variable” is misleading: A random variable
is not random, but a function that assigns, to every outcome ω in the sample
space S, a real number X(ω). Also, a random variable is not a variable, but
a function.
A random variable is neither random nor variable.
6.1.1 Flipping Three Coins
Assume we flip three coins. The sample space is
S = {HHH, HHT, HT H, HT T, T HH, T HT, T T H, T T T },
where, e.g., T T H indicates that the first two coins come up tails and the
third coin comes up heads.
Let X : S → R be the random variable that maps any outcome (i.e., any
element of S) to the number of heads in the outcome. Thus,
X(HHH) = 3,
X(HHT ) = 2,
X(HT H) = 2,
X(HT T ) = 1,
X(T HH) = 2,
X(T HT ) = 1,
X(T T H) = 1,
X(T T T ) = 0.
If we define the random variable Y to be the function Y : S → R that
• maps an outcome to 1 if all three coins come up heads or all three coins
come up tails, and
• maps an outcome to 0 in all other cases,
6.1. Random Variables 281
then we have
Y (HHH) = 1,
Y (HHT ) = 0,
Y (HT H) = 0,
Y (HT T ) = 0,
Y (T HH) = 0,
Y (T HT ) = 0,
Y (T T H) = 0,
Y (T T T ) = 1.
Since a random variable is a function X : S → R, it maps any outcome
ω to a real number X(ω). Usually, we just write X instead of X(ω). Thus,
for any outcome in the sample space S, we denote the value of the random
variable, for this outcome, by X. In the example above, we flip three coins
and write
X = the number of heads
and
1 if all three coins come up heads or all three coins come up tails,
Y =
0 otherwise.
6.1.2 Random Variables and Events
Random variables give rise to events in a natural way. In the three-coin
example, “X = 0” corresponds to the event {T T T }, whereas “X = 2”
corresponds to the event {HHT, HT H, T HH}. The table below gives some
values of the random variables X and Y , together with the corresponding
events.
value event
X=0 {T T T }
X=1 {HT T, T HT, T T H}
X=2 {HHT, HT H, T HH}
X=3 {HHH}
X=4 ∅
Y =0 {HHT, HT H, HT T, T HH, T HT, T T H}
Y =1 {HHH, T T T }
Y =2 ∅
282 Chapter 6. Random Variables and Expectation
Thus, the event “X = x” corresponds to the set of all outcomes that are
mapped, by the function X, to the value x:
Definition 6.1.2 Let S be a sample space and let X : S → R be a random
variable. For any real number x, we define “X = x” to be the event
{ω ∈ S : X(ω) = x}.
Let us return to the example in which we flip three coins. Assume that
the coins are fair and the three flips are mutually independent. Consider
again the corresponding random variables X and Y . It should be clear how
we determine, for example, the probability that X is equal to 0, which we
will write as Pr(X = 0). Using our interpretation of “X = 0” as being the
event {T T T }, we get
Pr(X = 0) = Pr(T T T )
= 1/8.
Similarly, we get
Pr(X = 1) = Pr({HT T, T HT, T T H})
= 3/8,
Pr(X = 2) = Pr({HHT, HT H, T HH})
= 3/8,
Pr(X = 3) = Pr({HHH})
= 1/8,
Pr(X = 4) = Pr(∅)
= 0,
Pr(Y = 0) = Pr({HHT, HT H, HT T, T HH, T HT, T T H})
= 6/8
= 3/4,
Pr(Y = 1) = Pr({HHH, T T T })
= 2/8
= 1/4,
Pr(Y = 2) = Pr(∅)
= 0.
6.2. Independent Random Variables 283
Consider an arbitrary probability space (S, Pr) and let X : S → R be
a random variable. Using (5.1) and Definition 6.1.2, the probability of the
event “X = x”, i.e., the probability that X is equal to x, is equal to
Pr(X = x) = Pr({ω ∈ S : X(ω) = x})
X
= Pr(ω).
ω:X(ω)=x
We have interpreted “X = x” as being an event. We extend this to more
general statements involving X. For example, “X ≥ x” denotes the event
{ω ∈ S : X(ω) ≥ x}.
For our three-coin example, the random variable X can take each of the
values 0, 1, 2, and 3 with a positive probability. As a result, “X ≥ 2”
denotes the event “X = 2 or X = 3”, and we have
Pr(X ≥ 2) = Pr(X = 2 ∨ X = 3)
= Pr(X = 2) + Pr(X = 3)
= 3/8 + 1/8
= 1/2.
6.2 Independent Random Variables
In Section 5.11, we have defined the notion of two events being independent.
The following definition extends this to random variables.
Definition 6.2.1 Let (S, Pr) be a probability space and let X and Y be two
random variables on S. We say that X and Y are independent if for all real
numbers x and y, the events “X = x” and “Y = y” are independent, i.e.,
Pr(X = x ∧ Y = y) = Pr(X = x) · Pr(Y = y).
Assume we flip three fair coins independently and, as in Section 6.1.1,
consider the random variables
X = the number of heads
284 Chapter 6. Random Variables and Expectation
and
1 if all three coins come up heads or all three coins come up tails,
Y =
0 otherwise.
Are these two random variables independent? Observe the following: If
Y = 1, then X = 0 or X = 3. In other words, if we are given some
information about the random variable Y (in this case, Y = 1), then the
random variable X cannot take, for example, the value 2. Based on this, we
take x = 2 and y = 1 in Definition 6.2.1. Since the event “X = 2 ∧ Y = 1”
is equal to ∅, we have
Pr(X = 2 ∧ Y = 1) = Pr(∅) = 0.
On the other hand, we have seen in Section 6.1.2 that Pr(X = 2) = 3/8 and
Pr(Y = 1) = 1/4. It follows that
Pr(X = 2 ∧ Y = 1) 6= Pr(X = 2) · Pr(Y = 1)
and, therefore, the random variables X and Y are not independent.
Now consider the random variable
1 if the first coin comes up heads,
Z=
0 if the first coin comes up tails.
We claim that the random variables Y and Z are independent. To verify
this, we have to show that for all real numbers y and z,
Pr(Y = y ∧ Z = z) = Pr(Y = y) · Pr(Z = z). (6.1)
Recall from Section 6.1.2 that Pr(Y = 1) = 1/4 and Pr(Y = 0) = 3/4.
Since the coin flips are independent, we have Pr(Z = 1) = 1/2 and Pr(Z =
0) = 1/2. Furthermore,
Pr(Y = 1 ∧ Z = 1) = Pr(HHH)
= 1/8,
Pr(Y = 1 ∧ Z = 0) = Pr(T T T )
= 1/8,
Pr(Y = 0 ∧ Z = 1) = Pr(HHT, HT H, HT T )
= 3/8,
Pr(Y = 0 ∧ Z = 0) = Pr(T HH, T HT, T T H)
= 3/8.
6.3. Distribution Functions 285
It follows that
Pr(Y = 1 ∧ Z = 1) = Pr(Y = 1) · Pr(Z = 1),
Pr(Y = 1 ∧ Z = 0) = Pr(Y = 1) · Pr(Z = 0),
Pr(Y = 0 ∧ Z = 1) = Pr(Y = 0) · Pr(Z = 1),
and
Pr(Y = 0 ∧ Z = 0) = Pr(Y = 0) · Pr(Z = 0).
Thus, (6.1) holds if (y, z) ∈ {(1, 1), (1, 0), (0, 1), (0, 0)}. For any other pair
(y, z), such as (y, z) = (3, 5) or (y, z) = (1, 2), at least one of the events
“Y = y” and “Z = z” is the empty set, i.e., cannot occur. Therefore, for
such pairs, we have
Pr(Y = y ∧ Z = z) = 0 = Pr(Y = y) · Pr(Z = z).
Thus, we have indeed verified that (6.1) holds for all real numbers y and z. As
a result, we have shown that the random variables Y and Z are independent.
Are the random variables X and Z independent? If X = 0, then all three
coins come up tails and, therefore, Z = 0. Thus,
Pr(X = 0 ∧ Z = 1) = Pr(∅) = 0,
whereas
Pr(X = 0) · Pr(Z = 1) = 1/8 · 1/2 6= 0.
As a result, the random variables X and Z are not independent.
We have defined the notion of two random variables being independent.
As in Definition 5.11.3, there are two ways to generalize this to sequences of
random variables:
Definition 6.2.2 Let (S, Pr) be a probability space, let n ≥ 2, and let
X1 , X2 , . . . , Xn be a sequence of random variables on S.
1. We say that this sequence is pairwise independent if for all real numbers
x1 , x2 , . . . , xn , the sequence “X1 = x1 ”, “X2 = x2 ”, . . . , “Xn = xn ” of
events is pairwise independent.
2. We say that this sequence is mutually independent if for all real numbers
x1 , x2 , . . . , xn , the sequence “X1 = x1 ”, “X2 = x2 ”, . . . , “Xn = xn ” of
events is mutually independent.
286 Chapter 6. Random Variables and Expectation
6.3 Distribution Functions
Consider a random variable X on a sample space S. In Section 6.1.2, we
have defined Pr(X = x), i.e., the probability of the event “X = x”, to be
Pr(X = x) = Pr({ω ∈ S : X(ω) = x}).
This defines a function that maps any real number x to the real number
Pr(X = x). This function is called the distribution function of the random
variable X:
Definition 6.3.1 Let (S, Pr) be a probability space and let X : S → R be a
random variable. The distribution function of X is the function D : R → R
defined by
D(x) = Pr(X = x)
for all x ∈ R.
For example, consider a fair red die and a fair blue die, and assume we
roll them independently. The sample space is
S = {(i, j) : 1 ≤ i ≤ 6, 1 ≤ j ≤ 6},
where i is the result of the red die and j is the result of the blue die. Each
outcome (i, j) in S has the same probability of 1/36.
Let X be the random variable whose value is equal to the sum of the
results of the two dies. The matrix below gives all possible values of X. The
leftmost column gives the result of the red die, the top row gives the result
of the blue die, and each other entry is the corresponding value of X.
1 2 3 4 5 6
1 2 3 4 5 6 7
2 3 4 5 6 7 8
3 4 5 6 7 8 9
4 5 6 7 8 9 10
5 6 7 8 9 10 11
6 7 8 9 10 11 12
6.4. Expected Values 287
As can be seen from this matrix, the random variable X can take any
value in {2, 3, 4, . . . , 12}. The distribution function D of X is given by
D(2) = Pr(X = 2) = 1/36,
D(3) = Pr(X = 3) = 2/36,
D(4) = Pr(X = 4) = 3/36,
D(5) = Pr(X = 5) = 4/36,
D(6) = Pr(X = 6) = 5/36,
D(7) = Pr(X = 7) = 6/36,
D(8) = Pr(X = 8) = 5/36,
D(9) = Pr(X = 9) = 4/36,
D(10) = Pr(X = 10) = 3/36,
D(11) = Pr(X = 11) = 2/36,
D(12) = Pr(X = 12) = 1/36,
whereas for all x 6∈ {2, 3, 4, . . . , 12},
D(x) = Pr(X = x) = 0.
In Sections 6.6 and 6.7, we will see other examples of distribution func-
tions.
6.4 Expected Values
Consider the probability space (S, Pr) with sample space S = {1, 2, 3} and
probability function Pr defined by Pr(1) = 4/5, Pr(2) = 1/10, and Pr(3) =
1/10. Assume we choose an element in S according to this probability func-
tion. Let X be the random variable whose value is equal to the element in S
that is chosen. Thus, as a function X : S → R, we have X(1) = 1, X(2) = 2,
and X(3) = 3.
The “expected value” of X is the value of X that we observe “on average”.
How should we define this? Since X has a much higher probability to take
the value 1 than the other two values 2 and 3, the value 1 should get a larger
“weight” in the expected value of X. Based on this, it is natural to define
the expected value of X to be
4 1 1 13
1 · Pr(1) + 2 · Pr(2) + 3 · Pr(3) = 1 · +2· +3· = .
5 10 10 10
288 Chapter 6. Random Variables and Expectation
Definition 6.4.1 Let (S, Pr) be a probability space and let X : S → R be
a random variable. The expected value of X is defined to be
X
E(X) = X(ω) · Pr(ω),
ω∈S
provided this summation converges absolutely1 .
6.4.1 Some Examples
Flipping a coin: Assume we flip a fair coin, in which case the sample
space is S = {H, T } and Pr(H) = Pr(T ) = 1/2. Define the random variable
X to have value
1 if the coin comes up heads,
X=
0 if the coin comes up tails.
Thus, as a function X : S → R, we have X(H) = 1 and X(T ) = 0. The
expected value E(X) of X is equal to
E(X) = X(H) · Pr(H) + X(T ) · Pr(T )
1 1
= 1· +0·
2 2
1
= .
2
This example shows that the term “expected value” is a bit misleading: E(X)
is not the value that we expect to observe, because the value of X is never
equal to its expected value.
Rolling a die: Assume we roll a fair die. Define the random variable X to
be the value of the result. Then, X takes each of the values in {1, 2, 3, 4, 5, 6}
with equal probability 1/6, and we get
1 1 1 1 1 1
E(X) = 1 · +2· +3· +4· +5· +6·
6 6 6 6 6 6
7
= .
2
1
P∞ P∞
The series n=0 an converges absolutely if the series n=0 |an | converges. If a series
converges absolutely, then we can change the order of summation without changing the
value of the series.
6.4. Expected Values 289
Now define the random variable Y to be equal to one divided by the result
of the die. In other words, Y = 1/X. This random variable takes each of the
values in {1, 1/2, 1/3, 1/4, 1/5, 1/6} with equal probability 1/6, and we get
1 1 1 1 1 1 1 1 1 1 1
E(Y ) = 1 · + · + · + · + · + ·
6 2 6 3 6 4 6 5 6 6 6
49
= .
120
Note that E(Y ) 6= 1/E(X). Thus, this example shows that, in general,
E(1/X) 6= 1/E(X).
Rolling two dice: Consider a fair red die and a fair blue die, and assume
we roll them independently. The sample space is
S = {(i, j) : 1 ≤ i ≤ 6, 1 ≤ j ≤ 6},
where i is the result of the red die and j is the result of the blue die. Each
outcome (i, j) in S has the same probability of 1/36.
Let X be the random variable whose value is equal to the sum of the
results of the two rolls. As a function X : S → R, we have X(i, j) = i+j. The
matrix below gives all possible values of X. The leftmost column indicates
the result of the red die, the top row indicates the result of the blue die, and
each other entry is the corresponding value of X.
1 2 3 4 5 6
1 2 3 4 5 6 7
2 3 4 5 6 7 8
3 4 5 6 7 8 9
4 5 6 7 8 9 10
5 6 7 8 9 10 11
6 7 8 9 10 11 12
290 Chapter 6. Random Variables and Expectation
The expected value E(X) of X is equal to
X
E(X) = X(i, j) · Pr(i, j)
(i,j)∈S
X 1
= (i + j) ·
36
(i,j)∈S
1 X
= (i + j)
36
(i,j)∈S
1
= · the sum of all 36 entries in the matrix
36
1
= · 252
36
= 7.
6.4.2 Comparing the Expected Values of Comparable
Random Variables
Consider a probability space (S, Pr), and let X and Y be two random vari-
ables on S. Recall that X and Y are functions that map elements of S to
real numbers. We will write X ≤ Y , if for each element ω ∈ S, we have
X(ω) ≤ Y (ω). In other words, the value of X is at most the value of Y ,
no matter which outcome ω is chosen. The following lemma should not be
surprising:
Lemma 6.4.2 Let (S, Pr) be a probability space and let X and Y be two
random variables on S. If X ≤ Y , then E(X) ≤ E(Y ).
Proof. Using Definition 6.4.1 and the assumption that X ≤ Y , we obtain
X
E(X) = X(ω) · Pr(ω)
ω∈S
X
≤ Y (ω) · Pr(ω)
ω∈S
= E(Y ).
6.4. Expected Values 291
6.4.3 An Alternative Expression for the Expected Value
In the last example of Section 6.4.1, we used Definition 6.4.1 to compute the
expected value E(X) of the random variable X that was defined to be the
sum of the results when rolling two fair and independent dice. This was a
painful way to compute E(X), because we added all 36 entries in the matrix.
There is a slightly easier way to determine E(X): By looking at the matrix,
we see that the value 4 occurs three times. Thus, the event “X = 4” has
size 3, i.e., if we consider the subset of the sample space S that corresponds
to this event, then this subset has size 3. Similarly, the event “X = 7” has
size 6, because the value 7 occurs 6 times in the matrix. The table below
lists the sizes of all non-empty events, together with their probabilities.
event size of event probability
X=2 1 1/36
X=3 2 2/36
X=4 3 3/36
X=5 4 4/36
X=6 5 5/36
X=7 6 6/36
X=8 5 5/36
X=9 4 4/36
X = 10 3 3/36
X = 11 2 2/36
X = 12 1 1/36
Based on this, we get
1 2 3 4 5 6
E(X) = 2 · +3· +4· +5· +6· +7· +
36 36 36 36 36 36
5 4 3 2 1
8· +9· + 10 · + 11 · + 12 ·
36 36 36 36 36
= 7.
Even though this is still quite painful, less computation is needed. What we
have done is the following: In the definition of E(X), i.e.,
X
E(X) = X(i, j) · Pr(i, j),
(i,j)∈S
we rearranged the terms in the summation. That is, instead of taking the
sum over all elements (i, j) in S,
292 Chapter 6. Random Variables and Expectation
• we grouped together all outcomes (i, j) for which X(i, j) = i + j has
the same value, say, k,
• we multiplied this common value k by the probability that X is equal
to k,
• and we took the sum of the resulting products over all possible values
of k.
This resulted in
12
X
E(X) = k · Pr(X = k).
k=2
The following lemma states that we can do this for any random variable.
Lemma 6.4.3 Let (S, Pr) be a probability space and let X : S → R be a
random variable. The expected value of X is equal to
X
E(X) = x · Pr(X = x).
x
Proof. Recall that the event “X = x” corresponds to the subset
Ax = {ω ∈ S : X(ω) = x}
of the sample space S. We have
X
E(X) = X(ω) · Pr(ω)
ω∈S
X X
= X(ω) · Pr(ω)
x ω:X(ω)=x
X X
= x · Pr(ω)
x ω:X(ω)=x
XX
= x · Pr(ω)
x ω∈Ax
X X
= x Pr(ω)
x ω∈Ax
X
= x · Pr (Ax )
x
X
= x · Pr(X = x).
x
6.5. Linearity of Expectation 293
When determining the expected value of a random variable X, it is usually
easier to use Lemma 6.4.3 than Definition 6.4.1. To use Lemma 6.4.3, you
have to do the following:
• Determine all values x that X can take, i.e., determine the range of
the function X.
• For each such value x, determine Pr(X = x).
• Compute the sum of all products x · Pr(X = x).
Expected value of a random variable X : S → R:
P
• Definition 6.4.1: E(X) = ω∈S X(ω) · Pr(ω). This is a
sum over all elements of the domain of X.
P
• Lemma 6.4.3: E(X) = x x · Pr(X = x). This is a sum
over all elements of the range of X.
6.5 Linearity of Expectation
In this section, we will present one of the most useful tools for determining
expected values. Consider a probability space (S, Pr), and let X and Y be
two random variables on S. Recall that X and Y are functions that map
elements of S to real numbers. Let a and b be two real numbers, and let
Z : S → R be the random variable defined by
Z(ω) = a · X(ω) + b · Y (ω)
for all elements ω in S. Thus, we combine the random variables X and Y ,
together with the real numbers a and b, into a new random variable Z on
the same sample space S. Usually, we just write this new random variable
as Z = aX + bY .
The Linearity of Expectation tells us how to obtain the expected value of
Z from the expected values of X and Y :
294 Chapter 6. Random Variables and Expectation
Theorem 6.5.1 Let (S, Pr) be a probability space. For any two random
variables X and Y on S, and for any two real numbers a and b,
E(aX + bY ) = a · E(X) + b · E(Y ).
Proof. We write Z = aX + bY . Using Definition 6.4.1, we get
X
E(Z) = Z(ω) · Pr(ω)
ω∈S
X
= (a · X(ω) + b · Y (ω)) · Pr(ω)
ω∈S
X X
= a X(ω) · Pr(ω) + b Y (ω) · Pr(ω)
ω∈S ω∈S
= a · E(X) + b · E(Y ).
Let us return to the example in which we roll two fair and independent
dice, one being red and the other being blue. Define the random variable
X to be the sum of the results of the two rolls. We have seen two ways to
compute the expected value E(X) of X. We now present a third way, which
is the easiest one: We define two random variables
Y = the result of the red die
and
Z = the result of the blue die.
In Section 6.4.1, we have seen that
1 1 1 1 1 1 7
E(Y ) = 1 · +2· +3· +4· +5· +6· = .
6 6 6 6 6 6 2
By the same computation, we have
7
E(Z) = .
2
Observe that
X = Y + Z.
6.5. Linearity of Expectation 295
Then, by the Linearity of Expectation (i.e., Theorem 6.5.1), we have
E(X) = E(Y + Z)
= E(Y ) + E(Z)
7 7
= +
2 2
= 7.
We have stated the Linearity of Expectation for two random variables.
The proof of Theorem 6.5.1 can easily be generalized to any finite sequence
of random variables:
Theorem 6.5.2 Let (S, Pr) be a probability space, let n ≥ 2 be an integer, let
X1 , X2 , . . . , Xn be a sequence of random variables on S, and let a1 , a2 , . . . , an
be a sequence of real numbers. Then,
n
! n
X X
E ai X i = ai · E (Xi ) .
i=1 i=1
The following theorem states that the Linearity of Expectation also holds
for infinite sequences of random variables:
Theorem 6.5.3 Let (S, Pr) be a probability space and let X1 , X2 , . . . be an
infinite sequence of random variables on S such that the infinite series
∞
X
E (|Xi |)
i=1
converges. Then,
∞
! ∞
X X
E Xi = E (Xi ) .
i=1 i=1
Proof. Define the random variable X to be
∞
X
X= Xi .
i=1
That is, as a function X : S → R, we have
∞
X
X(ω) = Xi (ω)
i=1
296 Chapter 6. Random Variables and Expectation
for all elements ω in S.
The derivation
P below uses Definition 6.4.1 and the assumption that the
infinite series ∞i=1 E (|Xi |) converges, which allows us to change the order
of summation without changing the value of the series:
∞
X ∞ X
X
E (Xi ) = Xi (ω) · Pr(ω)
i=1 i=1 ω∈S
XX ∞
= Xi (ω) · Pr(ω)
ω∈S i=1
X ∞
X
= Pr(ω) Xi (ω)
ω∈S i=1
X
= Pr(ω) · X(ω)
ω∈S
= E(X)
∞
!
X
= E Xi .
i=1
6.6 The Geometric Distribution
Let p be a real number with 0 < p < 1 and consider an experiment that is
successful with probability p and fails with probability 1 − p. We repeat this
experiment independently until it is successful for the first time. What is the
expected number of times that we perform the experiment?
We model this problem in the following way: Assume we have a coin that
comes up heads with probability p and, thus, comes up tails with probability
1 − p. We flip this coin repeatedly and independently until it comes up heads
for the first time. (We have seen this process in Section 5.15 for the case
when p = 1/2.) Define the random variable X to be the number of times
that we flip the coin; this includes the last coin flip, which resulted in heads.
We want to determine the expected value E(X) of X.
The sample space is given by
S = {T k−1 H : k ≥ 1},
6.6. The Geometric Distribution 297
where T k−1 H denotes the sequence consisting of k − 1 tails followed by one
heads. Since the coin flips are independent, the outcome T k−1 H has a prob-
ability of (1 − p)k−1 p = p(1 − p)k−1 , i.e.,
Pr T k−1 H = p(1 − p)k−1 .
Let us first verify that all probabilities add up to 1: Using Lemma 5.15.2, we
have
∞
X ∞
X
Pr T k−1 H p(1 − p)k−1
=
k=1 k=1
X∞
= p (1 − p)k−1
k=1
X∞
= p (1 − p)`
`=0
1
= p·
1 − (1 − p)
= 1.
6.6.1 Determining the Expected Value
We are going to use Lemma 6.4.3 to determine the expected value E(X).
We first observe that X can take any value in {1, 2, 3, . . .}. For any integer
k ≥ 1, X = k if and only if the coin flips give the sequence T k−1 H. It follows
that
Pr(X = k) = Pr T k−1 H = p(1 − p)k−1 .
(6.2)
By Lemma 6.4.3, we have
∞
X
E(X) = k · Pr(X = k)
k=1
X∞
= kp(1 − p)k−1
k=1
X∞
= p k(1 − p)k−1 .
k=1
How do we determine the infinite series on the right-hand side?
298 Chapter 6. Random Variables and Expectation
According to Lemma 5.15.2, we have
∞
X 1
xk = ,
k=0
1−x
for any real number x with −1 < x < 1. Both sides of this equation are func-
tions of x and these two functions are equal to each other. If we differentiate
both sides, we get two derivatives that are also equal to each other:
∞
X 1
kxk−1 = .
k=0
(1 − x)2
Since for k = 0, the term kxk−1 is equal to 0, we have
∞
X 1
kxk−1 = .
k=1
(1 − x)2
If we take x = 1 − p, we get
∞
X
E(X) = p k(1 − p)k−1
k=1
1
= p·
(1 − (1 − p))2
p
=
p2
1
= .
p
In Section 6.3, we have defined the distribution function of a random
variable. The distribution function of the coin-flipping random variable X is
given by (6.2). This function is called a geometric distribution:
Definition 6.6.1 Let p be a real number with 0 < p < 1. A random variable
X has a geometric distribution with parameter p, if its distribution function
satisfies
Pr(X = k) = p(1 − p)k−1
for any integer k ≥ 1.
6.7. The Binomial Distribution 299
Our calculation that led to the value of E(X) proves the following theo-
rem:
Theorem 6.6.2 Let p be a real number with 0 < p < 1 and let X be a
random variable that has a geometric distribution with parameter p. Then
E(X) = 1/p.
For example, if we flip a fair coin (in which case p = 1/2) repeatedly and
independently until it comes up heads for the first time, then the expected
number of coin flips is equal to 2.
6.7 The Binomial Distribution
As in Section 6.6, we choose a real number p with 0 < p < 1, and consider
an experiment that is successful with probability p and fails with probability
1−p. For an integer n ≥ 1, we repeat the experiment, independently, n times.
What is the expected number of times that the experiment is successful?
We again model this problem using a coin that comes up heads with
probability p and, thus, comes up tails with probability 1−p. We flip the coin,
independently, n times and define the random variable X to be the number
of times the coin comes up heads. We want to determine the expected value
E(X) of X.
Since our coin comes up heads with probability p, it is reasonable to guess
that E(X) is equal to pn. For example, if p = 1/2, then, on average, n/2
of the coin flips should come up heads. We will prove below that E(X) is
indeed equal to pn.
6.7.1 Determining the Expected Value
Since the random variable X can take any value in {0, 1, 2, . . . , n}, we have,
by Lemma 6.4.3,
X n
E(X) = k · Pr(X = k).
k=0
Thus, we have to determine Pr(X = k), i.e., the probability that in a sequence
of n independent coin flips, the coin comes up heads exactly k times.
300 Chapter 6. Random Variables and Expectation
To give an example, assume that n = 4 and k = 2. The table below gives
all 42 = 6 sequences of 4 coin flips that contain exactly 2 H’s, together with
their probabilities:
sequence probability
HHT T p · p · (1 − p) · (1 − p) = p2 (1 − p)2
HT HT p · (1 − p) · p · (1 − p) = p2 (1 − p)2
HT T H p · (1 − p) · (1 − p) · p = p2 (1 − p)2
T HHT (1 − p) · p · p · (1 − p) = p2 (1 − p)2
T HT H (1 − p) · p · (1 − p) · p = p2 (1 − p)2
T T HH (1 − p) · (1 − p) · p · p = p2 (1 − p)2
4
As can be seen from this table, each of the 2
sequences has the same
probability p2 (1 − p)2 . It follows that, if n = 4,
4 2
Pr(X = 2) = p (1 − p)2 .
2
We now consider the general case. Let n ≥ 1 and k be integers with
0 ≤ k ≤ n. Then, X = k if and only if there are exactly k H’s in the
sequence of n coin flips. The number of such sequences is equal to nk , and
each one of them has probability pk (1 − p)n−k . Therefore, we have
n k
Pr(X = k) = p (1 − p)n−k . (6.3)
k
As a sanity check, let us use Newton’s Binomial Theorem (i.e., Theorem 3.6.5)
to verify that all probabilities add up to 1:
n n
X X n k
Pr(X = k) = p (1 − p)n−k
k=0 k=0
k
= ((1 − p) + p)n
= 1.
We are now ready to compute the expected value of the random vari-
6.7. The Binomial Distribution 301
able X:
n
X
E(X) = k · Pr(X = k)
k=0
n
X n k
= k p (1 − p)n−k
k=0
k
n
X n k
= k p (1 − p)n−k .
k=1
k
Since
n n!
k = k·
k k!(n − k)!
(n − 1)!
= n·
(k − 1)!(n − k)!
n−1
= n ,
k−1
we get
n
n−1 k
X
E(X) = n p (1 − p)n−k .
k=1
k − 1
By changing the summation variable from k to ` + 1, we get
n−1
n − 1 `+1
X
E(X) = n p (1 − p)n−1−`
`=0
`
n−1
n−1 `
X
= pn p (1 − p)n−1−` .
`=0
`
By Newton’s Binomial Theorem (i.e., Theorem 3.6.5), the summation is equal
to
((1 − p) + p)n−1 = 1.
Therefore, we get
E(X) = pn · 1
= pn.
302 Chapter 6. Random Variables and Expectation
We have done the following: Our intuition told us that E(X) = pn. Then,
we went through a painful calculation to show that our intuition was correct.
There must be an easier way to show that E(X) = pn. We will show below
that there is indeed a much easier way.
6.7.2 Using the Linearity of Expectation
We define a sequence X1 , X2 , . . . , Xn of random variables as follows: For each
i with 1 ≤ i ≤ n,
1 if the i-th coin flip results in heads,
Xi =
0 if the i-th coin flip results in tails.
Observe that
X = X1 + X2 + · · · + X n ,
because
• X counts the number of heads in the sequence of n coin flips, and
• the summation on the right-hand side is equal to the number of 1’s
in the sequence X1 , X2 , . . . , Xn , which, by definition, is equal to the
number of heads in the sequence of n coin flips.
Using the Linearity of Expectation (see Theorem 6.5.2), we get
n
!
X
E(X) = E Xi
i=1
n
X
= E (Xi ) .
i=1
Thus, we have to determine the expected value of Xi . Since Xi is either 1
or 0, we have, using Lemma 6.4.3,
E (Xi ) = 1 · Pr (Xi = 1) + 0 · Pr (Xi = 0)
= Pr (Xi = 1)
= Pr(the i-th coin flip results in heads)
= p.
6.8. Indicator Random Variables 303
We conclude that
n
X
E(X) = E (Xi )
i=1
n
X
= p
i=1
= pn.
I hope you agree that this is much easier than what we did before.
The distribution function of the random variable X is given by (6.3).
This function is called a binomial distribution:
Definition 6.7.1 Let n ≥ 1 be an integer and let p be a real number with
0 < p < 1. A random variable X has a binomial distribution with parameters
n and p, if its distribution function satisfies
n k
Pr(X = k) = p (1 − p)n−k
k
for any integer k with 0 ≤ k ≤ n.
Our calculation that led to the value of E(X) proves the following theo-
rem:
Theorem 6.7.2 Let n ≥ 1 be an integer, let p be a real number with 0 <
p < 1, and let X be a random variable that has a binomial distribution with
parameters n and p. Then
E(X) = pn.
6.8 Indicator Random Variables
In Section 6.7, we considered the random variable X whose value is equal
to the number of heads in a sequence of n independent coin flips. In Sec-
tion 6.7.2, we defined a sequence X1 , X2 , . . . , Xn of random variables, where
Xi = 1 if the i-th coin flip results in heads and Xi = 0 otherwise. This
random variable Xi indicates whether or not the i-th flip in the sequence is
heads. Because of this, we call Xi an indicator random variable.
304 Chapter 6. Random Variables and Expectation
Definition 6.8.1 A random variable X is an indicator random variable, if
it can only take values in {0, 1}.
As we have already seen in Section 6.7.2, the expected value of an indi-
cator random variable is easy to determine:
Lemma 6.8.2 If X is an indicator random variable, then
E(X) = Pr(X = 1).
Proof. Since X is either 0 or 1, we have, using Lemma 6.4.3,
E(X) = 0 · Pr(X = 0) + 1 · Pr(X = 1)
= Pr(X = 1).
In the following subsections, we will see some examples of how indicator
random variables can be used to compute the expected value of non-trivial
random variables.
6.8.1 Runs in Random Bitstrings
Let n be a large integer. We generate a random bitstring
R = r1 r2 . . . rn
by flipping a fair coin, independently, n times. Let k ≥ 1 be an integer.
Recall from Section 5.14 that a run of length k is a consecutive subsequence
of R, all of whose bits are equal. Define the random variable X to be the
number of runs of length k.
For example, if R is the bitstring
0 0 1 1 1 1 1 0 0 0 1 1 0 0 0 0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
and k = 3, then X = 6, because R contains 6 runs of length 3, starting at
positions 3, 4, 5, 8, 13, and 14.
We want to determine the expected value E(X) of X.
6.8. Indicator Random Variables 305
A run of length k can start at any of the positions 1, 2, . . . , n − k + 1. Our
approach will be to define an indicator random variable that tells us whether
or not the subsequence of length k that starts at any such position is a run.
Thus, for any i with 1 ≤ i ≤ n − k + 1, we define the indicator random
variable
1 if the subsequence ri ri+1 . . . ri+k−1 is a run,
Xi =
0 otherwise.
Using Lemma 6.8.2, we get
E (Xi ) = Pr (Xi = 1) .
Since Xi = 1 if and only if all bits in the subsequence ri ri+1 . . . ri+k−1 are 0
or all bits in this subsequence are 1, we have
E (Xi ) = Pr (Xi = 1)
= (1/2)k + (1/2)k
= 1/2k−1 .
Since
n−k+1
X
X= Xi ,
i=1
the Linearity of Expectation (see Theorem 6.5.2) implies that
n−k+1
!
X
E(X) = E Xi
i=1
n−k+1
X
= E (Xi )
i=1
n−k+1
X
= 1/2k−1
i=1
n−k+1
= .
2k−1
Observe that the random variables X1 , X2 , . . . , Xn+k−1 are not mutually
independent. (Do you see why?) Nevertheless, our derivation is correct,
306 Chapter 6. Random Variables and Expectation
because the Linearity of Expectation is valid for any sequence of random
variables; independence is not needed.
For example, if we take k = 1 + log n, then 2k−1 = 2log n = n, so that
n − log n log n
E(X) = =1− .
n n
Thus, for large values of n, the expected number of runs of length 1 + log n
is very close to 1. This is in line with Section 5.14, because we proved there
that it is very likely that the sequence contains a run of length about log n.
If we take k = 1 + 21 log n, then
√ √
2k−1 = 2(log n)/2 = 2log n = n
and
n − 12 log n √ log n
E(X) = √ = n− √ .
n 2 n
Thus, for large values
√ of n, the expected number of runs of length 1 + 21 log n
is very close to n.
6.8.2 Largest Elements in Prefixes of Random Permu-
tations
Let n ≥ 1 be an integer and consider a sequence s1 , s2 , . . . , sn of n numbers.
The following algorithm computes the largest element in this sequence:
Algorithm FindMax(s1 , s2 , . . . , sn ):
max = −∞;
for i = 1 to n
do if si > max
then max = si (*)
endif
endfor;
return max
We would like to know the number of times that line (*) is executed,
i.e., the number of times that the value of the variable max changes. For
example, if the input sequence is
3, 2, 5, 4, 6, 1,
6.8. Indicator Random Variables 307
then the value of max changes 3 times, namely when we encounter 3, 5,
and 6. On the other hand, for the sequence
6, 5, 4, 3, 2, 1,
the value of max changes only once, whereas for
1, 2, 3, 4, 5, 6,
it changes 6 times.
Assume that the input sequence s1 , s2 , . . . , sn is a uniformly random per-
mutation of the set {1, 2, . . . , n}. Thus, each permutation has probability
1/n! of being the input. We define a random variable X whose value is equal
to the number of times that line (*) is executed when running algorithm
FindMax(s1 , s2 , . . . , sn ). We are interested in the expected value E(X) of
this random variable.
The algorithm makes n iterations. In each iteration, line (*) is either
executed or not executed. We define, for each iteration, an indicator random
variable that tells us whether or not line (*) is executed during that iteration.
That is, for any i with 1 ≤ i ≤ n, we define
1 if line (*) is executed in the i-th iteration,
Xi =
0 otherwise.
Since n
X
X= Xi ,
i=1
it follows from the Linearity of Expectation (see Theorem 6.5.2) that
n
!
X
E(X) = E Xi
i=1
n
X
= E (Xi )
i=1
Xn
= Pr (Xi = 1) .
i=1
How do we determine Pr (Xi = 1)? Observe that Xi = 1 if and only
if the maximum of the subsequence s1 , s2 , . . . , si is at the last position in
308 Chapter 6. Random Variables and Expectation
this subsequence. Since the entire sequence s1 , s2 , . . . , sn is a uniformly ran-
dom permutation of the set {1, 2, . . . , n}, the elements in the subsequence
s1 , s2 , . . . , si are in a uniformly random order as well. The largest element in
this subsequence is in any of the i positions with equal probability 1/i. In
particular, the probability that the largest element is at the last position in
this subsequence is equal to 1/i. It follows that
Pr (Xi = 1) = 1/i.
This can be proved in a more formal way as follows: By the Product Rule,
the number of permutations s1 , s2 , . . . , sn of {1, 2, . . . , n} for which si is the
largest element among s1 , s2 , . . . , si is equal to
n
(i − 1)!(n − i)! = n!/i.
i
(Do you see why?) Therefore,
n!/i
Pr (Xi = 1) = = 1/i.
n!
Thus,
n
X
E(X) = Pr (Xi = 1)
i=1
n
X
= 1/i
i=1
1 1 1
= 1+ + + ··· + .
2 3 n
The number on the right-hand side is called the harmonic number and de-
noted by Hn . In the following subsection, we will show that Hn is approx-
imately equal to ln n. Thus, the expected number of times that line (*) of
algorithm FindMax is executed, when given as input a uniformly random
permutation of {1, 2, . . . , n}, is about ln n.
As a final remark, the indicator random variables X1 , X2 , . . . , Xn that we
have introduced above are mutually independent; see Exercise 5.98. Keep in
mind, however, that we do not need this, because the Linearity of Expectation
does not require these random variables to be mutually independent.
6.8. Indicator Random Variables 309
6.8.3 Estimating the Harmonic Number
Consider a positive real-valued decreasing function f : [1, ∞) → R. Thus, if
1 ≤ x < x0 , then f (x) ≥ f (x0 ) > 0. For any integer n ≥ 1, we would like to
estimate the summation n
X
f (i).
i=1
For example, if we take f (x) = 1/x, then the summation is the harmonic
number Hn of the previous subsection.
For each i with 2 ≤ i ≤ n, draw the rectangle with bottom-left corner at
the point (i − 1, 0) and top-right corner at the point (i, f (i)), as in the figure
below.
y = f (x)
1 2 3 4 n−1 n
The area of the i-th rectangle is equal to f (i) and, thus,
n
X
f (i)
i=1
is equal to the sum of
• f (1) and
• the total area of the n − 1 rectangles.
Since f is decreasing, the rectangles are below the graph y = f (x). It follows
that the total area of the n − 1 rectangles is less than or equal to the area
between f and the x-axis, between x = 1 and x = n. We conclude that
Xn Z n
f (i) ≤ f (1) + f (x) dx. (6.4)
i=1 1
310 Chapter 6. Random Variables and Expectation
To obtain a lower bound on the summation, we modify the figure as
indicated below: For each i with 1 ≤ i ≤ n, draw the rectangle with bottom-
left corner at the point (i, 0) and top-right corner at the point (i + 1, f (i));
see the figure below.
y = f (x)
1 2 3 4 n n+1
In this case, the graph y = f (x) is below the top sides of the rectangles
and, therefore,
Xn Z n+1
f (i) ≥ f (x) dx. (6.5)
i=1 1
If we apply (6.4) and (6.5) to the function f (x) = 1/x, then we get
n
X 1
Hn =
i=1
i
n
dx
Z
≤ 1+
1 x
= 1 + ln n
and
n
X 1
Hn =
i=1
i
n+1
dx
Z
≥
1 x
= ln(n + 1)
≥ ln n.
We have proved the following result:
6.9. The Insertion-Sort Algorithm 311
Pn
Lemma 6.8.3 For any integer n ≥ 1, the harmonic number Hn = i=1 1/i
satisfies
ln n ≤ Hn ≤ 1 + ln n.
6.9 The Insertion-Sort Algorithm
InsertionSort is a simple sorting algorithm that takes as input an array
A[1 . . . n] of numbers. The algorithm uses a for-loop in which a variable i
runs from 2 to n. At the start of the i-th iteration,
• the subarray A[1 . . . i − 1] is sorted, whereas
• the algorithm has not yet seen any of the elements in the subarray
A[i . . . n].
In the i-th iteration, the algorithm takes the element A[i] and repeatedly
swaps it with its left neighbor until the subarray A[1 . . . i] is sorted. The
pseudocode of this algorithm is given below.
Algorithm InsertionSort(A[1 . . . n]):
for i = 2 to n
do j = i;
while j > 1 and A[j] < A[j − 1]
do swap A[j] and A[j − 1];
j =j−1
endwhile
endfor
We are interested in the total number of swaps that are made by this
algorithm. The worst-case happens when the input array is sorted in reverse
order, in which case the total number of swaps is equal to
n
1 + 2 + 3 + · · · + (n − 1) = .
2
Thus, in the worst case, each of the n2 pairs of input elements is swapped.
Assume that the input array A[1 . . . n] contains a uniformly random per-
mutation of the set {1, 2, . . . , n}. Thus, each permutation has probability
312 Chapter 6. Random Variables and Expectation
1/n! of being the input. We define the random variable X to be the total
number of swaps made when running algorithm InsertionSort(A[1 . . . n]).
We will determine the expected value E(X) of X.
Since we want to count the number of pairs of input elements that are
swapped, we will use, for each pair of input elements, an indicator random
variable that indicates whether or not this pair gets swapped by the algo-
rithm. That is, for each a and b with 1 ≤ a < b ≤ n, we define
1 if a and b get swapped by the algorithm,
Xab =
0 otherwise.
We observe that, since a < b, these two elements get swapped if and only
if in the input array, b is to the left of a. Since the input array contains a
uniformly random permutation, the events “b is to the left of a” and “a is to
the left of b” are symmetric. Therefore, we have
E (Xab ) = Pr (Xab = 1) = 1/2.
A formal proof of this is obtained by showing that there are n!/2 permutations
of {1, 2, . . . , n} in which b appears to the left of a and, thus, n!/2 permutations
in which a appears to the left of b. (See also Exercise 5.71.)
Since each pair of input elements is swapped at most once, we have
n−1 X
X n
X= Xab .
a=1 b=a+1
It follows from the Linearity of Expectation (see Theorem 6.5.2) that
n−1 X n
!
X
E(X) = E Xab
a=1 b=a+1
n−1 X
X n
= E (Xab )
a=1 b=a+1
n−1 n
X X 1
=
a=1 b=a+1
2
1 n
= .
2 2
Thus, the expected number of swaps on a uniformly random input array is
one half times the worst-case number of swaps.
6.10. The Quick-Sort Algorithm 313
6.10 The Quick-Sort Algorithm
We have already seen algorithm QuickSort in Section 1.3. This algorithm
takes as input an array A[1 . . . n] of numbers, which we assume for simplicity
to be pairwise distinct. A generic call QuickSort(A, i, j) takes two indices
i and j and sorts the subarray A[i . . . j]. Thus, the call QuickSort(A, 1, n)
sorts the entire array.
Algorithm QuickSort(A, i, j):
if i < j
then p = uniformly random element in A[i . . . j];
compare p with all other elements in A[i . . . j];
rearrange A[i . . . j] such that it has the following
form (this rearranging defines the value of k):
<p p >p
i k j
QuickSort(A, i, k − 1);
QuickSort(A, k + 1, j)
endif
The element p is called the pivot. We have seen in Section 1.3 that the
worst-case running time of algorithm QuickSort(A, 1, n) is Θ(n2 ). In this
section, we will prove that the expected running time is only O(n log n).
We assume for simplicity that the input array is a permutation of the set
{1, 2, . . . , n}. We do not make any other assumption about the input. In
particular, we do not assume that the input is a random permutation. The
only place where randomization is used is when the pivot is chosen: It is
chosen uniformly at random in the subarray on which QuickSort is called.
The quantity that we will analyze is the total number of comparisons
(between pairs of input elements) that are made during the entire execution
of algorithm QuickSort(A, 1, n). In such a comparison, the algorithm takes
two distinct input elements, say a and b, and decides whether a < b or a > b.
Observe from the pseudocode that the only comparisons being made are
between the pivot and all other elements in the subarray that is the input
to the current call to QuickSort. Since the operation “compare a to b” is
314 Chapter 6. Random Variables and Expectation
the same as the operation “compare b to a” (even though the outcomes are
opposite), we will assume below that in such a comparison, a < b.
We define the random variable X to be the total number of comparisons
that are made by algorithm QuickSort(A, 1, n). We will prove that the
expected value of X satisfies E(X) = O(n log n).
For each a and b with 1 ≤ a < b ≤ n, we consider the indicator random
variable
1 if a and b are compared to each other when
Xab = running QuickSort(A, 1, n),
0 otherwise.
Since each pair of input elements is compared at most once, we have
n−1 X
X n
X= Xab .
a=1 b=a+1
It follows from the Linearity of Expectation (see Theorem 6.5.2) that
n−1 X n
!
X
E(X) = E Xab
a=1 b=a+1
n−1
X n
X
= E (Xab )
a=1 b=a+1
n−1
X n
X
= Pr (Xab = 1) .
a=1 b=a+1
We consider two input elements a and b with 1 ≤ a < b ≤ n. We are
going to determine Pr (Xab = 1), i.e., the probability that the elements a and
b are compared to each other when running algorithm QuickSort(A, 1, n).
Consider the set
Sab = {a, a + 1, . . . , b}.
At the start of algorithm QuickSort(A, 1, n), all elements of the set Sab
are part of the input. Consider the first pivot p that is chosen. We observe
the following:
• Assume that p 6∈ Sab .
6.10. The Quick-Sort Algorithm 315
– If p < a, then after the algorithm has rearranged the input ar-
ray, all elements of the set Sab are to the right of p and, thus,
all these elements are part of the input for the recursive call
QuickSort(A, k+1, n). During the rearranging, a and b have not
been compared to each other. However, they may be compared
to each other during later recursive calls.
– If p > b, then after the algorithm has rearranged the input ar-
ray, all elements of the set Sab are to the left of p and, thus,
all these elements are part of the input for the recursive call
QuickSort(A, 1, k−1). During the rearranging, a and b have not
been compared to each other. However, they may be compared
to each other during later recursive calls.
• Assume that p ∈ Sab .
– If p 6= a and p 6= b, then after the algorithm has rearranged the
input array, a is to the left of p and b is to the right of p. During the
rearranging, a and b have not been compared to each other. Also,
since a and b have been “separated”, they will not be compared
to each other during later recursive calls. Thus, we have Xab = 0.
– If p = a or p = b, then during the rearranging, a and b have been
compared to each other. Thus, we have Xab = 1. (Note that in
later recursive calls, a and b will not be compared to each other
again.)
We conclude that whether or not a and b are compared to each other is
completely determined by the element of the set Sab that is the first element
in this set to be chosen as a pivot. If this element is equal to a or b, then
Xab = 1. On the other hand, if this element belongs to Sab \ {a, b}, then
Xab = 0. Since
• in any recursive call, the pivot is chosen uniformly at random from the
subarray that is the input for this call, and
• at the start of the first recursive call in which the pivot belongs to the
set Sab , all elements of this set are part of the input for this call,
each of the b − a + 1 elements of Sab has the same probability of being the
316 Chapter 6. Random Variables and Expectation
first element of Sab that is chosen as a pivot. It follows that
2
Pr (Xab = 1) = .
b−a+1
We conclude that
n−1 X
X n
E(X) = Pr (Xab = 1)
a=1 b=a+1
n−1 n
X X 2
=
a=1 b=a+1
b−a+1
n−1
X 1 1 1
= 2 + + ··· +
a=1
2 3 n−a+1
n−1
X 1 1 1
≤ 2 + + ··· +
a=1
2 3 n
n−1
X
= 2 (Hn − 1)
a=1
= 2(n − 1) (Hn − 1)
≤ 2n (Hn − 1) ,
where Hn is the harmonic number that we have seen in Sections 6.8.2 and 6.8.3.
Using Lemma 6.8.3, it follows that
E(X) ≤ 2n ln n.
6.11 Skip Lists
Consider a set S of n numbers. We would like to store these numbers in a
data structure that supports the following operations:
• Search(x): This operation returns the largest element in the set S
that is less than or equal to x.
• Insert(x): This operation inserts the number x into the set S.
• Delete(x): This operation deletes the number x from the set S.
6.11. Skip Lists 317
A standard data structure for this problem is a balanced binary search
tree (such as a red-black tree or an AVL-tree), which allows each of these three
operations to be performed in O(log n) time. Searching in a binary search
tree is straightforward, but keeping the tree balanced after an insertion or
deletion is cumbersome.
In this section, we introduce skip lists as an alternative data structure.
A skip list is constructed using the outcomes of coin flips, which result in a
structure that is balanced in the expected sense. Because of this, the insertion
and deletion algorithms become straightforward: We, as a programmer, do
not have to take care of rebalancing operations, because the coin flips take
care of this.
To define a skip list for the set S of n numbers, we first construct a
sequence S0 , S1 , S2 , . . . of subsets of S:
• Let S0 = S.
• For i = 0, 1, 2, . . ., assume that the set Si has already been constructed.
If Si is non-empty, we do the following:
– Initialize an empty set Si+1 .
– For each element y in the set Si , flip a fair and independent coin.
If the coin comes up heads, element y is added to the set Si+1 .
The process terminates as soon as the next set Si+1 is empty.
Let h be the number of non-empty sets that are constructed by this
process, and consider the sequence S0 , S1 , . . . , Sh of sets. Observe that h is a
random variable and each of the sets S1 , S2 , . . . , Sh is a random subset of S.
The skip list for S consists of the following:
• For each i with 0 ≤ i ≤ h, we store the sorted sequence of elements of
the set Si in a linked list Li .
– Each node u of Li stores one element of Si , which is denoted by
key(u).
– Each node u of Li stores a pointer to its successor node in Li ,
which is denoted by right(u). If u is the rightmost node in Li ,
then right(u) = nil .
– We add a dummy node at the beginning of Li . The key of this
node is nil and its successor is the node of Li whose key is the
smallest element in Si .
318 Chapter 6. Random Variables and Expectation
• For each i with 1 ≤ i ≤ h and each node u of Li , u stores a pointer to
the node u0 in Li−1 for which key(u0 ) = key(u). The node u0 is denoted
by down(u).
• There is a pointer to the dummy node in the list Lh . We will refer to
this node as the root of the skip list.
The value of h is called the height of the skip list. An example of a skip list
of height h = 3 for the set S = {1, 2, 3, 4, 6, 7, 9} is shown in the figure below.
L3 7
L2 3 7
L1 1 3 4 7 9
L0 1 2 3 4 6 7 9
6.11.1 Algorithm Search
The algorithm that searches for a number x keeps track of the current node u
and the index i of the list Li that contains u. Initially, u is the root of the
skip list and i = h. At any moment, if i ≥ 1, the algorithm tests if the key
of right(u) is less than x. If this is the case, then u moves one node to the
right in the list Li ; otherwise, u moves to the node down(u) in the list Li−1 .
Once i = 0, node u moves to the right in the list L0 and stops at the last
node whose key is at most equal to x. The pseudocode of this algorithm
Search(x) is given below.
6.11. Skip Lists 319
Algorithm Search(x):
// returns the rightmost node u in L0 such that key(u) ≤ x
u = root of the skip list;
i = h;
while i ≥ 1
do if right(u) 6= nil and key(right(u)) < x
then u = right(u)
else u = down(u);
i=i−1
endif
endwhile;
while right(u) 6= nil and key(right(u)) ≤ x
do u = right(u)
endwhile;
return u
The dashed arrows in the figure below show the path that is followed when
running algorithm Search(7). Note that if we replace “key(right(u)) < x”
in the first while-loop by “key(right(u)) ≤ x”, we obtain a different path that
ends in the same node: This path moves from the root to the node in L3
whose key is 7, and then it moves down to the list L0 . As we will see later,
using the condition “key(right(u)) < x” simplifies the algorithm for deleting
an element from the skip list.
L3 7
L2 3 7
L1 1 3 4 7 9
L0 1 2 3 4 6 7 9
6.11.2 Algorithms Insert and Delete
Algorithm Insert(x) takes as input a number x and inserts it into the skip
list. This algorithm works as follows:
320 Chapter 6. Random Variables and Expectation
• Run algorithm Search(x) and consider the node u that is returned.
We assume that key(u) 6= x and, thus, x is not in the skip list yet. Ob-
serve that the new number x belongs between the nodes u and right(u).
• Flip a fair and independent coin repeatedly until it comes up tails for
the first time. Let k be the number of flips.
• Add the new number x to the lists L0 , L1 , . . . , Lk−1 . Note that if k ≥
h + 2, we have to add new lists Lh+1 , . . . , Lk−1 to the skip list (each one
containing a dummy node and a node storing x), set h = k − 1, and
update the pointer to the root of the new skip list.
• When adding x to a list Li , we have to know its predecessor in this list.
– To find these predecessors, we modify algorithm Search(x) as
follows: Each time the current node u moves down, we push u
onto an initially empty stack. In this way, the predecessors that
we need are stored, in the correct order, on the stack.
– An easier way that avoids using a stack is to flip the coin and,
thus, determine k, before running algorithm Search(x). We then
modify algorithm Search(x): If i < k and the current node u
moves down, we add the new number x to Li between the nodes
u and right(u).
The figure below shows the skip list that results when inserting the number 5
into our example skip list. In this case, k = 3 and the new number is added
to the lists L0 , L1 , and L2 . The dashed arrows indicate the pointers that are
changed during this insertion.
L3 7
L2 3 5 7
L1 1 3 4 5 7 9
L0 1 2 3 4 5 6 7 9
Algorithm Delete(x) takes as input a number x and deletes it from the
skip list. This algorithm does the following:
6.11. Skip Lists 321
• Run a modified version of algorithm Search(x): Each time the current
node u moves down, test if key(right(u)) = x. If this is the case, delete
the node right(u) by setting right(u) = right(right(u)). Finally, delete
the node in L0 whose key is equal to x.
• At this moment, it may happen that some of the lists Lh , Lh−1 , . . . only
consist of dummy nodes. If this is the case, delete these lists, and
update the height h and the root of the new skip list.
Implementation details of skip lists and algorithms Search, Insert, and
Delete can be found in Pat Morin’s free textbook Open Data Structures,
which is available at http://opendatastructures.org/
6.11.3 Analysis of Skip Lists
In this subsection, we will prove that the expected size of a skip list is O(n)
and the expected running time of algorithm Search is O(log n). This will
imply that the expected running times of algorithms Insert and Delete
are O(log n) as well. Throughout this subsection, we assume for simplicity
that n is a power of 2, so that log n is an integer.
Consider again the lists L0 , L1 , . . . , Lh in the skip list. For the purpose of
analysis, we define, for each integer i > h, Li to be an empty list.
For each number x that is stored in the list L0 , we define the random
variable h(x) to be the largest value of i such that x is contained in the list Li .
Thus, x occurs in the lists L0 , L1 , . . . , Lh(x) , but not in the list Lh(x)+1 .
Lemma 6.11.1 For any number x that is stored in the list L0 ,
E(h(x)) = 1.
Proof. The value of h(x) is determined by the following process: flip a
fair coin repeatedly and independently until it comes up tails for the first
time. The value of h(x) is then equal to the number of flips minus one. For
example, if we flip the coin three times (i.e., obtain the sequence HHT ),
then x is contained in the lists L0 , L1 , and L2 , but not in L3 ; thus, h(x) = 2.
By Theorem 6.6.2, the expected number of coin flips is equal to two. As a
result, the expected value of h(x) is equal to one.
322 Chapter 6. Random Variables and Expectation
Lemma 6.11.2 For any number x that is stored in the list L0 and for any
i ≥ 0,
Pr (x ∈ Li ) = 1/2i .
Proof. The claim follows from the fact that x is contained in the list Li if
and only if the first i coin flips for x all result in heads.
Lemma 6.11.3 Let i ≥ 0 and let |Li | denote the number of nodes in the
list Li , ignoring the dummy node. Then,
E (|Li |) = n/2i .
Proof. We know from Lemma 6.11.2 that each number x in L0 is con-
tained in Li with probability 1/2i , independently of the other numbers in Li .
Therefore, |Li | is a random variable that has a binomial distribution with
parameters n and p = 1/2i . The claim then follows from Theorem 6.7.2.
Lemma 6.11.4 Let X be the random variable whose value is equal to the
total number of nodes in all lists L0 , L1 , L2 , . . ., ignoring the dummy nodes.
Then,
E(X) = 2n.
Proof. We will give two proofs. In the first proof, we observe that
h
X
X= |Li |
i=0
and, thus,
h
!
X
E(X) = E |Li | .
i=0
Observe that the number of terms in the summation on the right-hand side
is equal to h + 1, which is a random variable. In general, the Linearity of
Expectation does not apply to summations consisting of a random number
of terms; see Exercise 6.64 for an example. Therefore, we proceed as follows.
6.11. Skip Lists 323
Recall that, for the purpose of analysis, we have defined, for each integer
i > h, Li to be an empty list. It follows that
∞
X
X= |Li |.
i=0
Using the Linearity of Expectation (i.e., Theorem 6.5.3) and Lemmas 6.11.3
and 5.15.2, we get
∞
!
X
E(X) = E |Li |
i=0
∞
X
= E (|Li |)
i=0
∞
X
= n/2i
i=0
X∞
= n (1/2)i
i=0
= 2n.
In the second proof, we use the fact that each number x in L0 occurs in
exactly 1 + h(x) lists, namely L0 , L1 , . . . , Lh(x) . Thus, we have
X
X= (1 + h(x)) .
x
Using the Linearity of Expectation (i.e., Theorem 6.5.2) and Lemma 6.11.1,
we get
!
X
E(X) = E (1 + h(x))
x
X
= E (1 + h(x))
x
X
= (1 + E (h(x)))
x
X
= 2
x
= 2n.
324 Chapter 6. Random Variables and Expectation
Lemma 6.11.5 Recall that h is the random variable whose value is equal to
the height of the skip list. We have
E(h) ≤ log n + 1.
Proof. Since
h = max h(x),
x
we have
E(h) = E max h(x) .
x
It is tempting, but wrong, to think that this is equal to
max E (h(x)) ,
x
which is equal to 1 by Lemma 6.11.1. (In Exercise 6.63, you will find a simple
example showing that, in general, the expected value of a maximum is not
equal to the maximum of the expected values.)
To prove a correct upper bound on E(h), we introduce, for each integer
i ≥ 1, an indicator random variable
1 if the list Li stores at least one number,
Xi =
0 otherwise.
We observe that
∞
X
h= Xi .
i=1
Since Xi is either 0 or 1, it is obvious that
E (Xi ) ≤ 1. (6.6)
We next claim that
Xi ≤ |Li |. (6.7)
To justify this, if the list Li does not store any number, then (6.7) becomes
0 ≤ 0, which is a true statement. On the other hand, if the list Li stores
6.11. Skip Lists 325
at least one number, then (6.7) becomes 1 ≤ |Li |, which is again a true
statement. Combining (6.7) with Lemmas 6.4.2 and 6.11.3, we obtain
E (Xi ) ≤ E (|Li |) = n/2i . (6.8)
Using the Linearity of Expectation (i.e., Theorem 6.5.3), we get
∞
!
X
E(h) = E Xi
i=1
∞
X
= E (Xi )
i=1
log n ∞
X X
= E (Xi ) + E (Xi ) .
i=1 i=log n+1
If we apply (6.6) to the first summation and (6.8) to the second summation,
we get
log n ∞
X X n
E(h) ≤ 1+
i=1 i=log n+1
2i
∞
X n
= log n +
j=0
2log n+1+j
∞
X n
= log n +
j=0
n · 21+j
∞
X 1
= log n +
j=0
21+j
∞
1 X 1
= log n +
2 j=0
2j
1
= log n + ·2
2
= log n + 1.
326 Chapter 6. Random Variables and Expectation
Lemma 6.11.6 Let Y be the random variable whose value is equal to the
total number of nodes in all lists L0 , L1 , L2 , . . ., including the dummy nodes.
Then
E(Y ) ≤ 2n + log n + 2.
Proof. The total number of dummy nodes is equal to h + 1. Using the
notation of Lemma 6.11.4, we have
Y = X + h + 1.
Thus, using the Linearity of Expectation (i.e., Theorem 6.5.2) and Lem-
mas 6.11.4 and 6.11.5, we get
E(Y ) = E(X + h + 1)
= E(X) + E(h) + 1
≤ 2n + (log n + 1) + 1
= 2n + log n + 2.
Consider any number x. As we have seen in Section 6.11.1, algorithm
Search(x) starts at the root of the skip list and follows a path to the right-
most node u in the bottom list L0 for which key(u) ≤ x. We will refer to
this path as the search path of the algorithm. In the figure below, you see
the same skip list as we have seen before. The dashed arrows indicate the
search path of algorithm Search(7).
L3 7
L2 3 7
L1 1 3 4 7 9
L0 1 2 3 4 6 7 9
Lemma 6.11.7 For any number x, let N be the random variable whose value
is equal to the number of nodes on the search path of algorithm Search(x).
Then,
E(N ) ≤ 2 log n + 5.
6.11. Skip Lists 327
Proof. Consider the node u that is returned by algorithm Search(x), let v
be the second last node on the search path, and let P be the part of this
search path from the root to v. In the example above, u is the node in L0
whose key is 7, v is the node in L0 whose key is 6, and P is the part of the
dashed path from the root to v.
Let M be the random variable whose value is equal to the number of
nodes on P . Then, N = M + 1 and
E(N ) = E(M + 1) = E(M ) + 1.
Thus, it suffices to prove that
E(M ) ≤ 2 log n + 4.
Consider the following path P 0 in the skip list:
• P 0 starts at node v.
• At any node on P 0 , the path P 0 moves up one level if this is possible,
and moves one node to the left otherwise.
You should convince yourself that this path P 0 is the reverse of P and, there-
fore, M is equal to the number of nodes on P 0 . You should also convince
yourself that this may not be true, if we take for P the path from the root
to u.
For each i ≥ 0, let Mi be the random variable whose value is equal to the
number of nodes in the list Li at which the path P 0 moves one node to the
left. Then, M is the sum of
• h: these are the nodes on P 0 at which P 0 moves up one level,
• 1: this accounts for the last node on P 0 , which is the root, and
Ph
• i=0 Mi .
Thus,
h
!
X
E(M ) = E h + 1 + Mi
i=0
h
!
X
= E(h) + 1 + E Mi .
i=0
328 Chapter 6. Random Variables and Expectation
As in the proof of Lemma 6.11.4, the number of terms in the latter summation
is equal to h + 1, which is a random variable. Therefore, we cannot apply
the Linearity of Expectation to this sum. As in the proof of Lemma 6.11.4,
we proceed as follows. We first observe that
∞
X
M =h+1+ Mi .
i=0
As the figure below indicates, the random variable Mi can be interpreted as
being the number of tails obtained when flipping a fair coin until it comes
up heads for the first time. Since (i) the list Li may be empty (in which case
Mi = 0) or (ii) the portion of the path P 0 in Li may terminate because it
reaches the dummy node, Mi is in fact less than or equal to the number of
tails.
Li+1 :
Li : H T T T T T
Li−1 :
Therefore, by Lemma 6.4.2 and Theorem 6.6.2,
E (Mi ) ≤ 1. (6.9)
Also, since Mi is less than or equal to the size |Li | of the list Li (ignoring the
dummy node), we have, using Lemmas 6.4.2 and 6.11.3,
E (Mi ) ≤ E (|Li |) = n/2i . (6.10)
Using the Linearity of Expectation (i.e., Theorem 6.5.3), we get
∞
!
X
E(M ) = E h + 1 + Mi
i=0
∞
X
= E(h) + 1 + E (Mi )
i=0
log n ∞
X X
= E(h) + 1 + E (Mi ) + E (Mi ) .
i=0 i=log n+1
6.12. Exercises 329
We know from Lemma 6.11.5 that E(h) ≤ log n + 1. If we apply (6.9) to the
first summation and (6.10) to the second summation, we get
log n ∞
X X
E(M ) ≤ (log n + 1) + 1 + 1+ n/2i
i=0 i=log n+1
∞
X
= 2 log n + 3 + n/2i .
i=log n+1
We have seen the infinite series in the proof of Lemma 6.11.5 and showed
that it is equal to 1. Thus, we conclude that
E(M ) ≤ 2 log n + 4.
6.12 Exercises
6.1 Consider a fair coin that has 0 on one side and 1 on the other side. We
flip this coin once and roll a fair die twice. Consider the following random
variables:
X = the result of the coin,
Y = the sum of the two dice,
Z = X · Y.
• Determine the distribution functions of X, Y , and Z.
• Are X and Y independent random variables?
• Are X and Z independent random variables?
• Are Y and Z independent random variables?
• Are X, Y and Z mutually independent random variables?
330 Chapter 6. Random Variables and Expectation
6.2 Consider the set S = {2, 3, 5, 30}. We choose a uniformly random ele-
ment x from this set. Consider the random variables
1 if x is divisible by 2,
X =
0 otherwise,
1 if x is divisible by 3,
Y =
0 otherwise,
1 if x is divisible by 5,
Z =
0 otherwise.
• Is the sequence X, Y , Z of random variables pairwise independent?
• Is the sequence X, Y , Z of random variables mutually independent?
6.3 Let a and b be real numbers. You flip a fair and independent coin three
times. For i = 1, 2, 3, let
a if the i-th coin flip results in heads,
fi =
b if the i-th coin flip results in tails.
Consider the random variables
X = f1 · f2 ,
Y = f2 · f3 .
• Assume that a = b. Are the random variables X and Y independent?
• Assume that a = 0 and b 6= a. Are the random variables X and Y
independent?
• Assume that a 6= 0 and b = −a. Are the random variables X and Y
independent?
• Assume that a 6= 0, b 6= 0, a 6= b, and b 6= −a. Are the random
variables X and Y independent?
6.4 Lindsay and Simon want to play a game in which the expected amount
of money that each of them wins is equal to zero. After having chosen a num-
ber x, the game is played as follows: Lindsay rolls a fair die, independently,
three times.
6.12. Exercises 331
• If none of the three rolls results in 6, then Lindsay pays one dollar to
Simon.
• If exactly one of the rolls results in 6, then Simon pays one dollar to
Lindsay.
• If exactly two rolls result in 6, then Simon pays two dollars to Lindsay.
• If all three rolls result in 6, then Simon pays x dollars to Lindsay.
Determine the value of x.
6.5 You are given a fair coin.
• You flip this coin twice; the two flips are independent. For each heads,
you win 3 dollars, whereas for each tails, you lose 2 dollars. Consider
the random variable
X = the amount of money that you win.
– Use the definition of expected value to determine E(X).
– Use the linearity of expectation to determine E(X).
• You flip this coin 99 times; these flips are mutually independent. For
each heads, you win 3 dollars, whereas for each tails, you lose 2 dollars.
Consider the random variable
Y = the amount of money that you win.
Determine the expected value E(Y ) of Y .
r
6.6 Let r and b be positive integers and define α = r+b . A bowl contains r
red balls and b blue balls; thus, α is the fraction of the balls that are red.
Consider the following experiment:
• Choose one ball uniformly at random.
– If the chosen ball is red, then put it back, together with an addi-
tional red ball.
– If the chosen ball is blue, then put it back, together with an ad-
ditional blue ball.
332 Chapter 6. Random Variables and Expectation
Define the random variable X to be the fraction of the balls that are red,
after this experiment. Prove that E(X) = α.
6.7 The Ontario Lottery and Gaming Corporation (OLG) offers the follow-
ing lottery game:
• OLG chooses a winning number w in the set S = {0, 1, 2, . . . , 999}.
• If John wants to play, he pays $1 and chooses a number x in S.
– If x = w, then John receives $700 from OLG. In this case, John
wins $699.
– Otherwise, x 6= w and John does not receive anything. In this
case, John loses $1.
Assume that
• John plays this game once per day for one year (i.e., for 365 days),
• each day, OLG chooses a new winning number,
• each day, John chooses x uniformly at random from the set S, inde-
pendently from previous choices.
Define the random variable X to be the total amount of dollars that John
wins during one year. Determine the expected value E(X).
Hint: Use the Linearity of Expectation.
6.8 Assume we flip a fair coin twice, independently of each other. Consider
the following random variables:
X = the number of heads,
Y = the number of tails,
Z = the number of heads times the number of tails.
• Determine the expected values of these three random variables.
• Are X and Y independent random variables?
• Are X and Z independent random variables?
• Are Y and Z independent random variables?
6.12. Exercises 333
6.9 As of this writing2 , Ma Long is the number 1 ranked ping pong player in
the world. Simon Bose3 also plays ping pong, but he is not at Ma’s level yet.
If you play a game of ping pong against Ma, then you win with probability p.
If you play a game against Simon, you win with probability q. Here, p and
q are real numbers such that 0 < p < q < 1. (Of course, p is much smaller
than q.) If you play several games against Ma and Simon, then the results
are mutually independent.
You have the choice between the following two series of games:
1. MSM : First, play against Ma, then against Simon, then against Ma.
2. SMS : First, play against Simon, then against Ma, then against Simon.
For each s ∈ {MSM , SMS }, consider the event
As = “you play series s and beat Ma at least once and
beat Simon at least once”
and the random variable
Xs = the number of games you win when playing series s.
• Determine Pr (AMSM ) and Pr (ASMS ). Which of these two probabilities
is larger?
• Determine E (XMSM ) and E (XSMS ). Which of these two expected val-
ues is larger?
6.10 In order to attract more customers, the Hyacintho Cactus Bar and Grill
in downtown Ottawa organizes a game night, hosted by their star employee
Tan Tran.
After paying $26, a player gets two questions P and Q. If the player
gives the correct answer to question P , this player wins $30; if the player
gives the correct answer to question Q, this player wins $60. A player can
choose between the following two options:
1. Start with question P . In this case, the player is allowed to answer
question Q only if the answer to question P is correct.
2
November 2016
3
Jit’s son
334 Chapter 6. Random Variables and Expectation
2. Start with question Q. In this case, the player is allowed to answer
question P only if the answer to question Q is correct.
Elisa decides to play this game. The probability that Elisa correctly
answers question P is equal to 1/2, whereas she correctly answers question
Q with probability 1/3. The events of correctly answering are independent.
• Assume Elisa chooses the first option. Define the random variable X
to be the amount of money that Elisa wins (this includes the $26 that
she has to pay in order to play the game). Determine the expected
value E(X).
• Assume Elisa chooses the second option. Define the random variable
Y to be the amount of money that Elisa wins (this includes the $26
that she has to pay in order to play the game). Determine the expected
value E(Y ).
6.11 Assume we roll two fair and independent dice, where one die is red and
the other die is blue. Let (i, j) be the outcome, where i is the result of the
red die and j is the result of the blue die. Consider the random variables
X =i+j
and
Y = i − j.
Are X and Y independent random variables?
6.12 Assume we roll two fair and independent dice, where one die is red and
the other die is blue. Let (i, j) be the outcome, where i is the result of the
red die and j is the result of the blue die. Consider the random variables
X = |i − j|
and
Y = max(i, j).
Are X and Y independent random variables?
6.12. Exercises 335
6.13 Consider the sample space S = {1, 2, 3 . . . , 10}. We choose a uniformly
random element x in S. Consider the following random variables:
0 if x ∈ {1, 2},
X= 1 if x ∈ {3, 4, 5, 6},
2 if x ∈ {7, 8, 9, 10}
and
0 if x is even,
Y =
1 if x is odd.
Are X and Y independent random variables?
6.14 Consider the 8-element set A = {a, b, c, d, e, f, g, h}. We choose a uni-
formly random 5-element subset B of A. Consider the following random
variables:
X = |B ∩ {a, b, c, d}|,
Y = |B ∩ {e, f, g, h}|.
• Determine the expected value E(X) of the random variable X.
• Are X and Y independent random variables?
6.15 You roll a fair die repeatedly and independently until the result is an
even number. Consider the random variables
X = the number of times you roll the die
and
Y = the result of the last roll.
For example, if the results of the rolls are 5, 1, 3, 3, 5, 2, then X = 6 and
Y = 2.
Prove that the random variables X and Y are independent.
6.16 Consider two random variables X and Y . If X and Y are independent,
then it can be shown that
E(XY ) = E(X) · E(Y ).
In this exercise, you will show that the converse of this statement is, in
general, not true.
Let X be the random variable that takes each of the values −1, 0, and 1
with probability 1/3. Let Y be the random variable with value Y = X 2 .
336 Chapter 6. Random Variables and Expectation
• Prove that X and Y are not independent.
• Prove that E(XY ) = E(X) · E(Y ).
6.17 You are given two independent random variables X and Y , where
Pr(X = 1) = Pr(X = −1) = Pr(Y = 1) = Pr(Y = −1) = 1/2.
Consider the random variable
Z = X · Y.
Are X and Z independent random variables?
6.18 Jennifer loves to drink India Pale Ale (IPA), whereas Lindsay Bangs
prefers wheat beer. Jennifer and Lindsay decide to go to their favorite pub
Chez Connor et Simon. The beer menu shows that this pub has ten beers
on tap:
• Five of these beers are of the IPA style.
• Three of these beers are of the wheat beer style.
• Two of these beers are of the pilsner style.
Jennifer and Lindsay order a uniformly random subset of seven beers (thus,
there are no duplicates). Consider the following random variables:
J = the number of IPAs in this order,
L = the number of wheat beers in this order.
• Determine the expected value E(L) of the random variable L.
• Are J and L independent random variables?
6.19 You roll a fair die five times, where all rolls are independent of each
other. Consider the random variable
X = the largest value in these five rolls.
Prove that the expected value E(X) of the random variable X is equal to
14077
E(X) = .
2592
Hint: What are the possible value for X? What is Pr(X = k)?
6.12. Exercises 337
6.20 Consider the following algorithm, which takes as input a large integer
n and returns a random subset A of the set {1, 2, . . . , n}:
Algorithm RandomSubset(n):
// all coin flips are mutually independent
A = ∅;
for i = 1 to n
do flip a fair coin;
if the result of the coin flip is heads
then A = A ∪ {i}
endif
endfor;
return A
Define
the largest element in A if A 6= ∅,
max(A) =
0 if A = ∅,
the smallest element in A if A 6= ∅,
min(A) =
0 if A = ∅,
and the random variable
X = max(A) − min(A).
• Prove that the expected value E(X) of the random variable X satisfies
E(X) = n − 3 + f (n),
where f (n) is some function that converges to 0 when n → ∞.
Hint: Introduce random variables Y = min(A) and Z = max(A) and
compute their expected values. You may use
n
X x (n · xn+1 − (n + 1) · xn + 1)
k · xk = .
k=1
(x − 1)2
• Give an intuitive explanation why E(X) is approximately equal to n−3.
338 Chapter 6. Random Variables and Expectation
6.21 Let n ≥ 1 be an integer and let A[1 . . . n] be an array that stores a
permutation of the set {1, 2, . . . , n}. If the array A is sorted, then A[k] = k
for k = 1, 2, . . . , n and, thus,
n
X
|A[k] − k| = 0. (6.11)
k=1
If the array A is not sorted and A[k] = i, where i 6= k, then |A[k] − k| is equal
to the “distance” between the position of the value i in A and the position of
i in case the array were sorted. Thus, the summation in (6.11) is a measure
for the “sortedness” of the array A: If the summation is small, then A is
“close” to being sorted. On the other hand, if the summation is large, then
A is “far away” from being sorted. In this exercise, you will determine the
expected value of the summation in (6.11).
Assume that the array stores a uniformly random permutation of the set
{1, 2, . . . , n}. For each k = 1, 2, . . . , n, consider the random variable
Xk = |A[k] − k|,
and let
n
X
X= Xk .
k=1
• Assume that n = 1. Determine the expected value E(X).
• Assume that n ≥ 2. Is the sequence X1 , X2 , . . . , Xn of random variables
pairwise independent?
• Assume that n ≥ 1. Let k be an integer with 1 ≤ k ≤ n. Prove that
n + 1 k 2 − k − kn
E (Xk ) = + .
2 n
Hint: Assume A[k] = i. If 1 ≤ i ≤ k, then |A[k] − k| = k − i. If
k + 1 ≤ i ≤ n, then |A[k] − k| = i − k. For any integer m ≥ 1,
m(m + 1)
1 + 2 + 3 + ··· + m = .
2
6.12. Exercises 339
• Assume that n ≥ 1. Prove that
n2 − 1
E(X) = .
3
Hint:
n(n + 1)(2n + 1)
12 + 22 + 32 + · · · + n2 = .
6
6.22 Let n ≥ 2 be an integer. You are given n cider bottles C1 , C2 , . . . , Cn
and two beer bottles B1 and B2 . Consider a uniformly random permutation
of these n + 2 bottles. The positions in this permutation are numbered
1, 2, . . . , n + 2. Consider the following two random variables:
X = the position of the first cider bottle,
Y = the position of the first bottle having index 1.
For example, if n = 5 and the permutation is
B2 , C5 , C2 , C4 , B1 , C3 , C1 ,
then X = 2 and Y = 5.
• Determine the expected value E(X) of the random variable X.
• Determine the expected value E(Y ) of the random variable Y .
Pn+1 Pn+1 2
Hint: k=1 k = (n+1)(n+2)/2 and k=1 k = (n+1)(n+2)(2n+3)/6.
• Are X and Y independent random variables?
6.23 Let m ≥ 1 and n ≥ 1 be integers. You are given m cider bottles
C1 , C2 , . . . , Cm and n beer bottles B1 , B2 , . . . , Bn . Consider a uniformly ran-
dom permutation of these m + n bottles. The positions in this permutation
are numbered 1, 2, . . . , m + n. Consider the random variable
X = the position of the leftmost cider bottle.
• Determine the possible values for X.
340 Chapter 6. Random Variables and Expectation
• For any value k that X can take, prove that
n
m k−1
Pr(X = k) = · m+n .
k k
Hint: Use the Product Rule to determine the number of permutations
for which X = k. Rewrite your answer using basic properties of bino-
mial coefficients.
• For each i = 1, 2, . . . , n, consider the indicator random variable
1 if Bi is to the left of all cider bottles,
Xi =
0 otherwise.
Prove that
1
E (Xi ) = .
m+1
• Express X in terms of X1 , X2 , . . . , Xn .
• Use the expression from the previous part to determine E(X).
• Prove that
n+1 n
X k−1 m+n+1
m+n
= .
k=1 k
m(m + 1)
6.24 Let b ≥ 1, c ≥ 1, and w ≥ 1 be integers, and let n = b + c + w. You are
given b beer bottles B1 , B2 , . . . , Bb , c cider bottles C1 , C2 , . . . , Cc , and w wine
bottles W1 , W2 , . . . , Ww . Let m ≥ 1 be an integer with m ≤ b and m ≤ n − b.
All n bottles are in a box. From this box, you choose a uniformly random
subset consisting of m bottles. Consider the random variables
X = the number of beer bottles in the chosen subset,
Y = the number of cider bottles in the chosen subset,
Z = the number of wine bottles in the chosen subset.
• Determine the expected value E(X + Y + Z).
• Let k be an integer with 0 ≤ k ≤ m. Prove that
b
n−b
k m−k
Pr(X = k) = n
.
m
6.12. Exercises 341
• For each i = 1, 2, . . . , b and j = 1, 2, . . . , c, consider the indicator ran-
dom variables
1 if Bi is in the chosen subset,
Xi =
0 otherwise.
and
1 if Cj is in the chosen subset,
Yj =
0 otherwise.
Prove that
m
E (Xi ) = E (Yj ) = .
n
• Prove that m
n−b
X b bm n
k = .
k=0
k m − k n m
• Let i and j be integers with 1 ≤ i ≤ b and 1 ≤ j ≤ c. Are the random
variables Xi and Yj independent?
• Let i and j be integers with 1 ≤ i ≤ b and 1 ≤ j ≤ c. Determine
E (Xi · Yj ).
• Let i and j be integers with 1 ≤ i ≤ b and 1 ≤ j ≤ c. Is the following
true or false?
E (Xi · Yj ) = E (Xi ) · E (Yj ) .
6.25 Let m ≥ 1, n ≥ 1, and k ≥ 1 be integers with k ≤ m + n. Consider
a set P consisting of m men and n women. We choose a uniformly random
k-element subset Q of P . Consider the random variables
X = the number of men in the chosen subset Q,
Y = the number of women in the chosen subset Q,
Z = X − Y.
• Prove that
E(Z) = 2 · E(X) − k.
• Determine the expected value E(X).
Hint: Denote the men as M1 , M2 , . . . , Mm . Use indicator random
variables.
342 Chapter 6. Random Variables and Expectation
• Prove that
m−n
E(Z) = k · .
m+n
6.26 You are given four fair and independent dice, each one having six faces:
1. One die is red and has the numbers 7, 7, 7, 7, 1, 1 on its faces.
2. One die is blue and has the numbers 5, 5, 5, 5, 5, 5 on its faces.
3. One die is green and has the numbers 9, 9, 3, 3, 3, 3 on its faces.
4. One die is yellow and has the numbers 8, 8, 8, 2, 2, 2 on its faces.
Let c be a color in the set {red, blue, green, yellow}. You roll the die of
color c. Define the random variable Xc to be the result of this roll.
• For each c ∈ {red, blue, green, yellow}, determine the expected value
E (Xc ) of the random variable Xc .
• Let c and c0 be two distinct colors in the set {red, blue, green, yellow}.
Determine
Pr (Xc < Xc0 ) + Pr (Xc > Xc0 ) .
• Let c and c0 be two distinct colors in the set {red, blue, green, yellow}.
We say that the die of color c is better than the die of color c0 , if
Pr (Xc > Xc0 ) > 1/2.
– Is the red die better than the blue die?
– Is the blue die better than the green die?
– Is the green die better than the yellow die?
– Is the yellow die better than the red die?
– Explain why these dice are called non-transitive dice.
6.27 In this exercise, you are given a fair and independent coin. Let n ≥ 1
be an integer. Farah flips the coin n times, whereas May flips the coin n + 1
times. Consider the following two random variables:
X = the number of heads in Farah’s sequence of coin flips,
Y = the number of heads in May’s sequence of coin flips.
Let A be the event
A = “X < Y ”.
6.12. Exercises 343
• Prove that
n n+1
1 X X n n+1
Pr(A) = · .
22n+1 k=0 `=k+1
k `
• Consider the following two random variables:
X0 = the number of tails in Farah’s sequence of coin flips,
Y0 = the number of tails in May’s sequence of coin flips.
– What is X + X 0 ?
– What is Y + Y 0 ?
– Let B be the event
B = “ X 0 < Y 0 ”.
Explain in plain English why
Pr(A) = Pr(B).
– Express the event B in terms of the event A.
– Use the results of the previous parts to determine Pr(A).
• Prove that
n n+1
X X n n+1
· = 22n .
k=0 `=k+1
k `
6.28 Elisa Kazan’s neighborhood pub serves three types of drinks: cider,
wine, and beer. Elisa likes cider and wine, but does not like beer.
After a week of hard work, Elisa goes to this pub and repeatedly orders
a random drink (the results of the orders are mutually independent). If she
gets a glass of cider or a glass of wine, then she drinks it and places another
order. As soon as she gets a pint of beer, she drinks it and takes a taxi home.
When Elisa orders one drink, she gets a glass of cider with probability 2/5,
a glass of wine with probability 2/5, and a pint of beer with probability 1/5.
Consider the random variables
X = the number of drinks that Elisa orders,
Y = the number of different types that Elisa drinks.
344 Chapter 6. Random Variables and Expectation
If we denote cider by C, wine by W , and beer by B, then a possible sequence
of drinks is CCW CB; for this case X = 5 and Y = 3. For the sequence
W W W B, we have X = 4 and Y = 2.
• Determine the expected value E(X).
• Describe the sample space in terms of strings consisting of characters
C, W , and B.
• Describe the event “Y = 1” in terms of a subset of the sample space.
• Use the result of the previous part to determine Pr(Y = 1).
• Describe the event “Y = 2” in terms of a subset of the sample space.
• Use the result of the previous part to determine Pr(Y = 2).
• Determine Pr(Y = 3).
• Use the results of the previous five parts to determine the expected
value E(Y ).
• Consider the random variable
1 if Elisa drinks at least one glass of cider,
Yc =
0 otherwise.
Determine the expected value E (Yc ).
• Consider the random variable
1 if Elisa drinks at least one glass of wine,
Yw =
0 otherwise.
Determine the expected value E (Yw ).
• Express Y in terms of Yc and Yw .
• Use the results of the previous three parts to determine the expected
value E(Y ).
6.12. Exercises 345
6.29 You repeatedly flip a fair coin and stop as soon as you get tails followed
by heads. (All coin flips are mutually independent.) Consider the random
variable
X = the total number of coin flips.
For example, if the sequence of coin flips is HHHT T T T H, then X = 8.
• Determine the expected value E(X) of X.
Hint: Use the linearity of expectation.
6.30 In Section 6.6, we have shown that for −1 < x < 1,
∞
X x
kxk = .
k=1
(1 − x)2
In this exercise, you will prove this identity in a different way.
Consider the following infinite matrix:
x 0 0 0 0 0 ...
x2 x2 0 0 0 0 . . .
x3 x3 x3 0 0 0 . . .
x4 x4 x4 x4 0 0 . . .
x5 x5 x5 x5 x5 0 . . .
.. .. .. .. .. .. . .
. . . . . . .
We are going to add all elements in this matrix in two different ways. A
row-sum is the sum of all elements in one row, whereas a column-sum is the
sum of all elements in one column.
Note that the sum of all row-sums is equal to
∞
X
x + 2x2 + 3x3 + 4x4 + 5x5 + · · · = kxk .
k=1
• Using only the identity ∞ k 1
P
k=0 x = 1−x and algebraic manipulation,
prove that the sum of all column-sums is equal to
x
.
(1 − x)2
346 Chapter 6. Random Variables and Expectation
6.31 Let X be a random variable that takes values in {0, 1, 2, 3, . . .}. By
Lemma 6.4.3, we have
∞
X
E(X) = k · Pr(X = k).
k=1
As in Exercise 6.30, define an infinite matrix and use it to prove that
∞
X
E(X) = Pr(X ≥ k).
k=1
6.32 Let 0 < p < 1 and consider a coin that comes up heads with probability
p and tails with probability 1 − p. We flip the coin independently until it
comes up heads for the first time. Define the random variable X to be the
number of times that we flip the coin. In Section 6.6, we have shown that
E(X) = 1/p. Below, you will prove this in a different way.
• Let k ≥ 1 be an integer. Determine Pr(X ≥ k).
• Using only the identity ∞ k 1
P
k=0 x = 1−x , the expression for E(X) from
Exercise 6.31, and your answer for Pr(X ≥ k), prove that E(X) = 1/p.
6.33 By flipping a fair coin repeatedly and independently, we obtain a se-
quence of H’s and T ’s. We stop flipping the coin as soon as the sequence
contains either HH or T T . Define the random variable X to be the number
of times that we flip the coin. For example, if the sequence of coin flips is
HT HT T , then X = 5.
• Let k ≥ 2 be an integer. Determine Pr(X = k).
• Determine the expected value E(X) of X using the expression
X
E(X) = k · Pr(X = k).
k
P∞ x
Hint: Recall that, for −1 < x < 1, k=1 kxk = (1−x)2
.
• Determine Pr(X ≥ 1).
• Let k ≥ 2 be an integer. Determine Pr(X ≥ k).
6.12. Exercises 347
• According to Exercise 6.31, we have
∞
X
E(X) = Pr(X ≥ k).
k=1
Use this expression to determine the expected value E(X) of X.
6.34 Consider an experiment that is successful with probability 0.8. We
repeat this experiment (independently) until it is successful for the first time.
The first 5 times we do the experiment, we have to pay $10 per experiment.
After this, we have to pay $5 per experiment. Define the random variable X
to be the total amount of money that we have to pay during all experiments.
Determine the expected value E(X).
Hint: Recall that ∞ k−1
= 1/(1 − x)2 .
P
k=1 kx
6.35 When Lindsay and Simon have a child, this child is a boy with prob-
ability 1/2 and a girl with probability 1/2, independently of the gender of
previous children. Lindsay and Simon stop having children as soon as they
have a girl. Consider the random variables
B = the number of boys that Lindsay and Simon have
and
G = the number of girls that Lindsay and Simon have.
Determine the expected values E(B) and E(G).
6.36 Let p be a real number with 0 < p < 1. When Lindsay and Simon
have a child, this child is a boy with probability p and a girl with probability
1 − p, independently of the gender of previous children. Lindsay and Simon
stop having children as soon as they have a child that has the same gender as
their first child. Define the random variable X to be the number of children
that Lindsay and Simon
P∞ have. Determine the expected value E(X).
k−1
Hint: Recall that k=1 kx = 1/(1 − x)2 .
6.37 Let X1 , X2 , . . . , Xn be a sequence of mutually independent random vari-
ables. For each i with 1 ≤ i ≤ n, assume that
• the variable Xi is either equal to 0 or equal to n + 1, and
348 Chapter 6. Random Variables and Expectation
• E(Xi ) = 1.
Determine
Pr(X1 + X2 + · · · + Xn ≤ n).
6.38 The Ottawa Senators and the Toronto Maple Leafs play a best-of-seven
series: These two hockey teams play games against each other, and the first
team to win four games wins the series. Assume that
• each game has a winner (thus, no game ends in a tie),
• in any game, the Sens have a probability of 3/4 of defeating the Leafs,
• the results of the games are mutually independent.
Determine the probability that seven games are played in this series.
6.39 Let n ≥ 1 be an integer, let p be a real number with 0 < p < 1, and
let X be a random variable that has a binomial distribution with parameters
n and p. In Section 6.7.1, we have seen that the expected value E(X) of X
satisfies n
X n k
E(X) = k p (1 − p)n−k . (6.12)
k=1
k
Recall Newton’s Binomial Theorem (i.e., Theorem 3.6.5):
n
n
X n n−k k
(x + y) = x y .
k=0
k
• Use (6.12) to prove that E(X) = pn, by taking the derivative, with
respect to y, in Newton’s Binomial Theorem.
6.40 A block in a bitstring is a maximal consecutive substring of 1’s. For
example, the bitstring 1000111110100111 has four blocks: 1, 11111, 1, and
111.
Let n ≥ 1 be an integer and consider a random bitstring of length n that
is obtained by flipping a fair coin, independently, n times. Define the random
variable X to be the number of blocks in this bitstring.
• Use Exercise 4.43 to determine the expected value E(X) of X.
6.12. Exercises 349
• Use indicator random variables to determine the expected value E(X)
of X.
6.41 Let n ≥ 1 be an integer and consider a uniformly random permutation
a1 , a2 , . . . , an of the set {1, 2, . . . , n}. Define the random variable X to be the
number of indices i for which 1 ≤ i < n and ai < ai+1 .
Determine the expected value E(X) of X.
Hint: Use indicator random variables.
6.42 Let n ≥ 2 be an integer and let a1 , a2 , . . . , an be a permutation of the
set {1, 2, . . . , n}. Define a0 = 0 and an+1 = 0, and consider the sequence
a0 , a1 , a2 , a3 , . . . , an , an+1 .
A position i with 1 ≤ i ≤ n is called awesome, if ai > ai−1 and ai > ai+1 .
In words, i is awesome if the value at position i is larger than both its
neighboring values.
For example, if n = 6 and the permutation is 2, 5, 4, 3, 1, 6, we get the
sequence
value 0 2 5 4 3 1 6 0
position 0 1 2 3 4 5 6 7
In this case, the positions 2 and 6 are awesome, whereas the positions 1, 3,
4, and 5 are not awesome.
Consider a uniformly random permutation of the set {1, 2, . . . , n} and
define the random variable X to be the number of awesome positions. De-
termine the expected value E(X) of the random variable X.
Hint: Use indicator random variables.
6.43 Let n ≥ 1 be an integer and consider a permutation a1 , a2 , . . . , an of the
set {1, 2, . . . , n}. We partition this permutation into increasing subsequences.
For example, for n = 10, the permutation
3, 5, 8, 1, 2, 4, 10, 7, 6, 9
is partitioned into four increasing subsequences: (i) 3, 5, 8, (ii) 1, 2, 4, 10, (iii)
7, and (iv) 6, 9.
Let a1 , a2 , . . . , an be a uniformly random permutation of {1, 2, . . . , n}.
Define the random variable X to be the number of increasing subsequences
in the partition of this permutation. For the example above, we have X = 4.
In this exercise, you will determine the expected value E(X) of X in two
different ways.
350 Chapter 6. Random Variables and Expectation
• For each i with 1 ≤ i ≤ n, let
1 if an increasing subsequence starts at position i,
Xi =
0 otherwise.
For the example above, we have X1 = 1, X2 = 0, X3 = 0, and X8 = 1.
– Determine E (X1 ).
– Let i be an integer with 2 ≤ i ≤ n. Use the Product Rule to
determine the number of permutations of {1, 2, . . . , n} for which
Xi = 1.
– Use these indicator random variables to determine E(X).
• For each i with 1 ≤ i ≤ n, let
1 if the value i is the leftmost element of an increasing
Yi = subsequence,
0 otherwise.
For the example above, we have Y1 = 1, Y3 = 1, Y5 = 0, and Y7 = 1.
– Determine E (Y1 ).
– Let i be an integer with 2 ≤ i ≤ n. Use the Product Rule to
determine the number of permutations of {1, 2, . . . , n} for which
Yi = 1.
– Use these indicator random variables to determine E(X).
6.44 Lindsay Bangs and Simon Pratt visit their favorite pub that has 10
different beers on tap. Both Lindsay and Simon order, independently of each
other, a uniformly random subset of 5 beers.
• One of the beers available is Leo’s Early Breakfast IPA. Determine the
probability that this is one of the beers that Lindsay orders.
• Let X be the random variable whose value is the number of beers that
are ordered by both Lindsay and Simon. Determine the expected value
E(X) of X.
Hint: Use indicator random variables.
6.12. Exercises 351
6.45 Lindsay and Simon have discovered a new pub that has n different
beers B1 , B2 , . . . , Bn on tap, where n ≥ 1 is an integer. They want to try
all different beers in this pub and agree on the following approach: During
a period of n days, they visit the pub every day. On each day, they drink
one of the beers. Lindsay drinks the beers in order, i.e., on the i-th day, she
drinks beer Bi . Simon takes a uniformly random permutation a1 , a2 , . . . , an
of the set {1, 2, . . . , n} and drinks beer Bai on the i-th day.
Let X be the random variable whose value is the number of days during
which Lindsay and Simon drink the same beer. Determine the expected value
E(X) of X.
Hint: Use indicator random variables.
6.46 Consider the following recursive algorithm TwoTails, which takes as
input a positive integer n:
Algorithm TwoTails(n):
// all coin flips are mutually independent
flip a fair coin twice;
if the coin came up tails exactly twice
then return 2n
else TwoTails(n + 1)
endif
• You run algorithm TwoTails(1), i.e., with n = 1. Define the random
variable X to be the value of the output
of this algorithm. Let k ≥ 1
k
be an integer. Determine Pr X = 2 .
• Is the expected value E(X) of the random variable X finite or infinite?
6.47 Let A[1 . . . n] be an array of n numbers. Consider the following two al-
gorithms, which take as input the array A and a number x. If x is not present
in A, then these algorithms return the message “not present”. Otherwise,
they return an index i such that A[i] = x. The first algorithm runs linear
search from left to right, whereas the second algorithm runs linear search
from right to left.
352 Chapter 6. Random Variables and Expectation
Algorithm LinearSearchLeftToRight(A, x):
i := 1;
while i ≤ n and A[i] 6= x do i := i + 1 endwhile;
if i = n + 1 then return “not present” else return i endif
Algorithm LinearSearchRightToLeft(A, x):
i := n;
while i ≥ 1 and A[i] 6= x do i := i − 1 endwhile;
if i = 0 then return “not present” else return i endif
Consider the following algorithm, which again take as input the array A
and a number x. If x is not present in A, then it returns the message “not
present”. Otherwise, it returns an index i such that A[i] = x.
Algorithm RandomLinearSearch(A, x):
flip a fair coin;
if the coin comes up heads
then LinearSearchLeftToRight(A, x)
else LinearSearchRightToLeft(A, x)
endif
Assume that the number x occurs exactly once in the array A and let k
be the index such that A[k] = x. Let X be the random variable whose
value is the number of times the test “A[i] 6= x” is made in algorithm
RandomLinearSearch(A, x). (In words, X is the number of compar-
isons made by algorithm RandomLinearSearch(A, x).) Determine the
expected value E(X) of X.
6.48 Let n ≥ 3 be an integer and let p be a real number with 0 < p < 1.
Consider the set V = {1, 2, . . . , n}. We construct a graph G = (V, E) with
vertex set V , whose edge set E is determined by the following random process:
Each unordered pair {i, j} of vertices, where i 6= j, occurs as an edge in E
with probability p, independently of the other unordered pairs.
A triangle in G is an unordered triple {i, j, k} of distinct vertices, such
that {i, j}, {j, k}, and {k, i} are edges in G.
6.12. Exercises 353
Define the random variable X to be the total number of triangles in the
graph G. Determine the expected value E(X).
Hint: Use indicator random variables.
6.49 In Section 6.9, we have seen the following algorithm InsertionSort,
which sorts any input array A[1 . . . n]:
Algorithm InsertionSort(A[1 . . . n]):
for i = 2 to n
do j = i;
while j > 1 and A[j] < A[j − 1]
do swap A[j] and A[j − 1];
j =j−1
endwhile
endfor
Consider an input array A[1 . . . n], where each element A[i] is chosen inde-
pendently and uniformly at random from the set {1, 2, . . . , m}.
• Let i and j be two indices with 1 ≤ i < j ≤ n, and consider the values
A[i] and A[j] (just before the algorithm starts). Prove that
1 1
Pr(A[i] > A[j]) = − .
2 2m
• Let X be the random variable that is equal to the number of times the
swap-operation is performed when running InsertionSort(A[1 . . . n]).
Determine the expected value E(X) of X.
6.50 Let n ≥ 2 be an integer. Consider the following random process that
divides the integers 1, 2, . . . , n into two sorted lists L1 and L2 :
1. Initialize both L1 and L2 to be empty.
2. For each i = 1, 2, . . . , n, flip a fair coin. If the coin comes up heads,
then add i at the end of list L1 . Otherwise, add i at the end of the list
L2 . (All coin flips during this process are mutually independent.)
354 Chapter 6. Random Variables and Expectation
We now run algorithm Merge(L1 , L2 ) of Section 4.6. Define the random
variable X to be the total number of comparisons made when running this
algorithm: As in Section 4.6, X counts the number of times the line “if
x ≤ y” in algorithm Merge(L1 , L2 ) is executed. In this exercise, you will
determine the expected value E(X) of the random variable X.
• Prove that E(X) = 1/2 for the case when n = 2.
• Prove that E(X) = 5/4 for the case when n = 3.
• Assume that n ≥ 2. For each i and j with 1 ≤ i < j ≤ n, consider the
indicator random variable
1 if i and j are compared,
Xij =
0 otherwise.
Prove that E (Xij ) = (1/2)j−i .
Hint: Assume that i and j are compared. Can i and j be in the same
list? What about the elements i, i + 1, . . . , j − 1 and the element j?
• Determine E(X).
1−xk+1
Hint: 1 + x + x2 + x3 + · · · + xk = 1−x
.
6.51 Assume we have n balls and m boxes. We throw the balls independently
and uniformly at random in the boxes. Thus, for each k and i with 1 ≤ k ≤ n
and 1 ≤ i ≤ m,
Pr( the k-th ball falls in the i-th box ) = 1/m.
Consider the following three random variables:
X = the number of boxes that do not contain any ball,
Y = the number of boxes that contain at least one ball,
Z = the number of boxes that contain exactly one ball.
• Determine the expected values E(X), E(Y ), and E(Z).
• Assuming that m = n, determine the limits
1. limn→∞ E(X)/n,
6.12. Exercises 355
2. limn→∞ E(Y )/n,
3. limn→∞ E(Z)/n.
Hint: limn→∞ (1 − 1/n)n = 1/e.
6.52 Let 0 < p < 1 and consider a coin that comes up heads with probability
p and tails with probability 1 − p. For each integer n, let bn be the outcome
when flipping this coin; thus, bn ∈ {H, T }. The values bn partition the set of
integers into intervals, where each interval is a maximal consecutive sequence
of zero or more T ’s followed by one H:
... H T T T H T H T T H H T H ...
. . . −2 −1 0 1 2 3 4 5 6 7 8 9 10 . . .
• Consider the interval that contains the integer 0, and let X be its
length. (In the example above, X = 4.) Determine the expected value
E(X) of X.
Hint: Use the Linearity of Expectation. The answer is not 1/p, which
is the expected number of coin flips until the first H.
6.53 Your friend Mick takes a permutation of 1, 2, . . . , n, stores it in boxes
B1 , B2 , . . . , Bn (so that each box stores exactly one number), and then closes
all boxes. You have no idea what the permutation is.
Mick opens the boxes B1 , B2 , . . . , Bn , one after another. For each i with
1 ≤ i ≤ n, just before opening box Bi , you have to guess which number is
stored in it.
• Assume that, when you guess the number in box Bi , you do not remem-
ber the numbers stored in B1 , B2 , . . . , Bi−1 . Then, the only reasonable
thing you can do is to take a random element in {1, 2, . . . , n} and guess
that this random element is stored in Bi .
Assume that you do this for each i with 1 ≤ i ≤ n. Let X be the
random variable whose value is equal to the number of times that your
guess is correct. Compute the expected value E(X) of X.
• Now assume that your memory is perfect, so that, when you guess the
number in box Bi , you know the numbers stored in B1 , B2 , . . . , Bi−1 .
356 Chapter 6. Random Variables and Expectation
How would you make the n guesses such that the following is true: If
Y is the random variable whose value is equal to the number of times
that your guess is correct, then the expected value E(Y ) of Y satisfies
E(Y ) = Ω(log n).
6.54 Let n ≥ 1 be an integer.
• Consider a fixed integer i with 1 ≤ i ≤ n. How many permutations
a1 , a2 , . . . , an of the set {1, 2, . . . , n} have the property that ai = i?
• We choose a permutation a1 , a2 , . . . , an of the set {1, 2, . . . , n} uniformly
at random. Consider the random variable
X = |{i : 1 ≤ i ≤ n and ai = i}|.
Determine the expected value E(X).
Hint: Use indicator random variables.
6.55 Let n ≥ 2 be an integer. Consider a uniformly random permutation
a1 , a2 , . . . , an of the set {1, 2, . . . , n}. Define the random variable X to be
the number of ordered pairs (i, j) with 1 ≤ i < j ≤ n for which ai = j and
aj = i. Determine the expected value E(X) of X.
Hint: Use indicator random variables.
6.56 Let n ≥ 2 be an integer and consider n people P1 , P2 , . . . , Pn . Each of
these people has a uniformly random birthday, and all birthdays are mutually
independent. (We ignore leap years.) Consider the random variable
X = the number of indices i such that Pi and Pi+1 have the same birthday.
Determine the expected value E(X).
Hint: Use indicator random variables.
6.57 Let d ≥ 1 be the number of days in one year, let n ≥ 2 be an integer,
and consider a group P1 , P2 , . . . , Pn of n people. Assume that each person has
a uniformly random and independent birthday. Define the random variable
X to be the number of pairs {Pi , Pj } of people that have the same birthday.
Prove that
1 n
E(X) = .
d 2
Hint: Use indicator random variables.
6.12. Exercises 357
6.58 Nick wants to know how many students cheat on the assignments. One
approach is to ask every student “Did you cheat?”. This obviously does not
work, because every student will answer “I did not cheat”. Instead, Nick
uses the following ingenious scheme, which gives a reasonable estimate of the
number of cheaters, without identifying them.
We denote the students by S1 , S2 , . . . , Sn . Let k denote the number of
cheaters. Nick knows the value of n, but he does not know the value of k.
For each i with 1 ≤ i ≤ n, Nick does the following:
1. Nick meets student Si and asks “Did you cheat?”.
2. Student Si flips a fair coin twice, independently of each other; Si does
not show the results of the coin flips to Nick.
(a) If the coin flips are HH or HT , then Si is honest in answering the
question: If Si is a cheater, then he answers “I cheated”; otherwise,
he answers “I did not cheat”.
(b) If the coin flips are T H, then Si answers “I cheated”.
(c) If the coin flips are T T , then Si answers “I did not cheat”.
• Define the random variable X to be the number of students who answer
“I cheated”. Determine the expected value E(X) of X.
Hint: For each i, use an indicator random variable Xi which indicates
whether or not Si answers “I cheated”. If Si is a cheater, what is
E (Xi )? If Si is not a cheater, what is E (Xi )?
• Consider the random variable
Y = 2X − n/2.
Prove that E(Y ) = k. In words, the expected value of Y is equal to
the number of cheaters.
6.59 You roll a fair die repeatedly, and independently, until you have seen
all of the numbers 1, 2, 3, 4, 5, 6 at least once. Consider the random variable
X = the number of times you roll the die.
For example, if you roll the sequence
5, 5, 3, 5, 1, 3, 4, 2, 5, 2, 1, 3, 6,
358 Chapter 6. Random Variables and Expectation
then X = 13.
Determine the expected value E(X) of the random variable X.
Hint: Use the Linearity of Expectation. If you have seen exactly i different
elements from the set {1, 2, 3, 4, 5, 6}, how many times do you expect to roll
the die until you see a new element from this set?
6.60 Michiel’s Craft Beer Company (MCBC) sells n different brands of India
Pale Ale (IPA). When you place an order, MCBC sends you one bottle of
IPA, chosen uniformly at random from the n different brands, independently
of previous orders.
Simon Pratt wants to try all different brands of IPA. He repeatedly places
orders at MCBC (one bottle per order) until he has received at least one
bottle of each brand.
Define the random variable X to be the total number of orders that Simon
places. Determine the expected value E(X) of the random variable X.
Hint: Use the Linearity of Expectation. If Simon has received exactly i
different brands of IPA, how many orders does he expect to place until he
receives a new brand?
6.61 MCBC still sells n different brands of IPA. As in Exercise 6.60, when
you place an order, MCBC sends you one bottle of IPA, chosen uniformly at
random from the n different brands, independently of previous orders.
Simon Pratt places m orders at MCBC. Define the random variable X to
be the total number of distinct brands that Simon receives. Determine the
expected value E(X) of X.
Hint: Use indicator random variables.
6.62 You are given an array A[0 . . . n−1] of n numbers. Let D be the number
of distinct numbers that occur in this array. For each i with 0 ≤ i ≤ n − 1,
let Ni be the number of elements in the array that are equal to A[i].
Pn−1
• Show that D = i=0 1/Ni .
Consider the following algorithm:
6.12. Exercises 359
Algorithm EstimateD(A[1 . . . n]):
Step 1: Choose an integer k in {0, 1, 2, . . . , n − 1} uniformly at
random, and let a = A[k].
Step 2: Traverse the array and compute the number Nk of times
that a occurs.
Step 3: Return the value X = n/Nk .
• Determine the expected value E(X) of the random variable X.
Hint: Use the definition of expected value, i.e., Definition 6.4.1.
6.63 One of Jennifer and Thomas is chosen uniformly at random. The person
who is chosen wins $100. Consider the random variables
J = the amount that Jennifer wins,
T = the amount that Thomas wins.
Prove that
E (max(J, T )) 6= max (E(J), E(T )) .
6.64 Consider the sample space
S = {(123), (132), (213), (231), (312), (321), (111), (222), (333)}.
We choose an element u from S uniformly at random. For each i = 1, 2, 3,
let Xi be the random variable whose value is the i-th number in u. (For
example, if u = (312), then X1 = 3, X2 = 1, and X3 = 2.) Let N be the
random variable whose value is equal to that of X2 .
• Verify that Pr(Xi = k) = 1/3 for any i and k with 1 ≤ i ≤ 3 and
1 ≤ k ≤ 3.
• Verify that X1 , X2 and X3 are pairwise independent.
• Verify that X1 , X2 and X3 are not mutually independent.
P )
• Verify that E(N
i=1 E(Xi ) = 4.
P P
N E(N )
• Verify that E i=1 X i 6= i=1 E(Xi ).
360 Chapter 6. Random Variables and Expectation
6.65 Let k ≥ 0 be an integer and let T be a full binary tree, whose levels
are numbered 0, 1, 2, . . . , k. (The root is at level 0, whereas the leaves are
at level k.) Assume that each edge of T is removed with probability 1/2,
independently of other edges. Denote the resulting graph by T 0 .
Define the random variable X to be the number of nodes that are con-
nected to the root by a path in T 0 ; the root itself is included in X.
In the left figure below, the tree T is shown for the case when k = 3. The
right figure shows the tree T 0 : The dotted edges are those that have been
removed from T , the black nodes are connected to the root by a path in T 0 ,
whereas the white nodes are not connected to the root by a path in T 0 . For
this case, X = 6.
T T0
• Let n be the number of nodes in the tree T . Express n in terms of k.
• Prove that the expected value E(X) of the random variable X is equal
to
E(X) = log(n + 1).
Hint: For any ` with 0 ≤ ` ≤ k, how many nodes of T are at level `?
Use indicator random variables to determine the expected number of
level-` nodes of T that are connected to the root by a path in T 0 .
6.66 Let n ≥ 2 be a power of two and consider a full binary tree with n leaves.
Let a1 , a2 , . . . , an be a random permutation of the numbers 1, 2, . . . , n. Store
this permutation at the leaves of the tree, in the order a1 , a2 , . . . , an , from
left to right. For example, if n = 8 and the permutation is 2, 8, 1, 4, 6, 3, 5, 7,
then we obtain the following tree:
6.12. Exercises 361
2 8 1 4 6 3 5 7
Perform the following process on the tree:
• Visit the levels of the tree from bottom to top.
• At each level, take all pairs of consecutive nodes that have the same
parent. For each such pair, compare the numbers stored at the two
nodes, and store the smaller of these two numbers at the common
parent.
For our example tree, we obtain the following tree:
1
1 3
2 1 3 5
2 8 1 4 6 3 5 7
It is clear that at the end of this process, the root stores the number 1.
Define the random variable X to be the number that is not equal to 1 and
that is stored at a child of the root; think of X being the “loser of the final
game”. For our example tree, X = 3.
In this exercise, you will determine the expected value E(X) of the random
variable X.
• Prove that 2 ≤ X ≤ 1 + n/2.
• Prove that the following is true for each k with 1 ≤ k ≤ n/2: X ≥ k +1
if and only if
362 Chapter 6. Random Variables and Expectation
– all numbers 1, 2, . . . , k are stored in the left subtree of the root
– or all numbers 1, 2, . . . , k are stored in the right subtree of the
root.
• Prove that for each k with 1 ≤ k ≤ n/2,
n/2 n/2
k
k!(n − k)! k
Pr(X ≥ k + 1) = 2 · =2· n .
n! k
• According to Exercise 6.31, we have
∞
X
E(X) = Pr(X ≥ k).
k=1
Prove that
n/2
X
E(X) = Pr(X ≥ 1) + Pr(X ≥ k + 1).
k=1
• Use Exercise 3.67 to prove that
4
E(X) = 3 − .
n+2
6.67 If X is a random variable that can take any value in {1, 2, 3, . . .}, and
A is an event, then the conditional expected value E(X | A) is given by
∞
X
E(X | A) = k · Pr(X = k | A).
k=1
In words, E(X | A) is the expected value of X, when you are given that the
event A occurs.
You roll a fair die repeatedly, and independently, until you see the num-
ber 6. Define the random variable X to be the number of times you roll the
die (this includes the last roll, in which you see the number 6). It follows
from Theorem 6.6.2 that E(X) = 6. Let A be the event
A = “the results of all rolls are even numbers”.
Determine the conditional expected
P∞value E(X | A).
k−1
Hint: E(X | A) 6= 3. Recall that k=1 k · x = 1/(1 − x)2 .
6.12. Exercises 363
6.68 For any integer n ≥ 0 and any real number x with 0 < x < 1, define
the function
∞
X k k
Fn (x) = x .
k=n
n
(Using the ratio test from calculus, it can be shown that this infinite series
converges for any fixed integer n.)
• Determine a closed form expression for F0 (x).
• Let n ≥ 1 be an integer and let x be a real number with 0 < x < 1.
Prove that
x x2 0
Fn (x) = · Fn−1 (x) + · Fn−1 (x),
n n
0
where Fn−1 denotes the derivative of Fn−1 .
Hint: If k ≥ n ≥ 1, then nk = nk n−1
k−1
.
• Prove that for any integer n ≥ 0 and any real number x with 0 < x < 1,
xn
Fn (x) = ,
(1 − x)n+1
and
xn + n · xn−1
Fn0 (x) = .
(1 − x)n+2
• Let n ≥ 0 and m be integers with m ≥ n + 1. Prove that
min(n+1,m−n)
m−`
X
` n+1
(−1) = 0.
`=0
` n
Hint: You have shown above that
∞
n+1
X k k
(1 − x) x = (1 − x)n+1 · Fn (x) = xn . (6.13)
k=n
n
Use Newton’s Binomial Theorem to expand (1 − x)n+1 . Then consider
the expansion of the left-hand side in (6.13). What is the coefficient of
xm in this expansion?
364 Chapter 6. Random Variables and Expectation
6.69 Consider a fair red coin and a fair blue coin. We repeatedly flip both
coins, and keep track of the number of times that the red coin comes up
heads. As soon as the blue coin comes up tails, the process terminates.
A formal description of this process is given in the pseudocode below.
The value of the variable i is equal to the number of iterations performed
so far, the value of the variable h is equal to the number of times that the
red coin came up heads so far, whereas the Boolean variable stop is used to
decide when the while-loop terminates.
Algorithm RandomCoinFlips:
// both the red coin and the blue coin are fair
// all coin flips are mutually independent
i = 0;
h = 0;
stop = false;
while stop = false
do i = i + 1;
flip the red coin;
if the result of the red coin is heads
then h = h + 1
endif;
flip the blue coin;
if the result of the blue coin is tails
then stop = true
endif
endwhile;
return i and h
Consider the random variables
X = the value of i that is returned by algorithm RandomCoinFlips,
Y = the value of h that is returned by algorithm RandomCoinFlips.
Assume that the value of the random variable Y is equal to some integer
n ≥ 0. In this exercise, you will determine the expected value of the random
variable X.
Thus, we are interested in the conditional expected value E(X | Y = n),
which is the expected value of X (i.e., the number of iterations of the while-
6.12. Exercises 365
loop), when you are given that the event “Y = n” (i.e., during the while-loop,
the red coin comes up heads n times) occurs. Formally, we have
X
E(X | Y = n) = k · Pr(X = k | Y = n),
k
where the summation ranges over all values of k that X can take.
The functions Fn and Fn0 that are used below are the same as those in
Exercise 6.68.
• Let n ≥ 1 be an integer. Prove that
∞
X
Pr(Y = n) = Pr(Y = n | X = k) · Pr(X = k).
k=n
• Prove that
∞
X
Pr(Y = 0) = Pr(Y = 0 | X = k) · Pr(X = k).
k=1
• Let n ≥ 1 be an integer. Prove that
Pr(Y = n) = Fn (1/4).
• Prove that
1
Pr(Y = 0) = .
3
• Let n ≥ 1 be an integer. Prove that
Fn0 (1/4)
E(X | Y = n) = .
4 · Fn (1/4)
• Let n ≥ 1 be an integer. Prove that
4n + 1
E(X | Y = n) = .
3
• Prove that
4
E(X | Y = 0) = .
3
366 Chapter 6. Random Variables and Expectation
6.70 Let (S, Pr) be a probability space, and let X and Y be two identical
non-negative random variables on S. Thus, for all ω in S, X(ω) = Y (ω) ≥ 0.
Consider the new probability space (S 2 , Pr), where S 2 is the Cartesian
product S × S and
Pr (ω1 , ω2 ) = Pr (ω1 ) · Pr (ω2 )
for all elements (ω1 , ω2 ) in S 2 . (In words, we choose two elements ω1 and ω2
in S, independently of each other.)
Consider the random variable Z on S 2 defined by
Z (ω1 , ω2 ) = min (X (ω1 ))2 , (Y (ω2 ))2
for all (ω1 , ω2 ) in S 2 . Observe that the expected value of Z is equal to
X X
E(Z) = Z (ω1 , ω2 ) · Pr (ω1 , ω2 ) .
ω1 ∈S ω2 ∈S
• Let a and b be two non-negative real numbers. Prove that
min a2 , b2 ≤ ab.
• Prove that
E(Z) ≤ (E(X))2 .
6.71 Carleton University has implemented a new policy for students who
cheat on assignments:
1. When a student is caught cheating, the student meets with the Dean.
2. The Dean has a box that contains n coins. One of these coins has the
number n written on it, whereas each of the other n − 1 coins has the
number 1 written on it. Here, n is a very large integer.
3. The student chooses a uniformly random coin from the box.
4. If x is the number written on the chosen coin, then the student gives
x2 bottles of cider to Elisa Kazan.
6.12. Exercises 367
Consider the random variables
X = the number written on the chosen coin,
Z = the number of bottles of cider that Elisa gets.
(Note that Z = X 2 .)
• Prove that
E(X) = 2 − 1/n ≤ 2.
• Prove that
E(Z) = n + 1 − 1/n ≥ n.
• Prove that
E X2 =6 O (E(X))2 .
• By the arguments above, Elisa gets, on average, a very large amount
of cider. Since she cannot drink all these bottles, Carleton University
changes their policy:
1. The student chooses a uniformly random coin from the box (and
puts it back in the box).
2. Again, the student chooses a uniformly random coin from the box
(and puts it back in the box).
3. If x is the number written on the first chosen coin, and y is the
number written on the second chosen coin, then the student gives
min(x2 , y 2 ) bottles of cider to Elisa.
Consider the random variables
U = the number written on the first chosen coin,
V = the number written on the second chosen coin,
W = the number of bottles of cider that Elisa gets.
Use Exercise 6.70 to prove that
E(W ) ≤ 4.
368 Chapter 6. Random Variables and Expectation
Chapter 7
The Probabilistic Method
The Probabilistic Method is a very powerful and surprising tool that uses
probability theory to prove results in discrete mathematics. In this chapter,
we will illustrate this method using several examples.
7.1 Large Bipartite Subgraphs
Recall that a graph is a pair G = (V, E), where V is a finite set whose
elements are called vertices and E is a set whose elements are unordered
pairs of distinct vertices. The elements of E are called edges. Assume we
partition the vertex set V of G into two subsets A and B (thus, A ∩ B = ∅
and A ∪ B = V ). We say that an edge of E is between A and B, if one vertex
of this edge is in A and the other vertex is in B.
a b
e
d c
For example, in the graph above, let A = {a, d} and B = {b, c, e}. Then
four of the eight edges are between A and B, namely {a, b}, {a, e}, {d, c},
and {d, e}. Thus, the vertex set of this graph can be partitioned into two
subsets A and B, such that at least half of G’s edges are between A and B.
The following theorem states that this is true for any graph.
370 Chapter 7. The Probabilistic Method
Theorem 7.1.1 Let G = (V, E) be a graph with m edges. The vertex set V
of G can be partitioned into two subsets A and B such that the number of
edges between A and B is at least m/2.
Proof. Consider the following random process: Initialize A = ∅ and B = ∅.
For each vertex u of G, flip a fair and independent coin. If the coin comes
up heads, add u to A; otherwise, add u to B.
Define the random variable X to be the number of edges of G that are
between A and B. We will determine the expected value E(X) of X.
Number the edges of G arbitrarily as e1 , e2 , . . . , em . For each i with 1 ≤
i ≤ m, consider the indicator random variable
1 if e is an edge between A and B,
Xi =
0 otherwise.
Then
m
X
X= Xi
i=1
and
m
!
X
E(X) = E Xi
i=1
m
X
= E (Xi )
i=1
m
X
= Pr (Xi = 1) .
i=1
To determine Pr (Xi = 1), let ei have vertices a and b. The following ta-
ble shows the four possibilities for a and b; each one of them occurs with
probability 1/4.
a ∈ A, b ∈ A Xi =0
a ∈ A, b ∈ B Xi =1
a ∈ B, b ∈ A Xi =1
a ∈ B, b ∈ B Xi =0
7.2. Ramsey Theory 371
Since Xi = 1 in two out of the four cases, we have
Pr (Xi = 1) = 2/4 = 1/2,
and it follows that
m
X
E(X) = 1/2 = m/2.
i=1
Assume the claim in the theorem does not hold. Then, no matter how
we partition the vertex set V into A and B, the number of edges between
A and B will be less than m/2. In particular, the random variable X will
always be less than m/2. But then, E(X) < m/2 as well, contradicting that
E(X) = m/2.
7.2 Ramsey Theory
We return to a problem that we have seen in Section 1.1. Consider a complete
graph with n vertices, in which each vertex represents a person. Any pair of
distinct vertices is connected by an edge. Such an edge is solid if the two
persons representing the vertices of this edge are friends. If these persons are
strangers, the edge is dashed. Consider a subset S of k vertices. We say that
S is a solid k-clique, if any two distinct vertices in S are connected by a solid
edge. Thus, a solid k-clique represents a group of k mutual friends. If any
two distinct vertices of S are connected by a dashed edge, then we say that
S is a dashed k-clique; this represents a group of k mutual strangers.
In Section 1.1, we stated, without proof, Theorem 1.1.3. We repeat the
statement of this theorem and use the Probabilistic Method to prove it.
Theorem 7.2.1 Let k ≥ 3 and n ≥ 3 be integers with n ≤ b2k/2 c. There
exists a complete graph with n vertices, in which each edge is either solid or
dashed, such that this graph does not contain a solid k-clique and does not
contain a dashed k-clique.
Proof. We denote the complete graph with n vertices by Kn . Consider the
following random process: For each edge e of Kn , flip a fair and independent
coin. If the coin comes up heads, make e a solid edge; otherwise, make e a
dashed edge.
372 Chapter 7. The Probabilistic Method
Consider the event
A = “there is a solid k-clique or there is a dashed k-clique”.
We will prove below that Pr(A) < 1. This will imply that Pr A > 0, i.e.,
the event
A = “there is no solid k-clique and there is no dashed k-clique”
has a positive probability. This, in turn, will imply that the statement
in the
theorem holds: If the statement would not hold, then Pr A would be zero.
Thus, it remains to prove that Pr(A) < 1. The vertex set of Kn has
exactly nk many subsets of size k. We denote these subsets by Vi , i =
n n
1, 2, . . . , k . For each i with 1 ≤ i ≤ k , consider the event
Ai = “Vi is a solid k-clique or a dashed k-clique”.
k
Since the event Ai occurs if and only if the edges joining the 2
pairs of
vertices of Vi are either all solid or all dashed, we have
2
Pr (Ai ) = ;
2(2)
k
note that the denominator is equal to 2 to the power k2 .
Since A occurs if and only if A1 ∨ A2 ∨ · · · ∨ A(n) occurs, the Union Bound
k
(i.e., Lemma 5.3.5) implies that
Pr(A) = Pr A1 ∨ A2 ∨ · · · ∨ A(n)
k
()
X
n
k
≤ Pr (Ai )
i=1
(nk)
X 2
=
(k2)
i=1 2
2 nk
= .
2(2)
k
7.3. Sperner’s Theorem 373
If we can show that the quantity in the last line is less than one, then the
proof is complete. We have
2 nk
n(n − 1)(n − 2) · · · (n − k + 1) 2
= · (k2 −k)/2
2(2)
k
k! 2
nk 21+k/2
≤ · .
k! 2k2 /2
Since n ≤ b2k/2 c ≤ 2k/2 , we get
2 nk
(2k/2 )k 21+k/2
≤ · k2 /2
2(2)
k
k! 2
21+k/2
= .
k!
By Exercise 2.8, we have k! > 21+k/2 for k ≥ 3. Thus, we conclude that
2 nk
< 1.
2(2)
k
Take, for example, k = 20 and n = 1024. Theorem 7.2.1 states that there
exists a group of 1024 people that does not contain a subgroup of 20 mutual
friends and does not contain a subgroup of 20 mutual strangers. In fact, the
proof shows more: Consider a group of 1024 people such that any two are
friends with probability 1/2, and strangers with probability 1/2. The above
proof shows that Pr(A), i.e., the probability that there is a subgroup of 20
mutual friends or there is a subgroup of 20 mutual strangers, satisfies
21+k/2 211
Pr(A) ≤ = .
k! 20!
Therefore, with probability at least
211
1− = 0.999999999999999158,
20!
(there are 15 nines) this group does not contain a subgroup of 20 mutual
friends and does not contain a subgroup of 20 mutual strangers.
374 Chapter 7. The Probabilistic Method
7.3 Sperner’s Theorem
In Section 1.2, we considered the following problem. Let S be a set of size n
and consider a sequence S1 , S2 , . . . , Sm of m subsets of S, such that for all i
and j with i 6= j,
Si 6⊆ Sj and Sj 6⊆ Si . (7.1)
What is the largest possible value of m for which such a sequence exists?
The sequence consisting of all subsets of S having size bn/2c satisfies
n
(7.1). This sequence has length m = bn/2c . In Section 1.2, we stated,
without proof, that this is the largest possible value of m; see Theorem 1.2.1.
After stating this theorem again, we will prove it using the Probabilistic
Method.
Theorem 7.3.1 (Sperner) Let n ≥ 1 be an integer and let S be a set with
n elements. Let S1 , S2 , . . . , Sm be a sequence of m subsets of S, such that for
all i and j with i 6= j,
Si 6⊆ Sj and Sj 6⊆ Si .
Then
n
m≤ .
bn/2c
Proof. We assume that none of the subsets in the sequence S1 , S2 , . . . , Sm is
empty, because otherwise, m must be equal to 1, in which case the theorem
clearly holds.
We assume that S = {1, 2, . . . , n}. We choose a uniformly random per-
mutation a1 , a2 , . . . , an of the elements of S; thus, each permutation has
probability 1/n! of being chosen. Consider the following sequence of subsets
A1 , A2 , . . . , An of S: For j = 1, 2, . . . , n,
Aj = {a1 , a2 , . . . , aj }.
For example, if n = 4 and the permutation is 3, 1, 4, 2, then
A1 = {3},
A2 = {1, 3},
A3 = {1, 3, 4},
A4 = {1, 2, 3, 4}.
7.3. Sperner’s Theorem 375
Observe that the subsets A1 , A2 , . . . , An are random subsets of S, because
the permutation was randomly chosen.
Consider a subset Si in the statement of the theorem. We say that Si
occurs in the sequence A1 , A2 , . . . , An if there is an index j such that Si = Aj .
Define the random variable X to be the number of subsets in the sequence
S1 , S2 , . . . , Sm that occur in A1 , A2 , . . . , An . Since the subsets A1 , A2 , . . . , An
are properly nested, i.e.,
A1 ⊂ A2 ⊂ · · · ⊂ An ,
the assumption in the theorem implies that X is either 0 or 1. It follows that
the expected value of X satisfies
E(X) ≤ 1.
We now derive an exact expression for the value of E(X). For each i with
1 ≤ i ≤ m, consider the indicator random variable
1 if Si occurs in the sequence A1 , A2 , . . . , An ,
Xi =
0 otherwise.
Let k denote the size of the subset Si , i.e., k = |Si |. Then
Xi = 1 if and only if Si = Ak .
Since Ak = {a1 , a2 , . . . , ak }, Xi = 1 if and only if the first k values in the
permutation form a permutation of the subset Si :
a1, a2, . . . , ak ak+1, ak+2, . . . , an
permutation of Si
The Product Rule of Section 3.1 shows that there are k!(n − k)! many per-
mutations of S that have this property. Therefore, since we chose a random
permutation of S, we have
E (Xi ) = Pr (Xi = 1)
k!(n − k)!
=
n!
1
= n
k
1
= n
.
|Si |
376 Chapter 7. The Probabilistic Method
Thus, since
m
X
X= Xi ,
i=1
we get
m
!
X
E(X) = E Xi
i=1
m
X
= E (Xi )
i=1
m
X 1
= n
.
i=1 |Si |
If we combine this with our upper bound E(X) ≤ 1, we get
m
X 1
n
≤ 1.
i=1 |Si |
For a fixed value of n, the binomial coefficient nk is maximized when k =
bn/2c; i.e., the largest value in the n-th row of Pascal’s Triangle (see Sec-
tion 3.8) is in the middle. Thus,
n n
≤ ,
|Si | bn/2c
implying that
m
X 1
1 ≥ n
i=1 |Si |
m
X 1
≥ n
i=1 bn/2c
m
= n
.
bn/2c
We conclude that
n
m≤ .
bn/2c
7.4. The Jaccard Distance between Finite Sets 377
7.4 The Jaccard Distance between Finite Sets
Let X and Y be two finite and non-empty sets. We want to define a measure
that indicates how “close together” these two sets are. This measure should
be equal to 0 if the two sets are the same (i.e., X = Y ), it should be equal
to 1 if the two sets are disjoint (i.e., X ∩ Y = ∅), and it should be in the
open interval (0, 1) in all other cases.
The symmetric difference X Y is defined to be the “union minus the
intersection”, i.e.,
X Y = (X ∪ Y ) \ (X ∩ Y ).
From the Venn diagram below, it should be clear that
X Y = (X \ Y ) ∪ (Y \ X),
i.e., the set consisting of all elements in X that are not in Y and all elements
in Y that are not in X.
X Y
X \Y Y \X
If the symmetric difference X Y is “small” compared to the union X ∪Y ,
then the two sets X and Y are “pretty much the same”. On the other hand,
it X Y is “large” compared to X ∪ Y , then the sets X and Y are “very
different”.
Based on this, the Jaccard distance dJ (X, Y ) between the two finite and
non-empty sets X and Y is defined as
|X Y |
dJ (X, Y ) = . (7.2)
|X ∪ Y |
Since
|X Y | = |X ∪ Y | − |X ∩ Y |,
we have
|X ∩ Y |
dJ (X, Y ) = 1 − . (7.3)
|X ∪ Y |
The following claims are easy to verify:
378 Chapter 7. The Probabilistic Method
• 0 ≤ dJ (X, Y ) ≤ 1.
• dJ (X, Y ) = dJ (Y, X).
• dJ (X, X) = 0.
• If X ∩ Y = ∅, then dJ (X, Y ) = 1.
• If X 6= Y and X ∩ Y 6= ∅, then 0 < dJ (X, Y ) < 1.
In the rest of this section, we will prove that the Jaccard distance satisfies
the triangle inequality:
Theorem 7.4.1 Let X, Y , and Z be finite and non-empty sets. Then
dJ (X, Z) ≤ dJ (X, Y ) + dJ (Y, Z).
We will present two proofs of this result. The first proof uses “brute
force”: We consider the Venn diagram for the sets X, Y , and Z. Based on
this diagram, we transform the inequality in Theorem 7.4.1 into an equiv-
alent algebraic inequality. We then argue that the algebraic inequality is
valid. In the second proof, we show that the inequality in Theorem 7.4.1
can be rephrased as an inequality involving probabilities. The result then
follows by straightforward applications of Lemma 5.3.6 and the Union Bound
(Lemma 5.3.5).
7.4.1 The First Proof
In the figure below, you see the Venn diagram for the three sets X, Y , and Z.
The variables a, b, . . . , g denote the number of elements in the different parts
of this diagram. For example, d denotes the number of elements that are
in X and in Y , but not in Z, whereas g denotes the number of elements
that are in all three sets. Note that some of these variables may be equal
to 0. However, since none of the three sets X, Y , and Z is empty, we have
a + d + f + g > 0, b + d + e + g > 0, and c + e + f + g > 0.
7.4. The Jaccard Distance between Finite Sets 379
X Y
a d b
g
f e
c
Z
Using the definition of Jaccard distance in (7.2), the inequality in Theo-
rem 7.4.1 is equivalent to
a+c+d+e a+b+e+f b+c+d+f
≤ + ,
a+c+d+e+f +g a+b+d+e+f +g b+c+d+e+f +g
which we rewrite as
a+b+e+f b+c+d+f a+c+d+e
+ − ≥ 0.
a+b+d+e+f +g b+c+d+e+f +g a+c+d+e+f +g
After combining the three fractions into one fraction, and expanding the
three products in the numerator of the resulting fraction, we get1
N
≥ 0,
D
where
N = a2 b + ab2 + a2 c + 2abc + b2 c + ac2 + bc2 + a2 d + 2abd + b2 d +
2acd + 2bcd + ad2 + bd2 + 2abe + b2 e + 2ace + 2bce + c2 e +
ade + 2bde + cde + be2 + ce2 + a2 f + 4abf + 2b2 f + 4acf +
4bcf + c2 f + 4adf + 5bdf + 3cdf + 2d2 f + 3aef + 5bef + 4cef +
4def + 2e2 f + 3af 2 + 4bf 2 + 3cf 2 + 4df 2 + 4ef 2 + 2f 3 + 2abg +
2b2 g + 2acg + 2bcg + adg + 3bdg + 3beg + ceg + 3af g + 6bf g +
3cf g + 4df g + 4ef g + 4f 2 g + 2bg 2 + 2f g 2
and
D = (a + b + d + e + f + g)(b + c + d + e + f + g)(a + c + d + e + f + g).
Observe that D > 0. Moreover, all terms in the equation for N are non-
negative and they are connected by plus signs. It follows that N ≥ 0 and,
therefore, N/D ≥ 0. Thus, we have proved Theorem 7.4.1.
1
with some help from Wolfram Alpha
380 Chapter 7. The Probabilistic Method
7.4.2 The Second Proof
Consider the set X ∪ Y ∪ Z. Note that this is a set, so that there are no
duplicates. Let n = |X∪Y ∪Z| and consider a uniformly random permutation
x1 , x2 , x3 , . . . , xn
of the elements of X ∪ Y ∪ Z. Consider the random variables
i = min{` : x` ∈ X},
j = min{` : x` ∈ Y },
k = min{` : x` ∈ Z}.
In words, i is determined by walking along the sequence x1 , x2 , x3 , . . . , xn ,
from left to right. The value of i is the index of the first element that
belongs to the set X.
Consider the event
AXY = “i 6= j”.
We are going to determine the probability Pr (AXY ) that this event occurs.
Observe that
Pr (AXY ) = 1 − Pr AXY ,
where AXY is the event
AXY = “i = j”.
To determine Pr AXY , we do the following. Remove from the sequence
x1 , x2 , . . . , xn all elements that do not belong to X and do not belong to Y .
Then we are left with a uniformly random permutation of the set X ∪Y . The
event AXY occurs if and only if the first element of this new sequence belongs
to both X and Y . Since each element of X ∪ Y has the same probability of
being the first element in this new sequence, it follows that
|X ∩ Y |
Pr AXY =
|X ∪ Y |
and, thus, using (7.3),
|X ∩ Y |
Pr (AXY ) = 1 − = dJ (X, Y ).
|X ∪ Y |
7.5. Planar Graphs and the Crossing Lemma 381
If we consider the events
AXZ = “i 6= k”
and
AY Z = “j 6= k”,
then we have, by the same arguments,
Pr (AXZ ) = dJ (X, Z)
and
Pr (AY Z ) = dJ (Y, Z).
Thus, the inequality in Theorem 7.4.1 is equivalent to
Pr (AXZ ) ≤ Pr (AXY ) + Pr (AY Z ) .
Since
i 6= k ⇒ i 6= j ∨ j 6= k,
Lemma 5.3.6 implies that
Pr (AXZ ) ≤ Pr (AXY ∨ AY Z ) .
By applying the Union Bound (Lemma 5.3.5), we conclude that
Pr (AXZ ) ≤ Pr (AXY ) + Pr (AY Z ) .
Thus, we have completed our second proof of Theorem 7.4.1.
7.5 Planar Graphs and the Crossing Lemma
Consider a graph G = (V, E). Any one-to-one function f : V → R2 gives an
embedding of G:
1. Each vertex a of V is drawn as the point f (a) in the plane.
2. Each edge {a, b} of E is drawn as the straight-line segment f (a)f (b)
between the points f (a) and f (b).
Besides the function f being one-to-one, we assume that it satisfies the fol-
lowing three properties:
382 Chapter 7. The Probabilistic Method
1. For any two edges {a, b} and {a0 , b0 } of E, the intersection of the line
segments f (a)f (b) and f (a0 )f (b0 ) is empty or consists of exactly one
point.
2. For any edge {a, b} in E and any vertex c in V , the point f (c) is not
in the interior of the line segment f (a)f (b).
3. For any three edges {a, b}, {a0 , b0 }, and {a00 , b00 } of E, the line segments
f (a)f (b), f (a0 )f (b0 ), and f (a00 )f (b00 ) do not have a point in common
that is in the interior of any of these line segments.
For simplicity, we do not distinguish any more between a graph and its
embedding. That is, a vertex a refers to both an element of V and the point
in the plane that represents a. Similarly, an edge refers to both an element
of E and the line segment that represents it.
7.5.1 Planar Graphs
Definition 7.5.1 An embedding of a graph G = (V, E) is called plane, if no
two edges of E intersect, except possibly at their endpoints. A graph G is
called planar if there is a plane embedding of G.
Consider a plane embedding of a planar graph G. Again for simplicity, we
denote this embedding by G. This embedding consists of vertices, edges, and
faces (one of them being the unbounded face). For example, in the following
plane embedding, there are 11 vertices, 18 edges, and 9 faces.
In the rest of this section, we will use the following notation:
• G denotes a plane embedding of a planar graph.
• v denotes the number of vertices of G.
7.5. Planar Graphs and the Crossing Lemma 383
• e denotes the number of edges of G.
• f denotes the number of faces in the embedding of G.
How many edges can G have? Since G has v vertices, we obviously have
e ≤ v2 = Θ(v 2 ), an upper bound which holds for any graph with v vertices.
Since our graph G is planar, we expect a much smaller upper bound on e: If
G has Θ(v 2 ) edges, then it seems to be impossible to draw G without edge
crossings. Below, we will prove that e is, in fact, at most linear in v. The
proof will use Euler’s Theorem for planar graphs:
Theorem 7.5.2 (Euler) Consider any plane embedding of a planar graph
G. Let v, e, and f be the number of vertices, edges, and faces of this embed-
ding, respectively. Moreover, let c be the number of connected components of
G. Then
v − e + f = c + 1. (7.4)
Proof. The idea of the proof is as follows. We start by removing all edges
from G (but keep all vertices), and show that (7.4) holds. Then we add back
the edges of G, one by one, and show that (7.4) remains valid throughout
this process.
After having removed all edges, we have e = 0 and the embedding consists
of a collection of v points. Since f = 1 and c = v, the relation v −e+f = c+1
holds.
Assume the relation v − e + f = c + 1 holds and consider what happens
when we add an edge ab. There are two possible cases.
Case 1: Before adding the edge ab, the vertices a and b belong to the same
connected component.
b
a
When adding the edge ab,
• the number v of vertices does not change,
384 Chapter 7. The Probabilistic Method
• the number e of edges increases by one,
• the number f of faces increases by one (because the edge ab splits one
face into two),
• the number c of connected components does not change.
It follows that the relation v − e + f = c + 1 still holds after ab has been
added.
Case 2: Before adding the edge ab, the vertices a and b belong to different
connected components.
b
a
When adding the edge ab,
• the number v of vertices does not change,
• the number e of edges increases by one,
• the number f of faces does not change,
• the number c of connected components decreases by one.
It again follows that the relation v − e + f = c + 1 still holds after ab has
been added.
Usually, Euler’s Theorem is stated for connected planar graphs, i.e., pla-
nar graphs for which c = 1:
Theorem 7.5.3 (Euler) Consider any plane embedding of a connected pla-
nar graph G. If v, e, and f denote the number of vertices, edges, and faces
of this embedding, respectively, then
v − e + f = 2.
7.5. Planar Graphs and the Crossing Lemma 385
We now show how to use Euler’s Theorem to prove an upper bound on
the number of edges and faces of any connected planar graph:
Theorem 7.5.4 Let G be any plane embedding of a connected planar graph
with v ≥ 3 vertices. Then
1. G has at most 3v − 6 edges and
2. this embedding has at most 2v − 4 faces.
Proof. As before, let e and f denote the number of edges and faces of G,
respectively. If v = 3, then e ≤ 3 and f ≤ 2. Hence, in this case, we have
e ≤ 3v − 6 and f ≤ 2v − 4.
Assume that v ≥ 4. We number the faces of G arbitrarily from 1 to f .
For each i with 1 ≤ i ≤ f , let mi denote the number of edges on the i-th
face of G. Since
P each edge lies on the boundary of at most two faces, the
summation fi=1 mi counts each edge at most twice. Thus,
f
X
mi ≤ 2e.
i=1
On the other hand, since G is connected and v ≥ 4, each face has at least
three edges on its boundary, i.e., mi ≥ 3. It follows that
f
X
mi ≥ 3f.
i=1
Combining these two inequalities implies that 3f ≤ 2e, which we rewrite as
f ≤ 2e/3.
Using Euler’s formula (with c = 1, because G is connected), we obtain
e = v + f − 2 ≤ v + 2e/3 − 2,
which is equivalent to
e ≤ 3v − 6.
We also obtain
f ≤ 2e/3 ≤ 2(3v − 6)/3 = 2v − 4.
This completes the proof.
386 Chapter 7. The Probabilistic Method
7.5.2 The Crossing Number of a Graph
Consider an embedding of a graph G = (V, E). We say that two distinct
edges of E cross, if their interiors have a point in common. In this case, we
call this common point a crossing.
The example below shows an embedding of the complete graph K6 on six
vertices, which are denoted by black dots. In this embedding, there are three
crossings, which are numbered 1, 2, and 3.
2
3
1
Definition 7.5.5 The crossing number cr (G) of a graph G is defined to be
the minimum number of crossings in any embedding of G.
Thus, a graph G is planar if and only if cr (G) = 0. The example above
shows that cr (K6 ) ≤ 3.
In the rest of this section, we consider the following problem: Given a
graph G with v vertices and e edges, can we prove good bounds, in terms of
v and e, on the crossing number cr (G) of G?
A simple lower bound on the crossing number
Let G be any graph with v ≥ 3 vertices and e edges. Consider an embedding
of G having cr (G) crossings; hence, this embedding is the “best” one.
We “make” G planar, by defining all crossings to be vertices. That is, let
H be the graph whose vertex set is the union of the vertex set of G and the
set of all crossings in the embedding. Edges of G are cut by the crossings
into smaller edges, which are edges in the graph H.
The figure below shows the planar version of the embedding of K6 that
we saw before. This new graph has 9 vertices and 21 edges.
7.5. Planar Graphs and the Crossing Lemma 387
We make the following observations:
• The graph H is planar, because it is embedded without any crossings.
• The graph H has v + cr (G) vertices.
• How many edges does H have? Any crossing in G is the intersection
of exactly two edges of G; these two edges contribute four edges to H.
Hence, for any crossing in G, the number of edges in H increases by
two. It follows that H has e + 2 · cr (G) edges.
Since H is planar, we know from Theorem 7.5.4 that the number of its edges
is bounded from above by three times the number of its vertices minus six,
i.e.,
e + 2 · cr (G) ≤ 3(v + cr (G)) − 6.
By rewriting this inequality, we obtain the following result:
Theorem 7.5.6 For any graph G with v ≥ 3 vertices and e edges, we have
cr (G) ≥ e − 3v + 6.
the complete graph Kn on n vertices, where n ≥ 3.
For example, consider
n
Since this graph has 2 edges, we obtain
n 1 7
cr (Kn ) ≥ − 3n + 6 = n2 − n + 6. (7.5)
2 2 2
For n = 6, we get cr (K6 ) ≥ 3. Since we have seen before that cr (K6 ) ≤ 3, it
follows that cr (K6 ) = 3.
388 Chapter 7. The Probabilistic Method
Since Kn has n2 edges and any two of them cross at most once, we have
the following obvious upper bound on the crossing number of Kn :
n
cr (Kn ) ≤ 2 = O(n4 ). (7.6)
2
(Of course (7.6) holds for any graph with n vertices.)
To conclude this subsection, (7.5) gives an n2 -lower bound, whereas (7.6)
gives an n4 -upper bound on the crossing number of Kn . In the next section,
we will determine the true asymptotic behavior of cr (Kn ).
A better lower bound on the crossing number
As before, let G be any graph with v ≥ 3 vertices and e edges. Again
we consider an embedding of G having cr (G) crossings. In the rest of this
subsection, we will use the Probabilistic Method to prove a lower bound on
the crossing number of G.
We choose a real number p such that 0 < p ≤ 1. Consider a coin that
comes up heads with probability p and comes up tails with probability 1 − p.
Let Gp be the random subgraph of G, that is obtained as follows.
• For each vertex a of G, flip the coin (independently of the other coin
flips) and add a as a vertex to Gp if and only if the coin comes up
heads.
• Each edge ab of G appears as an edge in Gp if and only if both a and
b are vertices of Gp .
Recall that we fixed the embedding of G. As a result, this random process
gives us an embedding of Gp (which may not be the best one in terms of the
number of crossings.)
We denote the number of vertices, edges, and crossings in the embedding
of Gp by vp , ep , and xp , respectively. Observe that these three quantities are
random variables.
It follows from Theorem 7.5.6 that
cr (Gp ) − ep + 3vp ≥ 6,
provided that vp ≥ 3. This implies that
cr (Gp ) − ep + 3vp ≥ 0,
7.5. Planar Graphs and the Crossing Lemma 389
for any value of vp that results from our random choices.
Since cr (Gp ) ≤ xp , it follows that
xp − ep + 3vp ≥ 0.
The left-hand side is a random variable, which is always non-negative, no
matter what graph Gp results from our random choices. Therefore, its ex-
pected value is also non-negative, i.e.,
E (xp − ep + 3vp ) ≥ 0.
Using the Linearity of Expectation (i.e., Theorem 6.5.2), we get
E(xp ) − E(ep ) + 3 · E(vp ) ≥ 0. (7.7)
We are now going to compute these three expected values separately.
The random variable vp is equal to the number of successes in v inde-
pendent trials, each one having success probability p. In other words, vp has
a binomial distribution with parameters v and p, and, therefore, by Theo-
rem 6.7.2,
E(vp ) = pv.
To compute E(ep ), we number the edges of G arbitrarily from 1 to e. For
each i with 1 ≤ i ≤ e, define Xi to be the indicator random variable with
value
1 if the i-th edge is an edge in Gp ,
Xi =
0 otherwise.
Since an edge of G is in Gp if and only if both its vertices are in Gp , it follows
that
E(Xi ) = Pr(Xi = 1) = p2 .
Then, since ep = ei=1 Xi , we get
P
e
! e e
X X X
E(ep ) = E Xi = E(Xi ) = p2 = p2 e.
i=1 i=1 i=1
Finally, we compute the expected value of the random variable xp . Number
the crossings in the embedding of G arbitrarily from 1 to cr (G). For each i
with 1 ≤ i ≤ cr (G), define Yi to be the indicator random variable with value
1 if the i-th crossing is a crossing in Gp ,
Yi =
0 otherwise.
390 Chapter 7. The Probabilistic Method
Let ab and cd be the edges of G that cross in the i-th crossing2 . This crossing
appears as a crossing in Gp if and only if both ab and cd are edges in Gp .
Since the points a, b, c, and d are pairwise distinct, it follows that the i-th
crossing of G appears as a crossing in Gp with probability p4 . Thus,
E(Yi ) = Pr(Yi = 1) = p4 .
Pcr (G)
Since xp = i=1 Yi , it follows that
cr (G) cr (G) cr (G)
X X X
E(xp ) = E Yi =
E(Yi ) = p4 = p4 · cr (G).
i=1 i=1 i=1
Substituting the three expected values into (7.7), we get
p4 · cr (G) − p2 e + 3 · pv ≥ 0,
which we rewrite as
p2 e − 3pv
cr (G) ≥ . (7.8)
p4
Observe that this inequality holds for any real number p with 0 < p ≤ 1.
If we assume that e ≥ 4v, and take p = 4v/e (so that 0 < p ≤ 1), then
we obtain a new lower bound on the crossing number:
Theorem 7.5.7 (Crossing Lemma) Let G be any graph with v vertices
and e edges. If e ≥ 4v, then
1 e3
cr (G) ≥ .
64 v 2
Applying this lower bound to the complete graph Kn gives cr (Kn ) =
Ω(n4 ). This lower bound is much better than the quadratic lower bound in
(7.5) and it matches the upper bound in (7.6). Hence, we have shown that
cr (Kn ) = Θ(n4 ).
Remark 7.5.8 Let n be a very large integer and consider the complete
n
graph Kn with v = n vertices and e = 2 edges. Let us see what happens
if we repeat the proof for this graph. We choose a random subgraph Gp of
2
By our definition of embedding, see Section 7.5.1, there are exactly two edges that
determine the i-th crossing.
7.6. Exercises 391
Kn , where p = 4v/e = 8/(n − 1). The expected number of vertices in Gp
is equal to pn, which is approximately equal to 8. Thus, the random graph
Gp is, expected, extremely small. Then we apply the weak lower bound of
Theorem 7.5.6 to this, again expected, extremely small graph. The result is a
proof that in any embedding of the huge graph Kn , there are Ω(n4 ) crossings!
7.6 Exercises
7.1 Prove that, for any graph G with m edges, the sequence X1 , X2 , . . . , Xm
of random variables in the proof of Theorem 7.1.1 is pairwise independent.
Give an example of a graph for which this sequence is not mutually inde-
pendent.
7.2 Prove that Theorem 7.5.4 also holds if G is not connected.
7.3 Let K5 be the complete graph on 5 vertices. In this graph, each pair of
vertices is connected by an edge. Prove that K5 is not planar.
7.4 Let G be any embedding of a connected planar graph with v ≥ 4 vertices.
Assume that this embedding has no triangles, i.e., there are no three vertices
a, b, and c, such that ab, bc, and ac are edges of G.
• Prove that G has at most 2v − 4 edges.
• Let K3,3 be the complete bipartite graph on 6 vertices. The vertex set
of this graph consists of two sets A and B, both of size three, and each
vertex of A is connected by an edge to each vertex of B. Prove that
K3,3 is not planar.
7.5 Consider the numbers Rn that were defined in Section 4.8. In Sec-
tion 4.8.1, we proved that Rn = O(n8 ). Prove that Rn = O(n4 ).
7.6 Let n be a sufficiently large positive integer and consider the complete
graph Kn . This graph has vertex set V = {1, 2, . . . , n}, and each pair of
distinct vertices is connected by an undirected edge. (Thus, Kn has n2
edges.)
~ n be the directed graph obtained by making each edge {i, j} of Kn
Let K
~ n , this edge either occurs as the directed edge (i, j)
a directed edge; thus, in K
from i to j or as the directed edge (j, i) from j to i.
We say that three pairwise distinct vertices i, j, and k define a directed
~ n , if
triangle in K
392 Chapter 7. The Probabilistic Method
~ n or
• (i, j), (j, k), and (k, i) are edges in K
~ n.
• (i, k), (k, j), and (j, i) are edges in K
Prove that there exists a way to direct the edges of Kn , such that the
~ 1 n
number of directed triangles in Kn is at least 4 3 .
7.7 Let G = (V, E) be a graph with vertex set V and edge set E. A subset I
of V is called an independent set if for any two distinct vertices u and v in I,
(u, v) is not an edge in E. For example, in the following graph, I = {a, e, i}
is an independent set.
f
c i
a g
d
b e h
Let n and m denote the number of vertices and edges in G, respectively,
and assume that m ≥ n/2. This exercise will lead you through a proof of the
fact that G contains an independent set of size at least n2 /(4m).
Consider the following algorithm, in which all random choices made are
mutually independent:
Algorithm IndepSet(G):
Step 1: Set H = G.
Step 2: Let d = 2m/n. For each vertex v of H, with
probability 1 − 1/d, delete the vertex v, together with its
incident edges, from H.
Step 3: As long as the graph H contains edges, do the
following: Pick an arbitrary edge (u, v) in H, and remove
the vertex u, together with its incident edges, from H.
Step 4: Let I be the vertex set of the graph H. Return I.
7.6. Exercises 393
• Argue that the set I that is returned by this algorithm is an independent
set in G.
• Let X and Y be the random variables whose values are the number of
vertices and edges in the graph H after Step 2, respectively. Prove that
E(X) = n2 /(2m)
and
E(Y ) = n2 /(4m).
• Let Z be the random variable whose value is the size of the independent
set I that is returned by the algorithm. Argue that
Z ≥ X − Y.
• Prove that
E(Z) ≥ n2 /(4m).
• Argue that this implies that the graph G contains an independent set
of size at least n2 /(4m).
7.8 Elisa Kazan is having a party at her home. Elisa has a round table
that has 52 seats numbered 0, 1, 2, . . . , 51 in clockwise order. Elisa invites 51
friends, so that the total number of people at the party is 52. Of these 52
people, 15 drink cider, whereas the other 37 drink beer.
In this exercise, you will prove the following claim: No matter how the 52
people sit at the table, there is always a consecutive group of 7 people such
that at least 3 of them drink cider.
From now on, we consider an arbitrary (which is not random) arrange-
ment of the 52 people sitting at the table.
• Let k be a uniformly random element of the set {0, 1, 2, . . . , 51}. Con-
sider the consecutive group of 7 people that sit in seats k, k + 1, k +
2, . . . , k + 6; these seat numbers are to be read modulo 52. Define the
random variable X to be the number of people in this group that drink
cider. Prove that E(X) > 2.
Hint: Number the 15 cider drinkers arbitrarily as P1 , P2 , . . . , P15 . For
each i with 1 ≤ i ≤ 15, consider the indicator random variable
1 if Pi sits in one of the seats k, k + 1, k + 2, . . . , k + 6,
Xi =
0 otherwise.
394 Chapter 7. The Probabilistic Method
• For the given arrangement of the 52 people sitting at the table, prove
that there is a consecutive group of 7 people such that at least 3 of
them drink cider.
Hint: Assume the claim is false. What is an upper bound on E(X)?