So far we have seen the following sorting algorithms: InsertionSort, MergeSort and HeapSort. We start these Lecture notes with another sorting algorithm: Quicksort. Quicksort, like Mergesort, takes a divide and conquer approach, but on a different basis.
If we have done two comparisons among three keys and find that x < p and p < y, do we ever need to compare x to y? Where do the three belong relative to each other in the sorted array?
Quicksort uses this idea to partition the set of keys to be sorted into those less than the pivot p and those greater than the pivot. (It can be generalized to allow keys equal to the pivot.) It then recurses on the two partitions.
Compare this to Mergesort.
Quicksort performs well in practice, and is one of the most widely used sorts today.
To sort any subarray A[p .. r], p < r:
An array is sorted with a call to QUICKSORT(A, 1, A.length):
The work is done in the PARTITION procedure. A[r] will be the pivot. (Note that the end element of the array is taken as the pivot. Given random data, the choice of the position of the pivot is arbitrary; working with an end element simplifies the code):
PARTITION maintains four regions.
Three of these are described by the following loop invariants, and the fourth (A[j .. r-1]) consists of elements that not yet been examined:
Loop Invariant:
- All entries in A[p .. i] are ≤ pivot.
- All entries in A[i+1 .. j-1] are > pivot.
- A[r] = pivot.
It is worth taking some time to trace through and explain each step of this example of the PARTITION procedure, paying particular attention to the movement of the dark lines representing partition boundaries.
Continuing ...
Here is the Hungarian Dance version of quicksort, in case that helps to make sense of it!
Here use the loop invariant to show correctness:
The formal analysis will be done on a randomized version of Quicksort (see below). This informal analysis helps to motivate that randomization.
First, PARTITION is Θ(n): We can easily see that its only component that grows with n is the for loop that iterates proportional to the number of elements in the subarray).
The runtime depends on the partitioning of the subarrays:
The worst case occurs when the subarrays are completely unbalanced, i.e., there are 0 elements in one subarray and n-1 elements in the other subarray (the single pivot is not processed in recursive calls). This gives a familiar recurrence (compare to that for insertion sort):
One example of data that leads to this behavior is when the data is already sorted: the pivot is always the maximum element, so we get partitions of size n−1 and 0 each time. Thus, quicksort is O(n2) on sorted data. Insertion sort actually does better on a sorted array! (O(n))
The best case occurs when the subarrays are completely balanced (the pivot is the median value): subarrays have about n/2 elements. The reucurrence is also familiar (compare to that for merge sort):
It turns out that expected behavior is closer to the best case than the worst case. Two examples suggest why expected case won't be that bad.
Suppose each call splits the data into 1/10 and 9/10. This is highly unbalanced: won't it result in horrible performance?
We have log10n full levels and log10/9n levels that are nonempty.
As long as it's constant, the base of the log does not affect asymptotic results. Any split of constant proportionality will yield a recursion tree of depth Θ(lg n). In particular (using ≈ to indicate truncation of low order digits),
log10/9n = (log2n) / (log210/9) by formula 3.15
≈ (log2n) / 0.152
= 1/0.152 (log2n)
≈ 6.5788 (log2n)
= Θ(lg n), where c = 6.5788.
So the recurrence and its solution is:
A general lesson that might be taken from this: sometimes, even very unbalanced divide and conquer can be useful.
With random data there will usually be a mix of good and bad splits throughout the recursion tree.
A mixture of worst case and best case splits is asymptotically the same as best case:
Both these trees have the same two leaves. The extra level on the left hand side only increases the height by a factor of 2, and this constant disappears in the Θ analysis.
Both result in O(n lg n), though with a larger constant for the left.
Above, we had to assume a distribution of inputs, but we may not have control over inputs.
An "adversary" can always mess up our assumptions by giving us worst case inputs. (This can be a fictional adversary in making analytic arguments, or it can be a real one ...)
Randomized algorithms foil the adversary by imposing a distribution of inputs.
The modification to HIRE-ASSISTANT problem in Topic 05A is trivial: add a line at the beginning that randomizes the list of candidates.
Having done so, the above analysis applies to give us expected value rather than average case.
What is the relationship/difference between probabilistic analysis and randomized algorithms?
There are different ways to randomize algorithms. One way is to randomize the ordering of the input before we apply the original algorithm (as was suggested for HIRE-ASSISTANT above). A procedure for randomizing an array:
Randomize-In-Place(A) 1 n = A.length 2 for i = 1 to n 3 swap A[i] with A[Random(i,n)]
The text offers a proof that this produces a uniform random permutation. It obviously runs in O(n) time.
Another approach to randomization is to randomize choices made within the algorithm.
We expect good average case behavior if all input permutations are equally likely, but what if it is not?
To get better performance on sorted or nearly sorted data -- and to foil our adversary! -- we can randomize the algorithm to get the same effect as if the input data were random.
Instead of explicitly permuting the input data (which is expensive), randomization can be accomplished trivially by random sampling of one of the array elements as the pivot.
If we swap the selected item with the last element, the existing PARTITION procedure applies:
Now, even an already sorted array will give us average behavior.
Curses! Foiled again!
The analysis assumes that all elements are unique, but with some work can be generalized to remove this assumption (Problem 7-2 in the text).
The previous analysis was pretty convincing, but was based on an assumption about the worst case. This analysis proves that our selection of the worst case was correct, and also shows something interesting: we can solve a recurrence relation with a "max" term in it!
PARTITION produces two subproblems, totaling size n-1. Suppose the partition takes place at index q. The recurrence for the worst case always selects the maximum cost among all possible ways of splitting the array (i.e., it always picks the worst possible q):
Based on the informal analysis, we guess T(n) ≤ cn2 for some c. Substitute this guess into the recurrence:
The maximum value of q2 + (n - q - 1)2 occurs when q is either 0 or n-1 (the second derivative is positive), and has value (n - 1)2 in either case:
Substituting this back into the reucrrence:
We can pick c so that c(2n - 1) dominates Θ(n). Therefore, the worst case running time is O(n2).
One can also show that the recurrence is Ω(n2), so worst case is Θ(n2).
With a randomized algorithm, expected case analysis is much more informative than worst-case analysis. Why?
This analysis nicely demonstrates the use of indicator variables and two useful strategies.
The dominant cost of the algorithm is partitioning. PARTITION removes the pivot element from future consideration, so is called at most n times.
QUICKSORT recurses on the partitions. The amount of work in each call is a constant plus the work done in the for loop. We can count the number of executions of the for loop by counting the number of comparisons performed in the loop.
Rather than counting the number of comparisons in each call to QUICKSORT, it is easier to derive a bound on the number of comparisons across the entire execution.
This is an example of a strategy that is often useful: if it is hard to count one way (e.g., "locally"), then count another way (e.g., "globally").
Let X be the total number of comparisons in all calls to PARTITION. The total work done over the entire execution is O(n + X), since QUICKSORT does constant work setting up n calls to PARTITION, and the work in PARTITION is proportional to X. But what is X?
For ease of analysis,
We want to count the number of comparisons. Each pair of elements is compared at most once, because elements are compared only to the pivot element and then the pivot element is never in any later call to PARTITION.
Indicator variables can be used to count the comparisons. (Recall that we are counting across all calls, not just during one partition.)
Let Xij = I{ zi is compared to zj }
Since each pair is compared at most once, the total number of comparisons is:
Taking the expectation of both sides, using linearity of expectation, and applying Lemma 5.1 (which relates expected values to probabilities):
What's the probability of comparing zi to zj?
Here we apply another useful strategy: if it's hard to determine when something happens, think about when it does not happen.
Elements (keys) in separate partitions will not be compared. If we have done two comparisons among three elements and find that zi < x <zj, we do not need to compare zi to zj (no further information is gained), and QUICKSORT makes sure we do not by putting zi and zj in different partitions.
On the other hand, if either zi or zj is chosen as the pivot before any other element in Zij, then that element (as the pivot) will be compared to all of the elements of Zij except itself.
Therefore (using the fact that these are mutually exclusive events):
We can now substitute this probability into the analyis of E[X] above and continue it:
This is solved by applying equation A.7 for harmonic series, which we can match by substituting k = j - i and shifting the summation indices down i:
We can get rid of that pesky "+ 1" in the denominator by dropping it and switching to inequality (after all, this is an upper bound analysis), and now A7 (shown in box) applies:
Above we used the fact that logs of different bases (e.g., ln n and lg n) grow the same asymptotically.
To recap, we started by noting that the total cost is O(n + X) where X is the number of comparisons, and we have just shown that X = O(n lg n).
Therefore, the average running time of QUICKSORT on uniformly distributed permutations (random data) and the expected running time of randomized QUICKSORT are both O(n + n lg n) = O(n lg n).
This is the same growth rate as merge sort and heap sort. Empirical studies show quicksort to be a very efficient sort in practice (better than the other n lg n sorts) whenever data is not already ordered. (When it is nearly ordered, such as only one item being out of order, insertion sort is a good choice.)
Next week we will see how to select the pivots in Quicksort deterministically, so that we can guarantee that Quicksort always takes O(n lg n) time, even in the worst case.