Saturday, 10 August 2013

Priority Queues, Insertion Sort, Selection Sort


  • A Priority Queue ranks its elements by key with a total order relation.

  • Keys:

    • Every element has its own key.
    • Keys are not necessarily unique.
    • A key is often a numerical value but the concept is general, i.e., not limited to numerics.

  • Total Order Relation

    • Denoted by £ .
    • Reflexive:k£k.
    • Antisymetric: if k1£k2 and k2£k1 then k1 = k2.
    • Transitive: if k1£k2 and k2£k3 then k1£k3.

  • A Priority Queue supports these fundamental methods:

    • insertItem(Object k, Object e)
    • Object removeMinElement()
 Lafore Priority Queue Applet

Sorting with a Priority Queue

  • A Priority Queue P can be used for sorting by inserting a Sequence S of n elements and calling removeMinElement until P is empty.

Algorithm PriorityQueueSort(S,P):
Input: A sequence S storing n keys, on which a total order relation is defined, and a Priority Queue P that compares keys with the same relation. Output: The Sequence S sorted by the total order relation.     while !S.isEmpty() do         e ¬S.removeFirst()
        P.insertItem(e,e)
    while !P.isEmpty() do
        e ¬ P.removeMinElement()
        S.insertLast(e)


The Priority Queue ADT

  • A Priority Queue ADT must support the following methods:

    • size():
        • Return the number of elements in P.Input: None;      Output: integer

    • isEmpty():
        • Test whether P is empty.Input: None; Output: boolean
    Two other methods of Interface Container.
     
    • insertItem(k,e):
        • Insert a new element e with key k into P.Input: Objects k,e; Output: none

    • minElement():
        • Return but don’t remove an element of P with smallest key; an error occurs if P is empty.Input: none: Output: Object e





The Priority Queue ADT (cont.)

    • minKey():
        • Return the smallest key in P; an error occurs if P is empty.Input: none; Output: Object k

    • removeMinElement():
        • Remove from P and return an element with the smallest key; an error condition occurs if P is empty.Input: none; Output: Object e
 Interface SimplePriorityQueue

Class Item
Item.java

public class Item {
    private Object key, elem;
    public Item (Object k, Object e) {
        key = k;
        elem = e;
    }
    public Object key() { return key; }
    public Object element() { return elem; }
    public void setKey(Object k) { key = k; }
    public void setElement(Object e) { elem = e; }
}

Comparators

  • The most general and reusable form of a priority queue makes use of comparator objects. We don’t want to have to implement a different priority queue for every key type and every way of comparing them.

  • Comparator objects are external to the keys that are to be compared and compare two objects. If we required keys to compare themselves, ambiguities could result. For example, numerical vs. lexicographical comparison.

  • When the priority queue needs to compare two keys, it uses the comparator it is given to do the comparison.
  • Thus any priority queue can be general enough to store any object.

  • The Comparator ADT includes the following methods, all of which return boolean values:
    • isLessThan(Object a, Object b)
    • isLessThanOrEqualTo(Object a, Object b)
    • isEqualTo(Object a, Object b)
    • isGreaterThan(Object a, Object b)
    • isGreaterThanOrEqualTo(Object a, Object b)
    • isComparable(Object a)
 Interface Comparator


Implementation with an Unsorted Sequence

  • Let’s try to implement a priority queue with an unsorted sequence S.

  • The element-items of S are a composition of two objects, the key k and the element e.

  • We can implement insertItem() by using insertFirst() of the Sequence ADT. This would take O(1) time.


  • However, because we always insertFirst() regardless of the key value, out sequence is unordered.



Implementation with an Unsorted Sequence (cont.)

  • Thus, for methods such as minElement(), minKey(), and removeMinElement(), we need to look at all elements of S. The worst-case time complexity for these methods is Q(n).


Implementation with a Sorted Sequence

  • Another implementation uses a sequence S that is sorted by keys such that the first element of S has the smallest key.

  • We can implement minElement(), minKey(), and removeMinElement() by accessing the first element of S. Thus, these methods are O(1), assuming our sequence has O(1) front-removal.


  • However, these advantages come at a price. To implement insertItem(), we might now have to scan through the entire sequence. Thus inserItem() is O(n).



Cost of Operations
Sequence-Based Priority Queue



Method
Unsorted Sequence
Sorted
Sequence
size, isEmpty
O(1)
O(1)
insertItem
O(1)
O(n)
minElement, minKey, removeMinElement
Q(n)
O(1)

 Class SequenceSimplePriorityQueue
Class SequenceSimplePriorityQueue

  • A sorted-sequence implementation of the SimplePriorityQueue ADT.

  •    
     
     
    public class SequenceSimplePriorityQueue  implements SimplePriorityQueue {
    //# Implementation of a priority queue using a sorted sequence
        protected Sequence seq = new NodeSequence();
        protected Comparator comp;
        // auxiliary methods
        protected Object extractKey (Position pos) {
            return ((Item)pos.element()).key();
        }
        protected Object extractElem (Position pos) {
            return ((Item)pos.element()).element();
        }
        protected Object extractElem (Item kep) {
            return kep.element();
        }




Class SequenceSimplePriorityQueue (cont.)

    // methods of the SimplePriorityQueue ADT
    public SequenceSimplePriorityQueue (Comparator c) {
        this.comp = c;
    }     public int size () {return seq.size(); }
    public boolean isEmpty () { return seq.isEmpty(); }
    public void insertItem (Object k, Object e) throws InvalidKeyException {
        if (!comp.isComparable(k))
            throw new InvalidKeyException("The key is not valid");
        else
            if (seq.isEmpty())
                seq.insertFirst(new Item(k,e));
            else
                if (comp.isGreaterThan(k,extractKey(seq.last())))
                    seq.insertAfter(seq.last(),new Item(k,e));
                else {
                    Position curr = seq.first();
                    while (comp.isGreaterThan(k,extractKey(curr)))
                        curr = seq.after(curr);
                    seq.insertBefore(curr,new Item(k,e));
                }
    }


Class SequenceSimplePriorityQueue (cont.)

    public Object minElement () throws EmptyContainerException {
        if (seq.isEmpty())
            throw new EmptyContainerException("The priority queue is empty");
        else
            return extractElem(seq.first());
    }
    // methods minKey and removeMinElement are not
    // shown.
}


Selection Sort

  • Selection Sort is a variation of PriorityQueueSort that uses an unsorted sequence to implement the priority queue P.
  • Phase 1, the insertion of an item into P takes O(1) time.
  • Phase 2, removing an item from P takes time proportional to the number of elements in P.

Sequence S
Priority Queue P
Input
{7,4,8,2,5,3,9}
{}
Phase 1:
(a) (b) … (g)
{4,8,2,5,3,9}
{8,2,5,3,9}

{}
{7}
{7,4}

{7,4,8,2,5,3,9}
Phase 2:
(a) (b) (c) (d) (e) (f) (g)
{2}
{2,3}
{2,3,4}
{2,3,4,5}
{2,3,4,5,7}
{2,3,4,5,7,8}
{2,3,4,5,7,8,9}
{7,4,6,5,3,9}
{7,4,8,5,9}
{7,8,5,9}
{7,8,9}
{8,9}
{9}
{}



Selection Sort (cont.)

  • Clearly, a bottleneck exists in phase 2. The first call to removeMinElement takes Q(n), the second Q(n-1), etc., until the last removal takes Q(1) time.

  • The total time needed for phase 2 is then

.

  • Recall that .

  • The total time complexity of phase 2 is then Q(n2 ).

  • Thus the total time complexity of the algorithm is

.
Insertion Sort

  • Insertion sort is the sort that results when we perform a PriorityQueueSort implementing the priority queue with a sorted sequence.

  • The cost of phase 2 is improved to O(n).

  • However, phase 1 now becomes the bottleneck for the running time. The first insertItem takes O(1), the second O(2), until the last insertItem takes O(n).

  • The run time of phase 1 is

.

  • The overall run time of the algorithm is

O(n2 ) + O(n) = O(n2 ).
Insertion Sort (cont.)



Sequence S
Priority Queue P
Input
{7,4,8,2,5,3,9}
{}
Phase 1:
(a) (b) (c) (d) (e) (f) (g)
{4,8,2,5,3,9}
{8,2,5,3,9}
{2,5,3,9}
{5,3,9}
{3,9}
{9}
{}
{7} {4,7} {4,7,8} {2,4,7,8} {2,4,5,7,8} {2,3,4,5,7,8} {2,3,4,5,7,8,9}
Phase 2:
(a) (b) … (g)
{2}
{2,3}

{2,3,4,5,7,8,9}
{3,4,5,7,8,9}
{4,5,7,8,9}

{}


Comparison of Selection Sort and Insertion Sort



Phase 1 Phase 2 Total
Selection Sort O(n) + Q(n2 ) = Q(n2 )
Insertion Sort O(n2) + O(n) = O(n2 )
  • Selection sort and insertion sort both take O(n2 ).

  • Selection sort will always take W(n2 ) time, no matter what the input sequence.

  • The cost of insertion sort varies depending on the input sequence.

    • If we remove from the rear of an already sorted input sequence S or remove from the front of a reverse-ordered sequennce, then the (best-case) cost of phase 1 of insertion sort is O(n). Thus, nearly sorted sequences will perform better for insertion sort.

Huffman Coding & Dictonaries

Motivation: Compression and data coding

Data compression is a highly important topic in computer science and information theory. Many situations require the use of coding schemes for data compression, including:
  • Storage of a large file on disk.
  • Storage of a large array or other dataset in memory.
  • Transmission of a large piece of information over a bandwidth-limited channel such as a phone line.
Many coding schemes are in common use in today's computer systems:
  • JPEG images
  • MPEG movies
  • data compression algorithms in a modem
Data compression is an example of the space-time tradeoff. In order to save storage costs, we expend additional CPU time in running the compression algorithm.
In this lecture we limit our focus to Huffman coding, a variable-length prefix encoding algorithm for compression of character streams. Although more sophisticated algorithms are now available and more widely used, Huffman's algorithm has many desirable qualities and is an interesting application of the use of binary trees.

Bits and bytes

Digital computers store data in binary or base-2 format. A binary digit may have one of two values:
Value Alternate representations
0 off, false, 0 volts (TTL)
1 on, true, 5 volts (TTL)
A bit is a binary digit and is the atomic unit in a digital computer.
Most modern computers do not store 1-bit numbers, however. For example, a machine with a 32-bit operating system stores addresses as 32-bit words.
A byte is an 8-bit number and is typically the smallest word size on a computer. An 8-bit binary number is a base-2 representation of 8 digits. For example,

Longer words (such as 16-bit, 32-bit, and 64-bit words) are constructed from 8-bit bytes. Other commonly used numbering systems used in computer science are octal (base 8) and hexadecimal (base 16).

Unicode and ASCII

Java stores characters as 16-bit unsigned integers according to the Unicode character-set standard. The Unicode set stores 216=65,536 characters and is an international standard that includes Asian character sets.
All of the letters in the English language, as well as numbers, punctuation, and control characters are represented in the lowest 7 bits of the Unicode set, with the upper 9 bits set to zero. This subset of the Unicode set is known as the ASCII standard and includes 27=128 characters. All of the characters encountered in this course will be part of the ASCII standard (see next page).
The following fragment of Java code
char[] charVals = {'m','n','o'};
for(int i = 0; i < charVals.length; i++)
{
    System.out.println("Character '" +charVals[i]
        + "' is stored as " + +(int)charVals[i]
        + " in base-10.");
}
yields the output
Character 'm' is stored as 109 in base-10.
Character 'n' is stored as 110 in base-10.
Character 'o' is stored as 111 in base-10.

ASCII Table

+---------------------------------------------------------------+
|  0 NUL|  1 SOH|  2 STX|  3 ETX|  4 EOT|  5 ENQ|  6 ACK|  7 BEL|
|  8 BS |  9 HT | 10 NL | 11 VT | 12 NP | 13 CR | 14 SO | 15 SI |
| 16 DLE| 17 DC1| 18 DC2| 19 DC3| 20 DC4| 21 NAK| 22 SYN| 23 ETB|
| 24 CAN| 25 EM | 26 SUB| 27 ESC| 28 FS | 29 GS | 30 RS | 31 US |
| 32 SP | 33  ! | 34  " | 35  # | 36  $ | 37  % | 38  & | 39  ' |
| 40  ( | 41  ) | 42  * | 43  + | 44  , | 45  - | 46  . | 47  / |
| 48  0 | 49  1 | 50  2 | 51  3 | 52  4 | 53  5 | 54  6 | 55  7 |
| 56  8 | 57  9 | 58  : | 59  ; | 60  < | 61  = | 62    | 63  ? |
| 64  @ | 65  A | 66  B | 67  C | 68  D | 69  E | 70  F | 71  G |
| 72  H | 73  I | 74  J | 75  K | 76  L | 77  M | 78  N | 79  O |
| 80  P | 81  Q | 82  R | 83  S | 84  T | 85  U | 86  V | 87  W |
| 88  X | 89  Y | 90  Z | 91  [ | 92  \ | 93  ] | 94  ^ | 95  _ |
| 96  ` | 97  a | 98  b | 99  c |100  d |101  e |102  f |103  g |
|104  h |105  i |106  j |107  k |108  l |109  m |110  n |111  o |
|112  p |113  q |114  r |115  s |116  t |117  u |118  v |119  w |
|120  x |121  y |122  z |123  { |124  | |125  } |126  ~ |127 DEL|
+---------------------------------------------------------------+



Variable-length encoding

Unicode and ASCII are examples of fixed-length encoding schemes, since all characters require the same amount of storage (16 bits and 8 bits, respectively).
In contrast, Huffman coding is a variable-length encoding scheme. The number of bits required to store a coded character varies according to the relative frequency or weight of that character. Although little if any space is saved on infrequent characters, frequently used characters require very little space (say one, two, or three bits).
The encoding process is based on a Huffman coding tree, which is built from the observed frequencies of characters in the document. The document is scanned, and the number of occurrences of each character in the document is recorded. Next, a binary tree is built in which the external nodes store the characters and their frequency in the document. When the binary tree is built from the document being coded, Huffman's algorithm optimizes the length of the encoded document.
Pre-scanning a document and generating a customized Huffman coding tree may be impractical, however. Often, typical frequencies of characters in a given context are used instead of the specific frequencies in a particular document.

Building a Huffman coding tree

Suppose that the following character frequencies are observed in a string that we wish to encode:  
character C D E F K L U Z
frequency 32 42 120 24 7 42 37 2
The first step is to construct a priority queue and insert the character-frequency (element-key) pairs into it. Assuming that we are using a sorted sequence-based priority queue, the above table could be schematically represented by the illustration below.
Step 1:

 

Building a Huffman coding tree (2)

In step 2, the two items with the lowest keys are removed from the priority queue. A new binary tree is created with the lowest-key item as the left external node and the second-lowest-key item as the right external node. The new tree is then inserted back into the priority queue
Step 2:


This process continues until only one node (tree) is left in the priority queue.
Step 3:

Building a Huffman coding tree (3)

Step 4:

Step 5:
Building a Huffman coding tree (4)

Final tree (after n=8 steps)
Building a Huffman coding tree (5)

Algorithm Huffman(X):
    Input: String X of length n     Output: Coding tree for X
    Compute the frequency f(c) of each character c in X.
    Initialize a priority queue Q.
    for each character c in X do
        Create a single-node tree T storing c.
        insert T into Q with key f(c).
    whileQ.size() > 1 do
        f1 ¬ Q.minKey()
        T1 ¬ Q.removeMinElement()
        f2 ¬ Q.minKey()
        T2 ¬ Q.removeMinElement()
        Create a new tree T with left subtree T1 and right subtree T2.
        Insert T into Q with key f1 + f2.
    return tree Q.removeMinElement()

Decoding

A Huffman code is a prefix code, meaning that no code is the prefix of another. To decode a bit stream from the leftmost bit, start at the root node of the tree and move to the left child if the bit is "0" or the right child if the bit is "1". When an external node is reached, the character it stores is sent to the decoded string. The next bit is then decoded from the root of the tree.
Decode 1011001110111101

Encoding

Create a lookup table storing the binary code corresponding to the path to each letter. This can be done via a preorder traversal. If the Huffman tree is used to encode ASCII text, a 128-element array can be used for this purpose. For example, the following code fragment can be used to store the string representing the set of bit-wise operations required to encode the character 'C':
String[] encoder = new String[128];
encoder['C'] = "1110";
 

Character
frequency
code
# bits
C
32 1110
4
D
42 110
3
E
120 0
1
F
24 11111
5
K
7 111101
6
L
42 101
3
U
37 100
3
Z
2 111100
6
Encode the string DEED:

Analysis

Define fi = frequency of character ci, i=1,…,n.
li = path length = length of code for ci, i.e., number of bits.
Weighted path length .  
 
Can be shown that a Huffman code is the prefix code that minimizes the weighted path length.
Average expected cost per character = 
Example from table on previous page:
expected cost per character = 
» 2.57.

A fixed-length encoding on 8 characters would require 3 bits per character.


Dictionaries


  • The Dictionary ADT
  • Implementation with Sorted and Unsorted Sequences
  • Binary Search


Adapted from the Goodrich and Tamassia lecture on Searching

The Dictionary ADT

  • A dictionary is an abstract model of a database.

  • Like a priority queue, a dictionary supports key-element pairs.

  • The main operation supported by a dictionary is searching by key.

  • Methods inherited from the SimpleContainer ADT
    • size()
    • isEmpty()

  • Query methods:
    • Object findElement(Object k)
    • Enumeration findAllElements(Object k)

  • Update methods:
    • void insertItem(Object k, Object e)
    • Object remove(Object k)
    • Enumeration removeAll(Object k)

  • Special sentinel object:
    • NO_SUCH_KEY, returned by an unsuccessful search.
 Interface SimpleDictionary

Example Operations on a Dictionary



Operation Output Unordered Dictionary
insertItem(5,A) {(5,A)}
insertItem(7,B) {(5,A),(7,B)}
insertItem(2,C) {(5,A),(7,B),(2,C)}
insertItem(8,D) {(5,A),(7,B),(2,C),(8,D)}
findElement(7) B {(5,A),(7,B),(2,C),(8,D)}
findElement(4)
NO_SUCH_KEY
{(5,A),(7,B),(2,C),(8,D)}
insertElement(2,E) {(5,A),(7,B),(2,C),(8,D),(2,E)}
findAllElements(2) C,E {(5,A),(7,B),(2,C),(8,D),(2,E)}
removeAll(2) C,E {(5,A),(7,B),(8,D)}

Implementing a Dictionary with a Sequence

  • Unordered sequence:


    • Searching takes O(n) time.
    • Inserting takes O(1) time with a doubly linked-list, O(n) time with an array.

  • Ordered sequence


    • Searching takes O(log n) time with an array, O(n) time with a linked list.
    • Inserting takes O(n) time.

  • In the ordered sequence implementation, we can search faster if the sequence is array-based.




Cost of Sequence Dictionary Operations



Method
Unsorted Sequence 
Sorted Sequence
Array Doubly Linked List Array Doubly Linked List
size, isEmpty
O(1)
O(1)
O(1)
O(1)
findElement
O(n)
O(n)
O(log n)
O(n)
findAllElements
Q(n)
Q(n)
O(log n + s)
O(n)
insertItem
O(1)
O(1)
O(n)
O(n)
remove
O(n)
O(n)
O(n)
O(n)
removeAll
Q(n)
Q(n)
O(n)
O(n)
-- space --
O(N)
O(n)
O(N)
O(n)


Binary Search

  • Narrow down the search range in stages.

  • Instead of scanning the sequence we do bisections.

  • All candidate items have key(low) £k£ key(high)

  • Example: findElement(22)


Pseudo-Code for Binary Search

Algorithm BinarySearch(S, k, low, high)     if low > high then         returnNO_SUCH_KEY     else
        mid ¬ë low + highû / 2
        ifk = key(mid) then             return k         else ifk < key(mid)then             return BinarySearch(S, k, low, mid-1)         else
            return BinarySearch(S, k, mid+1, high)

  • The Dictionary ADT method findElement(k) on a sorted array can be implemented by calling BinarySearch(S, k, 0, S.size()-1).



Analysis of Binary Search

  • A constant number of primitive operations is involved at earch recursive level. Thus the running time is proportional the number of recursive calls.
  • The size of the candidate list is cut by at least half at each step (i.e., at each recursive level).

comparison
size of candidate list
0
n
1
£ n/2
2
£ n/4
i
£ n/2i
log2n
£ 1

The recursive calls stop when the size of the candidate list £ 1. The number of recursive calls is m such that

-->  Cost of binary search = O(log n).

The QuickSort Algorithm

QuickSort is a particularly effective and popular sorting algorithm. It is a divide-and-conquer algorithm, working by splitting the data into two parts and then sorting them recursively. Being recursive, it is often hard to visualise what the algorithm is doing.

Here is the pseudocode algorithm for QuickSort:


 
 quicksort(a[], p, r)
           if r>p then
      j=partition(a[], p, r)
      quicksort(a[], p, j-1)
           quicksort(a[], j+1, r)


      partition(a[], p, r)
          i=p
          j=r+1
          pivot=a[p]
          do { 
               do i=i+1 while (a[i]<pivot)
               do j=j-1 while (a[j]>pivot)
               if (i<j) exchange(a[i], a[j])
             }while (i<j)
          exchange(a[p], a[j])
          return j

The Partition method receives a list or sublist, and places the first element in its correct position within the list. It also ensures that all elements to the left of this are smaller, and all to the right are larger.

The best and average cases of Quicksort take O(nlogn) time.

Go to QuickSort demonstration

Data Structure Algorithm Examples

Enable JAVA in browser and  try out these demos. They are a good way of getting to grips with some of the algorithms involved with the data structures:

Complexity Related Papers by Oded Goldreich

The following papers can be obtained in PostScript (papers in REVERSE CHRONOLOGICAL ORDER):

Low quality PostScript files of very old papers

Introduction to Complexity Theory

Lecture 1: The P vs NP Question. We review the fundamental question of computer science, known as the P versus NP question: Given a problem whose solution can be verified efficiently (i.e., in polynomial time), is there necessarily an efficient method to actually find such a solution? Loosely speaking, the first condition (i.e., efficient verification) is captured in the definition of NP, and the second in that of P. The actual correspondence relies on the notion of self-reducibility, which relates the complexity of determining whether a solution exists to the complexity of actually finding one.
Lecture 2: NP-completeness and Self Reducibility. We prove that any relation defining an NP-complete language is self-reducible. This will be done using the SAT self-reducibility (proved in Lecture 1), and the fact that SAT is NP-Hard under Levin Reductions. The latter are Karp Reductions augmented by efficient transformations of NP-witnesses from the original instance to the reduced one, and vice versa. Along the way, we give a simple proof of the existence of NP-Complete languages (by proving that Bounded Halting is NP-Complete). NP-completeness.
Lecture 3: More on NP and some on DTIME. In the first part of this lecture we discuss two properties of the complexity classes P, NP and NPC: the first property is that NP contains problems which are neither NP-complete nor in P (provided NP not equal P), and the second one is that NP-relations have optimal search algorithms. In the second part we define new complexity classes based on exact time bounds, and consider some relations between them. We point out the sensitivity of these classes to the specific model of computation (e.g., one-tape versus two-tape Turing machines).
Lecture 4: Space Complexity. We define ``nice'' complexity bounds; these are bounds which can be computed within the resources they supposedly bound (e.g., we focus on time-constructible and space-constructible bounds). We define space complexity using an adequate model of computation in which one is not allowed to use the area occupied by the input for computation. Before dismissing sub-logarithmic space, we present two results regarding it (contrasting sub-loglog space with loglog space). We show that for ``nice'' complexity bounds, there is a hierarchy of complexity classes - the more resources one has the more tasks one can perform. One the other hand, we mention that this increase in power may not happen if the complexity bounds are not ``nice''.
Lecture 5: Non-Deterministic Space. We recall two basic facts about deterministic space complexity, and then define non-deterministic space complexity. Three alternative models for measuring non-deterministic space complexity are introduced: the standard non-deterministic model, the online model, and the offline model. The equivalence between the standard and online models and their exponential relation to the offline model are proved. We then turn to investigate the relation between the non-deterministic and deterministic space complexity (i.e., Savitch's Theorem).
Lecture 6: Non-Deterministic Logarithmic Space. We further discuss composition lemmas underlying previous lectures. Then we study the complexity class NL (the set of languages decidable within Non-Deterministic Logarithmic Space): We show that directed graph connectivity is complete for NL. Finally, we prove that NL = coNL (i.e., NL is closed under complementation).
Lecture 7: Randomized Computations. We extend the notion of efficient computation by allowing algorithms (Turing machines) to toss coins. We study the classes of languages that arise from various natural definitions of acceptance by such machines. We focus on probabilistic polynomial-time machines with one-sided, two-sided and zero error probability (defining the classes RP (and coRP), BPP and ZPP). We also consider probabilistic machines that uses logarithmic spaces (i.e., the class RL).
Lecture 8: Non-Uniform Polynomial Time (P/poly). We introduce the notion of non-uniform polynomial-time and the corresponding complexity class P/poly. In this (somewhat fictitious) computational model, Turing machines are provided an external advice string to aid them in their computation (on strings of certain length). The non-uniformity is expressed in the fact that an arbitrary advice string may be defined for every different length of input. We show that P/poly ``upper bounds'' the notion of efficient computation (as BPP} subseteq P/poly), yet this upper bound is not tight (as P/poly contains non-recursive languages). The effect of introducing uniformity is discussed, and shown to collapse P/poly to P. Finally, we relate the P/poly versus NP question to the question of whether NP-completeness via Cook-reductions is more powerful that NP-completeness via Karp-reductions. This is done by showing, on one hand, that NP is Cook-reducible to a sparse set iff NP subset P/\poly, and on the other hand that NP is Karp-reducible to a sparse set iff NP=P.
Lecture 9: The Polynomial Hierarchy (PH). We define a hierarchy of complexity classes extending NP and contained in PSPACE. This is done in two ways, shown equivalent: The first by generalizing the notion of Cook reductions, and the second by generalizing the definition of NP. We then relate this hierarchy to complexity classes discussed in previous lectures such as BPP and P/poly: We show that BPP is in PH, and that if NP subseteq P/poly then PH collapses to is second level.
Lecture 10: The counting class #P and Approximating it. The class NP captures the difficulty of determining whether a given input has a solution with respect to some (tractable) relation. A potentially harder question, captured by the class #P, refers to determining the number of such solutions. We first define the complexity class #P, and classify it with respect to other complexity classes. We then prove the existence of #P-complete problems, and mention some natural ones. Then we try to study the relation between #P and NP more exactly, by showing we can probabilistically approximate #P using an oracle in NP. Finally, we refine this result by restricting the oracle to a weak form of SAT (called uniqueSAT).
Lecture 11: Interactive Proof Systems. We introduce the notion of interactive proof systems and the complexity class IP, emphasizing the role of randomness and interaction in this model. The concept is demonstrated by giving an interactive proof system for Graph Non-Isomorphism. We discuss the power of the class IP, and prove that coNP subseteq IP. We discuss issues regarding the number of rounds in a proof system, and variants of the model such as public-coin systems (a.k.a.\/ Arthur-Merlin games).
Lecture 12: Probabilistically Checkable Proof (PCP). We introduce the notion of Probabilistically Checkable Proof (PCP) systems. We discuss some complexity measures involved, and describe the class of languages captured by corresponding PCP systems. We then demonstrate the alternative view of NP emerging from the PCP Characterization Theorem, and use it in order to prove non-approximability results for the problems max3SAT and maxCLIQUE.
Lecture 13: Pseudorandom Generators. Pseudorandom generators are defined as efficient deterministic algorithms which stretch short random seeds into longer pseudorandom sequences. The latter are indistiguishable from truely random sequences by any efficient observer. We show that, for efficiently sampleable distributions, computational indistiguishability is preserved under multiple samples. We related pseudorandom generators and one-way functions, and show how to increase the stretching of pseudorandom generators. The notes are augmented by an essay of Oded. 
Lecture 14: Pseudorandomness and Computational Difficulty. We continue our discussion of pseudorandomness and show a connection between pseudorandomness and computational difficulty. Specifically, we show how the difficulty of inverting one-way functions may be utilized to obtain a pseudorandom generator. Finally, we state and prove that a hard-to-predict bit (called a hard-core) may be extracted from any one-way function. The hard-core is fundamental in our construction of a generator.
Lecture 15: Derandomization of BPP (The NW-generator). We present an efficient deterministic simulation of randomized algorithms. This process, called derandomization, introduce new notions of pseudorandom generators. We extend the definition of pseudorandom generators and show how to construct a generator that can be used for derandomization. The new construction differ from the generator that constructed in the previous lecture in it's running time (it will run slower, but fast enough for the simulation). The benefit is that it is relying on a seemingly weaker assumption.
Lecture 16: Derandomizing Space-Bounded Computations. We consider derandomization of space-bounded computations. We show that BPL subseteq DSPACE(\log^2 n), namely, any bounded-probability Logspace algorithm can be deterministically emulated in O(\log^2 n) space. We further show that BPL subseteq SC, namely, any such algorithm can be deterministically emulated in O(\log^2 n) space and (simultaneously) in polynomial time.
Lecture 17: Zero-Knowledge Proof Systems. We introduce the notion of zero-knowledge interactive proof system, and consider an example of such a system (Graph Isomorphism). We define perfect, statistical and computational zero-knowledge, and present a method for constructing zero-knowledge proofs for NP languages, which makes essential use of bit commitment schemes. We mention that zero-knowledge is preserved under sequential composition, but is not preserved under the parallel repetition.
Lecture 18: NP in PCP[poly,O(1)]. The main result in this lecture is NP subseteq PCP(poly,O(1)). In the course of the proof we introduce an NPC language ``Quadratic Equations'', and show it to be in PCP(poly,O(1)). The argument proceeds in two stages: First assuming properties of the proof (oracle), and then testing these properties. An intermediate result that of independent interest is an efficient probabilistic algorithm that distinguishes between linear and far-from-linear functions.
Lecture 19: Dtime(t) contained in Dspace(t/log t). We prove that Dtime(t(\cdot)) subseteq Dspace(t(\cdot)/\log t(\cdot)). That is, we show how to simulate any given deterministic multi-tape Turing Machine (TM) of time complexity $t$, using a deterministic TM of space complexity $t/\log t$. A main ingrediant in the simulation is the analysis of a pebble game on directed bounded-degree graphs.
Lecture 20: Circuit Depth and Space Complexity. We study some of the relations between Boolean circuits and Turing machines. We define the complexity classes NC and AC, compare their computational power, and point out the possible connection between uniform-NC and ``efficient'' parallel computation. We conclude the discussion by establishing a strong connection between space complexity and depth of circuits with bounded fan-in.
Lecture 21: Communication Complexity. We consider Communication Complexity - the analysis of the amount of information that needs to be communicated betwen two parties that wish to reach a common computational goal. We start with some basic definitions, considering both deterministic and probabilistic models for the problem, and annotating our discussion with a few examples. Next we present a couple of tools for proving lower bounds on the complexity of communication problems. We conclude by proving a linear lower bound on the communication complexity of probabilistic protocols for computing the inner product of two vectors, where initially each party holds one vector.
Lecture 22: Circuit Depth and Communication Complexity. The main result presented in this lecture is a (tight) nontrivial lower bound on the monotone circuit depth of s-t-Connectivity. This is proved via a series of reductions, the first of which is of significant importance: A connection between circuit depth and communication complexity. We then get a communication game and proceed to reduce it to other such games, until reaching a game called FORK. We conclude that a lower bound on the communication complexity of FORK, to be given in the next lecture, will yield an analogous lower bound on the monotone circuit depth of s-t-Connectivity.
Lecture 23: Communication Complexity of the FORK Game. We analyze the FORK game, introduced in the previous lecture. We give tight lower and upper bounds on the communication needed in a protocol solving FORK. This completes the proof of the lower bound on the depth of monotone circuits computing the function st-Connectivity.
Lecture 24: Average-Case Complexity. We introduce a theory of average-case complexity which refers to computational problems coupled with probability distributions. We start by defining and discussing the classes of P-computable and P-samplable distributions. We then define the class DistNP (which consists of NP problems coupled with P-computable distributions), and discuss the notion of average polynomial-time (which is unfortunately more subtle than it may seem). Finally, we define and discuss reductions between distributional problems. We conclude by proving the existence of a complete problem for DistNP.
Lecture 25: Computational Learning Theory. We define a model of automoatic learning called probably approximately correct (PAC) learning. We define efficient PAC learning, and present several efficient PAC learning algorithms. We prove the Occam's Razor Theorem, which reduces the PAC learning problem to the problem of finding a succinct representation for the values of a large number of given labeled examples.
Lecture 26: Relativization. In this lecture we deal with relativization of complexity classes. In particular, we discuss the role of relativization with respect to the P vs NP question; that is, we shall see that for some oracle A, $P^A = NP^A$, whereas for another $A$ (actually for almost all other $A$'s) $P^A \neq NP^A$. However, it also holds that $IP^A \neq PSPACE^A$ for a random $A$, whereas $IP = PSPACE$