CS4018Formal Models of Computation

weeks 20-23

Computability and Complexity

Kees van Deemter

(partly based on lecture notes by Dirk Nikodem)

Second set of slides:Some uncomputable problems

• The halting problem

• An uncomputable integer function

• Noncomputability proofs by reduction

• Turing Machines

• Post’s Correspondence Problem

Some uncomputable problems

• Some problems must be uncomputable. (Stronger: some integer functions must be uncomputable.)

• But our diagonalisation proof was not constructive. Can we find examples of uncomputable problems/functions?

• First example (also historically first): the halting problem. Halt(P,I) means that program P halts on input I.

• This is a Boolean problem, so we will also speak of decidability instead of computability

Some appetisers (B.Russell)

• Suppose a barber, John, cuts everyone’s hair who does not cut his own hair (and noone else’s).

• Does John cut his own hair?

Some appetisers (B.Russell)

• Suppose a barber, John, cuts everyone’s hair who does not cut his own hair (and noone else’s).

• Does John cut his own hair?– Suppose he does. Then he’s the type of

person whose hair he doesn’t cut.– Suppose he does not. Then he’s the type of

person whose hair he does cut.

• Paradox!

Original home of Russell’s paradox: (old-fashioned) set theory

• Assumption 1: “Give me a property, then I’ll give you a set.”

• Assumption 2: Sometimes, a set is a member of itself. (E.g., the set of all sets is a set.) That’s a property!

• Let S =def {x: x x}• Is S S?• If yes (i.e., S S) then S S, so no

If no (i.e., S S) then S S, so yes

Halt(P,I): P terminates on input I

• We’ve seen examples of programs whose termination is unknown

• It does not follow that no algorithm exists

• What’s your intuition? Could there exist an algorithm for Halt(P,I)?

Halt(P,I): P terminates on input I

• Suppose P halts on I, then it halts in finite time. Consequently, we’ll discover by letting P run on I.

• We say that this makes Halt(P,I) partially decidable.

Halt(P,I): P terminates on input I

• Some problems are neither wholly nor partially decidable. (In fact, there’s a hierarchy of decidability, e.g., Harel Ch. 8)

• Example of such a highly undecidable problem: the totality problem, AlwaysHalts(P), which asks whether P halts on every input.

• Now let’s have a look at Halt(P,I)• Observe that programs may be applied to other

programs. (E.g., a compiler)

Proof of undecidability of Halt

• Suppose Halt(P,I) was decidable

• Then Halt(P,P) would also be decidable

• Supposing all this, let’s cook up a paradoxical algorithm

Proof of undecidability of Halt

• Suppose the programming language L contains a construction called loop, which loops forever. (E.g., loop = while True do;)

• Suppose L also contains a stop.

• Then here’s a paradoxical program S:

S(P) = If Halt(P,P) then loop else stop

Proof (ctd.)

S(P) = If Halt(P,P) then loop else stop

Q: Does S(S) halt? S(S) is …

If Halt(S,S) then loop else stop

• Suppose S(S) halts. Then Halt(S,S)=True, so loop is called, so S(S) loops.

• Suppose S(S) does not halt. Then Halt(S,S)=False, so stop is called, so S(S) halts.

Proof (ctd.)

We’ve derived a contradiction from theassumption that Halt(P,I) can beprogrammed.

S(P) = If Halt(P,P) then loop else stop

Compare Russell:• loop => not halt• stop => halt

Proof (ctd.)

How satisfying is this proof?

1. We now have an actual example of an undecidable problem.

2. Yet, the proof of its undecidability is rather indirect

A diagonalisation perspective:

consider a table matching programs P with

Inputs I, asking whether (P,I) halts:

P1 P2 P3 P4 P5 P1 y n y n n … P2 n n n y y … P3 n y y n y … P4 n y y n y … P5 y n y y n … … ………………………...… ...Diagonal records whether P(P) halts. Recall: S(P) says “If Halt(P,P) then loop else stop”S is the negation of the diagonal:S(P)=Halt iff P(P) does not halt: S = n y n y yS differs from every program in the list! It differs• … from P1 on argument P1, • … from P2 on argument P2, etc.

An uncomputable integer function

• Many non-computability results are inferred from others. This is called reduction (the third proof method after enumeration and diagonalisation).

• Many noncomputability results are inferred from the noncomputability of Halting

• Before explaining proofs by reduction, let’s perform diagonalisation once again

Defining an integer function

• Recall: Integer function F : NN• Recall: “most” of these functions must be uncomputable

(proven by enumeration)• Yet, examples are difficult to find• Enumerate strings, for example in lexicographical order:

wn = the n-th string

• F will hinge on the length of a program, so we define:

Hn= the set of programs of length n

Defining an integer function

• wn = the n-th string in the lexicographical ordering

• Hn= the set of programs of length n

----------------------------------------------------------------• P(n) = the number of the string that results

when P is applied to wn

• We’re interested in programs that are “applied to their own length”: Zn = { P(n): P Hn }

(Observe: for some n, Zn may be empty.)

• Un = MAX(Zn), or 0 if Zn is empty

• F(n) = U(n)+1 (i.e., 1 more than the maximum)

• Zn = { P(n): P Hn }

• Un = MAX(Zn), or 0 if Zn is empty

• F(n) = U(n)+1 (i.e., 1 more than the maximum)

--------------------------------------------------------------------------

Observe: Once the programming language is specified

and the ordering of strings, then F is a well-defined,

total, integer function.

Yet, no program can calculate this function.

Suppose program P did, while P Hj. .

How would P(j) = F(j) compare with U(j)?

U(j) is the largest element of Zj = { P(j): P Hj }.

But P(j)=U(n)+1, so it purports to be even larger.

Since P(j) Zj = { P(j): P Hj } as well,

this is a contradiction.

Our last proof method: reduction

Problem 1: How do you make tea, starting from an empty kettle? Answer:

(1) fill the kettle (2) put it on cooker, (3) let it boil, (4) pour water into teapot, (5) add tea, (6) wait 2 minutes. DONE

Our last proof method: reduction

Problem 2: How do you make tea, starting from a kettle that’s partly filled?

Our last proof method: reduction

Problem 2: How do you make tea, starting from a kettle that’s partly filled?

(1) Empty the kettle(2) Then do as in solution to Problem 1.

That’s the spirit of reduction!Reduction is important in computability,and in complexity as well.

Reduction (informally)

• Suppose we wonder whether problem P1 is computable• It sometimes suffices to find a mapping between P1 and

another problem P2, whose computablity is known• One direction: P2 is computable. Mapping should

transform every P1-problem into a P2-problem. Solution of P2 implies solution in P1.

• Main direction: P2 is not computable. Mapping should transform every P2-problem into a P1-problem. Non-existence of some P2 solutions implies non-existence of some P1-solutions. Hence, P1 must also be noncomputable.

Reduction (informally)

Terminology: • Hypothetical algorithm for P1

is called an oracle.• P2 is being reduced to P1.

(Easy to get confused!)• Also: P1 reduced from P2.

(Deduce contradiction from the assumption that an oracle exists)

Reduction (more formally)

• Target problem P1 used as oracle for uncomputable P2 (e.g., P2=Halting)

• P1 and P2 can have different domains(i.e., different possible inputs); both have Boolean output.

• Find function f: Dom(P2)Dom(P1), such that P2(x) is true iff P1(f(x)) is true

• f itself has to be computable

Reduction (more formally)

P2(x) is true iff P1(f(x)) is true

• Note the asymmetry:

• f is a function but not necessarily one-to-one (i.e., may not be a bijection)

• Hence, P1 can be used as oracle for P2, but not conversely (unless f is one-to-one)

• Many reductions are reductions from the Halting problem. (Halting is reduced to the new problem; hypothetical new algorithm is oracle)

• Halting is (hypothetically) reduced to the target problem: any solution to the target problem would imply a solution to Halting.

• Some simple examples. (Caveat: most reductions are much less expected than these.)

AlwaysHalt(P)

• Also know as the Totality Problem, or the Uniform Halting Problem

• AlwaysHalt(P) is true iff I(Halt(P,I))

• Intuitively, AlwaysHalt(P) is related to Halt(P), and it is harder. Therefore, it is natural to reduce it from Halt(P).

• Reduction is based on the equivalence

Halt(P,I) iff AlwaysHalt(f(P))

AlwaysHalt(P)

• Reduction is based on the equivalence Halt(P,I) iff AlwaysHalt(f(P))• f works by incorporating the input I into the

program P: instead of reading it “from outside”, it is represented in the programf(P) (See next slide for an example)

• f(P) does not take any input, hence AlwaysHalt(f(P)) just means that f(P) terminates.

Recall Algorithm B (input): var x:integer (1) while x<>1 do if x is even then do x:=x/2; else do x:=3x+1 (2) stop

To determine if Halt(B,29), check whether AlwaysHalt(f(B)):

Algorithm f(B):x:=29;(1) while x<>1 do if x is even then do x:=x/2; else do x:=3x+1(2) stop

Summary of argument:

1. Suppose Alwayshalt was computable2. Consider arbitrary program P, input I3. From (1): Alwayshalt(f(P)) would be comp.4. “f(P) halts for every input” means simply

that f(P) halts5. Outcome of Alwayshalt(f(P))

would tell you whether Halt(P,I)(N.B. I is made part of f(P))

6. This can’t be true, so (1) must be false.

Another reduction from Halting:Program verification [sketch]

• Definition: Correct(Program,Problem) iff Program correctly solves Problem.

• If Correct(Program,Problem) was computable then many computer scientists would be hunting for a solution! – Unfortunately it is not.

• [ Intuition: “To determine whether Correct(Program,Problem), you also have to termine when Program terminates. Hence if Correct(Program,Problem) was decidable then Halting would be decidable.” ]

Reduction is transitive

• If Halting is reduced to P1 andP1 is reduced to P2, then this is an indirect reduction of Halting to P2(thereby proving the uncomputability of P1 and P2).

• That’s how many results are proven: reductions to reductions to (…) reductions

Turing Machines

• Historically the first mathematical model of computation

• Still the most widely known model

• Computability can be studied by focussing on Turing machines completely

• We will only take a brief look

TMs as simple models of computation (following Harel 1987) • Advantage: a simple model makes it easy

to prove things

• Drawback: to express an algorithm in TMs is a lot of work– Simplifying data: strings of symbols on tape– Simplifying basic operations: transform one

symbol into another– Simplifying control: moving 1 place along

tape,left or right, depending on state and data. (State can encode memory!)

Turing machines (informal)

1. A head moving over a tape, readingand writing symbols

2. The head can move left as well as right 3. The head can “correct” the tape,

replacing one symbol with another4. Actions determined by the state of the

machine, and by the symbols encountered (= input)

5. Initial state; success states

Some observations:

• Start, Success• Halting states (4,5)• Role of B• Symmetry (states 1 and 2)

• Which strings end in a success state?• Give it input string abab

a. (0) BBababBB j. (3) BBxxxxBB

b. (1) BBxbabBB k. (3) BBxxxxBB

c. (3) BBxxabBB l. (3) BBxxxxBB

d. (3) BBxxabBB m. (0) BBxxxxBB

e. (0) BBxxabBB n. (0) BBxxxxBB

f. (0) BBxxabBB o. (0) BBxxxxBB

g. (0) BBxxabBB p. (0) BBxxxxBB

h. (1) BBxxxbBB q. (0) BBxxxxBB

i. (3) BBxxxxBB r. (4) Success!

Better names for states 0-5?

States as control/memory

• 4=success

• 5=failure

• 1= ||a found|| > ||b found||

• 2= ||b found|| > ||a found||

• 0 and 3: ||a found|| = ||b found||– 0 = looking for more (towards right edge)– 3 = moving back to the left edge

Turing machines (formal)TM = <Q,,,,q0,B,F>

• Q = set of states = set of allowable tape symbols• B is blank (or boundary) -{B} is finite input alphabet: Q x Q x x {+,-}• q0 Q is the initial state• F is the (nonempty) set of success states

(Note: may be a partial function)

Express example TM in these terms

• Function :– Please specify as pairs ((state,symbol),

(state,symbol,direction))– A representative sample of pairs will do

Example TM in formal terms

• Q = {0,1,2,3,4,5} = {a,b,x,B} = {a,b} = { ((0,a),(1,x,+)), ((1,a)(1,a,+)), … }

• q0 = 0

• F = {4}

(failure state 5 does not require explicit marking)

Does this formal apparatus remind you of anything?

Finite State Automata

• (Also called Deterministic Finite Automata)

Turing machines

• TMs are formal automata, like Finite State Automata and pushdown store automata

• TMs are the strongest automata that tend to be studied. Generating power equals that of Type-0rewriting systems: , stronger than regular grammars, context free grammars, and even context-sensitive grammars

Hopcroft & Ullman 1979, “Introduction to Automata Theory, Languages and Computation” – See also course on formal languages

FSA = <Q,,,q0,F>

• Q = set of states = finite input alphabet: Q x Q is a transition function

• q0 Q is the initial state

• F is the (nonempty) set of success states

Example

• abna (n>0)

FSA corresponds to a tape being read

from left to right: after a transition, move

to the symbol to the right on the tape

Turing Machines

1. The head can move left as well as right (as in two-way finite automata, seeHopcroft & Ullman 1979)

2. The head can “correct” the tape, replacing one symbol with another

Therefore, : Q x Q (in FSAs) becomes : Q x Q x x {+,-} (in TMs)

Play with TMs. E.g.,

• Build TM that recognises exactly all palindromes in the alphabeth {a,b}.(See Harel 87 for a solution)

Fiddling with TMs does not extend or diminish their generating power. E.g., – Multiple tapes (more)– Nondeterminism (more)– Multidimensionality (more)– More than one head (more)– At most three states (less)

(E.g., Hopcroft & Ullman 79, chapter 7)

“TM” a very robust notion

• Cf. the speed of light: given that nothing can go faster than light, there must be something important about it.

Another noncomputability result:Post’s Correspondence Problem (PCP)

• Another uncomputable problem

• Proven by reduction from Halting

• This reduction is less obvious

• Many other problems have been proven undecidable by reduction from PCP

• General statement of the problem (example in lecture notes):

Viewed as a puzzle:

Can pieces be joined in such a way that

top row symbols = bottom row symbols ?

1 2

XX

XXX

OXX

O

In this case yes (but it’s hard in general):

oxx xx xx = o xxx xxx

1

OXX

O

2

XX

XXX

2

XX

XXX

Post’s Correspondence Problem (PCP)

• Given n pairs of finite strings{ p1 = <s1

1,s12> ,…, pn = <sn

1,sn2> }

• n = size of problem• Notation: First(pi) = si

1, Second(pi) = si2

• Does there exist a sequence of pairs (possibly with iterations) <pi1,…,pik>, such thatFirst(pi1) +…+ First(pik)=Second(pi1) +…+ Second(pik)

Expressing example in this way:

• Strings from alphabeth {o,x}

• Size of problem=2:{ p1=<s1

1,s12>, p2=<s2

1,s22> } =

{ <oxx,o>, <xx,xxx> }

• Solution: <p1,p2,p2>, sinceoxx+xx+xx=o+xxx+xxx=oxxxxxx

• Does it matter that repetitions are allowed?

Looking ahead to complexity:

• If repetitions were not allowed then the problem would be finite and therefore decidable (for all sizes n).

• n pairs can be ordered in maximally n! ways, e.g., 3 pairs {a,b,c} can be ordered a+b+c, a+c+b b+a+c, b+c+a, c+a+b, c+b+a

• Worst case, there are n! orders to check

Post’s Correspondence Problem (PCP)

{ p1 = <s11,s1

2> ,…, pn = <sn1,sn

2> },

• Size of problem = n

• Decidability:– Size=2 is decidable– Size=7 is undecidable– Sizes in between are unknown

Lessons:

• Some reductions are quite unexpected

• Whether a problem is computable can be difficult to predict

Where this comes from, there is much more

E.g., Rice’s Theorem:

“Only syntactic properties of programs

are computable”

(Read Harel, one of the two books)

Remember high uncomputability

• AlwaysHalt

• Little brain teaser: Suppose we fix the program P and the input I. Does there exist an algorithm for determining whether Halt(P,I)?