CogSci 131
The problem of induction
Tom Griffiths
thought string = ‘computation’;disp(string); ≈
Minds and computers are both formal systems
Computational problems
• Problems of deduction and search: – arithmetic, algebra, chess
• We know what the underlying formal system should be for these problems – we know how computers can solve these
problems (at least in principle) – in many cases, computers can solve these
problems better than people
Computational problems
• Problems of deduction and search: – arithmetic, algebra, chess
• But what about: – learning and using language – sophisticated senses: vision, hearing – similarity and categorization – inferring causal relationships – scientific investigation
Outline
Inductive problems
Break
The problem of induction
Inductive problems
• Evaluating a set of hypotheses whose truth is underdetermined by the available data
• Examples: – learning and using language – sophisticated senses: vision, hearing – similarity and categorization – inferring causal relationships – scientific investigation
Learning language Red: Target language Blue: Current hypothesis
Multiple hypotheses can be consistent with the data
Learning language Hypotheses: (all consistent) • Rabbit • Dinner • Rabbit before t, dinner after • Undetached rabbit parts • Momentary rabbit-stage • Mass of rabbithood • Temporal cross-section of a
four-dimensional space-time extension of a rabbit
“Gavagai!”
Vision
• Two consistent hypotheses: – a cube – a cunningly shaded 2D shape
Vision
Vision
Vision
Categorization
Categorization
cat ⇔ small ∧ furry ∧ domestic ∧ carnivore
How do you find the appropriate rule?
Scientific discovery
76 y
ears
75
yea
rs
Halley, 1752
Inductive problems
P ⇒ Q P Q
P Q P Q P Q P ⇒ Q
P ⇒ Q Q P
deduction induction abduction
inductive problems
Causal induction
P Q P Q P Q P ⇒ Q
PushSwitch Light PushSwitch Light PushSwitch Light PushSwitch ⇒ Light
Causal reasoning
P ⇒ Q Q P
PushSwitch ⇒ Light Light PushSwitch
Inductive problems
P ⇒ Q P Q
P Q P Q P Q P ⇒ Q
P ⇒ Q Q P
deduction induction abduction
Philosophers: “a puzzle”, “a scandal”, “a myth”
Break
Up next: The problem of induction
Three problems • Plato’s problem
– how do we know so much? – why are our inductions so successful?
• Hume’s problem – induction can only be justified by induction
• Goodman’s “new riddle” – no simple syntactic rules for induction
Three problems • Plato’s problem
– how do we know so much? – why are our inductions so successful?
• Hume’s problem – induction can only be justified by induction
• Goodman’s “new riddle” – no simple syntactic rules for induction
Hume’s problem
• Inductive inferences assume that the future will be like the past
• What is the basis for this assumption?
• Induction can only be justified by induction
“It is impossible … that any arguments from experience can prove this resemblance from past to future; since all these arguments are founded on the supposition of that resemblance.”
The No Free Lunch Theorem (Wolpert)
• Averaged over all possible worlds, no learning algorithm is better than any other – e.g.sequence prediction: given x1, x2, predict x3
000 001 010 011 100 101 110 111
Worlds: Data:
00 00 01 01 10 10 11 11
Correct answer:
0 1 0 1 0 1 0 1
The No Free Lunch Theorem (Wolpert)
• In order for an algorithm to work better, the distribution over worlds must be constrained – e.g.the future is like the past
000 001 010 011 100 101 110 111
Worlds: Data:
00 00 01 01 10 10 11 11
Correct answer:
0 1 0 1 0 1 0 1
Anthropic argument
Goodman’s response
• Induction is no less justified than deduction – the formal system underlying deduction was
refined to confirm to our intuitions – the same process can yield rules for induction
• Instead of searching for justification, we should search for the rules of induction – what learning algorithm do people use?
• Some inductions are better than others…
Better and worse inductions
PushSwitch Light PushSwitch Light PushSwitch Light PushSwitch ⇒ Light
September17 Light September17 Light September17 Light September17 ⇒ Light
quality differs, despite the same syntax
suggests we’re missing some premises…
The “new riddle”
“Only a statement that is lawlike … is capable of receiving confirmation from an instance of it; accidental statements are not. Plainly, then, we must look for a way of distinguishing lawlike from accidental statements.”
What makes a statement lawlike (projectible)?
Grue
• Grue = “Green before t, blue after t” • Observe three green emeralds before t
• Both “all emeralds are green” and “all emeralds are grue” are equally confirmed
• So… why is green lawlike, but not grue?
Syntactic complexity
• Grue = “Green before t, blue after t” “This is a complicated property - perhaps
induction only works with simple properties?”
• Green = “Grue before t, bleen after t” – where bleen = “Blue before t, green after t”
Goodman’s conclusion “lawlike or projectible hypotheses cannot be distinguished on any merely syntactical grounds”
• There is some kind of extra knowledge (as to what is projectible) that enters into our inductive inferences
• So… there might be rules of induction, but they need to take this knowledge into account
The challenge for formal systems • Our best example of a formal system is
deductive logic, but induction has its own rules
• It’s not clear that assumptions like simple truth or falsehood apply – are you 100% sure this is the grammar? – are you 100% sure this is a cat? – are you 100% sure the comet will return?
The challenge for formal systems • Inductive problems:
– learning and using language – sophisticated senses: vision, hearing – similarity and categorization – inferring causal relationships – scientific investigation
• The situation differs from deductive problems: – what are the formal rules for induction? – how can computers solve these problems?
thought string = ‘computation’;disp(string); ≈
Minds and computers are both formal systems
Making computational models
Goodman suggests that we identify formal rules by iterative refinement
1. Develop models of inductive inferences 2. Test those models against human data 3. Modify models in light of data
“Why is a single instance, in some cases, sufficient for a complete induction, while in others myriads of concurring instances, without a single exception known or presumed, go such a very little way towards establishing a general proposition? Whoever can answer this question knows more of the philosophy of logic than the wisest of the ancients, and has solved the problem of Induction.”
John Stuart Mill (A System of Logic, 1843)
The challenge of induction
Next week
• Typicality and categorization – fuzzy borders and uncertainty
• Part II: Similarity, spaces, and features