Class Project Due at end of finals week Essentially anything you want, so
long as it’s AI related and I approve Any programming language you want In pairs or individual Email me by Wednesday, November
3
Projects Implementing Knn to Classify Bedform Stability
Fields Blackjack Using Genetic Algorithms Computer game players:Go, Checkers, Connect
Four, Chess, Poker Computer puzzle solvers: Minesweeper, mazes Pac-Man with intelligent monsters Genetic algorithms:
blackjack strategy Automated 20-questions player Paper on planning Neural network spam filter Learning neural networks via GAs
Projects Solving neural networks via backprop Code decryptor using Gas Box pushing agent (competing against
an opponent)
What didn’t work as well Too complicated games: Risk, Yahtzee, Chess, Scrabble,
Battle Simulation Got too focused in making game work I sometimes had trouble running the game Game was often incomplete Didn’t have time to do enough AI
Problems that were too vague Simulated ant colonies / genetic algorithms Bugs swarming for heat (emergent intelligence never happened) Finding paths through snow
AdaBoost on protein folding data Couldn’t get boosting working right, needed more time on
small datasets (spent lots of time parsing protein data)
Reinforcement Learning Game playing: So far, we have told
the agent the value of a given board position.
How can agent learn which positions are important? Play whole bunch of games, and
receive reward at end (+ or -) How to determine utility of states that
aren’t ending states?
The setup: Possible game states
Terminal states have reward Mission: Estimate utility of all possible game states
What is a state? For chess: state is a combination of
position on board and location of opponents Half of your transitions are controlled
by you (your moves) Other half of your transitions are
probabilistic (depend on opponent) For now, we assume all moves are
probabilistic (probabilities unknown)
Passive Learning Agent learns by “watching” Fixed probability of moving from one
state to another
Sample Results
Technique #1: Naive Updating Also known as Least Mean Squares
(LMS) approach Starting at home, obtain sequence of
states to terminal state Utility of terminal state = reward loop back over all other states
utility for state i = running average of all rewards seen for state i
Naive Updating Analysis Works, but converges slowly
Must play lots of games Ignores that utility of a state
should depend on successor
Technique #2: Adaptive Dynamic Programming
Utility of a state depends entirely on the successor state If a state has one successor, utility
should be the same If a state has multiple successors,
utility should be expected value of successors
) to from transition(
)()terminal()terminal(
)(
jiPM
UMiUrewardU
ij
isuccessorsjjij
Finding the utilities To find all utilities, just solve equations
Set of linear equations, solveable Changes each iteration as you learn
probabilities Completely intractable for large problems:
For a real game, it means finding actual utilities of all states
)(
)(isuccessorsj
jijUMiU
Technique 3: Temporal Difference Learning
Want utility to depend on successors, but want to solve iteratively
Whenever you observe a transition from i to j:
))()(()()()terminal()terminal(
iUjUiUiUrewardU
= learning rate difference between successive states =
temporal difference Converges faster than Naive updating
Active Learning Probability of going from one state
to another now depends on action ADP equations are now:
) to from transition(
max)(
)terminal()terminal(
)(
jiPM
UMiU
rewardU
ij
isuccessorsjj
aija
Active Learning Active Learning with Temporal Difference
Learning: works the same way (assuming you know where you’re going)
Also need to learn probabilities to eventually make decision on where to go
(terminal) (terminal)( ) ( ) ( ( ) ( ))
U rewardU i U i U j U i
Exploration: where should agent go to learn utilities?
Suppose you’re trying to learn optimal game playing strategies Do you follow best utility, in order to win? Do you move around at random, hoping to
learn more (and losing lots in the process)? Following best utility all the time can
get you stuck at an imperfect solution Following random moves can lose a lot
Where should agent go to learn utilities?
f(u,n) = exploration function depends on utility of move (u), and number
of times that agent has tried it (n) One possibility: instead of using utility
to decide where to go, use
Try a move a bunch of times, then eventually settle
otherwiseu
Nnifnumberbignuf ),(
Q-learning Alternative approach for temporal
difference learning No need to learn probabilities:
considered more desirable sometimes Instead, looking for “quality” of (state,
action) pair
'
( , terminal) (terminal)( , ) ( , ) (max ( ', ') ( , ))
where ' is the state that action a leads to from a
Q a rewardQ a s Q a s Q a s Q a s
s s
Generalization in Reinforcement Learning
Maintaining utilities for all seen states in a real game is intractable.
Instead, treat it as a supervised learning problem Training set consists of (state, utility) pairs
Or, alternatively, (state, action, q-value) triples Learn to predict utility from state This is a regression problem, not a classification
problem Radial basis function neural networks (hidden nodes are
Gaussians instead of sigmoids) Support vector machines for regression Etc…
Other applications Applies to any situation where
something is to learn from reinforcement
Possible examples: Toy robot dogs Petz That darn paperclip “The only winning move is not to play”