Date post: | 20-Jan-2016 |
Category: |
Documents |
Upload: | jerome-james |
View: | 218 times |
Download: | 0 times |
EE141
How to Motivate a Machine ?How to Motivate a Machine ?
Janusz StarzykSchool of Electrical Engineering and Computer Science, Ohio University, USA
www.ent.ohiou.edu/~starzyk
Cognitive ArchitectureCognitive Architecture
EE141
Traditional Artificial Intelligence Embodied Intelligence (EI) Embodiment of Mind EI Interaction with Environment How to Motivate a Machine Goal Creation Hierarchy GCS Experiment Motivated Learning Challenges of EI
We need to know how to organize it We need means to implement it We need resources to build and
sustain its operation Promises of EI
To economy To society
OutlineOutline
EE141
“…Perhaps the last frontier of science – its ultimate challenge- is to understand the biological basis of consciousness and the mental process by which we perceive, act, learn and remember..” from Principles of Neural Science by E. R. Kandel et al. E. R. Kandel won Nobel Price in 2000 for his work on physiological
basis of memory storage in neurons. “…The question of intelligence is the last great
terrestrial frontier of science...” from Jeff Hawkins On Intelligence. Jeff Hawkins founded the Redwood Neuroscience Institute devoted
to brain research
IntelligenceIntelligence
AI’s holy grailFrom Pattie Maes MIT Media Lab
EE141
Is what is intelligence?Is what is intelligence?
EE141
Various Definitions of IntelligenceVarious Definitions of Intelligence The American Heritage Dictionary:
The capacity to acquire and apply knowledge. The faculty of thought and reason.
Webster Dictionary: The act or state of knowing; the exercise of the understanding. The capacity to know or understand; readiness of comprehension;
Wikipedia – The Free Encyclopedia: The capacity to reason, plan, solve problems, think abstractly, comprehend ideas
and language, and learn. Kaplan & Sadock:
The ability to learn new things, recall information, think rationally, apply knowledge and solve problems.
On line dictionary dict.die.net The ability to comprehend; to understand and profit from experience
The classical behavioral/biologists: The ability to adapt to new conditions and to successfully cope with life situations.
Dr. C. George Boeree, professor in the Psychology Department at Shippensburg University: A person's capacity to (1) acquire knowledge (i.e. learn and understand), (2) apply
knowledge (solve problems), and (3) engage in abstract reasoning. Stanford University Professor of Computer Science Dr. John McCarthy, a pioneer in AI:
The computational part of the ability to achieve goals in the world. Scientists in Psychology:
Ability to remember and use what one has learned, in order to solve problems, adapt to new situations, and understand and manipulate one’s reality.
EE141From http://www.indiana.edu/~intell/map.shtml
Mainstream Science on Intelligence December 13, 1994: An Editorial With 52 Signatories, by Linda S. Gottfredson, University of Delaware
Intelligence is a very general mental capability that, among other things, involves the ability to reason, plan, solve problems, think abstractly, comprehend complex ideas, learn quickly and learn from experience.
IntelligenceIntelligence
EE141
Animals’ IntelligenceAnimals’ Intelligence Defining intelligence
through humans is not appropriate to design intelligent machines:
– Animals are intelligent too
Dog IQ test: Dogs can learn 165 words (similar to 2 year olds) Average dog has the mental abilities of a 2-year-old child (or better) They would beat a 3- or 4-year-old in basic arithmetic, Dogs show some basic emotions, such as happiness, anger and disgust “The social life of dogs is very complex - more like human teenagers -
interested in who is moving up in the pack, who is sleeping with who etc,“ says professor Stanleay Coren from University of British Columbia
Border collies, poodles, and german shepards are the smartest dogs
EE141
Traditional AITraditional AI Embodied Intelligence Embodied Intelligence Abstract intelligence
attempt to simulate “highest” human faculties:
– language, discursive reason, mathematics, abstract problem solving
Environment model Condition for problem
solving in abstract way “brain in a vat”
Embodiment knowledge is implicit in the
fact that we have a body– embodiment supports brain
development
Intelligence develops through interaction with environment Situated in environment Environment is its best model
EE141
Design principles of intelligent systemsDesign principles of intelligent systems
Design principlessynthetic methodologytime perspectivesemergencediversity/complianceframe-of-referencecomplete agent
principle
from Rolf Pfeifer “Understanding of Intelligence”
From: www.spectrum.ieee.org/.../biorobot11f-thumb.jpg
EE141
Design principles of intelligent systemsDesign principles of intelligent systemsfrom Rolf Pfeifer “Understanding of Intelligence”, 1999
Interaction with complex environment
ecological balance redundancy principle parallel, loosely
coupled processes asynchronous sensory-motor
coordination value principle cheap design Agent
Drawing by Ciarán O’Leary- Dublin Institute of Technology
EE141
The principle of “cheap design”The principle of “cheap design”
intelligent agents: “cheap” exploitation of ecological
niche economical (but redundant) exploitation of specific
physical properties of interaction with real world
EE141
Principle of “ecological balance”Principle of “ecological balance”
balance / task distribution between morphology neuronal processing (nervous
system) materials environment
balance in complexity given task environment match in complexity of sensory,
motor, and neural system
EE141
The redundancy principleThe redundancy principle
redundancy prerequisite for adaptive behavior
partial overlap of functionality in different subsystems
sensory systems: different physical processes with “information overlap”
EE141
Generation of sensory stimulation Generation of sensory stimulation through interaction with environmentthrough interaction with environment
multiple modalities constraints from
morphology and materials
generation of correlations through physical process
basis for cross-modal associations
EE141
The principle of sensory-motor The principle of sensory-motor coordinationcoordination
self-structuring of sensory data through interaction with environment
physical process —not „computational“
prerequisite for learning
Holk Cruse•no central control•only local neuronal communication•global communication through environment
neuronal connections
EE141
The principle of parallel, loosely The principle of parallel, loosely coupled processescoupled processes
Intelligent behavior emergent from agent-environment interaction
Large number of parallel, loosely coupled processes
Asynchronous Coordinated through agent’s –sensory-motor system–neural system–interaction with environment
EE141
So what is an Embodied So what is an Embodied Intelligence ?Intelligence ?
EE141
Embodied Intelligence Embodied Intelligence
Definition Embodied Intelligence (EI) is a mechanism that learns
how to survive in a hostile environment
– Mechanism: biological, mechanical or virtual agent
with embodied sensors and actuators– EI acts on environment and perceives its actions– Environment hostility is persistent and stimulates EI to act– Hostility: direct aggression, pain, scarce resources, etc– EI learns so it must have associative self-organizing memory– Knowledge is acquired by EI
EE141
EI mimics biological intelligent systems, extracting general principles of intelligent behavior and applying them to design intelligent agents.
Knowledge is not entered into such systems, but rather is a result of their successful interaction with the environment.
Embodied intelligent systems adapt to unpredictable and dynamic situations in the environment by learning, which gives them a high degree of autonomy.
Learning in such systems is incremental, with continuous prediction of the input associations based on the emerging models - only new information is registered in the memory.
Embodied IntelligenceEmbodied Intelligence
EE141
What is Embodiment of a Mind?What is Embodiment of a Mind?
EE141
Embodiment
Actuators
Sensors
Intelligence core
channel
channel
Embodiment
Sensors
Intelligence core
Environment
channel
channelActuators
Embodiment
Actuators
Sensors
Intelligence core
channel
channel
Embodiment
Sensors
Intelligence core
Environment
channel
channelActuators
Embodiment of a MindEmbodiment of a Mind Embodiment of a mind is a
part of environment under control of the mind
It contains intelligence core and sensory motor interfaces to interact with environment
It is necessary for development of intelligence
It is not necessarily constant or in the form of a physical body
Boundary of embodiment transforms modifying brain’s self-determination
EE141
Brain learns own body’s dynamic Self-awareness is a result of
identification with own embodiment Embodiment can be extended by
using tools and machines Successful operation is a function
of correct perception of environment and own embodiment
Embodiment of MindEmbodiment of Mind
EE141
Requirements for Embodied IntelligenceRequirements for Embodied Intelligence
State oriented Learns spatio-temporal patterns
Situated in time and space
Learning Perpetual learning
Screening for novelty
Value driven Pain detection
Pain management
Goal creation
Competing goals Emergence
artificial evolution
self-organization
EE141
INPUT OUTPUT
Simulation or
Real-World System
TaskEnvironment
Agent Architecture
Long-term Memory
Short-term Memory
Reason
ActPerceive
RETRIEVAL LEARNING
EI Interaction with EnvironmentEI Interaction with Environment
From Randolph M. Jones, P : www.soartech.com
EE141
Kandel Fig. 23-5
Sensory Inputs CodingSensory Inputs Coding
How do we process and represent sensory
information?
Richard Axel, 1995
Foot
Hip Trunk
ArmHand
Face
Tongue
Larynx
Kandel Fig. 30-1
Visual, auditory, olfactory, tactile, smell -> motor
EE141
Challenges of Embodied IntelligenceChallenges of Embodied Intelligence
EE141
Challenges of Embodied IntelligenceChallenges of Embodied Intelligence Development of sensory interfaces
Active vision Speech processing Tactile, smell, taste, temperature, pressure
sensing Additional sensing
– Infrared, radar, lidar, ultrasound, GPS, etc.– Can too many senses be less useful?
Development of pain sensors Energy, temperature, pressure,
acceleration level Teacher input
Development of motor interfaces Arms, legs, fingers, eye movement
Intelligence core
Embodiment
Sensors
ActuatorsEnvironmentEnvironment
EE141
Challenges of Embodied Intelligence (cont.)Challenges of Embodied Intelligence (cont.) Finding algorithmic solutions for
Association, memory, sequence learning, invariance building, representation, anticipation, value learning, goal creation, planning
Development of circuits for neural computing Determine organization of artificial
minicolumn Self-organized hierarchy of
minicolumns for sensing and motor control
Self-organization of goal creation pathway
EE141
V. Mountcastle argues that all regions of the brain perform the same computational algorithm
Groups of neurons (minicolumns) connected in a pseudorandom way
Same structure Minicolumns organized in
macrocolumns
VB Mountcastle (2003). Introduction [to a special issue of Cerebral Cortex on columns]. Cerebral Cortex, 13, 2-4.
Human Intelligence – Cortex Uniform StructureHuman Intelligence – Cortex Uniform Structure
EE141
“The basic unit of cortical operation is the minicolumn …
It contains of the order of 80-100 neurons except in the primate striate cortex, where the number is more than doubled.
The minicolumn measures of the order of 30-50 m in transverse diameter, separated from adjacent minicolumns by vertical, cell-sparse zones …
The minicolumn is produced by the iterative division of a small number of progenitor cells in the neuroepithelium.” (Mountcastle)
Mini ColumnsMini Columns
Copyright © 2006-2008, all rights reserved, Visualbiotech
EE141
Sensory neurons are responsible for representing environment receive inputs from sensors or sensory neurons on lower level represent the environment receive feedback input from motor and higher level neurons help to activate motor and reinforcement neurons
Motor neurons are responsible for actions and skills are activated by reinforcement and sensory neurons activate actuators or provide an input to lower level motor neurons provide planning inputs to sensory neurons
Reinforcement neurons are responsible for building the value system, goal creation, learning, and exploration receive inputs from reinforcement neurons on the lower level receive inputs from sensory neurons provide inputs to motor neurons initiate learning and force exploration
Artificial Minicolumn OrganizationArtificial Minicolumn Organization
EE141
How to Motivate a Machine ?How to Motivate a Machine ?
A fundamental question is what motivates an agent to do anything, and in particular, to enhance its own complexity?
What drives an agent to explore the environment and learn ways to effectively interact with it?
EE141
How to Motivate a Machine ?How to Motivate a Machine ? Pfeifer claims that an agent’s motivation should emerge
from the developmental process. He called this the “motivated complexity” principle. Chicken and egg problem? An agent must have a motivation to
develop while motivation comes from development?
Steels suggested equipping an agent with self-motivation. “Flow” experienced when people perform their expert activity well
would motivate to accomplish even more complex tasks. Humans get internal reward for activities that are slightly above their
level of development (Csikszentmihalyi). But what is the mechanism of “flow”?
Oudeyer proposed an intrinsic motivation system. Motivation comes from a desire to minimize the prediction error. Similar to “artificial curiosity” presented by Schmidhuber.
EE141
How to Motivate a Machine ?How to Motivate a Machine ?
Can a machine that only implements externally given goals be intelligent?
If not how these goals can be created?
•There is a need for a hierarchy of values.•Not all values can be predetermined by the designer.•There is a need for motivation to act, explore and learn.•As machine makes new observations about the environment, there is a need to relate them to goals and values and create new goals and values.
EE141
How to Motivate a Machine ?How to Motivate a Machine ?
Exploration is needed in order to learn and to model the
environment. But is this mechanism the only motivation we need to develop
intelligence? Can “flow” ideas explain goal oriented learning? Can we find a more efficient mechanism for learning?
I suggest a simpler mechanism to motivate a machine.
Although artificial curiosity helps to explore the environment, it leads to learning without a specific purpose. It may be compared to exploration in
reinforcement learning. internal reward motivates the machine to
perform exploration.
EE141
How to Motivate a Machine ?How to Motivate a Machine ? I suggest that it is the hostility of the environment, in the
definition of EI that is the most effective motivational factor. It is the pain we receive that moves us. It is our intelligence determined to reduce this pain that motivates us
to act, learn, and develop.
Both are needed - hostility of the environment and
intelligence that learns how to reduce the pain. Thus pain is good. Without pain there would be no intelligence. Without pain we would not be motivated to develop.
Fig. englishteachermexico.wordpress.com/
EE141
Motivated Learning Motivated Learning I suggest a goal-driven mechanism to motivate
a machine to act, learn, and develop. A simple pain based goal creation system is
explained next. It uses externally defined pain signals that are
associated with primitive pains. Machine is rewarded for minimizing the primitive
pain signals. Definition: Motivated learning (ML) is learning based on the
self-organizing system of goal creation in embodied agent. Machine creates higher level (abstract) goals based on the primitive
pain signals. It receives internal rewards for satisfying its goals (both primitive and
abstract). ML applies to EI working in a hostile environment.
EE141
EI Interaction with EnvironmentEI Interaction with Environment
INPUT OUTPUT
Simulation or
Real-World System
TaskEnvironment
EI Architecture
Goal Creation
ActPerceive Competing goals
Planning
Pain
INPUT OUTPUT
Simulation or
Real-World System
TaskEnvironment
EI Architecture
Goal Creation
ActPerceive Competing goals
Planning
Pain
EI machine interacts with environment using its three pathways
EE141
Goal Creation Experiment in MLGoal Creation Experiment in ML
Pain signals in CGS simulation
0 100 200 300 400 500 6000
1
Primitive Hunger
Pa
in
0 100 200 300 400 500 6000
0.5
Lack of Food
Pa
in
0 100 200 300 400 500 6000
0.5
Empty Gorcery
Pa
in
Discrete time
EE141
Goal Creation Experiment in MLGoal Creation Experiment in ML
Action scatters in 5 CGS simulations
0 100 200 300 400 500 6000
5
10
15
20
25
30
35
40Goal Scatter Plot
Go
al I
D
Discrete time
EE141
Goal Creation Experiment in MLGoal Creation Experiment in ML
The average pain signals in 100 CGS simulations
0 100 200 300 400 500 6000
0.5
Primitive Hunger
Pai
n
0 100 200 300 400 500 6000
0.10.2
Lack of FoodP
ain
0 100 200 300 400 500 6000
0.10.2
Empty Gorcery
Pai
n
0 100 200 300 400 500 6000
0.10.2
Lack of Money
Pai
n
0 100 200 300 400 500 6000
0.050.1
Lack of JobOpportunitites
Pai
n
Discrete time
EE141
Compare RL (TDF) and ML (GCS)Compare RL (TDF) and ML (GCS)
Mean primitive pain Pp value as a function of the number of iterations:
- green line for TDF -blue line for GCS.
Primitive pain ratio with pain threshold 0.1
EE141
Comparison of execution time on log-log scale TD-Falcon green GCS blue
Combined efficiency of GCS 1000 better than TDF
Compare RL (TDF) and ML (GCS)Compare RL (TDF) and ML (GCS)
Problem solved
Conclusion: embodied intelligence, with motivated learning based on goal creation system, effectively integrates environment modeling and decision making – thus it is poised to cross the chasm
EE141
Reinforcement LearningReinforcement Learning Motivated Learning Motivated Learning Single value function Measurable rewards
Can be optimized
Predictable Objectives set by
designer Maximizes the reward
Potentially unstable
Learning effort increases with complexity
Always active
Multiple value functions One for each goal
Internal rewards Cannot be optimized
Unpredictable Sets its own objectives Solves minimax problem
Always stable
Learns better in complex environment than RL
Acts when needed
EE141
How can we make human level How can we make human level intelligence?intelligence?
We need to know how We need means to
implement it We need resources to
build and sustain its operation
EE141From Ray Kurzwail, The Singularity Summit at Stanford, May 13, 2006
Resources – Evolution of ElectronicsResources – Evolution of Electronics
EE141From Ray Kurzwail, The Singularity Summit at Stanford, May 13, 2006
EE141
Software or hardware?Software or hardware?
Sequential Error prone Require programming Low cost Well developed
programming methods
Concurrent Robust Require design Significant cost Hardware prototypes
hard to build
Software Hardware
EE141
Promises of embodied intelligencePromises of embodied intelligence To society
Advanced use of technology– Robots– Tutors– Intelligent gadgets
Intelligence age follows– Industrial age– Technological age– Information age
Society of minds– Superhuman intelligence– Progress in science– Solution to societies’ ills
To industry Technological development New markets Economical growth
ISAC, a Two-Armed Humanoid RobotISAC, a Two-Armed Humanoid RobotVanderbilt UniversityVanderbilt University
EE141
Sounds like science fictionSounds like science fiction
If you’re trying to look far ahead, and what you see seems like science fiction, it might be wrong.
But if it doesn’t seem like science fiction, it’s definitely wrong.
From presentation by Feresight Institute
EE141
Embodied Artificial IntelligenceEmbodied Artificial IntelligenceBased on: [1] E. R. Kandel et al. Principles of Neural Science, McGraw-Hill/Appleton & Lange; 4 edition, 2000. [2] F. Inda, R. Pfeifer, L. Steels, Y. Kuniyoshi, “Embodied Artificial
Intelligence,” International seminar, Germany, July 2003.[3] R. Chrisley, “Embodied artificial intelligence, ” Artificial
Intelligence, vol. 149, pp.131-150, 2003.[4] R. Pfeifer and C. Scheier, Understanding Intelligence, MIT
Press, Cambridge, MA, 1999. [5] R. A. Brooks, “Intelligence without reason,” In Proc. IJCAI-91.
(1991) 569-595 .[6] R. A. Brooks, Flesh and Machines: How Robots Will Change
Us, (Pantheon, 2002).
[7] R. Kurzweil The Age of Spiritual Machines: When Computers
Exceed Human Intelligence, (Penguin, 2000).
EE141
Questions?Questions?
EE141