SERC Talks February 1, 2017 1
Today’s session will be recorded.
An archive of today’s talk will be available at: www.SERCuarc.org
Use the Q&A box to queue questions, reserving the chat box for comments, and
questions will be answered during the last 5-10 minutes of the session.
If you are connected via the dial-in information only, please email questions or
comments to Ms. Mimi Marcus at [email protected].
Any issues? Use the chat feature for any technical difficulties or other comments, or
email Ms. Mimi Marcus at [email protected].
WELCOME
What is the Self?
Grady Booch, IBM Almaden Research Laboratory
February 1, 2017 | 1:00 pm ET
SERC Talks February 1, 2017 2
The Systems Engineering Research Center (SERC) is a federally funded University Affiliated Research Center managed by Stevens Institute of Technology.
Any views, opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the United States Department of Defense, ASD(R&E), nor the SERC.
No Warranty. This Stevens Institute of Technology Material is furnished on an “as-is” basis. Stevens Institute of Technology makes no warranties of any kind, either expressed or implied, as to any matter including, but not limited to, warranty of fitness for purpose or merchantability, exclusivity, or results obtained from use of the material. Stevens Institute of Technology does not make any warranty of any kind with respect to freedom from patent, trademark, or copyright infringement.
This material has been approved for public release and unlimited distribution.
© 2017 IBM Corporation 1
What Is The Self?
Grady Booch
© 2017 IBM Corporation 2
Imagine unleashing Watson in the physical world. Give it eyes, ears, and touch, then let it act in that world with hands and feet and a face, not just as an action of force but also as an action of influence. This is embodied cognition: by placing the cognitive power of Watson in a robot, in an avatar, an object in your hand, or even in the walls of an operating room, conference room, or spacecraft, we take Watson's ability to understand and reason and draw it closer to the natural ways in which humans live and work. In so doing, we augment individual human senses and abilities, giving Watson the ability see a patient's complete medical condition, feel the flow of a supply chain, or orchestrate the tasks in a day in the life of an individual.
© 2017 IBM Corporation 3
Embodied Cognition Defined
An embodied cognition must– Be in and of the world– Reason– Learn– Have identity
An embodied cognition is not– Simply an STT -> NLC -> action pipeline
© 2017 IBM Corporation 4
Embodiment
Raw curiosity
More computational
power
Advances in AI algorithms and architectures
Better/smaller sensors and
actuators
Better batteries
3D printing
Advances in neural implants
New military doctrine
Demographic changes
Economic need
Advances in neural implants Technology push
Demand pullRaw curiosity
© 2017 IBM Corporation 5
An Agent May Be Embodied In A Robot…
© 2017 IBM Corporation 6
Dependent Autonomous
© 2017 IBM Corporation 7
Fixed place Mobile
© 2017 IBM Corporation 8
Thing-facing Human-facing
© 2017 IBM Corporation 9
Predictable Responsive Social
© 2017 IBM Corporation 10
Architecturally, robots may be classified along one of several dimensions
© 2017 IBM Corporation 11
Biologically-organized
Symbolically-organized
© 2017 IBM Corporation 12
Thin Client Thick Client
© 2017 IBM Corporation 13
Thin Client Thick Client
By way of context, Watson is largely a symbolically-organized thin client cognitive system
Biologically-organized
Symbolically-organized
© 2017 IBM Corporation 14
An Avatar
© 2017 IBM Corporation 15
A Space
© 2017 IBM Corporation 16
Or An Object
© 2017 IBM Corporation 17
Concierge/Greeter
Where is the elevator?
Home
Did I turn off the stove?
Hotel
Bring me some fresh towels
Retail
Where can I find this in my size?
Elder Care
Don’t forget to take your pills
Cobot
Get me a screwdriver
Fabrication
Watch me do this
Boardroom
Help me decide
Companion
Let’s play a game
Tutor
How do I solve this problem?
Get me a screwdriver
Transportation
Use Cases
© 2017 IBM Corporation 18
Embodied Cognition Differentiated
Embodiment offers far more than request/response behavior Embodiment is more than just conversation Agency Personalization Multi-modal Learns (not programed)
© 2017 IBM Corporation 19
Concierge/Greeter
Avatar, Robot, Device
Home
Space, Avatar, Robot, Device
Hotel
Space, Avatar, Robot, Device
Retail
Space, Avatar, Robot, Device
Elder Care
Space, Robot, Device
Cobot
Robot,Avatar
Fabrication
Robot
Boardroom
Space, Avatar, Device
Companion
Avatar, Robot, Device
Tutor
Avatar, Device
Space, Avatar
Transportation
Form Factors
© 2017 IBM Corporation 20
Self
20
Architecture
© 2017 IBM Corporation 21
Who said that?
Sensory Fusion
I feel sad.
Theory of Mind
I need the Phillips head screwdriver.
Context
Give me that one; no, I mean that one!
Embodied Conversation
It’s good to see you again, Sandia!
Learning
I’ve just picked up a fault in the AE35 unit.
Devices
In what room was Alyssa working yesterday?
Knowledge Representation
Technical Issues
You don’t have permission to do that
Security
Room 806 has a light out
Scale
© 2017 IBM Corporation 22
Intu Deployment
Access and configure Bluemixservices (via web)
Manage all Project Intu instances (via desktop/laptop)
SDK on devices (via Avatar, Raspberry Pi, Robots, Desktop/Laptop)
Project Intu Gateway Project Intu Manager Project Intu Kernel
© 2017 IBM Corporation 23
Research Portfolio
Core Technology Topic Project NamePlatform Self
Goals & Planning
Planning
Embodied Action
Multi-modal Contextual Interaction
Embodied Decision Support Systems
Models of the World, Others, and Self
Spatial Intelligence
People Intelligence
Spatial Awareness
Theory of Mind
Learning Learning by Example
Sensors & Perception/ActuatorsPhysical Interaction
Edge Intelligence
Platform & Meta Management Tooling
23
© 2017 IBM Corporation 24
Background
In November 2015, Rob High, Ray Chancey, and I were brainstorming about a suitable grand challenge for the 2016 Embodied Cognition GTO.
We settled on a very grand one: build a system to support mission control, flight operations, and the scientific agenda of NASA’s Mission to Mars; the requirements of this mission lead to the concept of Watson in the walls (an embodied cognition for the Orion spacecraft and the Mars habitat) together with robots and other devices powered by Watson (namely, Robonaut and its peers).
Recognizing this was a 20+ year research effort, I set out to devise an architecture that would not only support this Mission to Mars but the journey of which would yield a middleware that would attend to a variety of use cases, from concierge to retail to elder care and many others.
Ray named this architecture “Self”.
© 2017 IBM Corporation 25
Self Basic Principles
Augment human capabilities
Learn, don’t program
Theory of mind and social intelligence
Self understanding
Platform agnostic
Embodiment as a robot, an avatar, a device, or a space
Deployment as middleware with microservices in the cloud
© 2017 IBM Corporation 26
Self Significant Design Decisions
Self is a hybrid architecture, encompassing explicit symbolic computation in the center together with neural networks at the edges.
Inspired by Minsky’s Society of Mind, behavior takes place in the context of multiple concurrent agents that communicate opportunistically via blackboards and deterministically via peer to peer connections.
Inspired by Brooks’s subsumption architecture, behavior takes place in a hierarchy of cognition, from involuntary reflexes to voluntary skills to goals and planning.
We maintain a clear separation of concerns among perception, actuating, models, and behavior.
As much as possible, behavior is either taught or is learned, not programmed.
As much as possible – driven by these separation of concerns, the needs of packaging, and performance – all components are made manifest as RESTful microservices.
As much as possible, plans, skills, and reflexes are extensible.
Self is intentionally full of strange loops: components of Self are also parts of the models of itself.
Self is intentionally fractal: an instance of Self may have models of others, which themselves are other instances of Self.
© 2017 IBM Corporation 27
Self Markitecture
IBM Confidential 27
Sensors & Perception
Actuators
Agents
Goals & Planning
Platform Management Infrastructure
Meta Management
Models
Voluntary Behavior & Skills
Involuntary Behavior & Reflexes
Others
World
Self
© 2017 IBM Corporation 28
Embodied cognition represents the next generation of Watson
© 2017 IBM Corporation 29
Embodied cognition represents the extension of IT into the physical world, bringing cognition to the edge of the Internet of Things
© 2017 IBM Corporation 30
At this touch point we encounter the intersection of big data and human presence, mediated by cognitive systems that reason
© 2017 IBM Corporation 31
Text
Watson can be the back end to a variety of embodied cognitive systems.
© 2017 IBM Corporation 32
Text + Audio
© 2017 IBM Corporation 33
Text + Audio
This is the current state of an embodied Watson(the low hanging fruit, wherein platforms such as Pepper serve as animated speakers)
© 2017 IBM Corporation 34
Text + Audio
There is much, much more that is possible(if we extent the embodiment to encompass greater perception and action in the world)
© 2017 IBM Corporation 35
Text + Audio + Vision
© 2017 IBM Corporation 36
Text + Audio + Vision + Emotion
© 2017 IBM Corporation 37
Text + Audio + Vision + Emotion + Motion
© 2017 IBM Corporation 38
Text + Audio + Vision + Emotion + Motion + Physical interaction
© 2017 IBM Corporation 39
Text + Audio + Vision + Emotion + Motion + Physical interaction
Socially-intelligent cooperative robot
© 2017 IBM Corporation 40
SERC Talks February 1, 2017 3
Can Graphical Models Provide a Sufficient Basis for General Intelligence?Dr. Paul S. Rosenbloom, Institute for Creative Technologies, University of Southern CaliforniaApril 5, 2017 | 1:00 pm ET
UPCOMING TOPICS:
What Are Cyber-Social Learning Systems And How Will We Form Them?
Dr. Kevin Sullivan, University of Virginia
June 7, 2017 | 1:00 pm ET
Thank you for joining us! Please check back on the SERC website for today’s recording and future SERC Talks information!