Do it yourself, but not alone: Companion-Technology for DIY Assistance
A transfer project of the SFB/TRR 62 "A Companion-Technology for Cognitive Technical Systems"University of Ulm | Robert Bosch GmbH
Do it yourself, but not alone: Companion-Technology for DIY AssistancePage 2
A Digital Companion for DIY Projects
Support novice users with setting up DIY tools and completing DIY projects The DIY domain
► requires proficiency with electric and manual tools, attachments and different materials
► different possible courses of action towards a goal are possible► unforeseen events might occur during task execution
(e.g. mistake, tool breaks, etc.)
Do it yourself, but not alone: Companion-Technology for DIY AssistancePage 3
Underlying Technologies
We combine► (hierarchical) AI planning► ontology-based knowledge modeling► multimodal human-computer interaction
Do it yourself, but not alone: Companion-Technology for DIY AssistancePage 4
Application Scenario
Step-by-step instructions enable users to achieve different tasks► Planning techniques allow for adapting to individual situations and
preferences
Explanations enable users to learn about the application domain and the device(s):► How are devices and objects in the domain distinguished?► What characterizes them? ► What device can be used for which task?► Which settings, equipment, etc. are needed to perform a task?
Do it yourself, but not alone: Companion-Technology for DIY AssistancePage 5
Step-by-Step Assistance
Each step is illustrated by text, image and video Speech, text and touch commands for navigation and information requests Questions about concepts, appearance of objects, etc. supported
Do it yourself, but not alone: Companion-Technology for DIY AssistancePage 6
Robert ⎼ An Adaptive DIY Assistant
Do it yourself, but not alone: Companion-Technology for DIY AssistancePage 7
Robert ⎼ System Overview
Integration of procedural and declarative knowledge Ontology as (unified) underlying knowledge source for planning and dialogue Interaction realized using dialog agents and a multimodal GUI
Domain ModelDomain ModelHTN Planner
Ontology Manager
Planning Domain
Instructions Ontology
Domain OntologyReasoner
Verbalizer
User InterfaceUser Interface
Cloud-Based ASR
Content Slides
Cloud-Based TTS
InteractionInteraction
Cloud-Based NLU
HTTP API-Server Dialog Agents
Activity Recognition
Do it yourself, but not alone: Companion-Technology for DIY AssistancePage 8
System Components ⎼ Dialog and User Interface
Dialog management controls interaction between user and assistant components (planner, ontology):► Selection of appropriate system actions► Transformation of formalized concepts into natural speech► Initiation of proactive dialog
Domain ModelDomain ModelHTN Planner
Ontology Manager
Planning Domain
Instructions Ontology
Domain OntologyReasoner
Verbalizer
User InterfaceUser Interface
Cloud-Based ASR
Content Slides
Cloud-Based TTS
InteractionInteraction
Cloud-Based NLU
HTTP API-Server Dialog Agents
Activity Recognition
Do it yourself, but not alone: Companion-Technology for DIY AssistancePage 9
Robert's Dialog Management
Separation of user interface and dialogue control:► Client: multimodal user interface► Server: agent-based dialog management
Exploitation of cloud-based cognitive services:► Automatic speech recognition and text-to-speech synthesis (Google web speech)► Language understanding (RASA NLU)
User InterfaceUser Interface
Cloud-Based ASR
Content Slides
Cloud-Based TTS
InteractionInteraction
Cloud-Based NLU
HTTP API-Server Dialog Agents
Activity Recognition
Do it yourself, but not alone: Companion-Technology for DIY AssistancePage 11
Robert's Dialog Management
Task-specific Dialog Agents:► Individual Cloud-Based Natural Language Understanding (NLU) of user
input (RASA NLU)► Dialogue Agent with best NLU result selected► Activation of respective Dialog Handler for processing user request and
communication with assistant components
User InterfaceUser Interface
Cloud-Based ASR
Content Slides
Cloud-Based TTS
InteractionInteraction
Cloud-Based NLU
HTTP API-Server
„I want to build a key rack!“
Dialog Agents
Activity Recognition
Domain ModelDomain ModelHTN Planner
Ontology Manager
Planning Domain
Instructions Ontology
Domain OntologyReasoner
Verbalizer
Intent: startPlanning
Do it yourself, but not alone: Companion-Technology for DIY AssistancePage 13
System Components – Connected Tool Domain ModelDomain ModelHTN Planner
Ontology Manager
Planning Domain
Instructions Ontology
Domain OntologyReasoner
Verbalizer
User InterfaceUser Interface
Cloud-Based ASR
Content Slides
Cloud-Based TTS
InteractionInteraction
Cloud-Based NLU
HTTP API-Server Dialog Agents
Activity Recognition
Do it yourself, but not alone: Companion-Technology for DIY AssistancePage 14
Activity Recognition with the Connected Tool
Activity classifier Neural network architecture
based on CNNs and LSTMs (cf. Ordoñez & Roggen 2016)
Trained using labeled data from experiment (12 participants)
PSR18Li2 prototypically equipped with IMU (inertial
measurement unit) WiFi development
board
Recognized (disjoint) activity classes: drilling screwing running (idle) drill change battery change being moved off/resting
cloudmqttweb broker
Do it yourself, but not alone: Companion-Technology for DIY AssistancePage 15
Activity-Triggered Interaction
User picks up tool: ► “You seem to be working with the connected tool for the first time.► Don’t worry, I’ll guide you through the steps!”
As soon as battery change, drilling, drill change, or screwing are detected:► “I noticed that you were [changing the battery/drilling/changing the
attachment/screwing]. Did that work?”
Inactivity (3 minutes):► “I haven’t seen any tool activity by you in three minutes. Do you need help?”
Do it yourself, but not alone: Companion-Technology for DIY AssistancePage 16
Robert's Dialog Management – Connected Tool
Do it yourself, but not alone: Companion-Technology for DIY AssistancePage 17
System Components ⎼ Planner
Robert‘s Planner generates an individualized plan for the DIY project Considers the currently available tools and materials Adapts the plan exactly to the situation Provides a step-by-step plan
Domain ModelDomain ModelHTN Planner
Ontology Manager
Planning Domain
Instructions Ontology
Domain OntologyReasoner
Verbalizer
User InterfaceUser Interface
Cloud-Based ASR
Content Slides
Cloud-Based TTS
InteractionInteraction
Cloud-Based NLU
HTTP API-Server Dialog Agents
Activity Recognition
Do it yourself, but not alone: Companion-Technology for DIY AssistancePage 18
Hierarchical Planning
Planning is the task of finding a course of action which satisfies a desired goal when executed.
In hierarchical planning this is done by step-wise refining initially given abstract tasks into more fine-grained actions the user can execute.
Do it yourself, but not alone: Companion-Technology for DIY AssistancePage 19
Hierarchical Planning
Planning is the task of finding a course of action which satisfies a desired goal when executed.
In hierarchical planning this is done by step-wise refining initially given abstract tasks into more fine-grained actions the user can execute.
Do it yourself, but not alone: Companion-Technology for DIY AssistancePage 20
Robert – An Adaptive DIY Assistant
Robert generates a hierarchical plan Allows for
► showing the plan in different levels of granularity► adapting the plan to the knowledge of the user
Domain ModelDomain ModelHTN Planner
Planning Domain
Do it yourself, but not alone: Companion-Technology for DIY AssistancePage 21
Robert – An Adaptive DIY Assistant
Generated instruction might not be the preferred one of the user Robert allows for changing the plan Changes are performed such that the resulting plan is still sound and complete In the background the change request is translated into a formula in Linear
Temporal Logic
Domain ModelDomain ModelHTN Planner
Planning Domain
Do it yourself, but not alone: Companion-Technology for DIY AssistancePage 22
System Components – Ontology Manager
The ontology manager handles declarative knowledge
Domain ModelDomain ModelHTN Planner
Ontology Manager
Planning Domain
Instructions Ontology
Domain OntologyReasoner
Verbalizer
User InterfaceUser Interface
Cloud-Based ASR
Content Slides
Cloud-Based TTS
InteractionInteraction
Cloud-Based NLU
HTTP API-Server Dialog Agents
Activity Recognition
Do it yourself, but not alone: Companion-Technology for DIY AssistancePage 23
Modeling DIY Domain Knowledge
Domain OntologyModels general knowledge about DIY tools and materials:
spruce is-a softwoodsoftwood is-a woodwood is-a material
Domain ModelDomain ModelHTN Planner
Ontology Manager
Planning Domain
Instructions Ontology
Domain Ontology
Verbalizer
Reasoner
Do it yourself, but not alone: Companion-Technology for DIY AssistancePage 24
Knowledge about Actions and Media Contents
Instruction OntologyModels knowledge about the possible actions and their media contents in the DIY domain:
attach a battery to the PSR18Li2
Domain ModelDomain ModelHTN Planner
Ontology Manager
Planning Domain
Instructions Ontology
Domain Ontology
Verbalizer
Reasoner
Slide the battery onto the handle of your PSR18Li2 until it locks into place.
Stecken Sie den Akku auf den Griff Ihres PSR18Li2bis er einrastet.
Do it yourself, but not alone: Companion-Technology for DIY AssistancePage 25
System Architecture
Reasoning and background knowledge allows for inferring suitable alternatives if an exact match for an instruction is not available
insert the 3 mm bit into the PSR18Li2➡insert a bit into the drill
Domain ModelDomain ModelHTN Planner
Ontology Manager
Planning Domain
Instructions Ontology
Domain Ontology
Verbalizer
Reasoner
Do it yourself, but not alone: Companion-Technology for DIY AssistancePage 26
Enriching the Planning Model
Valid combinations of tools, materials and settings are inferred and passed to the planner
For establishing a screw connection in softwood with a 4 mm screw, pre-drilling with 3 mm diameter and a moderate speed is appropriate
Domain ModelDomain ModelHTN Planner
Ontology Manager
Planning Domain
Instructions Ontology
Domain Ontology
Verbalizer
Reasoner
Do it yourself, but not alone: Companion-Technology for DIY AssistancePage 27
Verbalization
Natural language explanations and descriptions are automatically generated
Spruce is softwood, thus, screwing a 3 mm screw in spruce applies to softwood. By definition, a 3 mm screw is a small screw. Screwing a small screw into softwood does not require pre-drilling. Therefore, screwing a 3 mm screw into spruce does not require pre-drilling.
Domain ModelDomain ModelHTN Planner
Ontology Manager
Planning Domain
Instructions Ontology
Domain Ontology
Verbalizer
Reasoner
Do it yourself, but not alone: Companion-Technology for DIY AssistancePage 28
Verbalization
Natural language explanations and descriptions are automatically generated
Domain ModelDomain ModelHTN Planner
Ontology Manager
Planning Domain
Instructions Ontology
Domain Ontology
Verbalizer
Reasoner