Overview of AURA: Automated User-centered … · Reasoning and Acquisition System Aura is a tool to...

March 5, 2009 Copyright 2009 Vulcan Inc. All Rights Reserved.

Overview of AURA: Automated User-centered Reasoning and Acquisition System

Presentation by

Mark Greaves

Vulcan Inc.http://www.vulcan.com

SRI’s AURA project website has papers: http://www.ai.sri.com/

project/aura


Acknowledgment

The project is funded by Vulcan Inc. – a Paul Allen CompanyVulcan Program Management

Mark GreavesDave GunningBenjamin Grosof


AURA Team

SRIVinay ChaudhriBrian Adair (Chemistry)Sunil MishraJohn PachecoAaron SpauldingJing Tien

The Boeing CompanyPeter ClarkJohn ThompsonPhil Harrison

University of Texas at Austin

Ken BarkerJason ChawBruce PorterDan Tecuci

ConsultantsRichard FikesArt Fortgang (Physics)Karen Hurst (Biology)Bonnie JohnShirin SohrabiShahin Zarafshar (Biology)


Motivation

``Read a Chapter of a text and answer questions at the back of the chapter’’

Raj Reddy on Three Open Problems in AI, JACM’03``Build a Knowledge Base by Reading a Textbook’’

Ed Feigenbaum on Some Challenges for Computational Intelligence, JACM’03


MotivationInspired by Dickson’s Final Encyclopedia, the HAL-9000, and the broad SF vision of computing

The “Big AI” Vision of computers that work with people

The volume of scientific knowledge has outpaced our ability to manage it

This volume is too great for researchers in a given domain to keep abreast of all the developmentsResearch results may have cross-domain implications that are not apparent due to terminology and knowledge volume

“Shallow” information retrieval and keyword indexing systems are not well suited to scientific knowledge management because they cannot reason about the subject matter

Example: “What are the reaction products if metallic copper is heated strongly with concentrated sulfuric acid?” (Answer: Cu2+, SO2(g), and H2O

Response to a query should supply the answer (possibly coupled with conceptual navigation) rather than simply list 1000s of possibly relevant documents


Outline

Research FrameworkRequirement AnalysisImplementation

Knowledge FormulationQuestion FormulationQuestion Answering

EvaluationFuture Work


Research Framework

Focus on fundamental hard sciences where knowledge is explicitly written down

Physics, Chemistry, and BiologyChoose a widely accepted test for competence

Advanced Placement TestThe AP test is merely a metric. The system capability should be general enough to answer a broader set of questions

Scope the problem to a manageable size50 pages of syllabus in each of the three domains


Research Framework

Automatic reading was kept out of (initial) scopeAutomatic techniques will not produce a fidelity of representation needed for AP question answeringFocus on the basic system and provide ways to incorporate automatically extracted information


AURA Concept of Operations

Domain Experts Enter Knowledge

Domain Experts add knowledge to theAURA Knowledge Base and imoprtKnowledge using the mapping tool

Users ask questions and get answers and explanations

AURA

AURA Answers Questions


AURAAutomated User-Centered Reasoning and Acquisition System

Aura is a tool to help users formalize knowledge Aura can then reason with that knowledgeSo users can ask questions and understand the answers.


Outline



EvaluationPotential ApplicationFuture Work


Requirement Analysis

Knowledge Representation and Reasoning (KR&R) RequirementsQuestion Asking Requirements


KR&R Requirements

We undertook a survey of the textbook knowledge and a sample of AP exams in the three domains

Systematically enumerated representation capabilities using a KR&R Ontology (Fikes 2008)Most frequent KR&R types

Structured ObjectsRulesMathematical equationsTablesDiagramsComputational Knowledge


KR&R Requirements

Rules

Structured Objects


KR&R Requirements

Mathematical equations


KR&R Requirements

Tables


KR&R Requirements

Diagrams

Mitochondrion

Vesicle

Nuclear Envelope

Lysosome


Knowledge Acquisition Design Approach

Structured Objects and RulesWell-known type of knowledgePrior experience with SHAKEN system

EquationsIndispensable in Physics, and Chemistry

TablesCould be stated using a conceptual knowledge interface, but sometimes very tedious

DiagramsVery common, yet a very hard problemMost often the same knowledge can be stated using text


Question Asking Requirements

The training requirement needs to be kept lowThe question asker should not have to know about how the knowledge is representedThe questions may contain scenarios


Outline





Knowledge Formulation

The textbook is embedded in the softwareProvides context, and starting point for formulation

The users never begin from an empty KBKnowledge Engineers provide a library of pre-built representations

The Component Library (CLIB) contains classes representing physical actions, e.g., Move, Attach, Penetrate, and semantic relations, e.g., agent, object, has-part (Barker, Clark, Porter, KCAP’01)

Some domain-specific knowledge is pump primed

User-centered Design for UI abstractionsConcept Maps

Based on extensive research in educationPresent a collection of rules instead of one rule at a time (Clark, et. al., KCAP 2001, Chaudhri, EKAW 2003, KCAP 2007)


Example Knowledge












Formulated Knowledge




Mapping Graphs to Axioms

Based on prior work with Shaken & CLIBClark et. al., KCAP’2001, Chaudhri, et. al. EKAW, 2003

(forall ?c(=> (instance-of ?c Eucaryotic-Cell)

(exists ?x ?y ?z(and(instance-of ?x Nucleus)(instance-of ?y Chromosome)(instance-of ?z Plasma-Membrane)(has-part ?c ?x) (has-part ?c ?y)(has-part ?c ?z) (is-inside ?y ?x)))))


Outline





Question Formulation

Formallanguage

Unrestrictednatural

languageComputer Processable Language

“A boulder is dropped”“Consider the following possible situation in which a boulder first…”

“∀x∃y B(x)∧R(x,y)→C(y)”

Too hard for the user

There lies a “sweet spot” between logic and full NL which is both human-usable and machine-understandable


Question Formulation Cycle

Originaltext

A boulder is dropped.The initial speed of the boulder is 0 m/s.The duration of the drop is 23 seconds.The acceleration of the drop is 7.9 m/s^2.What is the distance of the drop?.

CPL (Controlled english)

Question-Answering

Rewritingadvice

Graph & paraphrase ofsystem’s understanding

A boulder is the object of a dropping.The dropping has a duration of 23 seconds.The dropping has initial speed 23 seconds.The dropping has acceleratio 7.9 m/s^2.The dropping has a distance of unknownWhat is the distance?


Example of Question Formulation

A boulder is dropped.The initial speed of the boulder is 0 m/s.The duration of the drop is 23 seconds.The acceleration of the drop is 7.9 m/s^2.What is the distance of the drop?

An alien measures the height of a cliff by dropping a boulder from rest and measuring the time it takes to hit the ground below. The boulder fell for 23 seconds on a planet with an acceleration of gravity of 7.9 m/s2. Assuming constant acceleration and ignoring air resistance, how high was the cliff?

?


Example Feedback from the System


Outline





Question Answering

Reasoning ControlReasoning Engine

Knowledge Machine (Clark, Porter, 2009)Semantic Matching (Yeh, Porter, AAAI 2006)

Specialized Reasoning ModulesEquation solvingChemical compound recognition

Explanation GenerationEnglish generation from the knowledge base


Example Answer


Example AnswerA solution that has a pH greater than 7 will result when this substance is dissolved in

water. a. sulfur dioxideb. oxalic acidc. phosphoric acidd. carbonic acide. potassium nitrate

CPLa. there is an aqueous solution of SO2.what is the pH of the solution? AnswerpH = 0

CPLb. there is an aqueous solution of H_2C_2O_4.what is the pH of the solution?AnswerpH is less than 7


Example Answer

What is the relationship between Caveolin and Muscle Cell?(Inspired by Larry Hunter’s work on Biological Discovery)


Example Answer


Outline





Evaluation

How much of the knowledge in the three domains can be captured through a generic knowledge capture and reasoning capability and to what extent does it need to be specialized for each domain?

Scaling to a knowledge base of full textbookApplying the same technology to new domains


Basic AURA Performance

Domain experts in each domain constructed a KB for each of the three domainsThe KB was tested on a suite of questions

The questions were drawn from the 50 pages of syllabus, and did not cover un-implemented features such as diagrams

8 sec71%284189Biology40 sec73%235284Chemistry24 sec79%13736PhysicsResponseCorrectQuestionsClasses


Generality of Knowledge Formulation

Pump priming is needed for each domainKnowledge in prior chaptersKnowledge that cannot be entered using AURA

3621012Biology6567418804Chemistry91383659Physics17453335693CLIBRelationsClassesLOC



To what extent was the general knowledge in the CLIB useful across the three domains?

561602641327Biology4373223447078Chemistry55176926489PhysicsUniqueTotalUniqueTotal

RelationsClasses



Chemistry required the use of a reaction editor that was not used in any other domain


Generality of Question FormulationThe overall design of the question formulation dialog was the same across all three domainsSome customization was needed in the CPL across the three domains

Semantic role labeling rules: If a Living-Entity Moves an Object, then the Living-Entity is the agent of the MoveMetonymy rules: ``mixed with H2O should be interpreted as mixed with a substance whose basic unit is an H2O molecule

The overall design of the question formulation dialog was the same across all three domainsSome customization was needed in the CPL across the three domains


The overall design of the question formulation dialog was the same across all three domainsSome customization was needed in the CPL across the three domains


03Biology69Chemistry311Physics15102GeneralMetonymyRole Labeling


Generality of Question Answering

There are multiple question types implemented in the systemComputing the values of a slot is the most frequently used question type

The computation of the value of slot can vary greatly in complexity

900Relate individuals

060Example of a Class

27295Definition

400Count of Slot values

812Comparison

3811Superclass1658713True/False

33111116Compute Slot ValueBiologyChemistryPhysics


Generality Outside the Current Scope

Even though we built the KB only for 50 pages of syllabus, our requirement analysis covered the whole syllabus

With the features currently implemented, we expect to be able toanswer 50% of the questions on an AP Exam in the three Science domains

A preliminary exercise for encoding the whole Physics textbook required 36 specific extensions to CLIB

We have analyzed three new domainsMicro Economics

Similar to Physics, but also more qualitativeUS Government and Politics

Some similarities to Biology in the need for approximate matching of descriptions

Environmental SciencesSimilar to Physics and Biology, but requires qualitative reasoning

The 50% coverage generalizes to the new domains


Independent Evaluation by BBN

IdealReference

Near‐Term Target

Diagnostic(Interaction Effects)

UltimateTarget

Upper bound

Initial Indicator of KB Quality

• The evaluation employs a 2 x 2 design to test the effects of user experience on KF and QF

• In each cell users will author KBs for a section of an AP syllabus and other users will query the KBs to answer a set of AP questions in that domain


Outline



EvaluationFuture Work


Backdrop for Future Work

Direct Knowledge Entry by Domain

ExpertsUpper

OntologyMid-LevelTheories

Domain-SpecificTheories

KnowledgeRepresentation

(KR) Expert

DomainExperts

Parallel Development by

Distributed Teams

Knowledgeable

Usable

Embeddable

HPKB, Halo I

RKF, Halo II


Future Work

SILK KR for defaults, processes, higher-order Knowledge Factory in India Expand the expressiveness of the question formulation interface (Boeing)

Database of paraphrasesKR language specification for concept maps and composition operations (with Richard Fikes)Computational knowledge, qualitative knowledge, and diagrams Knowledge Debugging Applications, e.g., Bio-med


Summary

AURA is aimed to be a generic computational tool aimed at modeling knowledge in hard sciences

We have good results based on 50 pages of textbook syllabus and questions suite drawn from an advanced placement exam

For more info, e.g., papers: SRI’s AURA project website

http://www.ai.sri.com/project/auraAlso: SILK project website

http://silk.projects.semwebcentral.org


Disclaimer: The preceding slides represent the views of the author only. All brands, logos and products are trademarks or registered trademarks of their respective companies.

Thank You

Date post:	08-Oct-2018
Category:	Documents
Upload:	ngokhanh
View:	217 times
Download:	0 times

Overview of AURA: Automated User-centered … · Reasoning and Acquisition System Aura is a tool to...

Documents