AUTHORING WITH AURA WIKISemTechBiz 2013, San Francisco

… A CASE STUDY

Today we will be talking about…

• Populating a Symbolic AI – Aura• The spiraling cost structure for encoding data into a symbolic AI

• How do we bring low cost domain experts into the process?

• Creating a Semantic MediaWiki Installation• Importing a textbook into Semantic MediaWiki and marking up pages with properties

• Customizing the installation for annotating textbook sentences

AURA

3) Encoding Planning -- 35% time

Group Common UTs, ID KR/KE Issues,

ID Already Encoded, Write How to Encode

Pre-Planning, QA Check

Status Labeling: Encoding Complete, KR Issue (Closed)

2) Reaching Consensus -- 14% timeUniversal Truth Authoring, Concept Chosen QA Check

1) Determining Relevance -- 2% timeHighlighting, Diagram Analysis

QA Check

Status Labeling: Relevant, Irrelevant (Closed)

6) Question-Based Testing -- 14% timeUse Minimal Test Suite, Reasoning JIRA Issues Filed,

Encoder Fills KB GapsQA Check with Screenshots of “Passing" Comparison

and Relationship Questions

5) Key Term Review -- 25% timeKR Evaluated by Modeling Expert and Biologist,

Encoder Makes ChangesKR Evaluated by Modeling Expert and Biologist

QA Check

4) Encoding -- 10% timeEncode, File JIRA Issues

QA Check

Status Labeling: Encoding Complete, KE Issue

-- How to choose a concept given a UT?-- How to produce UTs from sentences?

Sentence

Sentence

UT

UT

UT

UT

Chapter

Chapter

KBBook

CMap

CMap

CMap

CMap

Chapter UT

2) Reaching Consensus -- 14% time

Univeral Truth Authoring, Concept Chosen

What is a Universal Truth?• “A Universal Truth is a stand-alone, unambiguous

declarative sentence about a textbook topic that expresses a single fact that is universally true”- AURA Knowledge Engineering Manual

• “Water is composed of two Hydrogen element molecules and one Oxygen element molecule with the chemical formula H20”• Water is composed of hydrogen• Water is composed of oxygen• Hydrogen is an element• Oxygen is an element• Water has the chemical formula H20

• Does: “Water is a compound” count?

Project Goals

• “Crowd Source Universal Truth Authoring”• Can Domain Experts Author Useful Universal

Truths?• Can We Speed Up Encoding a Textbook with Input

from Domain Experts?• Can We Create a UT Authoring Portal for Multiple

Textbooks?• Can Existing Social Networks Provide Domain

Experts Capable of UT Authoring?• Could Gamification be Applied to An Existing Portal

to Add Non-Domain Experts?

About the Domain Experts

• Students attending University of Washington or recent graduates

• All have a background in biology or life sciences• Native English speakers with excellent writing skills

• Each student read the chapters in question and was provided with an iPad running the Inquire application

• Students were paid for their time

A Semantic MediaWiki Portal

Storing a Text Book in Aura Wiki• The wiki was created with instances of page types

composed of textbook sentences• Sentence• Paragraph• Section• Chapter• Book

• The wiki also has imported resources to aid in the UT authoring process• Glossary Pages• Taxonomy Concepts• Universal Truths – Human and Machine

Navigating Aura Wiki

The text isn’t

readable

Is this the table of contents?

Why didn’t

I see this

first?Where’s the next

sentence?THIS PAGE HAS

NO FORMATTING

AND LOOKS

NOTHING LIKE

THE TEXTBOOK!

Navigating Aura Wiki

Removed

to Google

Analytics

Authoring Universal Truths

• Components : • Read Sentence• Access Sentence Context• Access Neighboring

Sentences• Check & Submit

Relevancy• Check & Submit Authoring

Status• Display Existing Universal

Truths• Author Universal Truths

Authoring Universal Truths

• Semantic Wiki Properties• Each page has a unique id

for the table of contents element

• The sentence itself is an element

• Elements pointing to the previous and next sentences.

• Elements pointing to top level entities

• Users can update the sentences relevancy and encoding status.

Sentence and Context View

Authoring Universal Truths

Input form for new UT. First two inputs are

required.

Authoring Universal Truths

• Semantic Wiki Properties• Reference sentence• The universal truth text• UT concept – AURA provided• UT context – AURA provided• Accuracy rating for the universal

truth• Date created, approved, and

when ratings were applied

Universal Truth

PROPOSALSUser Experience Review

… 5 of 5,000

Navigating Aura Wiki• Unregistered and Registered Main Pages

• Unregistered users are locked out• Registration is turned off for anonymous users

• Unique Extensions Proposed for Guided Authoring

How to View a Textbook Paragraph?Get rid of

title id

elements

Context needs to

include more

Click on

sentences in

context to

annotate next

Auto create triple format UTs from

sentence?

How to View a Universal Truth Page?

Can you edit

a UT you

didn’t create?

Is this edit another version

of the wiki page?

Is rating a UT

another

version of the

page?

How do we unify versions of the page for export

to AURA?

Knowledge Engineer Editing

Faceted

Browsing by

properties

Bulk editing by sentence /

propertyBulk

moderation of

universal

truths

Knowledge Engineer Editing

STUDENT REVIEWCan Experts Author Universal Truths?

Domain Expert Authoring Statistics• 6 University of Washington Students participated in the

test• Each received 45 minutes of training on creating

Universal Truths• Each was given 1 hour and a pre-selected list of

sentences on a user page to complete• The groups generated over 100+ Universal Truths each

session• They averaged 37 Universal Truths an hour per student• Students were frequently observed using their domain

experience to construct UTs not specifically worded in the source sentence (ie: “Water is a compound”)

CONCLUSION

Project Goals

• “Crowd Source Universal Truth Authoring”• Can Domain Experts Author Useful Universal Truths?

• Can We Speed Up Encoding a Textbook with Input from Domain Experts?

Project Goals

• “Crowd Source Universal Truth Authoring”• Can We Create a UT Authoring Portal for Multiple Textbooks?

Project Goals

• “Crowd Source Universal Truth Authoring”• Can Existing Social Networks Provide Domain Experts Capable of UT Authoring?

• Could Gamification be Applied to An Existing Portal to Add Non-Domain Experts?

QUESTIONS? COMMENTS?

THANK YOU(clap now)