1

Enhancing Interactions with To-Do Lists Using Artificial Assistants

Yolanda Gil, Timothy Chklovski

USC/Information Sciences Institute{gil, timc}@isi.edu

March 26, 2007

2

Structured statements about tasks and objects: - setting up a videoconference

camera: switch on computer: turn on, start up, link up microphone: turn on, test, adjust

Problems & remedies: - during a meeting

projector not available locate a portable projector

Learning Common Knowledge from Volunteers to Support Assistance

Learning about tasks

Learning to anticipate and repair

600,000+ statements collected over 12 months

Targets collection by topic and knowledge type

Learning about objects Learner

Proactively broadens coverage

Formulates relevant followup

questions in real time

Validates through other users

Guides knowledge entry

Learning paraphrases

3

To Do Lists

To Do lists are pervasive, and present large opportunity for assistance and learning We’ve been working with TOWEL, the To Do list

manager in CALO Glimpses / hints of users’ goals

To Do lists Have some regularity & structure Contents and surface form may vary widely Similar to the collected statements

4

To Do Lists: Related Work

Ethnographic studies look at usage of To Do lists Eg: V Bellotti, B Dalal, N Good, P Flynn, D Bobrow, N

Ducheneaut. What a To-Do: Studies of Task Management Towards the Design of a Personal Task List Manager. CHI 2004

Analysis of work activities and how tools may support it, cognitive aids

B Harrison. An Activity-Centric Approach To Context-Sensitive Time Management. In CHI 2004: Workshop on the Temporal Aspects of Work.

A Dey, G Abowd. CybreMinder: A Context-Aware System for Supporting Reminders. HUC, 2000

Some commercial tools support prioritization Based on activity type, urgency, time available Eg, Life Balance, http://www.llamagraphics.com/, Voo2Do,

“Remember the milk”, Tada.com (provides public to do lists) Not focused on NL interpretation or on automation, mostly

on human factors and usability

5

Assisting with To Do Lists: the Idea

Key idea: Develop interpretation & mapping of To Do entries to assistant-supported tasks

Exploit large knowledge repositories and preprocessed texts

Paraphrases to help interpret text Use knowledge repositories to interpret and connect user

knowledge to operationalized tasks in CALO

Build on prior work on volunteer collection of paraphrases to assist speech recognition / utterance identification [Chklovski, KCAP ’04 & KCVC ’04]

Build on prior work on extraction from large corpora [Chklovski & Pantel, EMNLP ’05], and volunteer validation

6

BEAM: “Broad-coverage Entity Analysis and Mapping”

To Do interface integrating BEAM and providing task interpretation and monitoring via CALO’s TOWEL

List is automatically updated when the “Plan conference travel” action is completed

To Do entry made by user

BEAM mapping to TOWEL task

User To Do EntriesFind hotel w/ pool for Joe

Reserve conf room for talk

Buy bread on the way home

Get talk abstract from Joe

CALO Task Ontology:Catalog of Automated

Procedures

Execution and Monitoring

InstrumentedDesktop

TaskLearning

Set airport pickup for Joe

Announce room for talk

Reserve accommodations

Host a visitor

Arrange meeting

Opportunities for Interpretation-based Assistance

Map entries to task procedures

Anticipate & suggest missing

Entries, sub-tasks

Assist with how activities are done in the organization

BEAM

Initiate and report execution

Group and organize entries

User To Do EntriesFind hotel w/ pool for Joe

Reserve conf room for talk Map entries to task procedures

Anticipate & suggest missing

Entries, sub-tasks

Assist with how activities are done in the organization

BEAM

Buy bread on the way home

Get talk abstract from Joe

Initiate and report execution

Group and organize entries

CALO Task Ontology:Catalog of Automated

Procedures

Execution and Monitoring

InstrumentedDesktop

TaskLearning

Set airport pickup for Joe

Announce room for talk

Reserve accommodations

Host a visitor

Arrange meeting

OntologiesNGrams

Repository“to schedule a meeting”

SubtasksRepository:

“Reserve X has-subtask

find X”

Organization-specificTask knowledge

“Airport pickup of visitorsis common, but not here”

Action ParaphrasesRepository:

“plan X schedule X”“lease car rent car”

Verb Relations

Repository:“schedule

happens-before reschedule”

From Text ExtractionFrom Web Volunteers From Volunteers in the Organization

From Knowledge Engineers

BEAM Knowledge Sources for NL Interpretation and Assistance

RepairsRepository:

“If projector not working,

try a new bulb”

Map entries to task procedures

Anticipate & suggest missing

Entries, sub-tasks

Assist with how activities are done in the organization

BEAM

Initiate and report execution

Group and organize entries

User To Do EntriesFind hotel w/ pool for Joe

Reserve conf room for talk

Added by BEAM

Added by user

Deletedby BEAMbecause unnecessary

Markedby BEAMas user-only

Executed& monitored

Buy bread on the way home

Get talk abstract from Joe

OntologiesNGrams

Repository“to schedule a meeting”

SubtasksRepository:

“Reserve X has-subtask

find X”

Organization-specificTask knowledge

“Airport pickup of visitorsis common, but not here”

Action ParaphrasesRepository:

“plan X schedule X”“lease car rent car”

Verb Relations

Repository:“schedule

happens-before reschedule”

From Text ExtractionFrom Web Volunteers From Volunteers in the Organization

From Knowledge Engineers

BEAM Knowledge Sources for NL Interpretation and Assistance

RepairsRepository:

“If projector not working,

try a new bulb”

CALO Task Ontology:Catalog of Automated

Procedures

Execution and Monitoring

InstrumentedDesktop

TaskLearning

Set airport pickup for Joe

Announce room for talk

Reserve accommodations

Host a visitor

Arrange meeting

Identify Omitted Actions

Identify Related Sub-tasks

Toleratesyntactic variety

Interpret entries

10

Our Approach

Extended existing semantic parsing techniques to take advantage of broad-coverage knowledge repositories

Based on the standard Semantic Parsing approach [Acero; Zue; Allen et al]

Syntactic chunking + identification of categories of entities + semantic parsing of annotated result

Our problem is simpler in some ways Allows simplifying assumptions about structure of entries,

speech acts present Our problem is harder in other ways: actions may not be

fully specified, new actions may be automated (learned) To assist with user requests, need to identify implied actions &

sub-actions Leverage large knowledge repositories Leverages paraphrase collection for speech system [Chklovski, K-

CAP05, KCVC-05]

11

BEAM in Year 3: What Can and Cannot Be Interpreted

Statements are processed into their semantic components

Some entries cannot be interpreted because content is not recognized

Paraphrasing knowledge allows rewriting of entries so they can be interpreted

12

BEAM’s Stages of Mapping a User’s To Do Entry

1. Syntactic parsing

2. Identification of semantic

components,present & implied

Volunteercontributed

paraphrases (also task-subtask

pairs)

User’s To Do entry

4. Automatic entry rewriting

yes

no

yes

no

Mapping failed

Rewrites available?

Mapped entry

3. Ontological mapping of semantic

components

Mapped?

Teraword textual frequency repository

ontology

knowledge from volunteers and text extraction

13

BEAM’s Stages of Mapping a User’s To Do Entry

1. Syntactic parsing

5. Identification of sub-tasks

2. Identification of semantic

components,present & implied

Volunteercontributed

paraphrases (also task-subtask

pairs)

User’s To Do entry

ontology

knowledge from volunteers and text extraction

4. Automatic entry rewriting

yes

no

yes

no

Mapping failed

Rewrites available?

Mapped entry

3. Ontological mapping of semantic

components

Mapped?

Teraword textual frequency repository

14

Using BEAM with a To Do Manager: BEAM API

15

Recent Developments: Smarter BEAM Identifies Likely Implied Actions

To Do entry does not specify an action

BEAM looks in 1012 word corpus for mentions of “to * a meeting”, etc, identifying actions

These actions are then mapped to the specific target ontology

http://seagull.isi.edu/cgi-bin/todo-mgmt/api2?q=a%20meeting%20with%20Yolanda;gv=1

16

Leveraging the Teraword Ngrams Source

“quarterly meeting on Monday” “to * a meeting” “presents for John” “to * a present” 4876 buy 642 send

3482 get 619 bring … 593 open 1751 give 499 wrap 1352 find http://seagull.isi.edu/cgi-bin/ngram-extract/query-ngrams.pl

But what if there is little or no data? (Eg, “TGW meeting”) We’re exploring backoff strategies

But what if there is noise in the data? Stoplists (eg, “be”, “have”) can provide some relief Validation & feedback could also help

17

Example of Volunteer-based Validation Completed in 2006

Snapshot of validation: 1,113 harvested statements were put through context-

directed validation for “likely to be purchased in an office categories of times”

Most were filtered out; 107 (9.6%) passed

Determined to be likely to be purchased in an office:0.977        'office supplies' > planners 0.922        'office supplies' > equipment0.674        Business & Industrial > Food Service & Retail >

Bar & Beverage Equipment > CoffeeCategories determined to be unlikely to be purchased in an

office:0.017       Home & Garden > Pet Supplies > Cats > Cat Toys0.000       'building supplies' > 'concrete finishing'0.000       cards > 'racing-nascar'

18

Towards Identifying Subtasks:Start with Large Corpus

“a meeting with Peter” QUERY1: “the [Y] for the [X]” QUERY2: (filter) “need the [Y]” QUERY3: “to [Z] the [X] [Y]”

“approve meeting agenda”, “set/change/confirm meeting time”

Similarly: for “flight to SFO” Top suggestions include: “buy/purchase flight ticket”, “make flight arrangements”

19

Towards Identifying Subtasks:Start with Large Corpus

“a meeting with Peter” QUERY1: “the [Y] for the [X]” QUERY2: (filter) “need the [Y]” QUERY3: “to [Z] the [X] [Y]”

“approve meeting agenda”, “set/change/confirm meeting time”

Similarly: for “flight to SFO” Top suggestions include: “buy/purchase flight ticket”, “make flight arrangements”

Under Development – Stay Tuned

20

2006 Test System: Mappings to Task Ontology Categories

Early integration done to support a test question not otherwise addressed by CALO:

Any small contribution to test results considered a success No lead time for targeted collection for relevant task entries

PQ0166: What instances of task type (choose one) {|sc:%Communicate|, |sc:%Decide|, |sc:%Obtain|, |sc:%PlanAndSchedule|} are on user’s to-do list?

Examples handled by BEAM: arrange travel #PlanAndSchedule::Plan office supplies #Obtain::Buy hire a car #Obtain::Rent respond to Mary's request #Communicate::Answer

This is more powerful than straightforward application of, eg, WordNet synonyms

BEAM handles some situations where there is no synonymy or is-a relation between terms, eg. (arrange travel #PlanAndSchedule::Plan)

BEAM handles some situations where action is not even present(office supplies #Obtain::Buy)

21

Evaluation of Paraphrase Component

Despite early integration, BEAM contributed to evaluation

582 To Do statements collected for CALO Y3 evaluation 31.1% were mappable using the paraphrase repository 24.6% without paraphrase repository

Paraphrase repository was collected without focus on these items specifically

Contained 3,114 items, but (we estimate) only ~100 related to the domain covered by the test question

Additional knowledge sources and larger repositories will support further improvement of performance

Now also have data from public online To Dos, “tada list”

22

Conclusions

Developed BEAM, first system to demonstrate To Do list interpretation to enable automation

Integrated with working ToDo list manager, CALO’s Towel

Extended existing semantic parsing techniques to take advantage of preexisting large knowledge sources

Leveraged volunteers-created paraphrase corpus to improve ToDo entry interpretation

Identify likely implied actions Subtask suggestion in the works

23

Ongoing & Planned Work: New Capabilities

Improve interpretation and mapping capabilities:

Evaluate support of To Do entries which have no verb – ability to identify the implied actions

Proactively identify automatable subtasks for To Do lists

Acquire knowledge about relevant subtasks

Use BEAM’s semantic frames to provide information for task arguments, Towel forms (eg, “travel to Boston”)

Validate with/acquire from volunteers knowledge about sub-tasks, mappings

To Do entry made by user

BEAM mapping to TOWEL task