Introduction to the TrindiKit

ESSLLI, Helsinki20th Aug 2001

Staffan Larssonsl@ling.gu.se

What is TrindiKit?

• a toolkit for – building and experimenting with dialogue move engines and systems, – based on the information state approach

• not a dialogue system!

• short demo• architecture & concepts• what’s in TrindiKit?• building a system

This lecture

Short demo

• web-based demo + simple GUI• information state view

– record– dialogue– update rules

• module view– active modules highlighted– asynchronous processing

Architecture & concepts

module1 module…

Total Information State (TIS)•Information state proper (IS)•Module Interface Variables•Resource Interface Variables

resource1

control

modulei modulej module… modulen

resource… resourcem

DME

• an abstract data structure (record, DRS, set, stack etc.)

• accessed by modules using conditions and operations

• the Total Information State (TIS) includes– Information State proper (IS)– Module Interface variables– Resource Interface variables

Information State (IS)

• module or group of modules responsible for – updating the IS based on observed moves– selecting moves to be performed

• dialogue moves are associated with IS updates using IS update rules– there are also update rules no directly associated with any

move (e.g. for reasoning and planning)

• update rules: rules for updating the TIS– rule name and class– preconditon list: conditions on TIS– effect list: operations on TIS

• update rules are coordinated by update algorithms

Dialogue Move Engine (DME)

• Modules (dialogue move engine, input, interpretation, generation, output etc.) – access the information state– no direct communication between modules

• only via module interface variables in TIS• modules don’t have to know anything about other modules• increases modularity, reusability, reconfigurability

– may interact with user or external processes

• Resources (device interface, lexicons, domain knowledge etc.)– hooked up to the information state (TIS) – accessed by modules– defined as object of some type (e.g. ”lexicon”)

Modules and resources

What’s in TrindiKit?

What does TrindiKit provide?• High-level formalism and interpreter for

implementing dialogue systems– promotes transparency, reusability, plug-and-play,

etc.– allows implementation and comparison of dialogue

theories – hides low-level software engineering issues

• GUI, WWW-demo • Ready-made modules and resources

– speech– interfaces to databases, devices, etc.– reasoning, planning

• a library of datatype definitions (records, DRSs, sets, stacks etc.)– user extendible

• a language for writing information state update rules

• GUI: methods and tools for visualising the information state

• debugging facilities– typechecking– logs of communication modules-TIS– etc.

TrindiKit contents (1)

• A language for defining update algorithms used by TrindiKit modules to coordinate update rule application

• A language for defining basic control structure, to coordinate modules

• A library of basic ready-made modules for input/output, interpretation, generation etc.;

• A library of ready-made resources and resource interfaces, e.g. to hook up databases, domain knowledge, devices etc.

TrindiKit contents (2)

Special modules and resources included with TrindiKit

• OAA interface resource– enables interaction with existing software

and languages other than Prolog

• Speech recognition and synthesis modules– TrindiKit shells for off-the-shelf recognisers– currently only ViaVoice, but more on the way

• Possible future modules:– planning and reasoning modules– multimodal input and output

Asynchronous TrindiKit

• Internal communication uses either– OAA (Open Agent Architecture) from SRI, or– AE (Agent Environment), a stripped-down

version of OAA, implemented for TrindiKit

• enables asynchronous dialogue management– e.g.: system can listen and interpret, plan

the dialogue, and talk at the same time

How to build a system

TRINDIKIT

dialogue theory(IS, rules, moves etc)

domain knowledge(resources)

domain-specificsystem

Relation TrindiKit – dialogue system

domain-independentDME

software engineering(basic types, control flow)

• Come up with a nice theory of dialogue• Formalise the theory, i.e. decide on

– Type of information state (DRS, record, set of propositions, frame, ...)

– A set of dialogue moves– Information state update rules, including

rules for integrating and selecting moves– DME Module algorithm(s) and basic control

algorithm – any extra datatypes (e.g. for semantics:

proposition, question, etc.)

Building a domain-independent Dialogue Move Engine

Domain independence of the Dialogue Move Engine

• The DME is domain independent, given a certain type of dialogue– information-seeking – instructional– negotiative– ...

• Domain independence of DME is not enforced by TrindiKit, but is good practice– promotes reuse of components– forces abstraction from domain-specific details,

resulting in a more general theory of dialogue

Specifying Infostate type

• the Total Information State contains a number of Information State Variables– IS, the Information State ”proper”– Interface Variables

• used for communication between modules

– Resource Variables• used for hooking up resources to the TIS, thus

making them accessible from to modules

• use prespecified or new datatypes

sample infostate type declaration

infostate_variable_of_type( is, IS ) :- IS = record( [

private : Private, shared : Shared ] ),

Shared = record( [ com : set( proposition ), qud : stack( question ), lm : set( move ) ] ),

Private = record( [ agenda: stack( action ), plan : stackset( action ), bel : set( proposition ),

tmp : Shared ] ) ] ).

resulting infostate type

PRIVATE :

PLAN : stackset( Action )

AGENDA : stack( Action )

SHARED :

BEL : set( Prop )

TMP : (same type as SHARED)

COM : set( Prop )

QUD : stack( Question )

LM: set( Move )

Sample interface variable type declarations

interface_variable_of_type( input, string ).

interface_variable_of_type( output, string ).

interface_variable_of_type( latest_speaker, speaker ).

interface_variable_of_type( latest_moves, set(move) ).

interface_variable_of_type( next_moves, set(move) ).

Specifying a set of moves

• amounts to specifying objects of type move (a reserved type)– there may be type constraints on the

arguments of moves

• preconditions and effects of moves– formalised in update rules, not in the move

definition itself– a move may have different effects on the IS

depending e.g. on who performed it

sample move specifications

% Social

of_type( quit, move ).

of_type( greet, move ).

of_type( thank, move ) .

% Q&A

of_type( ask(Q), move ) <- of_type( Q, question ).

of_type(inform(P), move ) <- of_type( P, proposition).

of_type( answer(R), move ) <-

of_type( R, proposition) or

of_type( R, ellipsis ).

Writing rules

• rule = conditions + operations– if the rule is applied to the IS and its

conditions are true, the operations will be applied to the IS

– conditions may bind variables with scope over the rule (prolog variables, with unification and backtracking)

A sample rule

rule( integrateUsrAnswer, [ $/shared/lu/speaker = usr,

assoc( $/shared/lu/moves, answer(R), false ),

fst( $/shared/qud, Q ),

$domain : relevant_answer( Q, R ),

$domain : reduce(Q, R, P)

], [

set_assoc( /shared/lu/moves, answer(R),true),

pop( /shared/qud ),

add( /shared/com, P ) ] ).

A sample rule (old syntax)

• rule( integrateUsrAnswer, [ – val#rec( shared^lu^speaker, usr ), – assoc#rec( shared^lu^moves, answer( R ), false ),– fst#rec( shared^qud, Q ), – domain :: relevant_answer( Q, R ), – domain :: reduce(Q, R, P) – ], [ – set_assoc#rec( shared^lu^moves, answer(R),true), – pop#rec( shared^qud ), – add#rec( shared^com, P ) ] ).

Writing rules

• available conditions and operations depend on the infostate type– the infostate is declared to be of a certain

(usually complex) type

• datatype definitions provide– relations: Rel(Arg1, …, ArgN)– functions: Fun(Arg1, …, ArgN,Result)– operations:

Op(ObjIn,Arg1, …, ArgN,ObjOut)

• New datatypes may be added

Writing rules: locations in TIS

• objects may be specified by giving a path to a location in the infostate; – paths are specified using selectors, which are similar

to functions• $Fun2($Fun1) ~ $Sel1/Sel2 • $fst($/shared/qud) ~ $/shared/qud/fst

– ”$” evaluates a path and gives the object at the location specified

• example: – is/shared/com is a path, pointing to a location in

the TIS– $is/shared/com is the object in that location– the is can be left out, giving $/shared/com

Writing rules: conditions (1)• conditions do not change the information state• if a condition fails, backtracking ensues• condition syntax (incomplete)

– Rel(Arg1, … , ArgN), e.g. • fst($/shared/qud,Q)

– Arg1:Rel(Arg2,…,ArgN), e.g. • $/shared/qud:fst(Q)• $domain:relevant_answer(Q,A)

– Arg1 = Arg2• Q = $fst($/shared/qud)

– Cond1 and Cond2– Cond1 or Cond2– not Cond1– forall(Cond1, Cond2)– (Arg is object or prolog variable)

Writing rules: conditions (2)

• quantification, binding and backtracking– if an instantiation a of a variable V in a condition C is

found that makes condition C true, V is bound to a– backtracking occurs until a successful instantiation of

all variables in the list of conditions has been found

• example list of conditionsfst($/shared/qud,Q), in($/shared/com,P),$domain:relevant_answer(P,Q)

• Explicit quantificationQ.P. fst($/shared/qud,Q) & in($/shared/com,P) & $domain:relevant_answer(P,Q)

Writing rules: operations

• operations change the information state• if an operation fails, an error is reported• variable bindings survive from conditions to

operations• operation syntax (incomplete)

– Op(Path,Arg1,…,ArgN)• push(/shared/qud, Q)

– Path : Op(Arg1, … ,ArgN)• /shared/qud : push(Q)

– Store := Fun(Obj,Arg1,…,ArgN)• /private/tmp/qud := push($/shared/qud,Q)

Specifying update algorithms

• uses rule classes• constructs include

– Rule– RuleClass– if Cond then S else T– repeat R until C– repeat R– try R– R orelse S– test C– SubAlgorithm

Sample update algorithmgrounding, if $latest_speaker == sys then

try integrate, try database, repeat downdate_agenda, store

else repeat

integrate orelse accommodate orelse find_plan orelse

if (empty#rec( private^agenda ) then manage_plan else downdate_agenda

repeat downdate_agendaif empty($/private/agenda))then repeat manage_plan

repeat refill_agendarepeat store_nim try downdate_qud

Specifying serial control algorithms

• serial constructs include– Module{:Algorithm}– if Cond then S else T– repeat R until C– repeat R– try R– R orelse S– test C– SubAlgorithm

Specifying concurrent control algorithms

• Agent1 | Agent2 | … | AgentN

• where Agenti is • AgentName : {

– import Module1 ,–

… – import Modulep ,– Trigger1 => SerialAlgoritm1 ,– …– Triggerm => SerialAlgoritmm }

• triggers:– condition(C) (C is a subset of the full condition set)– init– new_data(Stream)

Sample control algorithm (1)

repeat ( [ select,

generate,

output,

update,

test( $program_state == run ),

input,

interpret,

update ] )

Sample control algorithm (2)input: {

init => input:display_prompt, new_data(user_input) => input }

| interpretation: { import interpret, condition(is_set(input)) => [ interpret, print_state ] }

| dme: { import update, import select, init => [ select ], condition(not empty(latest_moves)) => [

update, if val(latest_speaker,usr) then select else []

] }

| generation: { condition(is_set(next_moves)) => generate }

| output: { condition(is_set(output)) => output } )).

From DME to dialogue system

Build or select from existing components: • Modules, e.g.

– input– interpretation– generation– output

• Still domain independent• the choice of modules determines e.g.

the format of the grammar and lexicon

Domain-specific system

Build or select from existing components:• Resources, e.g.

– domain (device/database) interface– dialog-related domain knowledge, e.g. plan

libraries etc.– grammars, lexicons

Building resources

• Resource– the resource itself; exports a set of predicates

• Resource interface– defines the resource as a datatype T, i.e. in terms of

relations, functions and operations

• Resource interface variable– a TIS variable whose value is an object of the type T

• By changing the value of the variable, resources can be switched dynamically– change laguage– change domain

sample resource variable type declarations (incl. resource interface)

resource_type( lexiconT ).

resource_variable_of_type( lexicon, lexiconT ).

of_type( lexicon_travel_english, lexiconT ).

of_type( lexicon_autoroute_english, lexiconT ).

of_type( lexicon_travel_svenska, lexiconT ).

of_type( lexicon_cellphone_svenska, lexiconT ).

resource_condition(lexiconT,input_form(Phrase,Move),Lexicon) :- Lexicon : input_form( Phrase, Move ).

resource_condition(lexiconT,output_form(Phrase,Move),Lexicon):-

Lexicon : output_form( Phrase, Move ).

• Explicit information state datastructure makes systems more transparent

• Update rules provide an intuitive way of formalising theories in a way which can be used by a system

• Domain knowledge encoded in resources; – the rest of the system is domain independent– resources can be switched dynamically

• Modular architecture promotes reuse

TrindiKit Features

Features, cont’d

• Allows both serial and asynchronous systems• Interfaces to OAA• Generic WWW interface • Runs on UNIX, Windows, Linux• Needs SICStus Prolog• Version 2 .0 is available, next version coming

soon (SIRIDUS)• www.ling.gu.se/projects/trindi/trindikit• Larsson & Traum: NLE Special Issue on Best Practice in

Dialogue Systems Design, 2000

• GoDiS – information state based on Questions Under Discussion (Larsson et al 2000)

• MIDAS – DRS information state, first-order reasoning (Bos & Gabsdil, 2000)

• EDIS – information state based on PTT (Matheson et al 2000)

• SRI Autoroute – information state based on Conversational Game Theory (Lewin 2000); robust interpretation (Milward 2000)

Systems developed using TrindiKit

Post-TRINDI applications

• SIRIDUS project (EU 2000-)– command and negotiative dialogues– Spanish, Swedish– GoDiS, SRI system

• D’Homme (EU 2001)– Dialogues in the Home Environment– GoDiS, SRI system

• Instruction Based Learning for mobile robots (U Edinburgh)– MIDAS

• Tutoring Electricity (U Edinburgh)– EDIS, Beetle

• ???selectors – selects an object (Obj) embedded in

another object (Arg)– selector(Sel,Arg,Obj,ArgWithHole,Hole).

– e.g. selector(fst,stack([E|S]),E,stack([H|S]),H).

– Every selector corresponds to a function function(Sel,[Arg],Object).

Datatype definitions

• relations– relations between objects; true or

false– format: relation(Rel,Args).– Example

• definition: relation(fst,[stack([E|S]),E]).• condition: fst($/shared/qud,Q)

Datatype definitions

• functions– functions from arguments to result– format: function(Fun,Args,Result).– Example

• definition: function(fst,[stack([E|S])],E). • in condition:

– Q = fst($/shared/qud)– Q = $/shared/qud/fst

• in effect: – next_move/content := fst($/shared/qud)

– every function corresponds to a relation relation(Fun,[Args@[Result]]).

Datatype definitions (3)

• operations– operation(Op,InObj,Args,OutObj).

– e.g. operation(push,stack(S),E,stack([E|S])).

– every operation corresponds to a relation relation(Op,[InObj|Args]@[OutObj]).

???Components of a system

• Infostate [domain independent]– type declaration– initial state

• Modules [domain independent]– rules– rule application algorithms– overall control algorithm

• Resources [domain dependent]– incl. resource interfaces