Date post: | 03-Jan-2016 |
Category: |
Documents |
Upload: | josephine-rachel-lewis |
View: | 220 times |
Download: | 3 times |
Dialogue Annotation at SUNY
Hilda Hardy
Amitiés Annotation WorkshopLIMSI-CNRS, France
July 8-10, 2002
2
What did we annotate?
Transcriptions of dialogues from 3 folders in the englishPlainTextDialogues.zip archive (dated March 28, 2002):
– 22mar02, 20 of 60 files (Uzo Enyinna)– 22mar02, 21 of 60 files (Hilda Hardy)– 26feb02, 30 of 64 files (Tom Palen) – 26mar02, 36 of 36 files (Hilda Hardy)
3
What approach did we use?
We annotated the Semantic and Functional layers of the dialogues (not the Stylistic layer), using XDMLTool version 2.0.
4
5
Why annotate Semantic and Functional Aspects of Real Call-Center Dialogues?• Get ideas and data for designing a better, more
human-like Dialogue Manager– Find out structures of dialogues
• Basic patterns of Dialogue Acts• Basic flow of the dialogue• What is essential in accomplishing the task?• What is parenthetical?
– Find out how information is exchanged• How is information broken down?• How is information ordered in a dialogue?
– Discover relationships between functional and semantic layers
6
Why annotate Semantic and Functional Aspects? Cont’d
– What are the most common questions customers ask agents?
– What facts do agents need to learn from customers? What facts do they give customers? We need supplementary information from the call centers to handle these aspects.
• Improve language generation– How do agents converse with customers? Look at
phrasing, length of turns
• Decide what can be automated
• Other: a well-annotated corpus is a valuable contribution to the research community
7
Semantic Annotation: Customer AccessFrames
Customer transactions: every dialogue has a purpose: at least one topic or task that the customer needs to have answered or satisfied.
Task Information
AcctBalanceStatement
Insurance
MakePayment
ChangeAddress
CancelDebit
CloseAccount
LostCard Fees
8
Semantic Annotation: Agent AccessFramesAgent transactions: tasks initiated by the agent in the process of satisfying the customer’s needs
GreetCaller
VerifyId
“Good afternoon, . . . How can I help?”
Information required for security checks: name, old post code, date of birth, usual method of payment, etc.
Closure “Is there anything else? . . . Goodbye”
9
Semantic Annotation: Attributes, Values and Modifiers
Most frames are associated with details that may be organized into key-value pairs. Examples:
AccessFrame Attribute Value Modifier
Name Paul Agent
PostCode AB1 2CD New
HouseNumber 10 New
GreetCaller
ChangeAddress
10
Semantic Annotation: General Shape of a Call Center Dialogue
A typical call center dialogue progresses with this general order of frames:
GreetCaller
ChangeAddress
VerifyId
Info-Insurance MakePaymentor or
Closure
11
A: Good afternoon customer services Paul speakingC: Hello I want to inform you of my change of address pleaseA: Certainly what name is it please?C: Name? It's Mrs Pauline SmithA+C: Right Mrs Hunter I'm just keying in the account number for your bear with me a momentUh hmmA: (pause)Could you confirm the, your previous postcode and telephone number please?C: Yeah the previous one was AB1 2CD01111 111111A: Thank you, you're speaking to Paul, good afternoonC: Uh hmmA: Right so it's Paul and PaulineC: (laughs)A: The new address, is it a house number Mrs Smith?C: Yeah it's number 3 A: And the postcode there is?C: W for Whiskey X for X-ray two ohA: Yes
Semantic Annotation: example
GreetCaller
ChangeAddress
VerifyId
GreetCaller
ChangeAddress
12
Semantic Annotation: example, cont’dC: Three Y for Yankee Z for ZuluA: And is that The Oaks?C: That's rightA: The Hall?C: That's itA: Do you have a new telephone number?C: Yes I do it's erm 01111A: YesC: 111A: YesC: 111A+C: And this is your Principles card isn't it?YeahA+C: Have you got any other shop cards that you need to changeNo, no I haven't noA: FineA+C: Is there anything else I can do for you?OkayC: That's all thank you very muchA: Thanks for your callC: Thank you byeA: Bye
(END OF CALL)
ChangeAddress
Closure
13
Semantic Annotation with XDMLTool
Semantic categories
14
Semantic Annotation with XDMLTool
• Semantic annotation is domain-dependent– But the categories AccessFrame, Attribute, Value,
and Modifier were designed to be domain-independent.
– For a new domain, use new lists of tags.
• Helpful to begin with a set of labels developed during preliminary annotation
• Necessary to allow the annotator to add to the lists as new information appears in the data
15
Semantic Annotation: Suggestions
• Compare annotators’ lists and merge labels that are clearly the same– ChangeAddress and UpdateAddress– VerifyId and VerifyCallerID– Card and NameofCard
• Discuss how to handle cases that are not so obvious: What level of detail is best?– Address = StreetAddr, Area/Town, City, County?
• Try to finalize lists as far as possible, leaving room for new labels
16
Functional Annotation with XDMLTool
Functional categories
17
Functional Annotation: General Approach
• Annotate dialogue act functions (dialogue-related speech acts, or DAs)
• Use DAMSL (Dialog Act Markup in Several Layers), tailored to reflect call center dialogue features– Both the categories and the lists are domain-
independent– Minor modifications to DAMSL are necessary– Annotation conventions must be discussed
18
DAMSL Amitiés/XDML (proposed)
Communicative Status (Features)Self-talk used occasionally
Uninterpretable not used (seems better suited
to transcription)
Abandoned used occasionally
Non-verbal (keyboard
noise, music, dog barking)
Third-party-talk
(side conversations)
19
DAMSL Amitiés/XDML (proposed)
Information-LevelTask annotated by default
Task-management not used (rare; agent reacts
to the customer)
Communication-mgt used as defined
Other-level Out-of-topic
(jokes, non-sequiturs, small talk, and any comments
or remarks that have no direct bearing on accomplishing
the task)
20
DAMSL Amitiés/XDML (proposed)
Forward Looking Function Statement
Assert used for simple assertions,
such as answers
Explanation (agent explains
policies, customer gives reasons)
Recap (agent summarizes, wraps
up, or recapitulates the task)
Reassert used as defined
Other-statement not used
21
DAMSL Amitiés/XDML (proposed)
Influence-on-listenerAction-directive used as defined
Open-option used as defined (when a course of
action is suggested with no obligation on the listener)
Info-request used only to request information,
not to elicit a yes/no response
Confirmation-request (needs a
yes/no response)
22
DAMSL Amitiés/XDML (proposed)
Influence-on-speakerOffer used as defined
Commit used as defined
ConventionalOpening used as defined
Closing used as defined
Explicit-performative
Exclamation Expression (thank you, okay,
Other-forward-function no problem, smashing, sorry)
23
DAMSL Amitiés/XDML (proposed)
Backward Looking Function Agreement
Accept
Accept-part
Maybe used as defined
Hold
Reject
Reject-part
I_don’t_know
24
DAMSL Amitiés/XDML (proposed)
UnderstandingSignal-non-understanding Non-understandingSignal-understanding
Acknowledge used as defined*Repeat-rephrase used as definedCompletion used as defined
Correct-misspeaking Correction
Answer used as defined*Same definition as “Backchannel”.NOTE: we recommend removing “Confirm” and
“Disconfirm” (same as Accept and Reject)
25
Annotation: Other Issues
• When should an annotator split (segment) a turn?– Semantic data must be annotated separately:
C: Post code AB1 2CD, phone 01111 111111
– One turn may have different functions:
C: Hi there erm I’ve lost my card and also I’ve changed address
– An overlapping turn has different speakers:
A+C: It’s Top Shop account card
Top Shop thanks
26
Annotation: Other Issues
• Leave the turn intact when it begins with an acknowledgment. Info-level is “Task” and not “Communication-mgt.”
A+C: Top Shop thanks
A: Okay what’s your surname?
A: Ok and your address, the old one that is?
27
Automatic tag selectionsSeveral of the DAMSL tags are contained in others or require other selections to be made. For example, "answers by definition will always be asserts." (Draft of DAMSL, p. 23). For added convenience, XDMLTool v. 2.0 makes the following additional selections by default. If exceptions arise, the user may change the automatic selections manually.
When annotator selects... ... XDMLTool selects...
any Backward Communicative Function Previous utterance number as "Response-to" value Conventional->Opening Info-level->Communication-mgt Conventional->Closing Info-level->Communication-mgt Understanding->Acknowledge Info-level->Communication-mgt Understanding->Backchannel Info-level->Communication-mgt Understanding->Non-understanding Info-level->Communication-mgt Answer Statement->Assert Agreement->Accept Answer, Statement->Assert Agreement->Accept-part Answer, Statement->Assert Agreement->Reject Answer, Statement->Assert Agreement->Reject-part Answer, Statement->Assert