+ All Categories
Transcript
Page 1: Ad hoc evaluation of SDS along project lifecylcle in Orange

Homonymy

Traceability

Nomadismof evaluation

methodologies !!

and Synonymy

Need : One evaluation

per evaluator

per situation

Difficult communicationbetween actors

No standard forinformation sharing,

cooperationand collaboration !

Local Contraints: Budget,

data collection, etc.

Different norms

depending on the

Communitiesof Practice

Need to refine indicators

+ collected data

Call-flow: entry, path, exit

Callerhistory Call

motive

Referencevalues

Analysis

Framing

Development

Experiments

Pilot

Generalisation

Exploitation

Specification of project scope, specifications and

project plan.

Feasibility evaluation.

The service is maintained by both the hosting

providers and the initial developers (for

corrections). Access is granted to the

overall flow of traffic. .

Service tested under real conditions, with a

fragment of the real flow of traffic.

The evaluation constitute a comprehensive

validation of the service before its access is

extended to all users. It includes the relevance of

the technical architecture.so as to detect and fix the last beta-solution issues.

Test the service so as to identify design issues

and refine the development to the users

practices. Alignment of observation with a priori

projections.

Tests organised on a panel of recruited users.

Development of a prototype integrating the initial design, linguistic, usability and technical

considerations.

Test within the development team to

detect major bugs and enrich the vocal

recongition model,

Analysis of end-users' needs and practices. Solutions benchmark.

Collect the users experience feedback in live conditions.

The service keeps constantky monitored. Dashboards containing high-level indicators are

sent daily to the various interested teams (number

EvaluationContexts

What is done?

Evaluation context?

Marketingexperts

Hostingproviders

Ergon

omic

s

expe

rts

Vocal

experts

Use

r ex

perie

nce

Pla

tfo

rm in

tera

ctio

ns

Tec

hn

ical

per

form

ance

SD

S- U

sers

inte

ract

ion

sS

erv

ice

%

46,2

3

12

3658

14

What is aSpoken

Dialogue System?

Viewpoints

Métier TechnicalDevelopers

ErgonomicsExperts Marketing

Experts

BusinessManagers

HostingProviders

GoalsEvaluationsIndicators

ContractingOwners

Hard/software

platform delivering

a service to

the customer

Automate routing service satisfying

the customer relationship

management

Hard / software platform

Interactive systemsimplying a human-amchine dialogue,

a QoS & an individualised

access to information

A technology basedservice tailored to

the various business customers' needs

Hard / software platform

Service automating

the human

operators low

added-value

tasks

Enhance the customer service with

automated vocal solutions

Deliver a service to the customer with respect to

the customer relationship policy

Ensure the respect of the predefined QoS

commitments and the end-user satisfaction.

ROI (project financing decisions), QoS,

dialogue performance, user satisfaction

Evaluate the service quality and its adequacy

to the organisation

Track the end-user experience

Expert evaluation campaign (field tests

and real users)

Measure of the service performance and the

users' perception.

Measure of the service performance, monitor

live services and technical support

Measure of the service performance and the

users' perception

Monitor the good functionning of the live

services they host

Responsible of the service hosting and

maintenance.

Enhance the organisation

Compare the adequation of the service

performance with the organisation needs

Compare the adequation of the service

performance with the users needs

Answer the users' expectations.

Tailor the development with a user-centric

development approach

Monitor the service perceived quality

Control the service performance, monitor

and correct anomalies.

Indicators linked to the service design

Develop the service

Marianne Laurent, Philippe Bretier, Carole Manquillet

Ad Hoc evaluations along the lifecycle of industrial spoken dialogues systems: heading to harmonisation?

[email protected], [email protected], [email protected] Labs, 2 avenue Pierre Marzin, 22307 Lannion, France

Top Related