Text vs. Speech A Comparison of Tagging Input Modalities

for Camera Phones

Research & Development

Mauro Cherubini, Xavier Anguera, Nuria Oliver, and Rodrigo de Oliveira

people do not want to tag their pictures

intro → hypotheses → methodology → results → implications

research question:

Assuming that users are willing to input at least one tag, which input

modality can help the production and retrieval of the pictures?

intro → hypotheses → methodology → results → implications

hypothesis 1

Speech is preferred to text as an annotation mechanism on mobile

phones (objective measure)

Support: - Mitchard and Winkles (2002)

intro → hypotheses → methodology → results → implications

hypothesis 1-bis

Speech annotations are preferred by users even if this means spending more time on the task (subjective measure)

Support: - Perakakis and Potamianos (2008)

intro → hypotheses → methodology → results → implications

hypothesis 2

The longer the tag the larger the advantage of voice over text for

annotating pictures on mobile phones

Support: - Hauptmann and Rudnicky (1990)

intro → hypotheses → methodology → results → implications

hypothesis 3

Retrieving pictures on mobile phones with speech is not faster than with text

(objective measure)

Support: - Mills et al. (2000)

intro → hypotheses → methodology → results → implications

the user study

intro → hypotheses → methodology → results → implications

field study (4 weeks)

controlled experiment

T1 - T2 - T3 - T4

3 experimental conditions: a. Speech only

b. Text only c. Speech and Text

intro → hypotheses → methodology → results → implications

MAMI

intro → hypotheses → methodology → results → implications

features of MAMI

•  processing is done entirely on the mobile phone

•  speech is not transcribed

•  to compare the waveforms of the audio tags, MAMI uses algorithm of Dynamic Time Warping

task 1: remember the tag

intro → hypotheses → methodology → results → implications

stimulus retrieval

Pictures taken during the field trial

task 2: remember the context

intro → hypotheses → methodology → results → implications

stimulus retrieval

TASK 2 PICTURE 1

three little bushes Garden Tree Stairs

task 3: remember the picture

intro → hypotheses → methodology → results → implications

stimulus retrieval

Text Audio tags were converted into

textual tags and vice versa

task 4: remember the sequence

intro → hypotheses → methodology → results → implications

assignment retrieval

TASK 4

Three pictures among the oldest and three pictures among the newest.

metrics

intro → hypotheses → methodology → results → implications

•  time to completion

•  false positives

•  retrieval errors

results H1

intro → hypotheses → methodology → results → implications

results H1-bis

All participants in the BOTH group felt that tagging with text was more effective than tagging with voice.

Voice: 3.33 [0.81], Text: 4.34 [0.81] (Mean [SD]) 1 = completely agree; 5 = completely disagree

intro → hypotheses → methodology → results → implications

results H2

intro → hypotheses → methodology → results → implications

results H3

intro → hypotheses → methodology → results → implications

results H3 - continued

take away 1: �speech is not a given

the advantage of audio as an input modality for tagging pictures on mobile phones is not a given

why? 1. retrieval precision

2. privacy

intro → hypotheses → methodology → results → implications

take away 2: �input mistakes

we address text input mistakes immediately. on the contrary mistakes in audio recordings are less

frequently addressed

intro → hypotheses → methodology → results → implications

take away 3: �memory

speech does not help memorizing the tags

intro → hypotheses → methodology → results → implications

implication 1:�allow multiple modalities

© Pixar, 2008

intro → hypotheses → methodology → results → implications

implication 2:�enable audio inspection

intro → hypotheses → methodology → results → implications

implication 3: �enable modality synesthesia

© Disney, 1940

intro → hypotheses → methodology → results → implications

end�thanks

martigan@gmail.com mauro@tid.es

http://www.i-cherubini.it/mauro/blog/ http://research.tid.es/multimedia/

Research & Development