Adapted typographical annotations for language learning: user-profiled design

solutions to problems of auditory perception and speech production

Anthony Stenton, Lairdil, Université Toulouse I

A3H@AH’06 anthony.stenton@univ-tlse1.fr

CNRS Project 2003-2005 : TCANTraitement Cognitif Apprentissage et NTIC

• The development of a prototype authoring system using visual cues for improving the perception of stress patterns in foreign languages, in particular in English

• SWANS = Synchronised Web Authoring Notation System

Fields of study of SWANS project specialists

• Linguistics• Auditory Perception • Cognitive Psychology• Computer Science• Document design specialists

Laboratories involved

• LAboratoire Interuniversitaire de Recherche en DIdactique des Langues, (LAIRDIL) Toulouse,

• Anthony Stenton, Anne Péchou, Gail Taillefer, Nicole Décuré, Antoine Toma, Christine Vaillant Sirdey

• Laboratoire Travail et CognitionUMR CNRS 5551, Toulouse

• André Tricot

• Laboratoire de Neuropsycholinguistique Jacques-Lordat, EA 1941 Toulouse

• Pascal Gaillard, Michel Billières, Angelika Rieussec

• Laboratoire d'Analyse et d'Architecture des Systèmes, LAAS CNRS 7, Toulouse

• Saïd Tazi

Equipe Automatique des Procédés Biotechnologiques, Laboratoire de Biotechnologies-Bioprocédés, INSA Toulouse

• Nabil Kabbaj

Project objectives

• Measurable progress in stress perception, and oral production

• The development of an authoring system for synchronising and annotating texts for an adaptive multilingual context

• Allowing teachers to annotate without having to master advanced programming skills

The problem of French student oral presentations in English at advanced levels

• Fluent

• Students find their words easily

• Appropriate use of language

• But…….

Random stress patterns

• Most francophone speakers do not consciously store information on stress patterns. During oral presentations , stress is placed randomly on the second or third syllable of the word “ deVELopment ” indicating not ‘deafness’ or a problem of production but negligence of memory storage.

• Communication depends on “tolerance thresholds”. It can breakdown after only a few stress errors.

The challenges

To define the rules of English stress (Alain Deschamps 94, d’après Guierre) and an appropriate learning method (Théo van Leeuwen : Visual Design, Rudolph Arnheim : Visual Weight)

To develop protocols for evaluating learning (Sweller, Mayer, Paivio)

To develop an ‘Intelligent Environment for Human Learning’ where visual perception can help improve auditory perception

To develop a prototype authoring system to make the process of textual annotation semi-automatic

70% to 90% of our French students get the accent wrong

/

–Why ?

Mother-tongue interference

• French stress is on the right (a codachrone language)

• English stress is on the left (a capochrone language)

• Reading and subvocalizing ( silent reading) can reinforce bad habits

A psychological barrier

• In terms of pronunciation, one might argue, the Hundred Years’ War is far from ended. Many French students refuse to leave the comfortable foothills of their own relatively low pitch frequencies for a roller coaster ride in English high frequencies. These are French or Latin words on the screen. Why on earth should I employ such unnatural stress patterns?

Are the problems with English stress patterns the same for all nationalities?

• No

• In Czech, Latvian, Hungarian, Swiss German (Bernese dialect), Finnish, and Swahili, stress is always placed on the first syllable.

• In French, Spanish, Portuguese, Turkish, and Polish stress is rarely placed on the first syllable.

Italian compound nouns

• Stress usally on the second element

• Transfer to English loan words leads to

• « airBAG » instead of « AIRbag »

Textual annotation must adapt to student mother-tongues

• Language centres must offer texts which place annotations on words where English the stress pattern will probably cause problems of perception for novice learners according to their mother-tongue

A battering ram of a solution

• It is suggested that adult L2 learners might counter L1 interference effects through managing to receive “exaggerated acoustic cues, multiples instances of many talkers, and massed listening experience” (McCandliss et al 2002; McClelland et al.1999)

Hypothesis of dual coding (Paivio)

• Dual coding (sound + text + paralinguistic markers) should improve learning for novice learners while having no effect or a negative effect for those who already know the place of the accent.

Textual annotation techniques for learning pronunciation

• How far can we go ?

• When does the brain start to complain because reading becomes too slow or because codes seem too obscure ?

• What constitutes cognitive overload ?

Robertson method circa 1850

The Robertson Method

Brazil 1994

Sounds Right 2002

Can typography change ?

• Is our software capable of offering new more memorable techniques for learning pronunciation ?

• Are we ready for a more flexible, animated and colourful alphabet ?

Annotation techniques

A text annotated with Swans

Question Time in Parliament annotated with SWANS

A Spanish text

German

Cimabue 1300 Rogier van der Weyden 1450

Does reading on a computer screen change learning ?

Simultaneous reading and listening

• Cinema : 1931 subtitles to the ‘Jazz Singer’ in Danish

• BBC television 1938, opera + subtitles

• Karaoke :1970’s

• Language Learning : Audio Partner Teleste 1995

How does memory behave after dual coding ?

Should synchronised reading be intensive or extensive ?

Does synchronised reading inhibit ‘L1 interference’ during silent reading?

How does memory behave when we change colours?

How does memory behave when text moves ?

Three « levels of representation for words in memory » (= triple storage):

• conceptual level • semantic-syntaxical level (lemmes)• formal level (lexèmes)• Willem Levelt, Modèle de la production

langagière • (résumé from de Bot, Paribakht & Wesche

1997, 312)

Levelt’s analysis is based on student reading of paper documents

• Toulouse tests of dual coding at the word level and at the sentence level constitute an early and limited analysis of the potential of the new plasticity in computer based reading

• Over 200 students participated

• Sound recordings of student pronunciation were stored to examine stress perception

Tests with students in Toulouse

LevelCoding techniques

A.Written test

B. Written test

C. Oral

‘Experts’2nd year under-

graduates

1. audio coding only

9,26 8,69 7,55*

2. Dual coding 7,45 6,50 5,19

Intermediary1st year under-

graduates

1. audio coding only

8,23 6,23 7,36

2. Dual coding 8,37 7,11 7,17*

‘Novices’High school

1. audio coding only

5,89 4,41 7,42

2. Dual coding 7,62 4,87 7,14

Problem : The adult brain is conditioned by typography

• « Freeing ourselves from chirographic and typographic bias is probably more difficult than any of us can imagine » Ong 1982

Annotation techniques

Synchronisation with SMIL

• The arrival of SMIL (Synchronised Multimedia Integration Language) in 1999 and the work of the W3 Consortium - ‘Timed-Text Initiative’ (2003) - constitute an important breakthrough. SMIL allows fine-tuned synchronisation of sound and text combined with a heightened degree of typographical plasticity for on screen textual annotation.

How do you synchronise sound and text ?

Problem : the absence of annotated and synchronised texts

• Programming ‘by hand’ with ‘Magpie’ and ‘Dreamweaver’, it takes two hours to annotate and synchronise a short text with a film or a sound file and to transfer the code into a web page.

• Early testing required 50 documents in 3 languages : English, Spanish & German

Solution 2005: SWANS

• Synchronised Web Authoring Notation System• The programme generates SMIL code which is

placed automatically in web pages to allow students to study stress by reading and listening

• The generation of annotated and synchronised web pages has been cut from 2 hours per one page document to 10 minutes.

The four stages of authoring with SWANS

SWANS : Synchronised Web Authoring Notation System

Annotation with SWANS

Can annotation become automatic?

• Like spelling correctors accuracy levels are variable but ‘acceptable’

• Method 1 rule-based expert system

• According to Deschamps we can deduce the stress pattern of a word from its spelling in 90% of cases. Some rules, such as those concerning endings with <_ic, itis, osis> or the <ION> rule, are close to a 100% (Deschamps p.33, 1994) and are relatively easy to implement. Accordingly all such character chains occurring in a given text can be temporarily highlighted in SWANS to aid annotation. Unstressed suffixes can be highlighted in red (e.g.  -able, -age, -ance, -ancy, ant, -cy, -ful, -hood, -ist, -ise, -ize, -less, ‘ly, ment, -ness, -or/-er, -ous) while stressed suffixes (-ade, -aire, -ee) are highlighted in blue if the user desires.

Can annotation be automatic ?

• Method 2 voice recognition techniques The integration of voice recognition techniques

giving raw and optimised phonemic transcriptions adapted from the PERL algorithms of the Aix-Marsec project is a more complex challenge. The Aix-MARSEC project time-aligns speech data at the phoneme and syllable level. Aix- MARSEC tools consist of a set of reference files (grapheme-phoneme conversion dictionaries) and multiplatform Praat and Perl scripts.

Can annotation become automatic?

• Method 3 text banks of annotated texts• a bank of pre-annotated scripts, from 2004-

2005 3-minute video news bulletins, offers an alternative to rule-based or dictionary based systems (e.g. 71,000 entries of the Oxford Advanced Learner’s using the SAMPA alphabet) but, of course, requires manual adjustment for ambiguous cases (“convict” = the verb // or the noun / ?) .

Conclusions

• Synchronised reading based on dual coding techniques opens promising new perspectives for working on pronunciation in an adaptive multilingual context.

• Mother-tongue based annotation techniques offer an adaptive solution to individual learning needs which deserves further exploration in university language centres.

The ‘extensive reading hypothesis’

• Extensive reading of synchronised text can help students to inhibit interference from the L1

• Perception of stress improves in novice learners exposed to sound and annotated text.

• Improved oral production cannot be demonstrated in the short term and may take time to appear. Explicit knowledge of stress patterns is a new springboard for turning savoir into savoir faire and savoir être.

Does SWANS announce the Swan-Song of the Gutenberg tradition for language learners

studying pronunciation ?

• Resistance to changing reading habits may ensure an erratic evolution of reform but we believe the way is open for new experiments with intensive and extensive reading on the computer screen with potentially important repercussions for perception and oral production.

Will teachers use SWANS ?

• The eternal debate … should teachers become authors of on-line resources ?

• Testing will take place in several European language centres affiliated to CERCLES in 2006