Melissa A+ BowlesUniversity of Illinois at Urbana-Champaign
Ronald P+ LeowGeorgetown University
The present study addresses the reactivity of two types of verbal pro-tocols in SLA research. It expands on the work of Leow and Morgan-Short (2004), who found nonmetalinguistic verbalization during asecond-language reading task to be nonreactive for beginning learn-ers’ text comprehension, intake, and production of the targeted mor-phological form. The present study investigated the reactivity of bothmetalinguistic and nonmetalinguistic protocols, using a syntactic struc-ture and advanced language learners of Spanish. Results indicatedthat neither type of verbalization significantly affected text com-prehension or written production of old or new exemplars of thetargeted structure when compared to a control group, although meta-linguistic verbalization appeared to cause a significant decrease intext comprehension over nonmetalinguistic verbalization. Further-more, both types of verbalization significantly increased the amountof time on task.
Since the 1980s, there has been a marked increase in the use of verbal reportsto study cognitive processes, so much so that “both concurrent and retrospec-tive verbal reports are now generally recognized as major sources of data on
Address correspondence to: Melissa A+ Bowles, Department of Spanish, Italian, and Portuguese, Uni-versity of Illinois, 4080 Foreign Language Building MC-176, 707 S+ Mathews Ave+, Urbana, IL 61801+
SSLA, 27, 415–440+ Printed in the United States of America+DOI: 10+10170S0272263105050187
subjects’ cognitive processes in specific tasks” ~Ericsson & Simon, 1993, p+ xi!+To gather information about subjects’ cognitive processing, thought processes,and strategies, the collection of verbal reports has become standard in manyfields, ranging from accounting ~Anderson, 1985! and marketing ~Biehal &Chakravarti, 1989! to software engineering ~Hughes & Parkes, 2003! and lan-guage research ~Cohen, 1986; Leow, 1997; Rosa & O’Neill, 1999!+ Such verbalaccounts, it is argued, are valuable because of the difficulty inherent in cap-turing the essence of mental processes+ Notwithstanding the apparent valueof verbal reports, the validity of such reports as a measurement of cognitiveprocesses has been debated in non-SLA literature for decades, especially inrelation to the issue of reactivity ~i+e+, the supposition that verbalization mightalter the cognitive process!+ One recent study ~Leow & Morgan-Short, 2004!empirically addressed the issue of reactivity in SLA research methodology andfound nonreactivity for concurrent nonmetalinguistic verbalization at the begin-ning level of foreign or second language ~L2! experience+ Given that the issueof reactivity is strongly related to the level of internal validity of studies prem-ised on the role of attention in SLA, the present study expands on Leow andMorgan-Short’s study by empirically investigating the role that type of verbalreport ~nonmetalinguistic vs+ metalinguistic! elicited during an L2 reading taskplays in the subsequent comprehension and written production of learnerswith an advanced stage of L2 experience+
Prior to the publication of Ericsson and Simon’s ~1984, 1993! seminal reviewof verbalization, many investigators tended to treat alike all studies instruct-ing subjects to verbalize their thoughts+ However, as Ericsson and Simonpointed out, not all verbalizations are alike; they differ according to the con-ditions under which they are obtained+ In their typology, Ericsson and Simoncategorized verbal reports as either concurrent or retrospective based on thetemporal frame in which the reports are collected+ Concurrent protocols arethose collected as subjects verbalize while performing the task in question,whereas retrospective protocols are collected when subjects verbalize sometime after performing the task+ Ericsson and Simon advised that retrospectiveprotocols be used with caution because it is impossible to “rule out the pos-sibility that the information @subjects# retrieve at the time of the verbal reportis different from the information they retrieved while actually performing theexperimental task” ~1993, p+ xii!+ They advised that concurrent protocols becollected whenever possible “to avoid this problem of accessing informationat two different times—first during the actual cognitive processing and thenat the time of report” ~1993, p+ xiii!+
In addition to categorizing verbal reports in terms of temporal space, Eric-sson and Simon ~1993! also distinguished between other types of protocols+After all, “being instructed to explain verbally each step of a solution is quitedifferent from focusing wholly on solving the problem efficiently while verbal-izing one’s thoughts concurrently” ~p+ xvii!+ Ericsson and Simon distinguishedbetween protocols that require subjects to verbalize their thoughts per se andthose that require subjects to verbalize specific information, such as reasons
416 Melissa A. Bowles and Ronald P. Leow
and justifications+ For the purposes of the present study, verbalizations ofthoughts per se will be referred to as nonmetalinguistic and those requir-ing verbalization of additional specific information will be referred to asmetalinguistic+1
Two major issues surrounding the validity of verbal reports surface fromthis typology+ One is veridicality—namely, whether the information in verbalreports accurately represents the thought process it is designed to capture+Some evidence for nonveridicality has been found for retrospective proto-cols, which can “yield substantial forgetting or fabrication in all tasks” ~Russo,Johnson, & Stephens, 1989!+ Concurrent protocols, on the other hand, are notsubject to the same critique, as they are collected during the task+ The sec-ond issue is reactivity—that is, whether the act of thinking aloud alters theend state of the cognitive process ~accuracy of task performance!+ Ericssonand Simon’s ~1993! model predicted that verbalizations that require subjectsto verbalize additional specific information ~metalinguistic verbalization! couldalter subjects’ thought processes and, therefore, potentially have reactiveeffects on task performance+ However, they cautioned that if such justifica-tions are “generated as part of the normal process of solution,” then suchverbalization should not have an effect on task performance ~p+ xxxiii!+ There-fore, implicit in this statement is the supposition that the requirements of atask might have a crucial impact on the reactivity ~or lack thereof! of metalin-guistic verbalizations+
VERBAL REPORTS IN SLA RESEARCH
Introspective methods, including verbal reports, have been used extensivelyas a data elicitation technique in SLA+ Since SLA was first studied systemati-cally in the early 1970s, there has been some debate over their use: Research-ers like Selinker ~1972! proposed that only learners’ production data shouldbe used in formulating theories and conducting research about SLA, whereasothers, such as Corder ~1973!, argued that production data from language learn-ers provide only a small piece of the language learning puzzle+ As Corderpointed out, in order to understand how language learning works, it mightalso be necessary to determine what learners think about their own produc-tion+ Corder believed that this type of information could only be gatheredthrough introspective methods+
Although the debate over the use of verbal reports continues today, manyresearchers in the field continue to use verbal reports—both concurrent andretrospective—in an effort to gain some information about learners’ cognitiveprocesses as they interact with the L2+ As Gass and Mackey ~2000! pointedout, “understanding the source of second language production is problematicbecause often there are multiple explanations for production phenomena thatcan only be assessed by exploring the process phenomena” ~p+ 26!+
Reactivity and Type of Verbal Report in SLA 417
Because verbal reports have the advantage of providing insight into learn-ers’ cognitive processes, they have been used in a number of fields in SLA,ranging from L2 reading and writing ~Cavalcanti & Cohen, 1990; Cohen, 1987;Cohen & Cavalcanti, 1987; Hosenfeld, 1976, 1977, 1979, 1984!, comparisonsbetween first language ~L1! and L2 strategies ~Chamot & El Dinary, 1999; Davis& Bistodeau, 1993; Nevo, 1989; Yamashita, 2002!, L2 test-taking strategies~Cohen, 2000; Norris, 1992; Warren, 1996!, translation ~Enkvist, 1995; Færch &Kasper, 1986; Jaaskelainen, 2000; Kern, 1994!, interlanguage pragmatics ~Cohen& Hosenfeld, 1981; Kasper & Blum-Kulka, 1993!, and oral interaction research~Mackey, Gass, & McDonough, 2000; Nabei & Swain, 2002; Philp, 2003!, to L2attention and awareness studies ~Leow, 1997, 1998, 2000, 2001a, 2001b; Rosa& Leow, 2004a, 2004b; Rosa & O’Neill, 1999!+
Verbal reports have been used as a data elicitation technique in SLA atten-tional research ~studies premised on the role of attention in SLA! for less thana decade; the first published mention of them is by Alanen ~1995!, who com-bined concurrent verbal reports with two other offline data collection mea-surements to address the role of noticing in her study+ Recently, severalempirical studies premised on the role of attention and awareness, or both, inL2 development have begun to employ concurrent ~online think-alouds in Leow,1997, 1998, 2000, 2001a, 2001b; Rosa & Leow, 2004a, 2004b; Rosa & O’Neill,1999! or retrospective verbal reports ~stimulated recalls in Mackey et al+, 2000!to gather data on learners’ cognitive processes while they interacted with theL2 data+ These data were used to establish the role of attention and aware-ness in L2 processing during exposure to the L2 before investigating its effectson L2 development+ However, the use of verbal reports in L2 acquisition hasbeen critiqued by a number of sources on the grounds that the requirementto think aloud might be reactive ~Ellis, 2001; Jourdenais, 2001!; that is, verbal-ization might alter the cognitive processes employed by learners while per-forming an L2 task+ Consequently, the validity of verbal reports in SLA researchmethodology has been questioned+ However, these claims appear to be basedlargely on anecdotal evidence, as there has been only one empirical SLA study~Leow & Morgan-Short, 2004! to date that investigated the reactivity of think-aloud protocols+
Leow and Morgan-Short ~2004! investigated the effects of nonmetalinguis-tic verbal reports during the reading process on 77 beginning Spanish learn-ers’ text comprehension of a Spanish article ~384 words! and intake and writtenproduction of a morphological structure ~the impersonal imperative in Span-ish!+ Participants in the control condition read and completed the tasks silently,whereas participants in the experimental condition read and completed thetasks while thinking aloud+ Results found nonreactivity for concurrent non-metalinguistic verbalization, as the two groups did not differ significantly oneither text comprehension ~measured by a multiple-choice recognition task!or postassessment task performance ~on a controlled written production task!+
Although Leow and Morgan-Short ~2004! were the first to empirically addressthe methodological issue of reactivity in SLA research methodology, whether
418 Melissa A. Bowles and Ronald P. Leow
type of verbal report ~metalinguistic vs+ nonmetalinguistic! plays a role in reac-tivity has not been empirically investigated in the field of SLA, and furtherexploration of this issue is clearly warranted+
PREVIOUS NON-SLA RESEARCH
Research on the effects of verbal reports on problem-solving and decision-making tasks has been conducted in the field of cognitive psychology since atleast the 1950s+ A comprehensive review of the non-SLA research into the reac-tivity of verbal reports is outside the scope of this article, but a synthesis ofdozens of studies in cognitive psychology that have investigated the reactiv-ity of nonmetalinguistic verbal reports indicates that such verbalizations donot seem to influence cognitive processes when compared to silent controlgroups ~Ericsson, 2002!+ This finding of nonreactivity suggests that nonmeta-linguistic verbalization might be a valid method of capturing internal thoughtprocesses+ However, the findings indicate that, overall, verbalization tends tobe reactive for latency ~solution time! because the additional time needed forverbalization increases the overall solution time+
With regard to metalinguistic verbalization, an early review of studies ~Eric-sson & Simon, 1993! found mixed results with regard to reactivity+ The major-ity of the studies involving metalinguistic verbal reports examined the effectsof metalinguistic think-alouds compared to a silent control group and foundmetalinguistic think-alouds to be reactive for accuracy in most cases ~Berry,1983; Berry & Broadbent, 1984; Bower & King, 1967; Davis, Carey, Foxman, &Tarr, 1968; McGeorge & Burton, 1989; Short et al+, 1991; Stanley, Mathews, Buss,& Kotler-Cope, 1989;Wilder & Harvey, 1971;Wilson & Schooler, 1991!+ For manyof the studies, time ~latency! was not reported, but Ericsson and Simon’s pre-diction that metalinguistic verbalization requires extra processing time ~as com-pared to other types of verbalization or to a control! was borne out in almostall of the studies that reported latency ~Ahlum-Heath & di Vesta, 1986; All-wood, 1990; Carpenter, Just, & Schell, 1990; Fidler, 1983; Gagné & Smith, 1962;K+ Robinson, 2001!+
Several more relevant studies were published before and after Ericsson andSimon’s ~1993! book; the results of these studies ~see Table 1! will be brieflysummarized in the following section+
Studies investigating the reactive effects of metalinguistic versus nonmet-alinguistic protocols have generally compared the following conditions: con-trol ~silent!, nonmetalinguistic verbalization, and concurrent metalinguisticverbalization+ Participants were usually undergraduate students, and experi-mental materials employed in the research designs included problem-solvingtasks ~e+g+, the Tower of Hanoi, Katona Card problem, computerized forest-firefighting simulator, etc+! and Vygotsky tests ~e+g+, Ladd and PAMs tests!+ Thefindings of these studies are inconclusive+ Three of the six studies ~Berardi-Coletta, Buyer, Dominowski, & Rellinger, 1995; Brunk, Collister, Swift, & Stay-
Reactivity and Type of Verbal Report in SLA 419
Tabl
e1.
Verb
alre
port
san
dre
activ
ity:O
verv
iew
ofem
piri
calr
esea
rch
stud
ies
com
pari
ngm
etal
ingu
istic
and
nonm
etal
ingu
istic
verb
alre
port
s
Stud
yPa
rtic
ipan
tsTa
sk~s!
Grou
psDe
sign
Reac
tivity
Non
reac
tivity
Bera
rdi-C
olet
ta,
etal+~
1995!
Betw
een-
subj
ects
Accu
racy,t
ime
Expe
rim
ent
110
9un
derg
radu
ate
stud
ents
Tow
erof
Han
oi2
TA@�
met
a#,
2TA@�
met
a#,
1co
ntro
l~si
lent!
Expe
rim
ent
264
unde
rgra
duat
est
uden
tsKa
tona
card
prob
lem
TA@�
met
a#,
TA@�
met
a#,
1co
ntro
l~si
lent!
Expe
rim
ent
340
unde
rgra
duat
est
uden
tsTo
wer
ofH
anoi
TA@�
met
a#,
cont
rol~
sile
nt!
Expe
rim
ent
415
unde
rgra
duat
est
uden
tsKa
tona
card
prob
lem
TA@�
met
a#,
TA@�
met
a#Br
ehm
er~1
974!
a64
unde
rgra
duat
est
uden
tsCu
e-pr
obab
ility
lear
ning
task
TA@�
met
a#,
TA@�
met
a#Be
twee
n-su
bjec
tsAc
cura
cy
Brun
ket
al+~
1958!
147
unde
rgra
duat
est
uden
ts2
Vygo
tsky
test
s~L
add,
PAM
s!Co
ncur
rent@�
met
a#ve
rbal
izat
ion,
conc
urre
nt@�
met
a#ve
rbal
izat
ion
Betw
een-
subj
ects
Accu
racy,t
ime
Dick
son
etal+
~200
0!a
60un
derg
radu
ate
stud
ents
Com
pute
rize
dfo
rest
fire
fight
ing
sim
ulat
orco
ntro
l~si
lent!,
TA@�
met
a#,T
A@�
met
a#Be
twee
n-su
bjec
tsAc
cura
cy~%
offo
rest
save
d!b
Num
ber
ofde
cisi
ons
mad
eRu
sso
etal+~
1989!
24un
derg
radu
ate
stud
ents
Mat
hem
atic
alte
st~m
enta
lad
ditio
n!,v
erba
ltas
k~a
nagr
ams!,n
umer
ical
task~g
ambl
es!,
pict
oria
lta
sk~R
aven
’sm
atri
ces!
Cont
rol~
sile
nt!,
conc
urre
ntTA
@�m
eta#,
retr
ospe
ctiv
eTA
@�m
eta#
Betw
een-
subj
ects
Accu
racy
for
TA@6
met
a#on
gam
bles
and
addi
tion;
time
for
conc
urre
ntTA
Accu
racy
for
TA@6
met
a#on
anag
ram
san
dRa
ven’
sm
atri
ces
Nor
ris~1
990!
a34
3hi
ghsc
hool
stud
ents
Mul
tiple
-cho
ice
test
ofcr
itica
lthi
nkin
g~P
art
Aof
test
for
appr
aisi
ngob
serv
atio
ns!
Cont
rol~
sile
nt!,
conc
urre
ntTA@�
met
a#,
retr
ospe
ctiv
eTA@�
met
a#,
crite
ria
orpr
obe
TA@�
met
a#
Betw
een-
subj
ects
Accu
racy
Note+T
A�
thin
k-al
oud;@�
met
a#�
met
alin
guis
tic;@
�m
eta#
�no
nmet
alin
guis
tic+
a Tim
eno
tre
port
ed+
bTh
em
etal
ingu
istic
verb
aliz
atio
ngr
oup
perf
orm
edsi
gnifi
cant
lyw
orse
than
the
cont
rolg
roup;t
heno
nmet
alin
guis
ticve
rbal
izat
ion
grou
ppe
rfor
med
betw
een
the
two
extr
emes,o
nly
slig
htly
wor
sein
term
sof
accu
racy
than
the
cont
rol+
420 Melissa A. Bowles and Ronald P. Leow
ton, 1958; Dickson, McLennan, & Omodei, 2000! found that verbalizationaffected task performance, whereas two of the six ~Brehmer, 1974; Norris, 1990!found that verbalization had no effect on task performance+ Only one study~Russo et al+, 1989! found mixed results, which indicated that verbalizationaffected performance for some tasks but not for others+ However, all of thestudies reviewed ~with the exception of Russo et al+, 1989! employed nonver-bal and problem-solving tasks, or both, limiting their relevance to SLA+
In fact, of all the studies on reactivity found in the literature, only six usedverbal tasks ~Lass, Klettke, Lüer, & Ruhlender, 1991; Mathews et al+, 1989; Rhe-nius & Deffner, 1990; Russo et al+, 1989; Short et al+, 1991; Stratman & Hamp-Lyons, 1994!+ For a brief overview of the methodologies of these studies, seeTable 2+ Of the six studies, only one ~Short et al+, 1991! found verbalization ~inthis case, metalinguistic! to be reactive for accuracy+ However, these studiesshowed great heterogeneity in terms of task type, with experimental tasks rang-ing from anagrams and verbal analogies to sentence assembly, text revision,and artificial grammar activities+ Furthermore, they included participants fromdiverse pools, ranging from elementary school children to college-age stu-dents+ As such, conclusive results cannot be drawn from this small sample ofstudies with regard to reactivity of verbal reports on L1 verbal tasks, althoughthe results in general support the finding of nonreactivity exhibited by non-verbal tasks+
Given the paucity of research involving L1 verbal tasks, it is not surprisingthat there has been even less research on reactivity of verbalization duringL2 verbal tasks+ To date, only one recent study ~Leow & Morgan-Short, 2004!of those cited previously has examined the reactivity of nonmetalinguistic ver-bal reports during the processing of L2 tasks+ However, the effects of meta-linguistic verbalization on L2 tasks and the differential effects of type ofverbalization ~both nonmetalinguistic and metalinguistic! on more advancedlanguage learners have yet to be examined+
The issue of time ~latency! also appears to be inconclusive+ In the studiesthat reported time ~Lass et al+, 1991; Rhenius & Deffner, 1990!, verbalization isusually shown to take longer than a silent group+ However, the majority ofstudies did not report times on task+
Given the inconclusive findings and heterogeneity of task types in reactiv-ity studies in the field of cognitive psychology and the lack of a base of researchwith L2 tasks, more studies are clearly needed to address the validity of ver-bal reports in SLA+
THE PRESENT STUDY
The present study, motivated by the previous research in cognitive psychol-ogy and the need to continue addressing the internal validity of researchdesigns in SLA studies, expanded on Leow and Morgan-Short’s ~2004! studyby empirically investigating the role that type of verbal report ~nonmetalin-
Reactivity and Type of Verbal Report in SLA 421
Tabl
e2.
Verb
alre
port
san
dre
activ
ity:O
verv
iew
ofem
piri
calr
esea
rch
stud
ies
empl
oyin
gve
rbal
task
s
Stud
yPa
rtic
ipan
tsTa
sk~s!
Grou
psDe
sign
Reac
tivity
Non
reac
tivity
Lass
etal+~
1991!
70un
derg
radu
ate
stud
ents
Anag
ram
sTA@�
met
a#,c
ontr
ol~s
ilent!
Betw
een-
subj
ects
Late
ncy
Accu
racy
Mat
hew
set
al+
~198
9,Ex
p+2!
a16
8un
derg
radu
ate
stud
ents
Artif
icia
lgra
mm
arac
tiviti
esTi
me
1&
time
2:si
lent;
,tim
e1
&tim
e2
~TA@�
met
a#!
time
3—al
lver
baliz
ed
Betw
een-
subj
ects
Accu
racy
Rhen
ius
&De
ffner~1
990!
TA@�
met
a#,c
ontr
ol~s
ilent!
Betw
een-
subj
ects
Late
ncyb
Accu
racy
Expe
rim
ent
124
stud
ents
Sent
ence
asse
mbl
yEx
peri
men
t2
48st
uden
tsW
ord
and
geom
etri
cal
puzz
les
Expe
rim
ent
321
stud
ents
Rave
n’s
mat
rice
sRu
sso
etal+~
1989!
24un
derg
radu
ate
stud
ents
Mat
hem
atic
alta
sk~m
enta
ladd
ition!,
verb
alta
sk~a
nagr
ams!,
num
eric
alta
sk~g
ambl
es!,
pict
oria
ltas
k~R
aven
’sm
atri
ces!
Cont
rol~
sile
nt!,
conc
urre
ntTA
@�m
eta#,r
etro
spec
tive
TA@�
met
a#
Betw
een-
subj
ects
Accu
racy
for
TA@6
met
a#on
gam
bles
and
addi
tion;
time
for
conc
urre
ntgr
oup
Accu
racy
for
TA@6
met
a#on
anag
ram
san
dRa
ven’
sm
atri
ces
Shor
tet
al+~
1991!a
94br
ight,a
vera
ge,
lear
ning
disa
bled,
and
deve
lopm
enta
llyha
ndic
appe
dfif
th-g
rade
rs
Spat
iala
ndve
rbal
anal
ogie
sTi
me
1:si
lent;
time
2:TA@�
met
a#W
ithin
-sub
ject
sAc
cura
cyfo
ral
lst
uden
tson
both
type
sof
anal
ogy
~mos
tpr
ofou
ndef
fect
sfo
rbr
ight
and
aver
age
stud
ents!
Stra
tman
&H
amp-
Lyon
s~1
994!
12un
derg
radu
ate
and
grad
uate
stud
ents
Revi
sion
ofte
xtTA@�
met
a#,s
ilent
With
in-a
ndbe
twee
n-su
bjec
tsN
ost
atis
tical
anal
ysis
bN
ost
atis
tical
anal
ysis
Note+T
A�
thin
k-al
oud;@�
met
a#�
met
alin
guis
tic;@
�m
eta#
�no
nmet
alin
guis
tic+
a Tim
eno
tre
port
ed+
b Abili
tyto
dete
ctan
dfix
orga
niza
tiona
lerr
ors
inth
ete
xtde
crea
sed
inth
eve
rbal
izat
ion
cond
ition+D
etec
tion
ofpr
onou
ner
rors
and
occu
rren
ceof
wor
d-le
vele
rror
sin
crea
sed+
422 Melissa A. Bowles and Ronald P. Leow
guistic vs+ metalinguistic! elicited during an L2 reading task plays in the sub-sequent text comprehension and written production of learners drawn froman advanced stage of L2 experience+ Given that previous research has foundthat reading strategies vary with proficiency level ~O’Malley, Chamot, Stewner-Manzanares, Kupper, & Russo, 1985!, advanced L2 learners were chosen forthis study to explore whether the previous finding of nonreactivity for begin-ning learners in Leow and Morgan-Short’s study would be replicated+ Addition-ally, the present study investigated the effects of verbal reporting on systemlearning ~the ability to apply recently learned grammatical information to newexemplars!+ Finally, the study sought to address the issue of time ~latency!;that is, whether the amount of time it took to complete the task was signifi-cantly affected by type of verbalization+ Specifically, the study was guided bythe following research questions:
Does the type of verbalization ~nonmetalinguistic vs+ metalinguistic vs+control!
1+ have an effect on advanced L2 readers’ comprehension of an L2 text?2+ have an effect on advanced L2 readers’ ability to produce old exemplars of the
targeted L2 structure?3+ have an effect on advanced L2 readers’ ability to produce new exemplars of the
targeted L2 structure?4+ significantly affect the amount of time it takes to complete the task?
Participants
The original pool of participants consisted of 70 college-level students enrolledin a fifth-semester Spanish course who received extra credit in exchange forparticipation+ At the time of the experiment, the targeted syntactic structure~the Spanish pluperfect subjunctive! had not been formally presented in classto the participants+ However, participants had almost certainly encounteredthe form at some point when reading authentic texts as part of the communi-cative language program+ To ensure that only participants with minimal knowl-edge of the targeted structure were included in the final sample, only thoseparticipants who scored 2 or less out of 10 on the pretest and who indicatedon a debriefing questionnaire at the end of the study that they did not ini-tially understand the rules for using the structure were included+ Participantswere also eliminated for ~a! not completing all parts of the experiment and ~b!not providing audible think-aloud protocols+ Of the original pool of 70 partici-pants, 25 were eliminated on the basis of these criteria, yielding a final sam-ple of 45 participants+ These participants were randomly assigned to either acontrol group or to one of the verbalization groups ~metalinguistic or nonmet-alinguistic!+ There were 17 participants in the control group, 18 in the nonmet-alinguistic group, and 10 in the metalinguistic group+
Reactivity and Type of Verbal Report in SLA 423
Targeted Structure
The targeted linguistic structure was the pluperfect subjunctive in Spanish+This structure was chosen because it poses difficulty even for advanced learn-ers of Spanish and had not been formally presented to the participants at thetime of the study+ Additionally, the experimental text provided numerous oppor-tunities for contextual guessing of the structure’s meaning and use+ To under-stand the basic message of the text, participants almost certainly had to realizethat the pluperfect subjunctive forms in the text conveyed contrary-to-factstatements in the past tense ~i+e+, Si George Walker Bush hubiera sido un argen-tino descendiente de algún inmigrante italiano, quizás hoy sería un tal JorgeÁngelo Fossati+ “If George Walker Bush had been an Argentine descendant ofsome Italian immigrant, perhaps today he would be named Jorge ÁngeloFossati+”!+
Materials
The text used in the present study was an 861-word article from the Argentinemagazine El Correo+ The article, titled Si Bush hubiera sido porteño ~“If Bushhad been Argentine”! contained 28 instances of the pluperfect subjunctive,making it ideal for this study ~see the Appendix!+
A computer audio recording program ~Audio HiJack version 1+2+4! was usedto record participants’ verbalizations as they read the text and completed thetasks+ Participants wore headsets during the experiment and spoke theirthoughts aloud into the microphone+ The software recorded their voices as64-bitrate MPEG Audio Layer 3 ~MP3! files, which could be easily stored andquickly retrieved for subsequent analysis+
Assessment Tasks
To measure participants’ comprehension of the text, a 10-item multiple-choicecomprehension task was designed to elicit pieces of information providedthrough the pluperfect subjunctive in the text+ The multiple-choice questionswere provided in the participants’ L1 ~English! to ensure that participants werebeing tested on their comprehension of the Spanish text alone and not ontheir comprehension of the questions themselves+
To measure participants’ written production of the targeted structure, twofill-in-the-blank written production tasks were prepared+ The first task wasdesigned to measure participants’ ability to produce the targeted structure infamiliar contexts+ This task consisted of 20 sentences, all taken directly fromthe Spanish text+ In each sentence, a verb was deleted and its English equiv-alent was provided in parentheses, as in ~1!+
424 Melissa A. Bowles and Ronald P. Leow
~1! Es probable que sus abuelos (would have arrived) ______ al país en la oleada de1880+“It is likely that his grandparents ______ to the country during the wave of 1880+”
Participants were instructed to write the appropriate verb form to completeeach sentence+ Of the 20 sentences, 10 required the targeted structure and 10were distracters that required other verb forms+
The second task was designed to measure participants’ ability to producethe targeted structure in new contexts+ This task consisted of 20 sentencesnot found in the text+ A contextualizing sentence was provided in Spanish,followed by a sentence with blanks in both the “if-clause” and the “resultclause,” as in ~2!+
~2! Juan trabajó veinte horas la semana pasada y ganó $600. Juan (GANAR) ______ másdinero la semana pasada si (TRABAJAR) ______ cuarenta horas+“Juan worked twenty hours last week and earned $600+ Juan ~EARN! ______ moremoney last week if he ~WORK! ______ forty hours+”
The infinitive of the appropriate verb for each blank was provided in paren-theses+ Of the 20 sentences, 10 required the use of the targeted structure and10 required the use of other verb forms+
Testing Procedure
The pretest, which consisted of the first written production task, was admin-istered 2 weeks before participants were exposed to the experimental input+During that 2-week period, participants were not formally exposed to the tar-geted structure+
On the day of the experimental exposure, participants were randomlyassigned to either the control group or to one of the verbalization groups+Participants in both verbalization groups reported to the language labora-tory, whereas participants in the control group remained in their classrooms+All participants were randomly assigned a packet containing ~a! an agreementform, ~b! instructions, ~c! the experimental text, ~d! the comprehension task,~e! the production task testing usage in familiar contexts, ~f! the productiontask testing usage in unfamiliar contexts, and, finally, ~g! the debriefing ques-tionnaire+ Materials were identical for all groups, except for the instructionsrelating to verbalization+ Participants in the control group were simplyinstructed, “You will read a text from an Argentine magazine and answer somequestions about it+ Please complete the questions in order+” Participants inthe two verbalization conditions were given the same instructions as the con-trol group, plus explicit instructions on how to think aloud+ In the nonmeta-linguistic condition, participants were instructed as follows:
Reactivity and Type of Verbal Report in SLA 425
As you read the article and answer the questions, please think your thoughtsaloud+ That is, say whatever passes through your mind while you read andanswer the questions+ You may speak in either English or Spanish+
Participants in the metalinguistic condition were instructed as follows:
While you read the text that follows, we would like you to verbalize everythought and every detail of your thought process, including what informa-tion you are looking at, what thoughts you are having about any piece ofinformation, how you evaluate different pieces of information, and why+And as you answer the questions that follow the text, please do the same,providing your reasoning or justification for each response+
Please complete the questions in order+ As you read the article andanswer the questions, please think your thoughts aloud+ That is, say what-ever passes through your mind while you read+ As you answer the ques-tions, give a justification out loud for your answers and explain your reasoning+You may speak in either English or Spanish+
As they read the text and completed the tasks, participants in the two ver-balization groups were reminded to continue thinking aloud+ The control groupfollowed the same procedure as both of the verbalization groups, with theexception of not having to provide think-aloud protocols+ Participants wereencouraged to speak in either L1 ~English! or L2 ~Spanish! as they wished sothat their think-alouds would not be constrained by their language proficiency+
Scoring Procedure
For the comprehension task, one point was awarded for each correct answerand zero points for each incorrect answer, for a total possible score of 10+ Forthe first production task, only those 10 sentences that necessitated the tar-geted structure were counted; one point was awarded for each correct answerand zero points for each incorrect answer, for a total possible score of 10+ Forthe second production task, again only the 10 sentences that necessitated thetargeted structure were counted+ Participants received one point if they cor-rectly produced the targeted verb form ~hubiera � past participle! in the appro-priate clause and zero points if they did not produce the targeted structure inthe appropriate clause+ Participants were not penalized for making spellingerrors or for overgeneralizing irregular past participles; that is, hubiera *volvido~for hubiera vuelto! would be scored correct+
Protocol Coding Procedure
To ensure that participants in each verbalization group accurately repre-sented their respective group in regard to the production of metalinguistic ornonmetalinguistic protocols, one-third of the protocols were randomly selectedand coded @6 metalinguistic# by two raters+ Given that interrater reliability
426 Melissa A. Bowles and Ronald P. Leow
was 100% for these protocols, the remaining protocols were subsequentlycoded by one rater+
RESULTS
Prior to conducting statistical analyses with participants’ test scores, reliabil-ity analyses were performed on the comprehension task and on the two pro-duction tasks+ In all cases, the reliability coefficient was computed usingCronbach’s alpha+ Reliability on the comprehension task was found to be fairlylow ~+45!, whereas reliability on the two production tasks was high ~+88 for theproduction of old exemplars and +87 for the production of new exemplars!+
Research Question 1
To measure the effects of type of verbalization on comprehension, raw com-prehension scores ~see Table 3! were submitted to a one-way analysis of vari-ance ~ANOVA! ~see Table 4!+ Results showed a significant main effect for group,F~2, 45!� 3+86, p , +05+ A post hoc Ryan test was performed to identify whichcontrasts contributed to the main effect+2 Results revealed that the significantmain effect was the result of a difference, significant at the +05 level, betweenthe nonmetalinguistic and metalinguistic groups, with the nonmetalinguistic
Table 3. Comprehension: Descriptivestatistics
Comprehension
Condition M SD
Control 5+59 1+938@� meta# 6+67 1+879@� meta# 4+80 1+135
Table 4. Comprehension: One-way ANOVA for type ofverbalization
Source dfSums ofsquares
Meansquares F
Type of verbalization 2 24+1935 12+0967 3+857*Within-groups 42 131+7176 3+1361Total 44
*p , +05+
Reactivity and Type of Verbal Report in SLA 427
group performing significantly better on the comprehension task than the meta-linguistic group+ In other words, when compared, participants who thoughtaloud nonmetalinguistically while reading an L2 text appeared to have com-prehended significantly more of the text content than participants who thoughtaloud metalinguistically+ However, no significant difference was found betweenthe control group and either experimental group on the comprehension task~p � +079 and p � +270 for the nonmetalinguistic and metalinguistic groups,respectively!+ As such, the presence of either type of verbal reports did notappear to have a detrimental effect on participants’ comprehension of the L2text when compared to a silent control group+
Research Question 2
To measure the effects of type of verbalization on participants’ ability to pro-duce old exemplars of the targeted structure, a one-way ANOVA was first con-ducted on raw pretest production scores to determine whether the threeexperimental groups were comparable at the outset of the study+ The resultsrevealed no significant differences between the groups, F~2, 42! , 1+ Conse-quently, it could be assumed that any gains in the production scores of oldexemplars from the pretest to the posttest were due to the type of verbaliza-tion rather than to preexisting differences between experimental groups+
The pretest and immediate posttest raw scores for each group were sub-sequently submitted to a 3 � 2 repeated-measures ANOVA ~for descriptive sta-tistics, see Table 5!+ Type of verbalization ~metalinguistic vs+ nonmetalinguisticvs+ control! was entered as the between-subject factor, and time ~pretest vs+posttest! was entered as the within-subject factor+ The ANOVA ~see Table 6!only showed a significant main effect for time, F~2, 42!� 56+72, p , +05, reveal-ing neither a significant main effect for type of verbalization, F~2, 42! , 1, nora significant interaction between time and type of verbalization, F~2, 42! , 1+In other words, although all three groups significantly improved from thepretest to immediate posttest, type of verbalization did not appear to havean effect on participants’ ability to produce old exemplars of the targetedstructure+
Research Question 3
To measure the effects of verbalization on participants’ ability to produce newexemplars of the targeted structure, raw scores on the new exemplars produc-tion task were submitted to a one-way ANOVA ~for descriptive statistics, seeTable 7!+ Results showed no significant main effect for type of verbalization,F~2, 45! , 1 ~see Table 8!+ In other words, as in research question 2, type ofverbalization did not appear to have an effect on participants’ ability to pro-duce new exemplars of the targeted structure+
428 Melissa A. Bowles and Ronald P. Leow
Research Question 4
Finally, to determine whether type of verbalization had a significant effect onthe amount of time it took participants to complete the reading task, timesspent on task were submitted to a one-way ANOVA ~for descriptive statistics,see Table 9!+ Results showed a significant main effect for type of verbaliza-tion, F~2, 45! � 8+74, p , +05 ~see Table 10!+ Post hoc Scheffé tests indicatedthat this main effect was due to significant differences between the controland both experimental groups+ Effect sizes calculated for the control0nonmetalinguistic comparison and control0metalinguistic comparison werelarge ~0+98 and 1+65, respectively!+ In other words, both experimental groups~metalinguistic and nonmetalinguistic! spent significantly more time process-ing the L2 text when compared to the control group+ The amount of time spentin the two verbalization groups did not differ significantly ~p � +405!+
DISCUSSION
Research Question 1
The first research question, which asked whether type of verbalization ~non-metalinguistic vs+ metalinguistic vs+ control! significantly affected readers’ text
Table 5. Production of old exemplars: Descriptivestatistics
Table 6. Production of old exemplars: ANOVA for type ofverbalization and time
Source dfSums ofsquares
Meansquares F
Type of verbalization 2 3+602 1+801 0+310Time 1 395+510 395+510 56+722*Type of verbalization � time 3 2+300 1+150 0+165
*p , +05+
Reactivity and Type of Verbal Report in SLA 429
comprehension, was answered affirmatively+ There was a statistically signifi-cant difference between the comprehension scores of the two experimentalgroups, with the nonmetalinguistic group performing significantly better thanthe metalinguistic group+ Asking even advanced L2 readers to verbally pro-vide additional information that might be extraneous to the reading processappears to have a reactive effect on their comprehension+ Indeed, this conclu-sion is supported by qualitative data from the think-aloud protocols of someparticipants in the metalinguistic group+ A few participants in this group ~whoscored low on comprehension and postassessment tasks! commented that itwas difficult to follow the meaning of the text while verbalizing their thoughtsand justifications+ One participant was particularly explicit, saying, “OK, I’mnot taking any of this in basically + + + I don’t know what any of this is going on,because I’m really not paying attention+ This is distracting to have to talk whileI’m doing this+”
No statistically significant difference was found between the control andeither experimental group, supporting Leow and Morgan-Short’s ~2004! find-ings at a lower language experience level that nonmetalinguistic verbalizationwas not reactive for text comprehension when compared to a control group+However, given that the comprehension task was found to have relatively lowreliability, these findings need to be viewed with some caution+
Table 7. Production of newexemplars: Descriptive statistics
New exemplars
Condition M SD
Control 3+06 2+86@� meta# 3+61 3+22@� meta# 2+40 3+56
Table 8. Production of new exemplars: One-way ANOVAfor type of verbalization
Source dfSums ofsquares
Meansquares F
Type of verbalization 2 9+5810 4+791 0+477Within-groups 42 421+6190 10+039Total 44
430 Melissa A. Bowles and Ronald P. Leow
Research Questions 2 and 3
The second and third research questions, which asked whether type of ver-balization significantly affected the production of old and new exemplars,respectively, were answered negatively+ No significant difference was foundamong any of the groups for production of either old or new exemplars of thetargeted structure+ These results support and build on the findings of Leowand Morgan-Short ~2004!, who similarly found no significant difference in per-formance between nonmetalinguistic and control groups on intake or produc-tion tasks for a morphological structure+ The present study, using a syntacticstructure rather than a morphological one, also found both metalinguistic andnonmetalinguistic verbalization to be nonreactive for production, in apparentcontradiction to Ericsson and Simon’s ~1993! model, which predicted that ver-balizing additional specific information such as justifications would affect taskaccuracy+ It appears that in this study, the requirement to verbalize justifica-tions did not have a significant impact on participants’ ability to produce eitherold or new exemplars of the targeted structure+
In addition to contradicting Ericsson and Simon’s ~1993! model, this find-ing of nonreactivity for the metalinguistic group seems counterintuitive+ Afterall, it stands to reason that if students—especially at this level of language
Table 9. Time on task ~latency!:Descriptive statistics
Time
Condition M SD
Control 34+65 5+30@� meta# 40+00 5+60@� meta# 42+80 4+26
Table 10. Time ~latency!: One-way ANOVA for type ofverbalization
Source dfSums ofsquares
Meansquares F
Type of verbalization 2 477+318 238+659 8+735*Within-groups 42 1147+482 27+321Total 44
*p , +05+
Reactivity and Type of Verbal Report in SLA 431
experience—are required to provide justifications for their thought processesas they read a text and are subsequently asked to provide reasons for each oftheir answers, they will be pushed to think more analytically and possiblyeven to come up with a rule for the use of the targeted structure+ However, inthis case, although participants were asked to provide such justifications, theirperformance on production tasks was neither improved nor hindered+
Data from the think-aloud protocols were helpful in providing further insightinto the similarities between the two verbalization groups+ Participants in eachgroup were sorted according to their performance on postassessment tasksto allow high and low scorers in each group to be compared+ Interestingly, theprotocols that the high and low scorers in each of the groups produced whilereading the text were remarkably similar+ High scorers from both the nonmet-alinguistic and metalinguistic groups shared a common trait: They all reportedawareness of the targeted structure at least at the level of meta-awareness asthey read the text+ Meta-awareness is an intermediate stage between aware-ness at the level of noticing and awareness at the level of understanding ~Leow,1997!+ According to Leow, a protocol is coded @� meta-aware# when partici-pants demonstrate a report of a cognitive change related to the targeted struc-ture but do not verbalize the underlying grammatical rule+ Compare, forexample, the following protocols in ~3! and ~4!+
~3! Metalinguistic high scorer@Reading the text aloud, and correctly translating into English, Es inevitable pregun-tarnos también qué hubiera pasado si Stalin. . .# “It’s inevitable to ask ourself whatwould have happened if Stalin + + + So hubiera is that would have stuff+”
~4! Nonmetalinguistic high scorer@Reading the text aloud, Si George Walker Bush hubiera sido un argentino descend-iente de algún inmigrante italiano. . .# “Wait, so what is this saying? If George WalkerBush + + + hubiera sido un argentino + + + what is hubiera sido? Oh, so maybe it’s ifGeorge Bush had been an Argentine + + + because, yeah, maybe this is about what ifhe was Argentine and not American+”
Similarly, the low scorers in both groups did not appear to report aware-ness even at the level of noticing, as they did not comment on the targetedform as they read the text+ This similarity between performers in both verbal-ization groups is reminiscent of P+ Robinson’s ~1995, 1997a, 1997b, 2001! sug-gestion that it might not be only the learning condition that affects allocationof attentional resources, level of awareness, and the extent of long-term learn-ing but, rather, a combination of variables, including learners’ differing cogni-tive abilities+ It also supports the findings of other studies conducted withinan attentional framework that find a facilitative role for awareness in L2 learn-ing ~Leow, 1997, 2000, 2001a; Rosa & Leow, 2004a; Rosa & O’Neill, 1999!+
The protocols of the high and low scorers in both groups on the postassess-ment tasks revealed another interesting trend+ Although only the high scorers
432 Melissa A. Bowles and Ronald P. Leow
in each group reported awareness of the targeted structure as they read thetext, as a group the participants in the metalinguistic condition ~both highand low scorers! generally showed some awareness of the function of theunknown structure as they verbalized their justifications for their answers tothe production tasks; that is, the requirement to verbalize justifications for allanswers on the production tasks pushed the participants to have a certainlevel of awareness+ Compare, for example, the following protocols from par-ticipants in the metalinguistic group as they completed the production tasks,in ~5! and ~6!+
~5! Metalinguistic low scorer@Reads the sentence, “Quizás (he would have returned) ______ todos los días a sucasa a almorzar” and says:# “Quizás he would have returned, well, first of all, Ithink quizás is @requires# subjunctive, but I’m not gonna mess with that because Idon’t really know+ Would be, umm + + + I think the best thing to do is the imperfect,umm although it sort of looks conditional because of the would, but it is in thepast for sure, so would have, wait, no, I think the best is + + + So here’s how we’regonna do it, just because I can do what I can + + + I think it’s habría ummm + + + volvido+”
~6! Metalinguistic high scorer“So you use it @the targeted structure# in the past tense when there is doubt oruncertainty, so it’s like, uh, maybe a past conditional+”
In the process of completing the production tasks and having to verbalizethe reasons for filling in each blank, both of these participants reported aware-ness of the function of the targeted structure+ However, the low scorer didnot report awareness of the targeted structure as he read the text, so he wasunable to use the targeted form from the text to fill in his gap of knowledge tocomplete the production tasks correctly+ He knew that the forms he was writ-ing were incorrect, as evidenced by his statement, “So here’s how we’re gonnado it, just because I can do what I can+” However, given that he did not dem-onstrate a relatively high level of awareness while processing the targetedstructure, this might have contributed to his inability to complete the produc-tion task with the correct forms+ This explanation is plausible because thehigh scorer did report awareness of the structure while reading the text andwas able to incorporate that knowledge with the justification to complete theproduction tasks correctly+
Research Question 4
The fourth research question, which asked whether type of verbalization ~non-metalinguistic vs+ metalinguistic vs+ control! significantly affected the amountof time on task, was answered affirmatively+ Both verbalization groups spentsignificantly more time on task than the silent control group+ Moreover, themetalinguistic group required more time on task than the nonmetalinguistic
Reactivity and Type of Verbal Report in SLA 433
group, although, in this study, the difference between the two was not statis-tically significant+ This is not surprising in light of the predictions of Ericssonand Simon ~1993! and the findings of reactivity studies in cognitive psychol-ogy ~e+g+, Berardi-Coletta et al+, 1995; Brunk et al+, 1958!+
LIMITATIONS AND FUTURE RESEARCH
One limitation of the present study is the relatively small sample sizes of par-ticipants and the uneven distribution of participants in the metalinguistic groupwhen compared to the other two groups+ Additionally, the reliability on thecomprehension task was found to be fairly low and caution is needed wheninterpreting the results of this task+
The finding that metalinguistic protocols are reactive for text comprehen-sion when compared to nonmetalinguistic protocols should be examined infuture studies, not just extrapolated to all situations+ The text used in thisstudy was quite long ~over 800 words! and also included a difficult syntacticstructure to which the participants had not been formally exposed+ These fac-tors could have had an impact on the results of the study, and future studiesshould therefore address factors such as text length and type, text difficulty,and linguistic structure+ Although Leow and Morgan-Short ~2004! and thepresent study have shed some light on the validity of verbal reports duringL2 reading, future studies are warranted to investigate the reactivity of suchprotocols on different types of tasks, particularly because verbal reports havealready been used in SLA research to measure learners’ attention and aware-ness during L2 problem-solving tasks ~Leow, 1997, 1998, 2000, 2001a; Rosa &Leow, 2004a, 2004b; Rosa & O’Neill, 1999!+ Such investigations into reactivitymight be even more crucially needed in light of the fact that several studiesin cognitive psychology have found differential effects for reactivity among acommon set of participants according to the task being performed ~Rhenius& Deffner, 1990; Russo et al+, 1989!+ This has led some researchers to suggestthat “the causes of reactivity are not general but due jointly to the demandsof the task and to verbalization” ~Russo et al+, p+ 763!+
In line with the idea that the causes of reactivity are not general is thenotion that individual differences might affect reactivity+ This is a promisingarea for future research to investigate, and moderator variables such as work-ing memory, cognitive speed, aptitude complexes, and learning style might befruitful sources of investigation in this methodological strand of investiga-tion+ As one anonymous SSLA reviewer pointed out, if learners’ patterns ofcognitive abilities or aptitudes differ, with some patterns facilitating and oth-ers inhibiting learning and performance under certain conditions, or on cer-tain tasks ~P+ Robinson, 2001!, it stands to reason that abilities and aptitudesmight also affect verbalization under different conditions and on different tasks+Empirical data are clearly warranted to support or refute this premise+
434 Melissa A. Bowles and Ronald P. Leow
CONCLUSION
The present study was designed to address the issue of reactivity of nonmet-alinguistic and metalinguistic protocols in SLA research methodology and itsrelated issue of time ~latency!+ It expands on the work of Leow and Morgan-Short ~2004!, who found that nonmetalinguistic verbalization during an L2 read-ing task did not have any significant effects on first-semester Spanish learners’text comprehension, intake, or production of the targeted morphological form+Using a syntactic structure rather than a morphological one and usingadvanced ~fifth-semester! Spanish learners, the present study investigated thereactivity of both nonmetalinguistic and metalinguistic protocols+ Results indi-cated that, compared to a control group, neither metalinguistic nor nonmeta-linguistic verbalization significantly affected either text comprehension orwritten production of old or new exemplars of the targeted form+ However,metalinguistic verbalization did appear to cause a significant decrease in textcomprehension when compared to nonmetalinguistic verbalization+ Finally,both metalinguistic and nonmetalinguistic verbalization had a significant effecton the amount of time spent processing the L2 text when compared to a con-trol group+
Although verbal reports have been used to measure attention and aware-ness in a number of SLA studies, the validity of these reports is just beginningto be examined+ If they are proven to be valid and to accurately representlearners’ mental processes, such protocols will provide insights into the lan-guage learning process, information that is unavailable in a classical pretest-posttest research design+ However, given the inconclusive findings even incognitive psychology, where studies on reactivity have been conducted forover 50 years, one should be careful in interpreting the findings of studiesthat employed verbal reports+ This is especially the case given that the reac-tivity of verbal reports might vary according to a number of factors, includingtask type, text length and type, text difficulty, linguistic structure, and individ-ual differences+ There is clearly a need for future research in this strand ofresearch methodology in SLA+
~Received 29 November 2004!
NOTES
1+ Given that participants in the metalinguistic group were requested to comment aloud on “everythought and every detail of @their# thought process” in addition to providing a justification for theirreasoning, the term metalinguistic could also be viewed in the present study to include metacogni-tive processes+
2+ As suggested by one anonymous SSLA reviewer, the more sensitive Ryan test was used insteadof the Scheffé test for the comprehension test because of its low alpha ~+45! and its less than 20within-cell sample size+
Reactivity and Type of Verbal Report in SLA 435
REFERENCES
Ahlum-Heath, M+ E+, & di Vesta, F+ J+ ~1986!+ The effect of conscious controlled verbalization of acognitive strategy on transfer in problem solving+ Memory and Cognition, 14, 281–285+
Alanen, R+ ~1995!+ Input enhancement and rule presentation in second language acquisition+ In R+Schmidt ~Ed+!, Attention and awareness in foreign language learning ~pp+ 259–302!+ Honolulu:University of Hawai‘i Press+
Allwood, C+ M+ ~1990!+ On the relation between justification of solution method and correctness ofsolution in statistical problem solving+ Scandinavian Journal of Psychology, 31, 181–190+
Anderson, M+ ~1985!+ Some evidence on the effect of verbalization on process: A methodologicalnote+ Journal of Accounting Research, 23, 843–852+
Berardi-Coletta, B+, Buyer, L+ S+, Dominowski, R+ L+, & Rellinger, E+ R+ ~1995!+ Metacognition and prob-lem solving: A process-oriented approach+ Journal of Experimental Psychology: Learning, Mem-ory, and Cognition, 21, 205–223+
Berry, D+ C+ ~1983!+ Metacognitive experience and transfer of logical reasoning+ Quarterly Journal ofExperimental Psychology, 35A, 39–49+
Berry, D+ C+, & Broadbent, D+ E+ ~1984!+ On the relationship between task performance and associ-ated verbalizable knowledge+ Quarterly Journal of Experimental Psychology, 36A, 209–231+
Biehal, G+, & Chakravarti, D+ ~1989!+ The effects of concurrent verbalization on choice processing+Journal of Marketing Research, 26, 84–96+
Bower, A+ C+, & King, W+ L+ ~1967!+ The effect of number of irrelevant stimulus dimensions, verbaliza-tion, and sex on learning bi-conditional classification rules+ Psychonomic Science, 8, 453–454+
Brehmer, B+ ~1974!+ Hypotheses about relations between scaled variables in the learning of probabi-listic inference tasks+ Organizational Behavior and Human Performance, 11, 1–27+
Brunk, L+, Collister, G+, Swift, C+, & Stayton, S+ ~1958!+ A correlation study of two reasoning problems+Journal of Experimental Psychology, 55, 236–241+
Carpenter, P+ A+, Just, M+ A+, & Schell, P+ ~1990!+ What one intelligence measures: A theoreticalaccount of the processing in the Ravens Progressive Matrices Test+ Psychological Review, 97,404–431+
Cavalcanti, M+ C+, & Cohen, A+ D+ ~1990!+ Comentarios em composições: Uma comparação dos pon-tos de vista do professor e do aluno @Comments in compositions: A comparison of the teach-er’s and student’s viewpoints# + Trabalhos em Lingüística Aplicada, 15, 7–23+
Chamot, A+ U+, & El Dinary, P+ B+ ~1999!+ Children’s learning strategies in language immersion class-rooms+ Modern Language Journal, 83, 319–338+
Cohen, A+ D+ ~1986!+ Mentalistic measures in reading strategy research: Some recent findings+ Englishfor Specific Purposes, 5, 131–145+
Cohen, A+ D+ ~1987!+ Recent uses of mentalistic data in reading strategy research+ Revista de Docu-mentação de Estudos em Lingüística Teorica e Aplicada, 3, 57–84+
Cohen, A+ D+ ~2000!+ Exploring strategies in test taking: Fine-tuning verbal reports from respondents+In G+ Ekbatani & H+ Pierson ~Eds+!, Learner-directed assessment in ESL ~pp+ 127–150!+ Mahwah,NJ: Erlbaum+
Cohen, A+ D+, & Cavalcanti, M+ C+ ~1987!+ Viewing feedback on compositions from the teacher’s andthe student’s perspective+ ESPecialist, 16, 13–28+
Cohen, A+ D+, & Hosenfeld, C+ ~1981!+ Some uses of mentalistic data in second language research+Language Learning, 31, 285–313+
Corder, S+ P+ ~1973!+ The elicitation of interlanguage+ In J+ Svartik ~Ed+!, Errata: Papers in error analysis~pp+ 36–48!+ Lund, Sweden: CKW Geerup+
Davis, J+, & Bistodeau, L+ ~1993!+ How do L1 and L2 reading differ? Evidence from think aloud proto-cols+ Modern Language Journal, 77, 459–472+
Davis, J+ H+, Carey, M+ H+, Foxman, P+ N+, & Tarr, D+ B+ ~1968!+ Verbalization, experimenter presence,and problem solving+ Journal of Personality and Social Psychology, 8, 299–302+
Dickson, J+, McLennan, J+, & Omodei, M+ M+ ~2000!+ Effects of concurrent verbalization on a time-critical, dynamic decision-making task+ Journal of General Psychology, 127, 217–228+
Ellis, R+ ~2001!+ Investigating form-focused instruction+ Language Learning, 51, 1–46+Enkvist, I+ ~1995!+ Intellectual and linguistic processes in foreign language students: Students’ devel-
opment during their first year of Spanish at a Swedish university+ In Studies of higher educationand research+ Washington, DC: ERIC Clearinghouse+ ~ERIC Document Reproduction Service No+ED390253!
436 Melissa A. Bowles and Ronald P. Leow
Ericsson, K+ A+ ~2002!+ Towards a procedure for eliciting verbal expression of non-verbal experiencewithout reactivity: Interpreting the verbal overshadowing effect within the theoretical frame-work for protocol analysis+ Applied Cognitive Psychology, 16, 981–987+
Ericsson, K+ A+, & Simon, H+ A+ ~1993!+ Protocol analysis: Verbal reports as data ~rev+ ed+!+ Cambridge,MA: MIT Press+
Færch, C+, & Kasper, G+ ~1986!+ One learner—two languages: Investigating types of interlanguage knowl-edge+ In J+ House & S+ Blum-Kulka ~Eds+!, Interlingual and intercultural communication ~pp+ 211–227!+ Tübingen, Germany: Gunter Narr+
Fidler, E+ J+ ~1983!+ The reliability and validity of concurrent, retrospective, and interpretive verbalreports: An experimental study+ In P+ Humphreys, O+ Svenson, & A+ Vari ~Eds+!, Analyzing andaiding decision processes ~pp+ 429–440!+ Amsterdam: Elsevier+
Gagné, R+ H+, & Smith, E+ C+ ~1962!+ A study of the effects of verbalization on problem solving+ Journalof Experimental Psychology, 63, 12–18+
Gass, S+ M+, & Mackey, A+ ~2000!+ Stimulated recall methodology in second language research+ Mahwah,NJ: Erlbaum+
Hosenfeld, C+ ~1977!+ A preliminary investigation of the reading strategies of successful and nonsuc-cessful second language learners+ System, 5, 110–123+
Hosenfeld, C+ ~1979!+ Cindy: A learner in today’s foreign language classroom+ In W+ Borne ~Ed+!, Theforeign language learner in today’s classroom environment ~pp+ 53–75!+ Montpelier, VT: North-west Conference on the Teaching of Foreign Languages+
Hosenfeld, C+ ~1984!+ Case studies of ninth grade readers+ In J+ C+ Alderson & A+ H+ Urquhart ~Eds+!,Reading in a foreign language ~pp+ 231–249!+ London: Longman+
Hughes, J+, & Parkes, S+ ~2003!+ Trends in the use of verbal protocol analysis in software engineeringresearch+ Behaviour & Information Technology, 22, 127–141+
Jaaskelainen, R+ ~2000!+ Focus on methodology in think-aloud studies on translating+ In S+ TirkkonenCondit & R+ Jaaskelainen ~Eds+!, Tapping and mapping the processes of translation and interpret-ing: Outlooks on empirical research ~pp+ 71–82!+ Amsterdam: Benjamins+
Jourdenais, R+ ~2001!+ Cognition, instruction, and protocol analysis+ In P+ Robinson ~Ed+!, Cognitionand second language instruction ~pp+ 354–375!+ New York: Cambridge University Press+
Kasper, G+, & Blum-Kulka, S+ ~1993!+ Interlanguage pragmatics+ Oxford: Oxford University Press+Kern, R+ G+ ~1994!+ The role of mental translation in second language reading+ Studies in Second Lan-
guage Acquisition, 16, 441–461+Lass, U+, Klettke, W+, Lüer, G+, & Ruhlender, P+ ~1991!+ Does thinking aloud influence the structure of
cognitive processes? In R+ Schmid & D+ Zambarbieri ~Eds+!, Oculomotor control and cognitiveprocesses ~pp+ 385–396!+ Amsterdam: Elsevier+
Leow, R+ P+ ~1997!+ Attention, awareness, and foreign language behavior+ Language Learning, 47,467–505+
Leow, R+ P+ ~1998!+ Toward operationalizing the process of attention in SLA: Evidence for Tomlin andVilla’s ~1994! fine-grained analysis of attention+ Applied Psycholinguistics, 19, 133–159+
Leow, R+ P+ ~2000!+ A study of the role of awareness in foreign language behavior: Aware versus unawarelearners+ Studies in Second Language Acquisition, 22, 557–584+
Leow, R+ P+ ~2001a!+ Attention, awareness, and foreign language behavior+ Language Learning, 51(Suppl. 1), 113–155+
Leow, R+ P+ ~2001b!+ Do learners notice enhanced forms while interacting with the L2? An online andoffline study of the role of written input enhancement in L2 reading+ Hispania, 84, 496–509+
Leow, R+ P+, & Morgan-Short, K+ ~2004!+ To think aloud or not to think aloud: The issue of reactivity inSLA research methodology+ Studies in Second Language Acquisition, 26, 35–57+
Mackey, A+, Gass, S+, & McDonough, K+ ~2000!+ How do learners perceive interactional feedback? Stud-ies in Second Language Acquisition, 22, 471–497+
Mathews, R+ C+, Buss, R+ R+, Stanley, W+ B+, Blanchard-Fields, F+, Cho, J+ R+, & Druhan, B+ ~1989!+ Role ofimplicit and explicit processes in learning from examples: A synergistic effect+ Journal of Exper-imental Psychology: Learning, Memory, and Cognition, 15, 1083–1100+
McGeorge, P+, & Burton, A+ M+ ~1989!+ The effects of concurrent verbalization on performance in adynamic systems task+ British Journal of Psychology, 80, 455–465+
Reactivity and Type of Verbal Report in SLA 437
Nabei, T+, & Swain, M+ ~2002!+ Learner awareness of recasts in classroom interaction: A case study ofan adult EFL student’s second language learning+ Language Awareness, 11, 43–63+
Nevo, N+ ~1989!+ Test-taking strategies on a multiple-choice test of reading comprehension+ LanguageTesting, 6, 199–215+
Norris, S+ P+ ~1990!+ Effect of eliciting verbal reports of thinking on critical thinking test performance+Journal of Educational Measurement, 27, 41–58+
Norris, S+ P+ ~1992!+ A demonstration of the use of verbal reports of thinking in multiple-choice crit-ical thinking test design+ Alberta Journal of Educational Research, 38, 155–176+
O’Malley, J+ M+, Chamot, A+ U+, Stewner-Manzanares, G+, Kupper, L+, & Russo, R+ P+ ~1985!+Learning strategies used by beginning and intermediate ESL students+ Language Learning, 35,21–46+
Philp, J+ ~2003!+ Nonnative speakers’ noticing of recasts in NS-NNS interaction+ Studies in Second Lan-guage Acquisition, 25, 99–126+
Rhenius, D+, & Deffner, G+ ~1990!+ Evaluation of concurrent thinking aloud using eye-tracking data+Proceedings of the Human Factors Society 34th Annual Meeting: Orlando ‘90—Countdown to the21st Century ~pp+ 1265–1269!+ Santa Monica, CA: Human Factors Society+
Robinson, K+ M+ ~2001!+ The validity of verbal reports in children’s subtraction+ Journal of Educa-tional Psychology, 93, 211–222+
Robinson, P+ ~1995!+ Aptitude, awareness, and the fundamental similarity of implicit and explicit sec-ond language learning+ In R+ Schmidt ~Ed+!, Attention and awareness in foreign language learning~pp+ 303–357!+ Honolulu: University of Hawai‘i Press+
Robinson, P+ ~1997a!+ Generalizability and automaticity of second language learning under implicit,incidental, enhanced, and instructed conditions+ Studies in Second Language Acquisition, 19,223–247+
Robinson, P+ ~1997b!+ Individual differences and the fundamental similarity of implicit and explicitadult second language learning+ Language Learning, 47, 45–99+
Robinson, P+ ~2001!+ Individual differences, cognitive abilities, aptitude complexes, and learning con-ditions in second language acquisition+ Second Language Research, 17, 368–392+
Rosa, E+, & Leow, R+ P+ ~2004a!+ Awareness, different learning conditions, and L2 development+ AppliedPsycholinguistics, 25, 269–292+
Rosa, E+, & Leow, R+ P+ ~2004b!+ Computerized task-based exposure, explicitness, type of feedback,and Spanish L2 development+ Modern Language Journal, 88, 193–217+
Rosa, E+, & O’Neill, M+ D+ ~1999!+ Explicitness, intake, and the issue of awareness+ Studies in SecondLanguage Acquisition, 21, 511–556+
Russo, J+ E+, Johnson, E+ J+, & Stephens, D+ L+ ~1989!+ The validity of verbal protocols+ Memory &Cognition, 17, 759–769+
Selinker, L+ ~1972!+ Interlanguage+ International Review of Applied Linguistics, 10, 209–231+Short, E+ J+, Schatschneider, C+, Cuddy, C+ L+, Evans, S+ W+, Dellick, D+ M+, & Basili, L+ A+ ~1991!+ The
effect of thinking aloud on the problem solving performance of bright, average, learning dis-abled, and developmentally handicapped students+ Contemporary Educational Psychology, 16,139–153+
Stanley, W+ B+, Mathews, R+ C+, Buss, R+ R+, & Kotler-Cope, S+ ~1989!+ Insight without awareness: Onthe interaction of verbalization, instruction, and practice in a simulated process control task+Quarterly Journal of Experimental Psychology, 41A, 553–577+
Stratman, J+ F+, & Hamp-Lyons, L+ ~1994!+ Reactivity in concurrent think-aloud protocols+ In P+ Smago-rinsky ~Ed+!, Speaking about writing: Reflections on research methodology ~pp+ 89–112!+ London:Sage+
Warren, J+ ~1996!+ How students pick the right answer: A ‘think aloud’ study of the French CAT+ Occa-sional Papers Applied Linguistics Association of Australia, 15, 79–94+
Wilder, L+, & Harvey, D+ J+ ~1971!+ Overt and covert verbalization and problem solving+ Speech Mono-graphs, 38, 171–176+
Wilson, T+ D+, & Schooler, J+ W+ ~1991!+ Thinking too much: Introspection can reduce thequality of preferences and decisions+ Journal of Personality and Social Psychology, 60,181–192+
Yamashita, J+ ~2002!+ Reading strategies in L1 and L2: Comparison of four groups of readerswith different reading ability in L1 and L2+ ITL, Review of Applied Linguistics, 135–136,1–35+
438 Melissa A. Bowles and Ronald P. Leow
APPENDIX
Experimental text
Si Bush hubiera sido porteño
¿Qué hubiera pasado en USA si George Bush hubiera nacido en La Boca? ¿Y si JoséStalin hubiera obedecido a su madre, quien lo obligó a ser seminarista a los 15 añoscuando su padre murió? Ni hablar si Adolfo Hitler hubiera sido aceptado en la Acade-mia de Bellas Artes de Viena+ Las vidas de estas personalidades no se puede cambiar,así como el fracaso de Vincent Van Gogh como pastor protestante y el sueño frustradode Leonardo Da Vinci de ser un excelente Chef+ Pero el futuro de la humanidad aún esincierto y está en nuestras manos+
Si George Walker Bush hubiera sido un argentino descendiente de algún inmigranteitaliano, quizás hoy sería un tal “Jorge Ángelo Fossati” o el “Sr. Jorge Luigi Marincioni.”+ + + es decir un tipo cualquiera sin trascendencia alguna ~salvo para su familia y la gentedel barrio!+
Es probable que sus abuelos hubieran llegado al país en la oleada de 1880, siendounos entre los miles de trabajadores expulsados por la primera Gran Depresión delsistema capitalista, a pesar de que desde 1850 se expandía a un ritmo galopante+ Lasriendas del liberalismo económico estaban en las manos de las potencias como GranBretaña, que iban conquistando a todo vapor nuevos mercados de ultramar, alimentandosu ambición con el prestigio que representaban las colonias, a las que luego de variosaños tuvieron que renunciar ~aunque hay que admitir que a algunas naciones imperi-alistas todavía les cueste aceptarlo!+
Imagínense que Bush hubiera sido fruto del amor entre algún grotesco italianoy una robusta gallega+ Si en vez de crecer en USA, bajo un techo republicano y pisode algodón, hubiera sido criado en el barrio portuario de La Boca, con su aromaparticular de siempre, tangos de arrabal y en un escenario rodeado de humeanteschimeneas+ + +
Si George Bush hubiera jugado a la pelota en la vereda, tomado mate en vez de milkshakes y hubiera visto ATC en vez de CNN, ¿Alguien conocería hoy a Bin Laden? Quizás“la nona” lo hubiera mandado al pequeño “Jorgito” a la escuela de la esquina conguardapolvo inmaculado y su padre le hubiera comprado con mucho esfuerzo un Man-ual Santillana para que estudiara las guerras mundiales ilustradas y las atrocidades dela historia universal condensadas en tan sólo unas hojas con mapas de colores+
Quizás, hubiera vuelto todos los días a su casa a almorzar, fascinado con los torpe-dos que lanzaban los submarinos alemanes a los buques norteamericanos en el Atlán-tico Norte durante 1917 o hasta le hubiera gustado jugar con soldaditos de plomo ~node carne y hueso! a la hora de la siesta+ Quizás se hubiera conmovido con la historiade las 1000 grullas, que forma parte de la tradición oriental y consiste en hacer plega-dos artesanales para salvar las vidas de las víctimas de la radiación que liberaron lasbombas nucleares en Hiroshima y Nagasaki ~1945!+
Tal vez, con el tiempo, el adolescente “Jorge” se hubiera estremecido al enterarsede la agonía de los 80+000 inocentes que fallecieron en los hospitales colmados de Tokio
Reactivity and Type of Verbal Report in SLA 439
y las camadas de jóvenes emprendedores que perdieron sus vidas en la encarnizadalucha de Vietnam+ Quizás no hubiera entendido la impaciencia de Truman por terminarla Segunda Guerra Mundial y mucho menos lo acordado por las potencias en Postdam,Alemania+
Es inevitable preguntarnos también qué hubiera pasado si Stalin le hubiera hechocaso a su madre, como la mayoría de los hijos solían hacer en ese entonces+ Sin embargo,el desafiante “José”, fue un georgiano huérfano quien se convirtió en el “salvador de lossoviéticos” y sucesor de Lenin en 1923 ~pese a que éste especificó en su testamento,que por prudencia, no quería entregarle el poder!+ ¿Qué sería de Rusia si en el con-vento en el que estudiaba para ser cura, no hubieran descubierto que participaba enmovimientos antizaristas en 1904? Es probable que Stalin terminara dando misa en unacapilla perdida de algún pueblo polvoriento de Georgia ~auque su temperamento y supoca fe en Dios, no lo hubieran permitido!+
También sería interesante saber qué hubiera pasado si a Adolfo Hitler lo hubieranaceptado en la Academia de Bellas Artes de Viena en Austria, su tierra natal+ Quizás sehubiera evitado que volcara su frustración en el campo militar y esa mezcla incandes-cente de resentimiento, paranoia y delirios de grandeza no hubiera entrado en erupción,tras el ridículo Tratado de Versalles que firmaron las potencias vencedoras en 1919,aplastando a los alemanes e instalando un aire de incertidumbre, y temor que acechóla paz del mundo+ ~Se podría decir que hoy se respira un aire parecido, gracias a laurgencia de Colin Powell y Bush!+
Si el joven George Bush hubiera vivido más cerca de lo que Nixon y Kissinger hicieronel martes 11 de septiembre de 1973 en la Casa de Gobierno chilena en la que estabaSalvador Allende, quizás hubiera entendido algunas de las razones por las que el indel-eble 11 de septiembre de 2001, se pagó con sangre inocente+ Fue atroz, no hay duda, elterrorismo no debería existir+ Pero existe y quizás deberíamos preguntarnos ¿por qué?
