— -~ r -r
-. —--,I,.__
,/ f Novembor 1978(:9 Report No. 1802
C ,1/
—-~~
Naomi Miyake and Donald A. Norman
TO ASK A QUESTION, ONE MUST KNOW ENOUGHTO KNOW WHAT IS NOT KNOWN
~~0
C..37 i~
-~ ~1 t
UNIVERSITY OF CALIFORNIA , SAN DIEGO CENTER FOR HUMAN INFORMATION PROCESSJN
This research was mpported by the Advanced Research Projects Age and the Office of Naval Research, Personnel and 7)viningResearch Programs and was monitored by ONR under Contract N000 28 NR 134-387, ~~~~ terms of ARPA OnierNo. 2284.The views and conclusions contained tn this document are thus o t ie author an u WôTlie interpreted ~iiiecess ily representingthc ~rnclal policies, either expressed or implied, of the Advanced Research Projects Agency, of the Office of Naval Research, or thethii’~’d States Government.Approved for public release; distribution unlimited. Reproduction in whole or part is permitted for any pw’pose of the UnitedSM:es Government.
IIn~~1~~~~4 flailsEcu~~I r’t CL. ASS IYICA f l O N Q~ TNIS PAGE (Wh mi Oat. tnI .r.~)
0 0~~~~T r~r~~ l , A ?IAIJ 0A(~ READ INSTRUCTI ONS
,I~i ~~ ~~~~~~~~ ‘~~ I ~~~~~ ~ BEFORE COMPL.ETIN G FORM_______
NUMBER ‘2 GOV1 ACCESSION NO 3. RECIPIENVS CAT A~ OO NUMSER
78,0 2/ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
r i~ii (mid SubUUs) .J~YPE O~ RV~ ñ 1 COVERED
E ~Techn I ca ~~~~~ ~~,~~~ To Ask A Question, One Must Know Enough I _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
to Know What Is Not Known _~~~~~~~ 6. PERFORMING ORG. REPORT dUMBER
7. AUT$~~R(,) -
S. CONTRAC T QR GRANT P4UMSER(.)
(3~ aom~i~~ ke~~~~OonaidAJNonMn / _~~ i/i/ A ~~~t9i i r~~~~~~’~~~ f9. P FORMING ORGA NIZA 11ON NAM E AND ADDRESS - ~ 1~ . PROGRAM CLEMENT. P!diC~T. ~~~ç~riter for Human Information Processing ~7 A REA & WO RK uNIT NUMBERS
University of California, San Diego MR 154 387La Jol la , CA 92093 ________________________
II. CONTROLLING OFFICE NAME AND AO ORESS Il. REPORT DATEPersonnel and Training Research Programs (‘~j ~~~~~~~ t~7i7 ~~Office of Naval Research (Code 458) ~~~~~ u.-,, uMIEw ~ ir-!r*ats- J
Arlington, VA 22217 2014. MONITORING AGENCY NAME 4 AOORESS(It ditf.rmi.t from Can •) IS. SECURITY CL ASS. (at tAd. ,.po$)
/ . Unclassifi edl5~ DECLASS I FICATIO H/ DOWNGRA DING
SCHEDULE
I4. QI5Tp~9U1~IQp4 S~TATEMEN1 (of this R.por t)
Approved for public release; distribution unlim i ted.
17. DISTR IBUT ION STATEMEN T (at A. .b.tr.eI antsr d In at.ck 20. ii diU.,mit from R.po?t)
IS. SUPPLEMENTARY HOlES
IL KEY WORDS (CantInu• an ,,at •. aide ii n.c.. .atv W td.nui ~ ~~ block nombat)
Question asking, complex learning, knowledge levels.
~~~~ASSTRA C t (C.nhInu. an ro’~~o. lid . It n~c•• v mid IdmIlSIV by bI..k I~~~~~at)
~ Is some appropriate level of prior ~~owledge a prerequisite for asking ques-tions about a new topic matter? The hypothesis that learners should ask themost questions when their ~~owledg. is well matched to the level of presentationtaa tested using two levels of background knowledge and two levels of difficultyqLffia ~~~~a~~~~~b~~~~~~ped ,~ . ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~vhile learning. Novice learners asked more questions on ihe easier material t~~~Learners with some training; trained learners asked more questions than noviceson the harder material. Accordingly , theories of question asking in learningshould address interactions between knowledge levels of askers and material.
DD , ~~~~~~ 1473 (DIllON OF I NOV SI IS OSIOt IT.SIN 0102.LP.0144101 tInr1~~cci flail
SECuRItY CL.AUIFICA 1IOH OF THIS PAGE (UP,.,, Dat. ~~Iat.I3
_________ — _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _- _ _ _
TO ASK A QUESTION ,
‘I ONE MUST KNOW ENOUGH TO KNOW
WHAT IS NOT KNOWN
Naomi Miyake and Donald A. Norman
Center for Human Information Processing Report No. 7802University of California , San Diego November 1978La Jolla , California 92093
The views and conclusions contained in this document are those of theauthors and do not necessarily reflect the policy of any agency of the UnitedStates Government. Partial, research support was provided by a contract f rcm theOffice of Naval Research and the Advanced Research Project s Agency , monitored byONR under contract NO0 1~ .-76—C—O628 , MR 15~1—387 . Support was also provided byNational Institutes of Mental Health grant MH—15828 to the Center for HumanInformation Processing. N. Miyake is supported by the Japan Society for thePromotion of Science.
The report is approved for public release ; distribution unlimited . Repro-duction in whole or in part is permitted for any purpose of the United StatesGovernment .
To Ask a Question , One Must Know Enough to Know What is Not Known
Naomi Miyake and Donald A. Norman
To ask a question of someone implies more than a need for informa-
tion. It also implies a proper structure of knowledge with which to
formulate the question and to int erpret the response. Thus , the ability
of a person to think of an appropriate question on a topic matter is a
complex function of the knowledge of that topic.
At a research seminar on computer techniques , we noted that
beginners at programming (to whom the seminar was addressed) asked few
questions and generated few comments. More expert programmers, however,
had many questions and, eventually, dominated the discussion. Reflec-
tion on this observation indicated that a general principle might be
involved: in order to be able to ask questions about new material, one’s
knowledge of the topic must be at an appropriate level: the number of
questions should be an inverted U—shaped function of the relationship
between student knowledge and task complexity.
Consider the novice exposed to expert level material: although
there should be many difficulties in understanding , the novice does not
even have the proper framework within which to ask questions. Thus ,
although at first glance , one might expect novices to be filled with
questions , there are strong theoretical reasons not to expect this.
Consider the expert exposed to novice material. Here, there should
be good structural framework for the material on the part of the expert,
- -~~~
- - -~~~~~
-~~~
-~~~~~~~~~~~
2
but a lack of questions: there are no gaps or inconsistencies in the
knowledge that need to be filled. The maximum number of’ questions
should occur when a person is well matched to the level of presentation.
There should be sufficient background knowledge, but the new material
should indeed be new, with inconsistencies or incomplete information
that causes the learner to question the interpretation, to seek more
information, or to confirm the existing understanding.
The ability to ask questions during the learning of new material,
therefore, seems to be a function of two variables: the existence of
appropriate knowledge structures, and the level of completeness of those
structures with respect to the new material. When there are two presen-
tation levels of the same material, easy and hard , the learner with the
ample knowledge structure should ask more questions on the harder
presentation and less on the easier one; the Learner with little
knowledge should ask most questions with the easier presentation and not
many with the harder presentation.
Considerable research has been done on question answering, but we
have been unable to find relevant experimental or theoretical studies on
the generation of questions. Some relevant issues are discussed in the
speech—act literature, but the relevance is indirect. Grosz (Note 1)
and L.evin and Moore (1977) have looked at a naturalistic dialogue situa-
tions, but they did not perform any analyses relevant to the issues
being raised here .
This paper tests the predicted interaction. The topic to be
learned is a computer text editor. This is a topic for which it is easy
I
3
to find subjects with no prior knowledge. Thus, it is relatively easy
to manipulate the amount of prior knowledge, as well as the difficulty
level of the material.
Method
Desian
We used two levels of background knowledge and two levels of diffi-
culty of the material to be learned. Subjects were preselected to have
little or no prior knowledge about text editors or computers. Trained
subjects received a pre—experimental training session. Other subjects
(novices) did not. There were two levels of the learning task: the
level called ~~~ was quite Less advanced than the level called ~~~~
There were tour groups of subjects, with 15 subjects per group:
trained, Easy; trained, Hard; novice, Easy; novice, Hard. The order of
the conditions were:
1. training (for subjects in trained conditions only);
2. a practice task;
3. a baseline task;
4. the learning task.
Trainini
During the training session , subjects learned a task related to the
final learning task. Using a simplified training manual , the subjects
in the two “trained ” groups learned three commands of the text editor
called ED: how to print, how to delete , and how to add a line of text.
After practicing these three commands on the computer terminal , the sub-
jects were given three new texts, each of which had mistakes marked:
they were asked to fix the mistakes . They could refer back to the
training manual. During the training , the experimenter was available
for help, but no more information than that given in the training manual
was provided . The amount of training was selected to be sufficient for
subjects to gain a sense of the text editor . The training session took
some 25 to 50 minutes, depending on the ability of each subject. All
the trained subjects reached the level where they could fix at least one
text out of three without any help .
The Dracti~e ~~~~~~~~ baseline tasks
The goal of the experiment was to collect freely occurring thoughts
and questions of the subjects who were learning different levels of
materials. Accordingly, it was felt important to train the subjects in
giving protocols naturally. (It takes a little bit of training to speak
thoughts and questions aloud freely, especially when the thoughts are
not responded to.) In addition , because different subjects would have
different baseline rates of asking questions and speaking , we wished to
get a calibration of their baseline rate.
All the materials for the practice , baseline , and learning tasks
were presented on individual index cards, held in a loose-leaf binder .
Each card had one or two sentences. Subjects read each card and were
instructed to say their thoughts and questions aloud.
The practice material was three sentences taken from an article in
a popular science magazine (Mahoney, 1978) . Each sentence was on a
~~~~~~~
_______ -~~~~~~~~~~~~~~
5
separate index card. After each subject finished reading and responding
to the third sentence, all the quesquestions up to that point were
answered by the experimenter. There was only one subject who did not
say anything, and so he was given the two most common questions and
their answers as examples.
The baseline task was to learn to play Owari , an African board and
counter game , quite different from most board games in the United
States. The explanation was made deliberately obscure. It consisted of
six very short paragraphs ( only one or two sentences each) , each on a
separate index card. Subjects’ questions were again answered at the end
of this session. (All subjects asked questions in the baseline task.)
I~t lear’nina ±.ask
Two versions of the instructional manual f o r the text editor NED
were constructed. The ~~~~ Manual was readable for people with no
experience with computers; it used only non—technical language, gave
detailed explanation for concepts, and taught five commands by first
presenting concrete examples and later more abstract explanations. The
~~~ Manual was more technical , written for experienced readers . It
taught seven commands and three special characters . It present ed
abstract conceptual explanations first , followed by a specification of
how to do each comnand . Both the Easy and Hard Manuals were divided
into 30 steps, each of which consisted of one or two sentences, each
step on a separate index card . (NED differs from ED, the editor used in
the training . The instruction manual used for ED was considerably dif-
ferent from either of the manuals used in the learning task for NED. )
_ _ _ ~~~~~~ -- -~~~~~
6
During the learning task, subjects were asked to read the material
page by page and to say aloud all the questions and thoughts which came
to mind. The instructions were specifically aimed at making the task
natural. Al]. the utterances were tape recorded. In order to keep the
amount of information available the same from subject to subject , the
experimenter never responded to these questions.
Sub lects
Sixty undergraduates in introductory psychology courses at the
University of California, San Diego served as subjects, either for
course credits or pay. They were pre—selected to have no knowledge of
either text editors or computers. It was hoped that no subjects would
be familiar with the base line task, Owari. However, 12 subjects recog-
n.tzed the game while reading the explanation. They were disqualified and
replaced with new subjects.
Results
Codina ~~~
Transcriets
All the protocols recorded during the baseline and learning tasks
were transcribed . There does not yet exist an adequate method for
analyzing naturally occuring questions. Considerable syntactic and
semantic variations were observed in these studies. Accordingly, we
first devised a rough classification of the utterances into ten dif—
ferent categories. It was difficult to make unambiguous classifications
of all the questions into these ten categories, however, and so these
categories were grouped into four composite, more conceptual categories
r~ ~~~~~~~~~~~~ - -
— -~~~~~~~ -~~
7
(plus a fifth category, looking back) . Four of the transcripts , one
from each experimental group, were randomly selected and coded by an
independent judge and the results were compared with the codings by one
of the authors (N.M.). Between coder reliability was only about 0.7 on
the ten categories, but was 0.95 on the composite classifications.
Examples of the questions and the categorizations used are shown in
Table 1. Only the five categories are considered in this paper. They
are:
1. Overall questions: The total number of questions, including
repetitions , paraphrases of previously asked questions , and inargi—
rial questions (an interrogative , but with no substantial content) .
2. Number ~~ ~onceots: The number of concepts for which further
clarification was asked . Operationally , this is the total number
of questions , minus repetitions , paraphrases and narginal ques-
tions.
3. Rvnotheses: The number of specific hypotheses or subjects ’ own
interpretations——a measure of how easily subjects could speculate
about the possible answers to their own questions.
The composite categories “number of concepts” and “hypotheses”
are both subsets of the composite category “overall questions”:
moreover , the two overlap in sharing some common subcategories .
~4. Confirmations: The number of utterances (other than questions)
in which subjects stated their own interpretation of the material.
Mere read ing aloud was not included . This category reflects sub—
je cts’ understanding of the material , but with an apparent need for
confirmation of the interpretation.
r~~~~r~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
8
Table 1
Coding Categories, Example statements , and Composite Categories.
(In parentheses are shown previously asked questions.)
~~~~~2ries Example statements
Questions
With contents
Original
With hypotheses “Is NED a person?”
Without hypotheses “What does NED stand for?”
Paraphrased
With hypotheses “(Is NED a person?) Or a computer?”
Without hypotheses “(What is the commnad?) What type of
thing do you want to type in, that
would be considered as conmand?”
Repeated “(What is the buffer?) Again, what is
the buffer?”
Without ccntents “What...?”
Judgements “I am confused .”
Confirmations “Oh , so insert goes backward , instead
of forward like append does. ”
Verbatim reading aloud
Composite Categories:
Overall questions = Al]. statements categorized as “Questions” above.
Number of concepts = All statements categorized as “Questions -
With contents — Original” above. - —
Hypotheses = “Questions — With contents — Original — With hypotheses”
+ “Questions - With contents — Paraphrased — With hypotheses.”
Confirmations = All statements categorized as “Confirmations” above.
- --
~
- . - .- - - -~~—-~~~~~~~~~~~~--- _ _ _
r _ _ _ _
9
5. J.,~g~—backs: The total number of the times subjects turned back
to previous pages . The number of pages turned back was riot
counted. (The data were recorded by the experimenter during the
baseline and learning tasks.)
Ad lustment ~~~ tendency ~g j~g~ auestions
There were considerable individual differences in the tendency to
speak aloud. To compensate for this propensity to speak , the rr.imber of
utterances in the experimental condition was adjusted by using the
number of questions observed in the baseline condition . The argument
for this adjustment is simple. In~the baseline condition , all subjects
were novices. Therefore , differences in the number of questions could
only result from individual differences in speaking styles. Moreover,
there were no significant differences in mean baseline frequency among
the tour experimental groups. The correlation between overall questions
in the baseline condition and overall questions in the learnIng task was
.66 . The regression equation for the number of overall questions was :
TQ 6.2 4. 1 . 7 BQ,
where TQ is the learning task questions and SQ is the baseline ques-
tions. Accordingly, the slope of the regression equation was used to
determine the adjustment factor: the number of overall questions , number
of concepts, and hypotheses were divided by 1.78Q . All statistical ana-
lyses were done on both raw scores and adjusted scores , and both values
will be reported. (We used the slope of the regression equation for our
The mean number of overall baseline questions was J4 .7 for the novicesubjects with the Easy Manual , ~4.9 for the trained subjects with theEasy Manual, 7.2 for the novice subjects with the Hard Manual, 5.5 forthe t rained subjects with the Hard Manual .
_ _ _ _ _
______ -
10
correction because it provided a satisfactory and easy estimate. Note,
howev er , that our results are not sensitive to the form of the adjust-
ment used . The primary point of the paper is confirmed by using the raw
data , without adjustment . Performing the adjustment does allow some
secondary results to reach significance.)
The number of “confirmations” and “look—backs ” were not adjusted
because the baseline trequencie5 for more than t~O subjects were zero
(this is about 2/3 of the subjects). Moreover, the correlation between
the baseline and the learning phase for these two categories was low Cr
= .23 and .38, for confirmations and look—backs respectively).
Findinas
The results for questions confirm expectations. All the adjusted
questioning scores show significant interactions between the training
conditions and the levels of the instructional manuals: with the easier
manual , novice subjects asked more questions than trained ; with the
harder manual , t rained subjects asked more questions than novices. Fig-
ure 1 shows the pattern based on the number of concepts , the most rigid,
qualitative measure of questioning. The L values for the interactions
calculated from the 2 X 2 analysis of variance (training conditions by
manual levels) are , £ (1 , 56) = 32.3k, ~~. < .01 for overall questions; .F.
(1 , 56) = 32.35 , ~ < .01 for the number of concepts ; F. (1 , 56) 6.59, .~~.
< .05 for hypotheses. 1
1. MaIn effects were not significant , except for training conditions onoverall questions (.~. (1, 56) = 6 . 93 , ~~. ( .05) and on concepts (.~~ (1 , 56)= ~~~~ ~ < .05). Main effects themselves do not convey much informa-tion , because the experimental hypotheses clearly call for the signifi-cant interactions.
—
~
-
~
-- .- --—---- -.-
~
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
r.~Y - --- -
11
3 0-a4,
VI
a
0. 2.0~~ • Trained
~
..-.--‘.- ._~
___•
— -~
Navice
I
Easy Ha rd
Manual
Figure 1. Interactional pattern of adjusted scores for number ofconcepts asked during learning task.
_ _ _ _ _ _ _ _ _ _ _ _ _
______
12
The mean questioning scores (both raw and adjusted) are shown in
Table 2. With the exception of the hypotheses, all the raw frequencies
show the interacting pattern in the predicted order. The £ values for
the interactions are £ (1, 56) = 12.39, ~ ( .01 for overall questions; £(1 , 56) 11.72, ~ < .01 for the number of concepts; ~ (1 , 56) = 1.90,
for hypotheses. 2
Simple main effects for the factor of manual levels are mostly sig-
nificant. All, the questioning scores on the Easy Manual are signifi-
cantly greater for novices than for trained subjects. The £ values are
£ (1 , 56) = 34.61, ~~~, < .01 for overall questions; Z. (1 , 56) = 30.50, .a <
.01 for concepts; £ (1 , 56) = 13.76 , ~, < .0 1 for hypotheses. On the
Hard Manual , where the question is whether the trained subjects asked
more questions than the novices, only the category of hypotheses fails
to reach the significant level difference. The £ values for overall
questions, concepts, and hypotheses are £ (1 , 56) = ~4. 73 , ,~ < .05 ; Z (1 ,
56) 6.39 , ~ < .05 ; and ~ (1 , 56) = O.~5, respectively.
Mean number of confirmations and look-backs are shown in Figure 2
and Table 3 (these are raw, unadjusted data) . Although the interaction
of confirmation does not reach significance, CL (1 , 56) = 2.22) , the
data do show the appropriate pattern of interaction. Look-backs have a
unique pattern, with neither significant interaction CL (1 , 56) = 0.30),
nor any significant differences between the novices and the trained sub-
jects . The Hard Manual elicits more look-backs than the Easy Manual
(main ,~~~ for manual levels is £ (1, 56) = 4.92 , ~~. < .05).
2. Only one main effect , the training condition factor on overall ques-tions (L (1, 56) 5.44, ~g. < .05) is significant.
-~~~~~~- - - - -—~~~~~~~~~~~—-~~~
,~~~
______ .- . . — - -~~
-- •~~
- ---
~~~1~~
13
Table 2
Mean Number of Questions Asked
While Learning the Easy Manual or the Hard Manual.
(Standard deviations are given in parentheses.)
Easy Manual Hard Manual
Novice Trained Novice Trained
Raw frequencies
Overall 20.9 6.5 16.2 19.1
(11.9) C L~.5) (10.1) (10.1)
Concepts 15.1 5.9 12.5 15.3
( 7.3) ( 3 .9) ( 7.6) ( 7.6)
Hypotheses 7.0 1.5 7.5 7.2
C 7.5) C 1 . 7) ( 9.1) C 8.5)
Adjusted scores
Overall 2.8 0.9 1.5 2.2
( 1.2) C 0.7) ( 0.5) C 0.9)
Concepts 2.1 0.8 1.1 1.8
( 0.8) ( 0.6) C 0.4) ( 0.7)
Hypotheses 0.9 0.2 0.6 0.7
C 0.9) ( 0.2) ( 0.3) C 0.8)
~
14
Table 3
Mean Number of Confirmations and Look—backs.
(Standard deviations are given in parentheses.)
Easy Manual Hard Manual
Novice Trained Novice Trained
Confirmations 2.5 1.3 1.0 2.9
C 4.5) C 2.4) C 2.1) C 6.0)
Look—backs 1.3 1.7 2.7 3.9
C 1.3) ( 2.2 ) C 1.9) C 5.3)
__________________________ - - _ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - - —
15
A3 0 ’
• • Tra ined
E 2.0
1.0’ ~~~
Novice
-
Easy HardManual
4.0 ’
~~~ Trai ned
~~.1
~ 3.0C) Novice
I ——
—-p
o 20’ p
0_. w ——
10
Easy HardManual
Figure 2. Interactional patterns of confirmations and look—backs .A: Confirmations. R: Look—backs.
—.— - - - - . - —— - _
L. -
• -‘~~ -~~~~-~~~~~~—--_ - _
—~~
--- • --- . - - — • — -
~---
__________
16
Discussion
The most interesting result is that novices do not ask many ques-
tions on material that is too difficult. This finding shows that a
theory of questioning that suggests that people ask questions to fill in
their knowledge structures is too simplistic. People do not appear to
be able to cope with material, too far beyond their present knowledge.
The number of overall questions gives a quantitative measure of the
general tendency to ask questions. The number of concepts and
hypotheses can be viewed as reflecting more qualitative aspects of ques-
tioning. To ask about a concept, the subjects must know what is missing
and what is necessary for further understanding; questions asking about
concepts thus imply the asker is at an appropriate distance from the
given material. To create a hypothesis, subjects should nave scme
expectation or inferred understanding beyond the given material;
hypotheses thus imply the asker is active in constructing a knowledge
structure. In this sense, the novices who learned on the easier manual
and the trained subjects on the harder manual had the greater tendency
to ask questions, not only quantitatively but qualitatively.
Because the two inatruotional manuals differed in structural pro-
perties, it is possible to inter what subjects in each group were doing.
The Easy Manual first presented concrete examples and later , more
abstract explanations; the Hard Manual presented the abstract concepts
first and then , later , the detailed specification of how to use each
co and.
~~~~~~~~~~~~~~~~~~~~~~~~ -
j
17
The novices given the Easy Manual started without knowledge of the
text editor task , but the manual provided concrete examples which could
be used as the “core” to start learning. Thus, these subjects could
follow the manual, constantly expanding the scope of their knowledge.
The novices on the Hard Manual also started without any specific
knowledge, but had to cope immediately with the abstract framework. It
was as if they did not know what they should know to know further. By
the time they came to more concrete examples, it was too late.
Look—backs can be regarded either as a measure of the complexity of
the material or as a measure of memory loss . There should be more
memory loss for the novices on the Hard Manual , because they had the
greatest trouble in following the material. Similarly, trained subjects
on the Hard Manual should need to check back on the materials , beca use
they were taking in the largest amount of information. These two types
of look—backs are indistinguishable with the present data. The data for
look—backs are consistent with either interpretation. However, the
relatively high score of the novices with the Hard Manual implies that
they were actually trying to understand the material, not just flipping
the page with boredom; they simply did not know what they should ask.
This paper is one of the first experimental attempts to manipulate
question asking in a learn.tng task. The results show that at least some
aspects of complex learning are accessible through the measures of ques—
tion asking. The results also show that any theory of question asking
in learning cannot simply use the gaps in a person’s knowledge as the
source of questions. To ask a question, knowing too little is just as
disadvantageous as knowing too much.
- - -.--
~
--— --
~
- -— -~~~~ -- - .-- - — ---~~~~- ---~~.--— -- --- - — - - _ •-
18
Reference Note
1. Grosz, 8. J. ~~~~~~~ reoresentation ~~~~~~~~ use ,Q~~ focus ,j~ dialoaue
imderstandiria. (Technical Note 15 1) . Menlo Park , Calif.: Stanford
Research Institute, July 1977.
19
References
Levin, J. A., & Moore, .J. A. Dialogue—games: Metacommunication struc-
tures for natural language interaction. Cognitive Science, 1977 ,
.1, 395—420.
Mahoney, M. J. Publish and perish. Human Behavior, February, 1978.
_ _ _ _ -- - -- - ---- -- . • - I
~~-~~-•-~~- ~~—.- -~~~~ •-,— —
• 20
Acknowledgement
We thank the LNR Research Group for their support and guidance.
Partial research support was provided by a contract from the Office of
Naval Research and the Advanced Research Projects Agency, monitored by
ONR under contract N0014—76—C—0628, NR 154—387. Support was also pro-
vided by a National Institutes of Mental Health grant MH—15828 to the
Center for Human Information Processing. N. Miyake is supported by the
Japan Society for the Promotion of Science. Address reprint requests to
Naomi Miyake, Department of Psychology, C—009, University of California
San Diego, L.a Jolla, California 92093. -
~~~~ -~~~~~~~~~~~
-~~~~~~~~ -••~~~~~~~~ _ _
r~~~~v - -
~~~~~~~~~~
504? 41? Ft•*
I (0’ 04 01104 I P*73 10(Ij; 3 *01 000,3.0147 0004rOry 3 0,’ . 00...1l 5000•( I 04? . 0 IIorr.* 0 *1* 0 4005234 1~ -••fl 700001(002 II~ /0*000 447101 44 (*9(2. 004424*08 144404 s.o.s. 30/00 3 0,8000,0. for S,7o,..01.. 010.01,. 64~4,. 1 •, 007 0. _fl - , 31.040 - 4 - - 07 0 0 / 0 4 4— 00045 . /4 52032 22 0 0.0 04flLu 4 .440 40000 *42 40., *0, 5 6 3 4 2 004000.4 *43 Of 1.0110730. . 00/. 000(o 4 lo . r .ooy l I to’...,,Lola . S. 0375 (~~ _4 2.. 4020. 44 92093
~,._,, 44 s on, .j , :4/ . •3 0?. J90A 001.0093 3 0~ . 5. 4. 00(31*4*0P?50S0 I *0.0,010 D0~0, I P0130.104004 920.04./u 1 0,. Jot, S. to..U.S. 0*092 P0019204o.0. 02000 0 440 I?8~0l 0f711• fl1041.PUflh J500 (It , 14 0 3 * 3 1 *•154 1*0/ 4..OIL - ,..l00,, 00. 404.03/SI. 4* 4 ,4 340* 0 0030 8401 00,00 50,0,4 537 3 0,730. 4* 00 *0.4 , ,,s ,,, — , /,*4*00009 . /4 9 0 3 0 ) ~~~. oat. . 7*4. 4200 . 4 00 003 ..~~~o ,.., *•0 1 00 1I 0,• ‘0• Or 000004 000*0* 0*03 Li AIfl Pro/U5000.10110 000.40 *., UIiSL,PLPIIO 3*401000 2*0 . I a. c. vociog nposa ( 1 so - lu, o, • - : 00 .31 0 0 0 7,00 JUL00 3/f 04500 0500(304 4 0L1.10400 API . Al *3229 4 I CAT SIC ‘,00/ 7.3 . 0,. 4 / .0/4*01. 220 9 0,0.041110 04.1,05 O,ooop( to~~~ 0(10U0Ofr fl_ ALA , SO01~ 2 /104.1. . 13 .4 ..o’:’~o . , 0t~ ’t,o ,, S,ot04.014*3.00.. 904*05, I 0?. 0000 C. 00.0(000 ) AII L,A3I I 000 59 37474 03. 0 0 1 / 0 0 4 0 7 1 0 * 0,. . ’ O . O . ’ / O f . 0 . . ’ . 4 1 . / A• 00. 04000 • .0.5. 4,0 008 P0 0*01.., , .4 04303 0004001 ‘P.10.0000 40. (4,0. 07 40031 0.l~~O ,~~/ 0 4 4 0 .40.03/ ( 0 0 / / P s L400fltos’, 0000 0540 3 0 00‘4 0 5 4 : / ~~0.t /•3/0~ ? I (.44 . 0000013/0 . 00*0000.0*0 *04 31.41..
3 00 0 0,1 — 0 - • .. .0 , • .0 -, 0 10’ 0. -• , 0_.410 0/. ’ (0P 020 • 0 *4’ *00101 (*3/nAy ( 810*0/fl) %n0400000 rO3/ 0.40 . 5 • - . l~~~~ ,• . — . .oflo,001004. *0*, 01 oI.44 - 1.0. 3.00.4’,? .‘I, •_~ 4~ L0~~ On *0 0.4 .000*44144(10. . 00 20310 00101*40 02 31 0*01. 9*11 0 / 4 * 411/0 3/.’.. 0 0 0 - . •000 / 140.0 01 20007
004*10.00 40. 01 7 00 III 04 f I .0. 00 ~ 044*4 ~ 04 —a *44/ ,34400* 044. 00 ‘0 00 03/000
:0. 15 0003. Ci... •— ,~~ Is 05*4 0? *0 20332 *14 000 (~o4 403 *0 /0 /0
.0 3 05 00* 2U °‘ 0-’ 0 ’- ~°’:O~ /0441 03/5.. •0 . ‘C_ ‘ .4 0 /0 00033 b0 / , , , * , 1 0 0 10.4 .2,7/50.50/ 0000/0 ’. *1, .0.00/ -- u*(?0L10 . 00 . 402 I 92. 11.35.53 134,0013 I 00 40020,, Ca.., “ C’ O0 i•001. .10. 0.34/ 000. 4-~~~~~ * 00*010*0 ~1034*C0 00044013 92P(C3 1.00002 64 4 405* 1 . 01/ U . /4 0 0! 00304033..A000/0000 . 304 o .‘ ‘ o fl 741 073.. . ) *00.0.0(01 1000 (ILS0( *0.00 . 000 3/ 00.040/000 0 4 0 , 0 0 . 0 3 0 c_A Ol10
~ ~,, 1000 ~~ 209 .0/o0 0 ....... 44. ... ’ .. 3000 04 . / 3 0 / 0 . P/So. PD Il 0*103/.. 000 .000010# COlt.? / 0, *40, /23 /0 /0/ 2000 00400.04 0,05‘4” /4
0/ /10* 09 4/~ .0. 000 4
~ 4 4CII
34(00 03/2 00 30 9 00*, .3/ •~ .13 4 . ...
1*3/00 • 0 73. 12504 000
~~~~~ 73. 0250* P0041073 I0~~~ 3 92 ~~~~~~~ A / SOPS 1401.0 . .2 ‘ o oI “~~~~~~~~
‘ - l~
- /4• 04 300/513/0 41L140044 , 0* 22200 Co 0$• 0 5*00 ‘‘ 0 01 (*100( 3 / / t o o I I •I 4. 0. 040000. 5900.?000y 07 0003.00.7 0.50,0*03/I 01 / 0 7 0 . 4 3
00 .0.00 7 (004 pg C~~ 400 4. 0. .0
10 ~~~~ 9 00000* 10 P 40/4/ ~~ 00 003/0. *1 4 to, CO11.09330., 1* .0.07/’ ~~~ 04 02 52 0/rI o. or 00.1., 3007ot*,j Of 0.?,o*.
to, ,....,‘ot & r.soo..rorg I 0’ hh1~~ 0. _~~ ‘~ ‘ I 0. 00,10 *0.,.. • —• ol 007 00 40 o OP.. *0 *0 00 0003/ 0.4. 93 ,* 07
10 so
‘1 4* *00 °0~ 4(0/ 31 00 2 4 10 40
0/ *0 01303 0*30 0.03/ *40 00 /4 50 Mt 44
34 40 00 Or 310400 03 7000 3/4 /0 0.(0 44 •_ 00 00 .304 ~~
~~~03/ ~• 00 00
04 .3/ 9.0*, 40104 0020037
04 3400 Ctat0.. 5 Th ola l J O . 30/ . 000 4.0011400,, 0/ 2020* 3 0,. *400,3 3/1.7(00 00 )0 /30 / /030/0 / /100
/0*00 0.000.. *30 nO 0 0.0* 30*, P 00*4 .3/0?. 0(001 II, 000.10 *,t /.00., 0 04000 03/ 0( 0.0.4,. *40.3/40103 3
• — .4*3. 097
P01.30,0 .. , 0 .~~~ . •~~. (.000,01 0.0.43/. P0004* 0103/ • 090400 -700.00 .5 10 05 030 00 .. PD a• or 00/04/0 L 93 • ,D 0000004o , 00 20013 0 0 / 0 4 / 4 / 0 4 07 _ 0 . . f . • , ,o .01 ‘ lolL 3,3 00 ,070 ’ .4-. /1,0/1:00 2 5 0?’ 04
0* 0 11.0 4,0 0 ~~ 3/0 00• — ,.0*0 03 .100’ ‘ ‘ 4.0031 44003/ , 0’, 2020* £4301.4 3*34.000, 0400 - .4000/103 P40 ,3/ / I . ,
‘00 09 0* 01 0/ 0,3/
000 4 9
0 0 * 0 0
01
4000 3(10 *0 2~5~~ 70 1000 7 114
~.
500
3 *01 *0 014.11100 0 5 5 00 00 /2 “30 0 0 / 0 / 000, . 13 5 , . 0 0 , 1 1 / 3 . 04 0 2 1 ) 1 VIP? . /7 7s30001043 50/,.?’ sp o t 4 ( , MD 2031.
P05*20.31.. 3/ o, , ,, .0 (300 /0 34004.’ 0 1 ) 0 ./ 0 • x ‘ 0 3 . - 1? ‘ 0000 *0*1*4’ 04.2 1041310* 0*164*0103 I 0 . . 2030 4 . r,.3.,o n,,
3/ 4’09 30 3 5 8 43/ O0 ,o. o. 15* ~,,
Dr 4.0 0 *3/ 53/ 3/40 40 344 38 0~s 3136400r’,.,0l 07 P0 1333/104, ‘00, - .
‘ 0/ 00/0/10 4 ‘*4/ 0 , 2 ’ - toront o 110m 04100r.lt, 0 0!. P0*0*7 001007 P.0,1., . 0. 0 0 , . 4)50 /10 * .. * p030.00? ,, . PA 1 52 ) 4 1300
400 10• 500 3 0/5030 , 000 4 340/1 04134*3073 01 3130000303 ‘ ‘ ‘~~ -.P., 33 3 444 75643/ 3039 II4~4 ~~
03 C-.40100/ 0*3 *7 007* 07730*107* P 5 3• *7. 23/0530 90,0 00 0*8000 P1030. C_I07 4000 £o, 4 * 3 0 , 00 0 / , /0.0. *4.3 *0,00roO / 0 3 0 1 / l o t . 19 33 0, 18*0111* 0/0*0* I 0 4 . 1?. 0000.0.4.- * .- 500 1 *00*0*000, *0.30. ~~~~493/43 CO *0310 i.r.s 04*0 , , p. n- - • 3 0/000021 10 . 04 223 33 4 333 Corol. 3010 01. ‘ °“•‘
~ 0I.’ ’ 000/01 3 / 7 3 )01/ ’o -
010 ,, ,., P.73/I.00010540 0.00*100 07414 Polo 0000 . /4 9033/0 0*3/3/0.0 n
-‘ . 0 (0 4 / 1 / 0 3 0.0030*
0.70. of 0.1.30. (47.7 0(10*.) ~~1~~~ l01 L I
1, / O r 0(301 , ACT 2000 , 00,3/1.10. IPX 0, 00*090
C 0r0440, 34 02)39- . 204. 73/ 0*3/
0(03000033 0? 1070754109393 7 400 00 Co 0...
~~ 4. 4 939 0 370141 0’ 000443041 0 . 0( 22 3 11,I**r5037 0? 4104(0 1, P0ft3W10033, 75 ‘724 3 0, 0034 1
30 0*7 I I 100! 0 °‘ 0*0.4 444 1 330
‘~ “‘ 00 104 00 00100 44S. 09 00 0 7*711.1.00000130700 . 00 20380 3 *3’ . *1,002.. A. 03.0 4*3 •1. 0* 00002 20 3? 0. ~~~1VT 7403*03O 7,,040..1 0 00./3/33 !000~~ ol 1700( 0 100.000000 0.90 . of P.?00.01747 10000070*. 740 22207• 5240 459/ 30l,~~0410 34.3. 001014. 2 0 4 . wboro 0.3*0.3,030 .. 905 J .? 34,0.1 64. 0*05 Co,’p.,.IIo. I *0 . P37*0 0,1544IP. ’ l•4 , 0 . £ 0 .0 50 J SOPII J . 1140*. CIII? 003r 00410 . 1* 33013 3330 9007 *or.*t 0*17 . 0? 7410740.043
000*OPCO (04.1/621 3.01. It*oo~~, 08 80000 0(1/0*3433 3/1 004045910(30 10(0* MS I 03’ . 032. 00.040 *00.004! . 00 40342
((3310/744, DC 20*50 7*74,3.07 * 4? P0?3S0O(%/M030’lIt? If 441033004000040? , 00 00702
- “ _ _ - - - - , —---.~~~~-- .. .~~~~~~~~~~, ~~~~ . —---