COGNITIVE SCIENCE Vol22 (4) 1998, pp. 517-546 ISSN 0364-0213 Copyright 0 1998 Cognitive Science Society, Inc. All rights of reproduction in any form reserved.

Pertinence Generation

in Radiological Diagnosis:

Spreading Activation and

the Nature of Expertise

ERIC RAUFASTE

HELENE EYROLLE

CLAUDETTE MARINE

Universite’ de Toulouse Le Mirail

An empirical study of human expert reasoning processes is presented. Its pur- pose is to test a model of how a human expert’s cognitive system learns to detect, and does detect, pertinent data and hypotheses. This process is called pertinence generation. The model is based on the phenomenon of spreading activation within semantic networks. Twenty-two radiologists were asked to produce diagnoses from two very difficult X-ray films. As the model predicted, pertinence increased with experience and with semantic network integration. However, the experts whose daily work involved explicit reasoning were able, in addition, to go beyond and to generate more pertinence. The results sug- gest that two qualitatively different kinds of expertise, basic and super, should be distinguished. A reinterpretation of the results of Lesgold et al. (1988) is proposed, suggesting that apparent nonmonotonicities in performance ore not representative of common radiological expertise acquisition but result from the inclusion of basic and super expertise on the same curve.

I. INTRODUCTION

Problem solving consists of finding a path that gradually reduces the gap between a current

state and a goal state (Newell & Simon, 1972). However, in diagnosis the goal state is not

given and must be identified first by the physician: “Medical problem solving proceeds by

selecting a number of diagnostic hypotheses as possible goals, and then testing to see

whether one or more of the hypotheses selected can be justified.” (Elstein, Shulman, &

Sprafka, 1978, p. 21). Early diagnostic hypotheses guide data gathering (Barrows, Nor-

man, Neufeld, & Feightner, 1982; Lesgold, Rubinson, Feltovich, Glaser, Klopfer, & Wang,

Direct all correspondence to: Eric Raufaste, Laboratoire Travail et Cognition, UMR 5551 du CNRS / UniversitC

Toulouse Le Mirail, Maison de la Recherche, 5, All&s A. Machado, 31058 Toulouse Cedex, France.

517

518 RAUFASTE, EYROLLE, AND MARINk

1988) in such a way that further processes tend to eliminate early diagnoses rather than add

new ones (Elstein et al., 1978; Joseph & Patel, 1990; Sebillotte, 1984). Consequently, the

problem of how early hypotheses are selected is crucial in medical reasoning. More specif-

ically, given the large number of possible diagnostic schemata, and the fact that diagnosti-

cians often make use of irrelevant information (Doherty, Schiavo, Mynatt, & Tweney,

1981), an important question is how are pertinent rather than non-pertinent hypotheses

selected. Acquisition of expertise in medical diagnosis could also be related to the ability

to select appropriate information and hypotheses (Custers, Boshuizen, & Schmidt, 1996).

The present study was designed to investigate what we call the problem of pertinence gen-

eration. This involves both the question of how pertinence increases with experience and

the question of how pertinence is created by the expert’s mental processes. After reviewing

the related work, we present a semantic-network-based model of expertise acquisition and

an empirical study of radiologists’ diagnostic activity.

Levels of Expertise and Pertinence Generation

We begin with three questions. The first, how expertise develops, relates to the nature of

expertise. The second, how pertinence is generated, addresses the issue of the processes

that allow early selection of pertinent hypotheses. Third, we examine how recognition of

pertinence can be expected to develop.

1. How does Expertise Develop?

Expertise (and thus the cognitive processes underlying expert performance) is thought to

develop in various ways, depending on the authors’ assumptions. Dreyfus and Dreyfus

(1986) proposed a five-step model of expertise development. At the first level, the novice

uses context-independent rules without taking into account the pertinent cues that exist

within the context. The next two steps are rule-based reasoning refinements. At the fourth

level, the subject goes beyond analytical reasoning and uses holistic recognition, although

if necessary, she or he may determine a goal structure and return to analytic processing.

Finally, the expert uses intuitions that cannot really be explained. Even if deliberating, the

expert does not use reasoning but proceeds by comparisons between the current situation

and his reservoir of memorized situations. Thus, Dreyfus and Dreyfus’ model tends to pro-

mote a conception of expertise acquisition where automaticity is what characterizes the

real expert.

Chase and Simon (1973a, 1973b) described an unconscious and fast perceptual pro-

cessing that structures perception in chess players and retrieves memorized patterns for

subsequent slow conscious processing. According to Lesgold et al. (1988), the first kind of

processing in radiological diagnosis is perceptual processing. The outcome of perceptual

processing is a set of pointers directed toward the patterns that are most compatible with

the cues contained in the X-ray film. However, one problem in perceptual processing is dis-

crimination insufficiency, particularly in radiological diagnosis. Several diagnoses are

often compatible with all the data, so a simple recognition process is inadequate. Accord-

ing to Lesgold, deeper cognitive-based processing is then necessary. In this second step,

the cognitive apparatus delays the action that would be based solely on probabilities and

tries to resolve ambiguity by obtaining more information. Such cognitive processing would

be triggered by the outcomes of the perceptual process. Only experts would develop a

plain, cognitive way of reasoning because cognitive processing uses the outcomes of per-

ceptual processing as input, and because prior contact with a large number of experiences

would be necessary to develop perceptual processes. Concerning perceptual and cognitive

processing, Lesgold et al. (1988) proposed a three-stage model of expertise development.

They observed that intermediate individuals sometimes demonstrate worse performance

than novices and experts. Lesgold et al. attributed this temporary loss of performance to a

transition phase in the three-stage cognitive development. The three stages hypothesized

by Lesgold and his colleagues were: (1) Subsymbolic processes develop with the first

experiences. (2) Perceptual processes begin to produce outcomes. These outcomes trigger

cognitive processes which may start to gain importance. But at that time, they may still be

in conflict with the perceptual processes. (3) Cognitive processing is sufficiently developed

and the subjects are now able to tune their schemata and test them, so that perceptual and

cognitive processes are no longer in conflict.

Lesgold’s model is the opposite of Dreyfus and Dreyfus’s model. It emphasizes percep-

tual processing in novices, and then both perceptual and cognitive processing in experts. In

the Dreyfus and Dreyfus view, cognitive processing comes first, and then reasoning disap-

pears, leaving only perceptual processing. However, these models have in common the

point that both processes exist within some period of the acquisition of expertise.

2. How is Pertinence Generated?

Pertinence might be generated by perceptual or cognitive processes. The two kinds of mod-

els will now be reviewed.

In connectionist models (Rumelhart & McClelland and the PDP Research Group,

1986), each concept is represented by a schema that can be viewed as a processor with an

activation level. Only the most activated schemata have a chance to control behavior.

When a cue matches parts of a schema, this increases its activation. Now, the most appro-

priate schemata are usually more likely to match environmental cues and thus to be

selected. In a connectionist view, generating pertinence may also result from connections

between schemata. When the activation of a schema goes over a certain threshold, it

spreads to the surrounding schemata. Balota and Paul (1996) found that the facilitation pro-

duced on a node by preliminary activation of several linked nodes (multiple priming) is

additive. Consequently, schemata that match several cues receive a large amount of activa-

tion, so they are reinforced more than non-pertinent schemata, which tend to be eliminated.

This competition among schemata could explain empirical studies showing that not all

available data are used by physicians (e.g., Hoffman, Slavic, & Rorel, 1968 for radiolo-

gists; Weber, Biickenholt, Hilton, & Wallace, 1993 for general practitioners). In the

experts studied by Lesgold et al., pertinent diagnostic schemata appeared to be triggered

within the first two seconds. According to Lesgold et al., this could result from a pandemo-

nium decision process (Selfridge, 1959) based on the probabilities associated with the dis-

eases, and resident training is supposed to make those probabilities more accurate. In the

520 RAUFASTE, EYROLLE, AND MARINk

same way, Medin and Edelson (1988) found an effect of base-rate knowledge acquired

from experience on diagnostic judgments, and according to Hasher and Zacks (1984), the

frequency of occurrences associated with facts is spontaneously coded by memory. Never-

theless, several studies have shown that a human’s estimation of probabilities is biased

(e.g., Smedslund, 1963; Kahneman & Tversky, 1973). Moreover, Medin and Edelson

found some heuristics used by subjects that enable them to take rare cases into account, for

example, when confronted with specific cues. According to Weber et al. (1993), physicians

make a compromise between likelihood and clinical severity because they cannot afford to

miss diagnoses with severe consequences.

The previous examples suggest that perceptual processes are not sufficient for solving

every case. So, how might cognitive processing explain pertinence generation?

Pate1 and Groen (1986) suggested that relevant causal information is accessed from the

initial representation of the case. Using a propositional analysis, they rebuilt causal net-

works, and showed that expert physicians demonstrated a better ability than novices to dis-

tinguish relevant from irrelevant material. Moreover, experts made more inferences from

the relevant material. Lemieux and Bordage (1992) found that diagnosticians who pro-

vided correct diagnoses had a deeper representation of the problem, mainly because they

were able to transform elementary meanings into rich and well organized semantic net-

works. When they collected data, they used a more pertinent and diversified set of semantic

axes that enabled them to think in terms of opposing properties. Lemieux and Bordage

hypothesized that successful diagnosticians have a greater “ability to recognize pertinent

relationships between abstract properties” (1992, p. 198). The abstract properties are

thought to be linked inside a network of formal qualities that are then used to drive induc-

tive inferences. Successful diagnosticians would be able to follow a vertical pathway of

abstraction from the case, whereas unsuccessful physicians would not. The pathway would

be built by using formal qualities which allow the physician, when confronted with a

branch, to decide what direction has to be taken.

Because early generation of hypotheses takes only a few seconds, we should consider

mechanisms that can generate pertinence automatically rather than by controlled process-

ing (Shiffrin and Schneider, 1977). The Lemieux and Bordage approach is appealing but

its limitation lies in the fact that it is based upon deliberate choices by the subjects, so it

cannot explain automated processing such as early hypothesis generation.

3. How can Pertinence be Expected to Develop?

Few studies have addressed the issue of pertinence development. Experts have regularly

been found to be more able to recognize pertinence than novices (Pate1 & Groen, 1986;

Lesgold et al., 1988; Lemieux & Bordage, 1992; Shanteau, 1992) but we know nothing

about the monotonicity of the pertinence curve. In perceptual processing, pertinence comes

from the interdependence of schemata within networks. In novice radiologists, where per-

ceptual processes are dominant (Lesgold et al., 1988), one can expect pertinence to grow

correlatively with that interdependence. On the other hand, when cognitive processing is

dominant, we do not know how pertinence develops: it might grow monotonically with the

progressive acquisition of mastery in cognitive reasoning, or it might temporarily decrease

due to a transition period during which subjects would no longer use genuine perceptual

processing and would not yet be able to use cognitive processing. Pate1 and Groen (1991)

reported three experiments where intermediates used a greater number of both relevant and

non-relevant inferences than novices and experts. Experts used fewer irrelevant inferences

than both novices and intermediates. Nevertheless, the experts in these studies were resi-

dents. In radiology, beginners already are residents and it would be hazardous to generalize

any conclusions without first knowing more about what mental processes are involved.

Connectionist models seem to be oriented toward perceptual processing, and symbolic

models seem to be oriented toward cognitive processing. But Lesgold et al. (1988) empir-

ically showed the necessity of finding a framework of radiological expertise that can inte-

grate both types. We shall now present a conception of expertise acquisition, which is

based on spreading-activation theory (Collins and Loftus, 1975), and could fulfill that con-

dition.

Toward a Model of Pertinence Generation

The schemata (and the links between those schemata) that have been explicitly acquired

through a university education will be called canonical schemata and canonical links. Our

main assumption is that explicitly acquired canonical knowledge constitutes a pre-struc-

tured semantic network to which further acquisitions will be attached. An analogy could be

a small crystal in a salt solution: the crystalline network develops around the initial crystal.

In other words, semantic networks of explicit knowledge can be the basis for further

implicit acquisitions. We will examine two questions: how experience affects semantic

networks, and how this could explain previous experimental results about expertise.

How does Experience Affect Semantic Networks?

According to Rabinowitz and McAuley (1990), connectionist approaches toward concep-

tual knowledge acquisition involve three aspects: the number of available concepts, the

number of units interconnected by associative links, and the potential for knowledge struc-

tures to activate the units so they can be retrieved. Applying this view to the acquisition of

medical expertise, we suggest that in spite of the initial existence of canonical schemata

and links in the semantic network, poor spreading activation potential does not allow nov-

ices to easily access the concepts (Rabinowitz & Chi, 1987) that are necessary to establish

a pertinent diagnosis. Each time a resident manages to see the association between a com-

bination of features and the correct diagnosis given by the professor, a subsymbolic asso-

ciation is created or reinforced within the semantic network. When a feature matches one

or more low-level concepts in the semantic network, the correct diagnosis (the one pro-

vided by the professor) is also activated. This simultaneous activation is repeated several

times so that subsymbolic associations can link two concepts in the semantic network. The

same result is obtained when the correct solution is given, not by a professor, but by further

examination, another x-ray, or anything that can confirm the real solution. Such an associ-

ation may also occur with abstract concepts, between a non-observable sign and a disease

for example, or between a disease and a complication. Such a process is plausible, given

522 RAUFASTE, EYROLLE, AND MARINE

that Frick and Lee (1995) demonstrated that implicit learning of concepts may occur by

mere exposure to the instances of the concepts.

As Rabinowitz and Chi (1987) stated, “the amount of activation that spreads from any

node is dependent on both the strength of the associative link and the level of activation of

a node” (p. 92). These notions are not easily measured in human knowledge bases such as

the ones experts have. It is easier to determine, when a schema is triggered, whether acti-

vation spreads or does not spread to surrounding ones. We defined as activable, links that,

through experience, became easy to activate, we called this property activability. As such,

the associative mechanisms produce two types of links that do not result from explicit

teaching alone (activable canonical links and operative links). Activable canonical links

are canonical links that already exist in the network but have been reinforced by experi-

ence. Operative links are new links, created only by an associative process. They connect

concepts that, canonically, were not directly connected. For instance, a radiologist sees a

film of a patient with an ablated lung and immediately starts to search for secondary loca-

tions of cancer. There is no direct causal relation between lung ablation and metastases.

The relation is indirectAancer is a possible, although not unique, cause of lung ablation.

The search for metastases is a normal procedure when a cancer is suspected. Experience

has created a new direct link between lung ablation and metastases, an operative link. This

point of view is an extension of the experimental results obtained by Dagenbach, Horst,

and Carr (1990), who showed that “very extensive episodic association of some types can

result in an addition to semantic memory that will function similarly to existing structures,

at least with respect to producing automatic semantic priming of a lexical decision” (p.

589). Considering that retrieval of a concept within a semantic network is a probabilistic

function that depends on the level of activation of the concept, Rabinowitz and McAuley

(1990) implemented a computer simulation of how knowledge structure affects free recall.

They found that variations in the strengths of associations between concepts determine the

amount of information available for retrieval. On this basis, we hypothesize that activable

and operative links make canonical knowledge accessible and useful for calculation.

Figure 1 illustrates these concepts by depicting a part of a hypothetical semantic net-

work showing canonical schemata and the links between them. The link between schemata

al and b is weakly activable. We assume that, in such a case, no automatic spreading of

activation will occur. Thus, the only way for schema b to be activated is through deliberate

attention on the subject’s part. On the other hand, the link between schemata a2 and b is

highly activable because it has been reinforced by one or more experiences. If activated,

schema a2 will automatically activate schema b. Furthermore, in this example an operative

link had previously been created between a2 and c, which is two steps away. This operative

link will enable the subject to directly activate schema c after seeing a2. For a less-experi-

enced subject who has not developed that link, only b would be activated and the activation

of c would need a deliberate inference from b.

If experience affects semantic networks by adding and reinforcing links, a second ques-

tion is: How could these changes in semantic networks explain empirical results concern-

ing expertise?

Knowledge Base

Explicit Learning

canonical schemata

canonical links a&able canonical links

non-activable canonical link l low-level schema (e.g., Rgural feature)

- activable canonical link 0 intermediate schema (e.g., semiological concept)

- operative link (shortcut) l high-level schema (e.g., diagnosis)

Figure 1. A Model of How Experience Affects the Knowledge Base

Working memory can be seen as the activated part of long-term memory (Anderson,

1983; Ericsson & Kintsch, 1995; and for empirical evidence Cantor & Engle, 1993). In this

point of view, only subjects who have a lot of activated canonical links and operative links

will have a rich and deep representation: rich, because the number of activated links deter-

mines the number of concepts that will be included in the representation; deep, because

concepts having a higher level of abstraction will be included. This mechanism seems suf-

ficient to explain why studies have shown that experts’ representations are deeper and

richer than those of novices.

In order to represent a global property of semantic networks that could account for the

interdependence of the elements in the representation, we define a numeric function called

integration. The desired function has to fulfill two requirements: (1) for a constant number

of schemata, integration should increase with the number of links; and (2) for a constant

number of links, integration should decrease with the number of schemata. A cluster (Les-

gold et al., 1988) is defined as a set of schemata such that each schema in the cluster is con-

nected to at least one other schema in the same cluster. Each separate schema is counted as

one cluster. An integration mark (I) for each cluster is defined by taking the ratio of the

number of links (L) within the cluster to the number of schemata (S). We can generalize

that formula for the whole representation:

Such a function has the advantage of being richness-independent: 20 schemata con-

nected by 20 links (I= 1) are equivalent to five schemata connected by five links. The result

is equal to half the mean number of links departing from or arriving at a schema.

As we reported earlier, experts seem to demonstrate better pertinence than novices in

their selection of cues and diagnostic hypotheses. From a connectionist standpoint, perti-

nence could come from the interrelations between the elements in the network-the more

schemata are linked together, the more pertinence will be demonstrated because pertinent

schemata will receive multiple primes and then receive more activation (Balota & Paul,

1996). Non-pertinent schemata could also be inhibited by concurrent ones, but here we

524 RAUFASTE, EYROLLE, AND MARINk

chose not to treat this issue. It is important to note that this relation between network inte-

gration and pertinence is not really causal: in novices, when integration increases, activa-

tion is roughly distributed among categories. In experts, more links connect schemata.

Many more combinations are then available and the distribution of activation among sche-

mata may be more precise and differentiated. Consequently, in a competition based upon

the level of activation, and for a given number of schemata, pertinent schemata are more

likely to be selected when more relations exist in the network. This is why, over all, we

can expect pertinence to increase with integration. However this relation is not causal

because there could be cases where a novice activates only accurate categories, leading to

one hundred-percent pertinence despite a sparse network. Conversely, the same configura-

tion of cues might activate a non-pertinent hypothesis in a particular expert, despite a

dense network. Nevertheless, these cases should be unlikely enough that if experience

increases the number and the activability of links then pertinence can be expected to grow

accordingly.

From that global framework, we can derive seven predictions:

1.

2.

3.

4.

5.

6.

Experience increases the number of activable canonical links. With experience, each

link is used more. Consequently, due to classical subsymbolic learning processes, its

activability will grow. The increased activability of canonical links should allow sche-

mata to be more easily retrieved. Therefore, a greater number of canonical links should

appear within the verbalizations.

Experience increases the number of operative links. This hypothesis is justified by the

same subsymbolic mechanisms as the previous one. The expected difference between

the two kinds of links comes from the fact that most canonical links exist before expe-

rience because they are explicitly acquired. Subsymbolic processes do not create them

but just make them more activable. On the other hand, operative links are supposed to

be acquired genuinely by experience-they result from the frequent simultaneous acti-

vation of two schemata that canonically are not directly linked.

A correlation exists between the number of operative and activable canonical links.

Because the numbers of activable canonical links and operative links are expected to

grow due to the effect of experience, we can expect them to be correlated.

Experience increases the integration of the representation. Because the accessibility

of schemata is expected to grow with experience, there should be, given a constant

number of features, a greater number of relations among the elements of the represen-

tation. This should produce a rise in integration: that is, an increase in the ratio of links

to schemata.

Pertinence is related to integration. Because pertinence is expected to come solely

from the effect of spreading activation within the semantic networks, there should be a correlation between integration and pertinence. In particular, pertinent schemata

should receive more activation because they are more compatible with the whole set

of data. Moreover, if pertinence results from the network structure, then the correlation

between pertinence and integration should be independent of experience.

The pertinence of the representation increases with experience. Because experience is expected to increase integration, and integration is expected to generate pertinence, more experience will lead to greater pertinence.

7. Accuracy depends on pertinence. Because pertinence is expected to determine the ini-

tial set of schemata, and given that prior studies have shown that subsequent process-

ing essentially consists of eliminating superfluous diagnoses, pertinence should be an

important factor for accuracy.

II. EXPERIMENTAL CONDITIONS

In order to test these hypotheses, we asked radiologists to examine some x-ray films and

make a diagnosis. We used a method similar to that designed by Lesgold et al. (1988) in

which several radiologists examined films of real cases.

Subjects

Our treatment of the subjects’ levels of experience was based on the work of Lesgold and

his colleagues (1988). Lesgold’s experts were considered as “outstanding” by colleagues

and had at least 10 years of experience after residency. But as not all experienced radiolo-

gists are outstanding, one could ask if this group was truly representative of the overall

population of radiologists. Consequently, we used four groups of subjects instead of three.

The first group consisted of four experts who had practiced radiology for at least 13 years

after residency. They were not only diagnosticians but also had an institutional role as

researchers and as teachers and trainers of residents. They were recognized by colleagues

as outstanding, among the top French specialists, so they were called super experts. They

had the kind of expertise that Lesgold et al. called “expert”. The second group consisted of

four other experts who had practiced radiology for at least six years after residency.

Because their professional roles involved neither teaching nor research, they were called

basic experts, that is, they were more representative of the overall population of radiolo-

gists. They were genuine practitioners. The third group consisted of eight “novices” who

were first- and second-year residents. The fourth group consisted of six intermediates who

were third- and fourth-year residents.

Task

Each physician was asked to examine two cases. In each case, the experimenter had the

physician follow a sequence of three steps.

1. In the first step, the physician was asked to produce a diagnosis from a posteroanterior

chest x-ray (the most typical type of x-ray). A radiologist typically has access to exter-

nal information, such as a clinical description, that constrains the exploration (Kundel

& Wright, 1969). In the experiment, however, we were interested in the cognitive con-

straints on exploration, so the x-rays were the only source of data provided to the par-

ticipating diagnosticians. The subjects were asked to think aloud and their verbalizations were recorded simultaneously on a Dictaphone and a tape recorder.

Radiologists are familiar with the use of a Dictaphone; the only difference here was

that subjects were asked to leave the recording button on the ‘run’ position. This way,

everything was recorded twice, as needed for the third step.

526 RAUFASTE, EYROLLE, AND MARINk

2. In the second step the x-ray was taken away from the physician, who was asked to

draw it on a paper and to name everything he or she could remember that was relevant

to analyzing the case. The verbalizations were tape-recorded. This step allowed the

experimenter to obtain the kind of information that is contained in topographical

aspects of the subject’s representation (Denis, 1989). These aspects can play a role in

reasoning, especially in the field of medical imagery.

3. In the third and final step, the subject commented on the playback of the Dictaphone

recording from the first step, while the film was presented again. The subject was

asked to explain what he or she had said in the first step. During this third step, the sub- ject might intervene spontaneously or answer the experimenter’s questions. As in the

previous steps, verbalizations were tape-recorded.

X-ray Films

Using appropriate psychological criteria (discussed below), and with the helpful coopera-

tion of a confederate expert radiologist, we chose two films that were difficult enough to

generate differences among groups. The films were selected so as to present different kinds

of problems. The first was chosen because of salient cues that usually lead to another diag-

nosis which was in fact wrong. The second was expected to be difficult due to the simulta-

neous presence of several common but independent pathologies.

The first film (“Film 1”) showed a case of lymphangitic carcinomatosis with a left lower

lobe atelectasis. This type of cancer often causes nodules that are commonly visible on the

x-rays. On our film, the nodules existed but were very hard to discern because one was hid-

den by the heart and the images of the others were extremely faded. Typically, the salient

clues in this film, given the fact that no clinical data was available, evoke bronchial disease

and lead to a completely different kind of diagnosis such as chronic bronchitis.

The second film (“Film Y) showed a complex case where four independent pathologies

could be diagnosed. The most salient was a lung ablation that had occurred two months

before, after cancer. The second pathology was an infection in the remaining lung. The fea-

tures that could lead to that diagnosis were not very hard to see but could be misinterpreted

when compared with pneumonectomy features. The third pathology was heart failure, and

the fourth was a hydatid cyst in the liver. The key aspect of this film was the overwhelming

salience of one of the pathologies, in this case the pneumonectomy.

Building Semantic Networks

Radiologists use a technical language with a codified vocabulary describing precise medi-

cal concepts. These concepts could be identified in the verbalizations. Our confederate

expert, who knew the films and all the charts, helped us list these schemata and determine

the logical relationships among them. The schemata constituted the nodes of networks, and

their relations provided the links. Semantic networks were built according to several rules;

a detailed example of semantic network building is provided in the Appendix.

Some of the rules involved data selection. A specific semantic network was built for

each film and subject, using the verbalizations from the three steps. Non-medical com-

ments and systematic evaluations were ignored. For example, reports about absent pathol-

ogies that are supposed to be verified systematically on that kind of film were ignored.

Technical evaluations of the film were ignored unless they were used in an inference

related to a suspected pathology. Residents are taught to systematically verify such things,

regardless of the particular case they have to diagnose. However, radiologists do not ver-

balize technical evaluations, and do not even do all the systematic verifications (Carmody,

Kundel, & Nodine, 1984; Lesgold et al., 1988). Taking them into account would have inap-

propriately penalized novices’ pertinence marks.

Graphical rules were also used. The nodes, which were the words or expressions corre-

sponding to the medical concepts, were typewritten. Each relationship between two nodes

(link) was depicted as a line, without regard to its inhibiting or activating nature. The sche-

mata describing diagnoses were enclosed in ellipses. Rejected diagnoses were crossed out.

The topography of the semantic networks was oriented vertically in order to represent the

depth of semantic categories in canonical medical knowledge. The visual clues (such as

lines, borders, and opacities) were near the top of the networks, and diagnostic schemata

were further down. Thus, an inductive inference was drawn as a line connecting two type-

written schemata, with the deepest schema lower down.

Data Coding

All of medical concepts expressed were considered as canonical schemata and all of the

links expressed (through explicit or implicit inferences) were counted as canonical links.

The shortcuts (i.e., inferences in the first phase between two schemata that canonically

were not directly linked) were counted as operative links. Intermediary schemata that had

remained implicit even after the last phase were restored. For instance, an expert said “a

North African patient”. Only one thing could enable him to say that: the name written on

the film was “Mohammed X”. Consequently, that cue was assumed to have been taken into

account by the expert and was introduced into his semantic network.

As previously discussed, the integration score for a subject and a film was calculated by

dividing the number of links in the verbalizations by the number of schemata.

We now turn to the issue of methods for evaluating pertinence. Pate1 and Groen’s

(1986) definition of relevancy was based on the matching of the findings to the correct

diagnosis, and what led to that diagnosis. They considered a proposition to be relevant

when it matched a canonical structure (given by their confederate expert) that led to the

right solution. With this definition, all propositions leading to other diagnoses are irrele-

vant. Our problem here, however, was less that of understanding how rules can lead to the

correct solution rather than to understand how relevant rules are selected. Pate1 and Groen

considered that a rule is applied when either the premise or the conclusion of the rule

matches a clinical datum. Actually, a single clinical cue may match with the premise of

numerous rules, and most of the time, a physician can yield only a set of possible diag-

noses. Consequently, all diagnoses are relevant until new data can eliminate some of them.

Therefore, we preferred to evaluate relevancy from the standpoint of stimuli rather than

correct diagnoses: a representation was considered as more pertinent when it contained

more clearly visible features or legitimately inferable cues from the film. For instance, on

the lymphangitis film, the diagnosis of chronic bronchial disease was incorrect. But this

528 RAUFASTE, EYROLLE, AND MARINE

diagnosis could be legitimately evoked because the film contained salient clues that were

relatively typical of that pathology, and did not contain any true incompatibilities. We

developed a list of schemata used by each subject. For each schema we decided, with our

confederate expert, whether it was legitimate to evoke that schema. The pertinence value

was determined by calculating the proportion of pertinent schemata in the verbalizations.

In contrast to pertinence, accuracy was taken to express whether the subject had found

the correct diagnoses. Our confederate expert established a comprehensive list of correct

diagnoses. For x-ray 1, two diagnoses were correct: carcinomatous lymphangitis and left

lower lobe atelectasis. For x-ray 2, the four correct diagnoses were pneumonectomy,

hydatid cyst in the liver, pneumonia, and heart disease. The level of accuracy was deter-

mined by tallying the number of correct diagnoses each subject had stated.

Finally, in order to compare our results with Lesgold’s reference study (1988, p. 3 18-

3 19), we measured the longest chain of reasoning (the longest sequence of links), the num-

ber of findings in the biggest cluster, the number of different clusters, and the percentage

of findings connected to other findings.

In order to evaluate coding reliability, two confederate expert radiologists built seman-

tic networks from eight protocols (one per group and per x-ray), using the rules presented

above. Each schema and each link that was cited by at least one coder received a value for

each coder: “1” if cited, “0” if not. Coders also rated schema pertinence (“1” if pertinent,

“0” if not). For each of these three variables, we computed the statistic G, developed by

Light (1971) to compare the joint agreement of several observers with a standard. In the

present study, the standard was the initial coding.

Results

First we report the results related to the hypotheses derived from our model. Then we

present data for comparison with the study by Lesgold et al. (1988).

Effects of Experience on Canonical and Operative Links

The index of agreement for schema coding was Light’s G(226) = 3.82 (p < .OOOl). The

index for link coding was G(245) = 11.986 (p < .OOOl). As shown in Table 1, and as

expected with experience, there was a monotonic growth in the number of activable canon-

ical links (F(3,18) = 7.160;~ c .003) and in the number of operative links (F(3,18) = 8.13;

p c .002) demonstrated by subjects.

TABLE 1 Numbers of Canonical and Operative Links, by level of Expetfise

Level of Expertise

Novices Intermediates

Mean (SD) Mean (SD)

Basic Experts

Mean (SD)

Super Experts

Mean (SD)

Number of canonical links 17.3 (9.2) 22.2 (6.4) 33.75 (12.7) 42.5 (11.7)

Number of operative links 0.25 (0.4) 0.5 (0.8) 1.25 (0.6) 2.125 (0.9)

TABLE 2 Correlations between Canonical and Operative Links, by Level of Expertise

Level of Expertise

Novices Intermediates Basic Experts Super Experts

Correlations between r = -.0975 r = .5261 r = .8527 r = .7490

canonical and operative links

Intergroup comparisons showed that means significantly differed only for noncontigu-

ous groups. For instance, novices differed from basic experts (p < .OOS) but did not differ

significantly from intermediates. The absolute number of operative links remained small,

even in super experts-only the comparisons between (a) novices and super experts, and

(b) intermediates and super experts showed a significant difference (p < .05).

Over all, the existence of the expected correlation between the numbers of canonical

and operative links was proven by Pearson’s coefficient analysis (t-(20, = .765; p < .OOl).

Correlations involving groups of four subjects cannot be regarded as reliable. Neverthe-

less, after analysis of within-group correlations (see Table 2), the correlation between the

numbers of canonical and operative links did not even exist in novices (r-(6, = -.0975). It

was higher in intermediates (r(4) = .5261), although not statistically significant. It was rel-

atively high in experts (r(6) = ,814; p = .014), although not significant if we consider the

basic and super experts separately.

This apparent growth of the correlation between the numbers of canonical and operative

links requires validation by additional research. However, our model suggests a possible

interpretation: For novices, many canonical links are not very activable and there are very

few operative links. Therefore, canonical links are used so they can be reinforced. Con-

versely, the generation of operative links requires sufficient prior activability of canonical

links. In intermediates, many canonical links become activable so that the initial operative

links can be built. Because experience reinforces both types of links, a correlation gradu-

ally emerges. In experts, the networks are fully formed. They are dense and highly activa-

ble. The subjects can demonstrate both types of links, and the correlation produced by

experience is more fully manifested.

Effects of Experience on the Integration and Pertinence of the Representation

The expected continuous growth of integration, shown in Table 3, was highly significant

(F(3,lB) = 9.777; p c .OOOS). This suggests that as experience grows, subjects make more

inferences. This growth appears in Figure 2, which shows that from the same stimulus, the

experienced radiologists built representations that were both richer and more integrated.

The index of agreement for coding pertinence was G(zo7) = 10.126 (p < .OOOl). As

expected, pertinence also rose with experience (F(3,lB) = 8.586; p c .OOl). Although Table

3 seems to show that basic experts demonstrated a lower mean pertinence than intermedi-

ates, this difference was not significant. This lack of a difference suggests that the effi-

ciency of the mechanisms (whatever they might be) that generate pertinence in levels of

530 RAUFASTE, EYROLLE, AND MARlNi

Figure 2. Examples of Semantic Networks on Film 1

expertise ranging from novice to basic expert reaches a limit beyond which only super

experts go. The nature of this limit, and the solutions super experts find in order to go

beyond it, will be discussed below.

Relationship between Integration and Pertinence

As expected, the global correlation between integration and pertinence was confirmed

(r(*u) = .776; p < .OOl). The correlation between integration and pertinence might be

explained by the genuine effect of experience. Studies have shown that experts usually

TABLE 3 Mean Values of Integration and Pertinence, by Level of Expetiise

Level of Expertise

Value of Integration

Novices Intermediates Meon (SD) Mean (SD)

0.783 (0.162) 0.993 (0.104)

Basic Experts

Mean (SD)

1.071 (0.92)

Super Experts

Mean (SD)

1 .162 (0.90)

Value of Pertinence 0.745 (0.13) 0.927 (0.058) 0.910 (0.024) 0.978 (0.020)

make more inferences than novices. Experts also demonstrate greater pertinence (e.g.,

Pate1 & Groen, 1986). These two characteristics may be sufficient to generate a correlation.

An analysis of covariance was conducted in order to obtain the real effect of experience,

controlling for integration. In these conditions, pertinence was no longer significantly

dependent upon experience (F(3,17) = 1.536; p = .241), and integration seemed sufficient

to explain the variations in pertinence (F(3,17) = 5.384; p = .033). This result is interesting

because it suggests that pertinence may be an effect of integration rather than experience.

Nevertheless, if we consider the within-group correlation, we find a monotonic decrease as

experience increases: from ‘(6) = +.62 in novices (p = .OSO) to ~(2) = -.77 in super experts.

Even if correlations within such small groups are unreliable, our results suggest that inte-

gration may be a source of pertinence in novices but not in experts. These ideas will be

developed in the discussion.

Origins of Precision and Accuracy

Film 1 turned out to be too difficult to generate differences among the subjects. Only one

subject in each group, even for super experts, found the correct diagnosis. All the other

subjects found none. Therefore, we used only Film 2 for statistical tests involving accu-

racy. Nevertheless, an analysis of the diagnoses proposed by the subjects for Film 1 con-

firmed the expected sources of error. The critical clues were both atypical and not very

salient, and conversely, salient clues typically led to an incorrect solution.

On Film 2, we would have been able to say that accuracy increased with experience had

the basic experts performed a little better (F(3,18) = 3.135; p = .0512). As Table 4 shows,

their performance was lower than that of the novices.

The qualitative analysis of the protocols showed that the basic experts were affected by

the overwhelming salience of some clues, so they did not detect more subtle features. This

finding is particularly clear when we examine the detection rate for the hydatid cyst, on

Film 2. Fifty percent of the novices saw it, compared to only 17 percent of the intermedi-

ates and none of the basic experts. In contrast, 75 percent of the super experts saw it. After

the experiment, when the critical clues were pointed out, the basic experts immediately

interpreted them correctly. They had not seen them before because they were masked by

the image of pneumonectomy. Novices, as they are taught to be, were globally more sys-

tematic in their exploration. They saw the clue but they had difficulty interpreting it. We

TABLE 4 Accuracy and Precision on Film 2, by level of Expertise

Level of Expertise

Novices Intermediates

Mean (SD) Mean (SD)

Basic Experts

Mean (SD)

Super Experts

Mean (SD)

Number of accurate 1.63 (1.06) 2.33 (1.03) 1.5 (0.58) 3.25 (0.96)

diagnoses

Precision of diagnoses 43.13 (30,Ol) 49.37 (16,29) 91.68 (16,65) 49.10 (9,69)

532 RAUFAETE, EYROLLE, AND MARINk

TABLE 5 Diagnoses Proposed for Film 2 (with percentages of subiects within groups)*

Diagnoses Novices Interm. B. Experts S. Experts

Pneumonectomy Pneumonectomy

Cancer

Metastases

Atelectasis

Right pleural effusion

Hematic effusion

Surgical complication

Mcleod’s Syndrome

Focus Focus Pneumonia

Cancer or metastases

Inhalation

Edema

Bronchopleural fistula

Tuberculosis

Gynecomastia

Heart failure Heart failure

Hydatid intracardiac Dissemination

Pericorditis

Tumor invading the pericardial sac

Hydatid cyst in the liver Hydatid cyst

Amebic cyst

No interpretation

Other Left pleural effusion

Left susclavicular nodule

62.5

25

12.5

37.5

62.5

12.5

12.5

37.5

12.5

12.5

25 33.3 25

83.3 100 100

16.7 75

16.7

100

16.7

50 25

33.3

16.7 25

16.7

16.7

37.5 16.7

12.5

25

12.5

12.5

75

25

50

25

25

50

50

25

25

75

Note. *Correct diagnoses are shown in italics

can also examine how the salient clues of pneumonectomy were processed. All of the

experts found the right diagnosis (Table 5) nearly instantaneously. Among the residents,

two intermediates out of six rejected the diagnosis and three novices out of eight did not

even evoke it. These two examples show that novices detected more clues than basic

experts, probably due to their more systematic exploration, but also that their problem was

interpreting the clues they had found. In contrast, experts processed features in a better

way, but sometimes did not detect all critical clues.

Sixteen diagnoses were proposed by novices, 12 by intermediates, only 4 by basic

experts, and 12 by super experts. However, this variation appears to depend on the type of

pathology involved. For features involving pneumonectomy, only two diagnoses were

offered by super experts, and one by basic experts. These were correct diagnoses. In con-

trast, diversity among residents came from the inability to discard inappropriate diagnoses.

TABLE 6 Correlations between Pertinence or Richness, and Accuracy

Level of Expertise

Correlation between

Accuracy and Novices Intermediates Basic Experts Super Experts

Pertinence

Richness

.3105 -.4665

.5809 .9055

p = .066 p = .006

-.9853

p = ,007

.9642

p = .018

.1741

.9619

p = .019

Six diagnoses were proposed by novices and live by intermediates. For intermediates,

pleural effusion was the most frequently cited diagnosis for the pneumonectomy image,

and it was wrong. As one can see in Table 5, the main variation among super experts came

from the right diagnosis, or from the activation of knowledge about rare complications of

the right diagnoses: pneumonectomy (e.g., bronchopleural fistula), cancer (e.g., tumor

invading the pericardial sac), or hydatid disease (e.g., hydatid intracardiac dissemination).

Only basic experts had a highly focused analysis. In order to evaluate accuracy relative to

variation, we computed a new variable, called “precision”, which was the ratio of the num-

ber of correct diagnoses made by a subject to the total number of diagnoses she or he made

(Table 4). Precision varied with experience (F (3,18) = 4.691; p = .0137), and basic experts

obtained the best performance (p’s were below .05 on Student-Newman-Keuls tests com-

paring them with every group). This probably means that basic experts restrict their infer-

ences to typical diagnoses, in contrast to super experts who tend to explore all possibilities.

The overall correlation between accuracy and pertinence was poor (r(~u) = .3586, p =

0.05 1). As Table 6 shows, the correlation increased negatively with experience, except in

super experts. The gap between basic and super experts was considerable.

In order to find better determinants of accuracy, we also hypothesized that in addition to

pertinence, the richness of the representation would play a role in accuracy. Pertinent find-

ings were the ones that, given the available data, could be assumed to lead to accurate diag-

noses. If among the salient features, the pertinent findings were not sufficient for the

diagnostician to reach an accurate diagnosis, one can assume that accuracy came from the

deliberate search for new features on the film, or from the deliberate search for non-obvi-

ous diagnoses in memory. Both strategies can be expected to lead to a greater number of

activated schemata and expressed links. To test this idea, the number of schemata and the

number of links were totaled to make the “richness” variable. Thus, a representation was

considered rich when it included many schemata and links.

As expected, we found (see Table 6) a significant correlation between the richness of the

representation and accuracy (r(zo) = . 7266; p < .OOl). Because accuracy appeared to

increase with experience, we analyzed the variance of accuracy produced by experience

while controlling for richness. We found that richness was the main source of accuracy

(FC3,t8) = 30.06; p < .OOOl). Experience alone was not significant (F(3,ls) = 2.662; p =

.OSl). Nevertheless, the film was very difficult because neither the obvious cues nor the

534 RAUFASTE, EYROLLE, AND MARINk

TABLE 7 Results for Comparison with the Lesgold et al. (1988) Reference Study

Level of Expertise

Novices Intermediates

Mean (SD) Mean (SD)

Basic Experts

Mean (SD)

Super Experts

Mean (SD)

Number of findings

Longest chain

Biggest cluster

Number of different

clusters

Percentage of findings

connected to others

21.4 (11.1) 22.8 (7.5) 32.3 (9.5) 37.8 (8.0)

2.9 (0.7) 3.4 (0.6) 3.9 (0.6) 6.0 (1.1)

10.9 (4.6) 15.1 (2.7) 25.8 (6.7) 36.1 (8.5)

5.1 (2.9) 2.9 (1.5) 2.5 (0.7) 1.8 (0.5)

89.5 (6.7) 97.9 (1.2) 97.2 (2.2) 99.42 (0.5)

pertinence were sufficient to find accurate diagnoses. It was necessary to explore the film

in order to find new clues that could not be produced inferentially. Consequently, more typ-

ical x-rays do not need such a level of richness for proper diagnosis. We agree with Weber

et al. (1993), who suggested that richness is useful for solving uncommon cases, but auto-

matic generation of hypotheses is sufficient for common cases.

Results for Comparison with the Reference Study (Lesgold et al., 1988)

Because the low number of subjects may have biased our results, especially in the two

expert groups, it would be useful to see how the study matches previous results in the field

of radiological expertise. Consequently, before our general discussion, we will present

some quantitative and qualitative results that enable comparison with Lesgold et al.‘s ref-

erence study.

Most of the relations shown in Table 7 are similar to those found in Lesgold et al.

(1988). Several variables increased monotonically with experience. These were the mean

number of findings (Fc3,18) = 3.278; p = .0388), the longest chain of reasoning (F(3,t~ =

16.637;~ < .OOOl), the biggest cluster size (Fc3,1~) = 21.834; p < .OOOl), and the percentage

of connected findings (F(,,,,) = 6.891; p = .0028). There were two discrepancies, however:

1) Even though the overall trends were similar, the absolute values of the measured vari-

ables were consistently higher in our study. For example, the longest chain was 2.03 in

Lesgold’s experts whereas ours was 6.0. The explanation probably lies in the particularities

of the stimuli: we chose x-rays with numerous visible abnormalities, leading to more find-

ings, from which more hypotheses were plausible and thus had to be discussed by radiolo-

gists. This effect was probably reinforced by the simultaneous existence of several diseases

(two on Film 1 and four on Film 2). The combination of possibilities might have led sub-

jects to make more attempts to connect findings, especially among the super experts. All of

the super experts explicitly evoked rare complications of diseases in order to make all the

data fit with only one disease (e.g., one of them declared “I think he may have had a pneu-

monectomy (...) I think he has a hydatid disease. Is there a connection between the two? It

seems logical or else it is afive-legged sheep”). Differences in coding rules may also have

played a small role. We chose to include only reported abnormalities in the semantic net-

works, and thus, in our calculations. We also included normal features when their normal-

ity made sense with regard to the interpretation of abnormalities (for example, when a

normal feature allowed the diagnostician to discard a hypothesis). Most of the time, the

eliminated elements were disconnected findings (e.g., “the scapulas are normal”), which

might have resulted in a higher percentage of connected findings in our study than in Les-

gold’s. Consequently, the discrepancies in the percentage of connected findings are proba-

bly well explained by specific stimulus-characteristic differences and coding differences.

2) The number of different clusters decreased monotonically (F(3,18) = 3,330; p = .043),

whereas in Lesgold’s study it increased. With the rates of connected findings we found

(ranging from 89,5% in novices to 99,42% in super experts), it is reasonable to find a

reduction in the number of clusters, as greater integration tends to make clusters merge.

Qualitative analysis of protocols showed that subjects used several reasoning patterns.

We could identify all the elementary reasoning steps reported by Lesgold et al.: schema

triggering, testing, tuning and changing. At first a schema was triggered by features, and

then tested. Sometimes the schema did not fit the features well, so the subjects evoked a

more specific diagnosis (what Lesgold et al. called “tuning”). The subjects tuned and dis-

carded schemata but we observed only one case where the subject completely changed the

schema (what Lesgold et al. called “flexibility”). All subjects manifested schema trigger-

ing and testing. On Film 1, only triggering and testing were really used by subjects. On

Film 2, the tuning rate was 37.5% for the novices, 67% for the intermediates, 75% for the

basic experts, and 100% for the super experts. These results are consistent with those

reported by Lesgold et al. In addition, we found a new complex pattern of forward reason-

ing, related to independent disease processing. We call it “integrating.” Having selected

and tested a diagnosis, some subjects tried to relate this diagnosis to every available fea-

ture. While doing this, they evoked rare complications of diseases. This pattern was used

by all of the super experts on Film 2, and by no one else. On Film 1, only one basic expert

and one super expert used this pattern.

III. DISCUSSION

After an evaluation of the above results, we will examine their implications with regard to

the nature of expertise and pertinence generation.

Evaluation of the Results

The results confirmed most of the predictions derived from the model. Experience

increased the mean number of activable canonical and operative links, and the correlation

between them. Integration and pertinence also increased continuously with experience, but

while the effect of experience upon pertinence disappeared when integration was con-

trolled, the correlation between integration and pertinence remained significant. Unexpect-

edly, experience had little effect on overall accuracy, mainly because basic experts did not

detect every critical feature; super experts avoided this problem. Richness was more highly

536 RAUFASTE, EYROLLE, AND MARlNi

correlated with accuracy than with pertinence. However, qualitative analysis revealed dif-

ferences in data processing. Residents had more problems than experts in retrieving good

diagnoses and discarding poor ones. Super experts used a specific kind of deliberate rea-

soning. Most of the variables used in Lesgold’s study were found to behave similarly in our

study: with experience, the number of findings grew, as did the longest chain, the biggest

cluster, and the percentage of findings connected to others.

Despite the fact that the experimental testing confirmed the predictions derived from the

model, we are aware that we did not provide any manipulation that clearly addresses sub-

symbolic processes. Consequently, other explanations of our data might exist. For exam-

ple, there is evidence of the existence of mediated priming in semantic networks (Balota &

Larch, 1986), that is, facilitation of a schema that is not directly connected to the prime.

This could explain the emergence of shortcuts in protocols without recourse to operative

links. Nevertheless, some authors claim that “mediated priming can be said to be priming

between directly related weak associates” (McKoon & Ratcliff, 1992, p. 1165).

A second issue arises out of the idea that experts possess much more knowledge than do

novices. In this view, the higher number of findings and links could be a consequence of

differences in knowledge availability rather than being an effect of knowledge accessibil-

ity. Actually, the real question is why intermediates who have knowledge that would

enable them to reach the correct diagnoses do not use that knowledge? In a recent article

about medical diagnosis, Custers et al. said that “It remains a challenge . . . to discover why

subjects at intermediate levels of expertise do possess the relevant knowledge, but are

unable to use it in many diagnostic situations” (1996, p. 395). In Lesgold’s and our studies,

even the novices were already doctors, with experience in external medicine. Intermediates

were 3- and 4-year residents. It is extremely unlikely that they were unaware of pneu-

monectomies or hydatid cysts. This phenomenon is demonstrated in the following tran-

script, from the verbalizations of an intermediate who missed the pneumonectomy:

(Experimenter) You rejected pneumonectomy because there was no-

(Subject) Attraction (Experimenter) Then you kept pleural effusion. Do you have other signs for that? (Subject) No. I don’t. Actually in pneumonectomies, just after, there may be no mediastinal

attraction (. . .). (Experimenter) But did you think of it at that time? (Subject) No.

In our model, such an effect might be explained in terms of the weakness of canonical

link activability, that is, knowledge accessibility.

A third potential bias might result from knowledge that has been activated but not ver-

balized. This would matter if it affected the groups differently. In fact, many studies have

shown that acquisition of expertise is accompanied by greater automatization. This means

that we might expect omissions to affect more experts than intermediates and novices; this

would have led to a greater underestimation of the number of findings and links in experts

than in novices. This bias probably made it more difficult to confirm hypotheses about dif-

ferences in the richness of the representation. It is unclear what effects such a bias might

have with regard to pertinence.

All these criticisms might be summarized by noting that verbal protocols cannot consti-

tute sufficient proof of a spreading activation system; rule-based reasoning could lead to

the same verbal protocols. Thus, we have no data to prove that spreading activation is the

cause of the observed results, even if the predicted hypotheses were globally corroborated.

Nevertheless, our results have implications with respect to the nature of expertise and per-

tinence generation.

Implications for the Nature of Expertise

We claim that the addition of the basic expert group reveals that what we have called basic

and super expertise are qualitatively different, so that outstanding radiologists (super

experts) are not representative of common radiological expertise because of qualitative

gaps in daily practice and cognitive processes.

Expertise can be considered either as increased automation of perceptual processing

(e.g., Dreyfus & Dreyfus, 1986) or, in contrast, as a greater capacity to proceed in a more

flexible and deliberate manner in both perceptual and cognitive processing (e.g., Lesgold

et al., 1988). Two facts suggest that Dreyfus and Dreyfus’s position does not apply to radi-

ologists. First, the mean longest chain for super experts was 6.0 whereas it was 2.9 in nov-

ices. Second, the integrating pattern of reasoning we found in all super experts on Film 2

was clearly deliberate. Therefore cognitive processing was involved in super experts’ per-

formance. They did not proceed only by intuition.

However, the existence of perceptual processing in radiological expertise is well docu-

mented (e.g., Kundel & Nodine, 1983; Lesgold et al., 1988). Considering their findings

showing that intermediates sometimes provided less accurate diagnoses than did novices,

Lesgold et al. (1988) attempted to apply a three-stage model to expertise acquisition in

adults. The loss of accuracy was attributed to the conflict between perceptual and cognitive

processes. Our own results showed that salient features (namely of the pneumonectomy)

led to a monotonic increase in the performance curve with experience. The results also

showed that subtle features led to a U-shaped performance curve (namely those associated

with a hydatid cyst). The point here is that the non-monotonicity can be ascribed to the

super experts. If we had considered only three groups (with no super experts), we would

have found two monotonic performance curves: increasing for salient features and

decreasing for subtle features. But if we had considered three groups with no basic experts,

we would have found the same results as Lesgold et al. (Figure 3). So, if basic and super

experts represent two different kinds of expertise then we have no reason to place these

subjects on the same curve. Basic expertise acquisition curves (including novices, interme-

diates, and basic experts) are compatible with models where expertise acquisition is a

gradual process leading to automaticity (e.g., Anderson 1983, 1992; Sternberg & Frensch,

1992): with the salient and typical features, performance monotonically increased, and

with the inconspicuous or atypical features, performance monotonically decreased. This

result is compatible with the qualitative findings showing that basic experts used a type of

cognitive processing which was limited to testing diagnostic hypotheses and tuning sche-

538 RAUFASTE, EYROLLE, AND MARINE

Qualitative gap

Novices Super experts

Intermediates

Novices ’ Super expert5

lntermedfetes

Basic experts

Lesgold ‘s model Our findings

Figure 3. Reinterpretation of the Lesgald et al. (1988) Performance Curve an Atypical Films*

*The left figure of the diagram shows Lesgold’s point of view. It can be reconstructed from our data if basic experts are withdrawn and super experts included in the curve. The right-hand figure shows our point of view: basic experts are representative of the overall radiologist population, and super experts are not.

mata. The basic expert level probably matches what Olsen and Rasmussen (1989) called

“highly skilled performer” whereas super experts correspond more closely to the descrip-

tion of the “reflective expert” provided by these authors. The longest chain of reasoning,

which might be taken as a cue for deliberate reasoning, was significantly higher in super

experts than in any other group, including basic experts. At the same time, there was no

significant difference between intermediates and basic experts. Furthermore, as we

reported, super experts also demonstrated a deliberate ‘integrating’ strategy that was not

found in the other groups. Consequently, super experts probably produced higher perfor-

mance due to their greater use of cognitive reasoning. This raises two new questions. First,

why did all the basic experts fail to detect a feature (hydatid cyst) that was noticed by inter-

mediates, and even better by novices? Their more advanced perceptual skills should have

given them a better chance at detection. Second, why did super experts not miss this

hydatid feature?

Berbaum and colleagues (1990) reported that having previously detected features rel-

evant to a specific diagnosis may increase the difficulty at detecting features related to

different diagnoses, which they called the satisfaction of search phenomenon (SOS). Using eye-position tracking, Samuel, Kundel, Nodine, and Toto (1995) found that

SOSs occur when the attention is captured by the first features detected. Another expla-

nation involves the use of a schema-driven exploration strategy, which in our protocols,

was used by all basic experts. Studying the integration of clinical data in radiological

diagnosis, Norman, Brooks, Coblentz, and Babcook (1992) found that feature calls

depend on expected categories. In our study, the feature for hydatid cyst could be

expected from nothing else.

The most obvious explanation of why super experts did not fail to detect the hydatid

cyst would be to attribute it to better perceptual skills. But this explanation is not suffi-

cient, because if only perceptual processes were responsible for this detection then basic

experts would have performed better than residents. Furthermore, Samuel and colleagues,

who studied nodule detection, reported that “most missed nodules were fixated.” (1995,

p. 895).

We propose another explanation based on differences in usual professional practices.

The basic expert has to make the best diagnosis, in a limited time. As the job becomes

familiar, processes that require a high level of conscious attention are needed less and less.

In real-life practice, basic experts have several x-rays of the same patient. They often have

a clinical description, which has been proven to reduce the SOS phenomenon (Berbaum et

al., 1993). In case of doubt, they proceed with further examinations using other tech-

niques. In contrast, super experts are not only practitioners in radiology but also profes-

sors, which implies an ability to observe, justify and explicitly describe the methods

leading to accurate diagnoses. Furthermore, super experts also are researchers. It means

they have to devote conscious effort making their results explicit and publishing in scien-

tific journals. Finally, as our confederate expert radiologist suggested, super experts usu-

ally see much more complex cases than other radiologists. They are often asked to give

advice about difficult or non-standard films. So, as a consequence of their different attri-

butions and practices, they are highly trained to maintain high levels of attention in their

diagnostic activities, and to organize and explain the information with which they are con-

fronted. Their usually higher level of attention may partially protect them against the SOS

phenomenon. In another task domain, Bryan and Harter (1897) showed that Morse code

operators normally reached a plateau in their skill acquisition but that performance could

still increase with deliberate effort. According to Ericsson, Krampe, and Tesch-RGmer

(1993), “eminent performance” is directly related to “the amount of deliberate practice

related to that goal” (p. 392), and it is probable that a significant proportion of super

experts’ daily activities constitutes deliberate practice. So, if super expertise comes from a

specific professional activity that is clearly different from the activity of “normal” radiolo-

gists, super expertise can no longer be considered as the natural end of expertise develop-

ment in radiology.

More practically, we suggest that daily activity could be used as a criterion in order to

determine groups that vary in processing level: experienced subjects whose normal profes-

sional activity mainly involves skill-based processing could be classified as highly skilled

performers, and experienced subjects whose normal activity involves explicit reasoning

could be classified as reflective experts. Validating such a criterion requires further exper-

imentation, but if correct, this distinction could be useful for testing hypotheses that imply

control over the level of data processing.

540 RAUFASTE, EYROLLE, AND MARINk

I

Pertinence ,

J coming fron controlled processing

b

coming fron automatic

proC6SSillg

\i/

intermediates

basic experts

novices

Experience

Figure 4. Hypothetical Curve of Pertinence Acquisition

On the Origins and Role of Pertinence

The fact that experience increases the interdependence between representation elements-

what we called integration-is consistent with numerous studies showing that experts have

more integrated representations (e.g., Lesgold et al., 1988 in radiology; AmergC & MarinC,

1992 in ergonomic diagnosis; Chase & Simon, 1973a; Freyhof, Gruber, & Ziegler, 1992 in

chess playing). The present study shows that pertinence in schema triggering is related to

integration. Ericsson and Smith stated, “In domains with complex stimuli, such as medi-

cine . . . it is clear that part of the integration of the presented information involves identifi-

cation of the relevant and critical information., .” ( 199 1, p. 24). In our model, identification

of relevant and critical information is not part of the integration process but rather is the

result of that process. Integration of data into the network occurs before pertinence detec-

tion: information is deemed pertinent after it has been integrated. This is globally compat-

ible with a connectionist point of view for the acquisition of pertinence through the basic

expert level. Nevertheless, the pertinence curve drawn from our results (Figure 4) showed

an increase only until a plateau was reached in the intermediates. Only super experts went

beyond. Actually, in the super experts the pertinence continued to grow while the correla-

tion between pertinence and integration disappeared. Consequently, other mechanisms are

necessary to explain why a plateau is reached and how pertinence can still increase.

Rabinowitz (1991) found that strategic processing is important for learning “medium typi-

cal” nouns because spreading activation within semantic networks is not sufficient to con-

strain the activation of related knowledge, whereas strategic processing is sufficient. Such

an explanation could be transferred to our model: even if, as experience grows, the possi-

bility for activation to spread within the semantic network grows, integration of the net-

work as the source of pertinence lessens. Thus, for typical cases, spreading activation could

be assumed to be wide enough in basic experts for pertinent diagnoses to be quickly

retrieved, but processing would become unconstrained for less typical cases. Only super

experts, who maintain deliberate reasoning activity in everyday work, would trigger suffi-

cient levels of deliberate reasoning, the only kind of reasoning able to constrain possibili-

ties optimally in such cases.

IV. CONCLUSION

Holyoak (1991) stated that “The most salient gap in the current models is that none

addresses the crucial issue of learning. Nonetheless, it seems reasonable to expect that

learning models can be developed within the symbolic connectionist paradigm” (p. 324).

The model of medical reasoning presented in this article falls within this paradigm. The

present work is somewhat preliminary: much more testing is necessary to give this

model a solid empirical background. Nevertheless, the results appear promising. The

model proposes a simple mechanism by which declarative knowledge is triggered and

integrated into a representation. Because it involves both declarative and subsymbolic

knowledge, it supplies a basis for understanding how likelihood (provided by experi-

ence) and probabilities (provided by college teaching) are used. Using spreading activa-

tion mechanisms, the model describes how pertinent diagnostic hypotheses would be

triggered and integrated into a representation. The model also provides an explanation of

how experience enhances performance for typical cases. Basic experts appeared to con-

strain their inferences to typical diagnoses, in contrast with super experts who tended to

explore all possibilities. Solutions to atypical cases probably require controlled process-

ing. Super experts, who demonstrated more deliberate reasoning than any other group,

and whose daily activity involves more deliberate reasoning, also performed better than

any other group. Finally, the question of the difference between basic and super exper-

tise, which was raised in this study, is of practical importance because the value of an

expert lies in his or her ability to find solutions not only for typical cases but for atypical

cases as well.

Acknowledgments: We thank Paul Feltovich, Mitchell Rabinowitz, and an anony-

mous reviewer for their comments on earlier drafts of this article. We also thank our

consulting expert radiologists, Daniele Verderi-Raufaste, Andreas Schulz, and Jacques

Bernier.

APPENDIX

Example of Semantic Network Building

Verbalizations

The following protocol (Table 8) was verbalized by a novice (first year resident). Trans-

lated from French.

542 RAUFASTE, EYROLLE, AND MARINk

TABLE 8 Verbal Protocols and Commentaries

Concomitant Protocol Self-Confrontation

Well, good, the second film... well, I’m looking

at the skeleton. So the x-ray, it was taken in bed,

it’s written on the film. Clavicles.. . I look at the

scapulas, it’s OK, ribs, you can follow the ribs,

mainly on the left side because on the right you

can see nothing. Left costo-diaphragmatic

angle. then, . . . . . . so, I’m missing a little

trachea..

Well, left diaphragmatic cupola OK.. Mmm left

costo-diaphragmatic angle.. . OK. Right, so,

what can we... what can we say? The whole

field, the right pulmonary field is white. Well,

. ..er...

and left pulmonary field, I’ve a feeling I see little

micro-nodules... so are they cut vessels?

. ..Mmm... may be vessels.

Well... There you are, so to conclude, field...

white lung on the right or . ..er... lung absence,

prior pneumonectomy to con... well to confront

with clinical data.

No deviation of the mediastinum, neither right

attraction nor right-to-left shift.

No left pleural effusion, no visible evolution-

ary-like pieuro-parenchymatous lesion on the

left....mmm... on skeleton, no visible lesion

. ..er... in the soft tissues... no well, that’s all.

What does “missing a little trachea” mean to

you?

Well no, it’s because I cannot follow it until tra-

cheal bifurcation because it’s white.. . it does not

mean anything to me.

Finally I think not, there are no micro-nodules,

finally I don’t think so.

Yes, it must be a pneumonectomy because I

think I’m seeing the . . . [points to surgical clips]

Buf you didn’f see it a few minutes ogo?

No. I didn’t.

Well, what did you try fo verify?

so, precisely, if there’s a pneumonectomy, nor-

mally the porenchyma there expands a little and

therefore it pushes away mediastinum... It

seems to me. Well, here it’s not so.

Then it would be against pneumonectomy?

Yes.

Verbalizations During Drawing

Well, I’m drawing the skeleton, these are thoracic outlines, two clavicles, trachea that can

be followed until.. . until above the aortic notch, no left lung abnormality but either little

nodules or cut vessels . . . should ask the boss, then a whole white lung there.

PERTINENCE GENERATION IN RADIOLOGICAL DIAGNOSIS

Extracted Schemata

in bed film

no visible left evolutionary-like pleuro-parenchymatous lesion

no pleural effusion

no right attraction of mediastinum

no right-to-left shift of mediastinum

micro-nodules

pneumonectomy

right white lung

missing a little trachea

cut vessels

Comments on the Choices

TABLE 9 Commentaries on the Choices in Semantic Network Building

543

Verbalization Comment

Well, good, the second film

it wos taken in bed,

clavicles... I’m looking at the scapulas, it’s OK,

ribs, you can follow the ribs

then , . . , . . . so, I’m missing a little trachea...

What does u missing a little trachea a mean to

you? Well no, it’s because I cannot follow it until tra-

cheal bifurcation because it’s white... it does not

mean anything to me.

. ..little micro-nodules... so are they cut vessels?

. ..Mmm... may be vessels.

. . . white lung on the right or . ..er... lung

absence, prior pneumonectomy to con... well to

confront with clinical data.

Yes, it must be a pneumonectomy because I

think I’m seeing the . . . [she points to surgical

clips] Buf you didn’t see it a few minutes ago?

No. I didn’t.

This is a commentary which has nothing to do

within the semantic network.

In spite of the fact that this sentence seems dis-

connected from everything else, we kept it

because this characteristic is pertinent informa-

tion relative to the patient’s state.

We didn’t choose to retain these schemata

because they are, here, parts of a systematic

examination of anatomy that does not mean

anything by itself: novices are taught to do this.

Two schemata are connected here: the subject

explains the lost trachea as a consequence of

the white lung. That’s why we drew a line

between the two.

The subject hesitates between two semiological

interpretotions of an image. Both are non-perti-

nent. We drew a line between them because the

subject clearly tries to reinterpret micro-nodules

as vessels.

The subject suspects a white lung then changes

her mind towards the hypothesis of a a pneu-

monectomy (lung absence is the same thing

here).

While commenting on the verbal protocols, the

subject discovered a critical cue not seen before.

We did not include that late finding which would

have been an artifact.

544

Semantic Network

in bed film

lost trachea

no right attraction of mediastinum

\

RAUFASTE, RYROLLE, AND MARINk

micro-nodules- cut vessels

no right-to-left shift of medjastinum

Figure 5. Resulting Semantic Network

REFERENCES

Amerge, C., & Marine, C. (1992). Etude comparative expert-debutant lors de l’elaboration d’un prediagnostic

ergonomique. Le Travail Humain, S(2), 97-117. Anderson, J. R. (1983). The architecture of cognition. Cambridge, MA: Harvard University Press.

Anderson, J. R. (1992). Automaticity and the ACT* theory. American Joumal of Psychology, 105(2), 165-180.

Balota, D. A., & Larch, R. F. (1986). Depth of automatic spreading activation: Mediated priming effects in pro-

nunciation but not in lexical decisions. Journal of Experimenfal Psychology: Learning, Memory, and Cog- nition, 12, 336-345.

Balota, D. A., & Paul, S. T. (1996). Summation of activation: Evidence from multiple primes that converge and

diverge within semantic memory. Journal of Experimental Psychology: Learning, Memory, and Cogni- tion, 22(4), 827-845.

Barrows, H. S., Norman, G. R., Neufeld, V. R., & Feightner, J. W. (1982). The clinical reasoning of randomly

selected physicians in general medicine practice. Clinical and Investigative Medicine, 5.49-55. Berbaum, K. S., Franken, E. A., Dorfman, D. D., Rooholamini S. A., Kathol , M. H., Barloon, T. J., Behike, F.

M., Sato, Y, Lu C. H., El-Khoury G. Y., Fhckinger, F. W., & Montgomery, W. J. (1990). Satisfaction of

search in diagnostic radiology. Investigative Radiology, 25, 133-140. Berbaum, K. S., Franken, E. A., Anderson, K. L., Dorfman, D. D., Erkonen, W. E., Farrar, G. P., Geraghty, J. J.,

Gleason, T. J., MacNaughton, M. E., Phillips, M. E., Renfrew, D. L., Walker, C. W., Whitten, C. G., &

Young, D. C. (1993). The influence of clinical history on visual search with single and multiple abnormal-

ities. Investigative Radiolagy, 28, 19 l-201.

Bryan, W. L., & Harter, N. (1897). Studies in the physiology and psychology of the telegraphic language. Psycho- logical Review, 4,27-53.

Cantor, J., & Engle, R. W. (1993). Working-memory capacity as long-term memory activation: An individual-dif-

ferences approach. Journal of Experimental Psychology: Learning, Memory and Cognition, 19(5), 1 lOI- 1114.

Carmody, D. P., Kundel, H. L., & Nodine C. F. (1984). Comparison scans while reading chest images: Taught but

not practiced. Investigative Radiology, 19(5), 462-466.

Chase, W. G., & Simon, H. A. (1973a). Perception in Chess. Cognirive Psychology, 4.55-81. Chase, W. G., & Simon, H. A. (1973b). The mind’s eye in chess. In W. G. Chase (Ed.), Visual information pro-

cessing (pp. 215-281). New York: Academic Press.

Collins, A. M., & Loftus, E. F. (1975). A spreading activation theory of semantic processing. Psychological Review, 82, 407-428.

Custers, E. J. M. F., Boshuizen, H. P. A., & Schmidt, H. G. (1996). The influence of medical expertise, case typ-

icality, and illness script component on case processing and disease probability estimates. Memov and Cognition, 24,3, 384-399.

Dagenbach, D., Horst, S., & Carr, T. H. (1990). Adding new information to semantic memory: How much leam-

ing is necessary to produce automatic priming. Joumai of Experimenful Psychology: Learning, Memory. and Cognition, 16(4), 58 1-59 1.

Denis, M. (1989). Image et cognition. 2nd edition 1994. Paris : PUF.

Doherty, M. E., Schiavo, M., Mynatt, C. R., & Tweney, R. D. (1981). The influence of feedback and diagnostic

data on pseudodiagnosticity. Bulletin of the Psychonomic Society, 18(4), I91 -194. Dreyfus, H. L., & Dreyfus, S. E. (1986). Mind over machine: The power of human intuition and expertise in the

era of the computer. New York: The Free Press.

Elstein, A. S., Shulman, L. S., & Sprafka, S. A. (1978). Medicalproblem solving: An analysis of clinical reason- ing. Cambridge, MA, & London: Harvard University Press.

Ericsson, K. A., Krampe, R. T., & Tesh-RBmer, C. (1993). The role of deliberate practice in the acquisition of

expert performance. Psychological Review, 100(3), 363-406. Ericsson, K. A., & Smith, J. (1991). Prospects and limits of the empirical study of expertise. In K. A. Ericsson &

J. Smith (Eds.), Toward a general theory of expertise-prospects and limits (pp. l-38). Cambridge: Cam-

bridge University Press.

Ericsson, K. A., & Kintsch, W. (1995). Long-term working memory. Psychological Review, 4, 21 l-245. Freyhof, H., Gruber H., & Ziegler, A. (1992). Expertise and hierarchical knowledge representation in chess. Psy-

chological Research, 54.32-37. Frick, R. W., & Lee, Y. -S. (1995). Implicit learning and concept learning. The Quarterly Journal of Experimenral

Psychology, 48A(3), 762-782. Hasher, L., & Zacks, R. T. (I 984). Automatic processing of fundamental information: The case for frequency of

occurrence. American Psychologist, 39, 1372-1388.

Hoffman, P., Slavic, R., & Rorel, L. (1968). An analysis of variance model for the assessment of configural cue

utilization in clinical judgment. Psychological Bulletin, 69, 338-349.

Holyoak, K. J. (1991). Symbolic connectionism: Toward third-generation theories of expertise. In K. A. Ericsson

& J. Smith (Eds.), Toward a general rheory of expertise-prospecrs and limits (pp. 301-335). Cambridge:

Cambridge University Press.

Joseph, G.-M., & Patel, V. L., (1990). Domain knowledge and hypothesis generation in diagnostic reasoning.

Medical Decision Making, 10, 3 l-46.

Kahneman, D., & Tversky, A. (1973). On the psychology of prediction. Psychological Review, 80, 237-251. Kundel, H. L., & Nodine, C. F. (1983). A visual concept shapes image perception. RudioIogy, 146(2), 363.368. Kundel, H. L., & Wright, D. J. (1969). The influence of prior knowledge on visual search strategies during the

viewing of chest radiographs. Radiology, 93(8), 315-320. Lemieux. M., & Bordage, G. (1992). Propositional versus structural semantic analyses of medical diagnostic

thinking. Cognitive Science, 16, 185-212.

Lesgold, A. M., Rubinson, H., Feltovich, P., Glaser, R., Klopfer, D., & Wang, Y. (1988). Expertise in a complex

skill: Diagnosing X-Ray Pictures. In M. T. H. Chi, R. Glaser, & M. J. Farr (Eds.), The nature of expertise (pp. 31 l-342). Hillsdale, NJ: Lawrence Erlbaum.

Light, R. J. (1971). Measures of response agreement for qualitative data: Some generalizations and alternatives.

Psychological Bulletin, 76(5), 365-377. McKoon, G., Ratcliff, R. (1992). Spreading activation versus compound-cue accounts of priming: Mediated

priming revisited. Journal of Experimental Psychology: Learning, Memory, and Cog&ion, Z&6), 1155-

1172.

Medin, D., & Edelson, S. (1988). Problem structure and the use of a base-rate information from experience. Jour- nal of Experimental Psychology: General, II 7( I), 68-85.

Newell, A., & Simon, H. A. (1972). Human problem solving. Englewood Cliffs, NJ: Prentice-Hall.

546 RAUFASTE, RYROLLE, AND MARlNi

Norman, G. R., Brooks, L. R., Coblentz, C. L., & Babcook, C. J. (1992). The correlation of feature identification

and category judgment in diagnostic radiology. Memory and Cognition, 20(4), 344-355.

Olsen, S. E., & Rasmussen, J. (1989). The reflective expert and the prenovice: Notes on skill-, rule- and knowl-

edge-base performance in the setting of instruction and training. In L. Bainbridge & S.A. Ruiz-Quintanilla

(Eds.), Developing skills with information technology (pp. 9-33). Chichester: John Wiley.

Pate], V. L., & Green, G. J. (1986). Knowledge base solution strategies in medical reasoning. Cognitive Science, IO, 91-l 16.

Patel, V. L., & Green, G. J. (1991). The general and specific nature of medical expertise: A critical look. In K. A.

Ericsson & J. Smith (Eds.), Toward a general theory of expertise-prospects and limits (pp. 93-125).

Cambridge: Cambridge University Press.

Rabinowitz, M. (1991). Semantic and strategic processing: Independent roles in determining memory perfor-

mance. American Journal of Psychology, 104(3), 427-431. Rabinowitz, M., & Chi, M. T. H. (1987). An interactive model of strategic processing. In S. J. Ceci (Ed.), Hand-

book of the cognitive, social, and physiological characteristics of learning disabilities (Vol. 2, pp. 84-

102). NJ: Erlbaum.

Rabinowitz, M., & McAuley, M. (1990). Conceptual knowledge processing: an oxymoron? In W. Shneider & F.

E. Weinert (Eds.), Interactions among aptitudes, strategies, and knowledge in cognitive performance (pp.

117-133). New York: Springer-Verlag.

Rumelhart, D. E., McClelland J. L., & The PDP Research Group (1986). Parallel distributed Processing: Explo- rations in the microstructure of cognition. (Volumes 1 & 2). Cambridge, MA: MIT Press.

Samuel, S., Kundel, H. L., Nodine, C. F., & Toto, L. C. (1995). Mechanism of satisfaction of search: Eye position

recordings in the reading of chest radiographs. Radiology, I94(3), 895-902. Sebillotte, S. (1984). La resolution de probltme en situation de diagnostic. Un exemple: le diagnostic medical.

Psychologie Francaise, 29(3/4), 273-277. Selfridge, 0. G. (1959). Pandemonium: A paradigm for learning. In Symposium on the mechanization of thought

processes, London, HMSO.

Shanteau, J. (1992). How much information does an expert use? Is it relevant? Acta Psychologica, 81, 75-86. Shiffrin, R. M., & Schneider, W. (1977). Controlled and automatic human information processing, II: Perceptual

learning, automatic attending and a general theory. Psychological Review, 84, 127-190. Smedslund, J. (1963). The concept of correlation in adults. Scandinavian Journal of Psychology, 4, 165-173. Stemberg R. J., & Frensch P. A (1992). On being an expert: A cost-benefit analysis. In R. R. Hoffman (Ed.), The

psychology of expertise--cognitive research and empirical AI (pp. 191-203). New York: Springer-Verlag.

Weber, E. U., Bijckenholt, U., Hilton, D. J., & Wallace, B. (1993). Determinants of diagnostic hypothesis gener-

ation: Effects of information, base rates and experience. Journal of Experimental Psychology: Learning, Memory, and Cognition, I9(5), 1151-1164.