[ s R L CL E T I N IDENTIFICATION OF SUB-SKILLS OF READING COMPREHENSION BY MAXIMUM LIKELIHOOD FACTOR ANALYSIS Donald Spearritt RB-72-4 This Bulletin is a draft for interoffice circulation. Corrections and suggestions for revision are solicited. The Bulletin should not be cited as a reference without the specific permission of the author. It is automati- cally superseded upon formal publication of the material. Educational Testing Service Princeton, New Jersey February 1972
Transcript
~[s~8R LC L~ E
TIN
IDENTIFICATION OF SUB-SKILLS OF READING COMPREHENSION
BY MAXIMUM LIKELIHOOD FACTOR ANALYSIS
Donald Spearritt
RB-72-4
This Bulletin is a draft for interoffice circulation.
Corrections and suggestions for revision are solicited.
The Bulletin should not be cited as a reference without
the specific permission of the author. It is automati
cally superseded upon formal publication of the material.
Educational Testing Service
Princeton, New Jersey
February 1972
IDENTIFICATION OF SUB-SKILLS OF READING CO~ITREHENSION BY MP~IMUM
LIKELIHOOD FACTOR ANALYSIS
Donald Spearritt
University of Sydney
Abstract
Differing interpretations have been made of the results obtained by
Davis in a large-scale study of the mental skills involved in reading
comprehension among twelfth grade students. On the basis of a uniqueness
analysis and a varimax-rotated principal components analysis, Davis maintained
that five of his eight postulated skills were experimentall~ distinguishable.
A refactorization of Davis's data by R. L. Thorndike suggested that except
for word knowledge, the reading skills were not separately distinguishable.
The present study consists of a further refactorization of Davis's data}
taking more advantage of their factor analytic design possibilities} and
employing the more comprehens ive procedures now available for maximum
likelihood factor analysis. Word knowledge and three other skills were
shown to be separately distinguishable, but the latter three skills were
very highly correlated and thus could be predominantly measuring a single
bas ic ability.
IDENTIFICATION OF SUB-SKILLS OF READING CQMPREHENSIONBY ~~IMUM
LIKELIHOOD FACTOR ANALYSIS1
Donald Spearriti
University of Sydney
1. Introduction
The nature of the mental skills involved in understanding what
one reads was considered by E. L. Thorndike in 1917, and continues to be a
subject of speculation, investigation, and disagreement. Thorndike (1917)
concluded that reading comprehension was basically a process of reasoning.
Educators and. reading sp~cialists, however, have continued to draw up
lists and. taxonomies of the skills involved in reading comprehension,
and the early efforts of measurement specialists assumed that there
were separate skills which were worth measuring. Gates (1927), for
ir~tance, included in his battery measures of reading to appreciate general
significance, reading to understand precise directions, reading to note
details, and reading to predict the outcome of given events. From research
associated with the development of the Cooperative Reading Comprehension
Tests, Davis (1944) concluded that there were nine distinguishable skills
in reading. His :findings were challenged by Thurstone (1946), whose
reanalysis of Davis's correlations, employing communalities in the
diagonal cells, indicated that the correlations could be adequately
explained in terms of one common factor, presumably representing dif:ferences
in a general reading ability.
Later experimental evidence relating to the differentiability
of the skills involved in reading comprehension has been obtained by
Derrick (1953) and by Vernon (1962), among others. Derrick was unable to
-2-
differentiate reliably among the tasks involved in answering factual
questions, inferential questions, and questions requiring critical
judgments. Vernon obtained a clear differentiation between vocabulary
and reading comprehension tests, and at least an indication that ractual
and inferential questions may be measuring dirrerent skills.
2. Recent Analyses of Experimental
Data on Reading Comprehension Sub-Skills
The nature of the mental skills involved in reading comprehension
has been investigated again recently by Davis (1968), in a comprehensive
and well-designed study. Following a review of earlier experimental
studies of reading comprehension, he selected eight skills and constructed
items to measure each of these skills, each item being based on a different
passage for comprehension in order to avoid the problem of experimental
dependence among responses to items. At'ter appropriate item analyses,
two sets of twelve items were selected to represent each of the rollowing
skills:
"recalling word meanings,"
"drawing inferences about the meaning or a word from context,"
"finding answers to questions answered explicitly or in paraphrase,"
"weaving together ideas in the content,"
"drawing inferences from the content,"
"recognizing a writer's purpose, attitUde, tone, and mood,1l
"identifying a writer's techniques,"
"following the structure of a passage."
One set of twelve items in each of these categories formed Form C of the
experimental test, which was administered without time limit to twelfth
grade pupils in academic high schools. The other eight subsets of
-3-
twelve items were incorporated into an equivalent form of the test, Form
D, which was administered to the same pupils after an interval of one or
two days. Analyses of the data were based on the scores of 988 pupils,
designated by Davis as Sample 3, who had taken both forms of the test.
For cross-validation purposes, the group '..ras subdivided into Sample 1
(494 pupils) and Sample 2 (494 pupils) by using alphabetical lists of
names within schools to assign one pupil to Sample 1, the next to Sample 2,
and so on.
Davis employed a uniqueness-analysis technique to analyze his data,
as he claimed that I1highly refined statistical techniques must be employed
if tiny amounts of variance unique to each skill in a set are to be
detected" (Davis, 1968, p. 509). The technique consists, in effect, of
determining the proportion of variance of a particular skill which can
be accounted for by the best-weighted combination of the other seven
skills,the weights being determined by multiple-regression procedures
in a cross-validating sample eitner across samples (Sample 1 vs, Sample 2),
across days (Form C vs. Form D), or across various combinations of both
days and samples, correcting this proportion for attenuation to take
account of unreliability in both the test itself and the weighted composite
score based on the other skills, and subtracting the adjusted proportion
f'rom unity. The percentage of unique variance in the nonerror variance
of the eight reading skills for Davis's Sample 3 is shown in the first
two data colu~ns of Table 1; the values in the first column were arrived at
Insert Table 1 about here
after the appropriate cross-validating procedures were applied with
correlation matrices obtained within single testing sessions, while
those in the second column involve correlations across days, i.e., correlations
-4-
between Form C and Form D tests, and vice versa. From these resu.1Lts, Davis
concluded that reading comprehension among mature readers is not a unitary
trait, and that substantial parts of the mental abilities used in the
eight skills are independent of one another. He claims in effect that
Skills 1, 3, 5, 6, and 8 are experimentally distinguishable.
In a subsequent research report, Davis (1971) presents the results
of a principal components analysis of the same data, after rotation to the
normalized varimax criterion. '11his type of analysis was applied independently
to two correlation matrices. '1be first matrix was formed from the inter
correlations of sub-test Cl with sub-tests D2 through DB, sub-test C2
with sub-tests D3 through DB, and so on, while the second consisted of
the intercorrelations of sub-test Dl with sub-tests C2 through C8, and
so on. Davis identified the following factors in each of the two matrices:
knowledge of word meanings (Skill 1), recognition of writer's purpose,
attitude, tone and mood (Skill 6), ability to get literal sense meaning of
details (Skill 3) and a combination of weaving together ideas in the content,
and drawi!~ inferences from the content (Skills 4 and 5), apparently a
reasoning ability underlying both deductive and inductive processes.
The skills identifying distinguishable factors in these analyses are
indicated in Columns 3 and 4 of Table 1.
In an address on "Reading as Reasoning," given to Division 15 of
the Ameri.can Psychological Association in September,. 1971, Robert L.
Thorndike (1971) presented the results of his refactorization of Davis's
data using reliability coefficients in the diagonal cells, and showed
that the non-chance variance in the sets of eight tests could be completely
accounted for by three factors. He pointed out that Davis, in his principal
components analysis, had analyzed error variance as well as reliable
variance. 'I'horndike I s first factor accounted for more than 9310 of the
-5-
non-chance variance. His rotated factor 10adingSI which are presented in
Table 1 1 showed that the word knowledge test (Skill 1) could be distinguished
from the other reading tests in terms of its factor pattern, but none of
the other eight skills was separately distinguishable. His paper emphasizes
the importance of the first factor in tests of reading comprehension,
which he believes to be largely a measure of reasoning.
The same data, then, appear to lead to very different results and
conclusions depending on the technique employed in their analysis. This
state of affairs must be disconcerting to uniqueness analysts and factor
analysts alike, quite apart from the uncertainty it creates about the
differentiability of reading skills.
3. A Further Refactorization of Davis's 1968 Data
The aim of the present paper is to examine further this question of
differentiability by exploiting the factor analytic approach more fully
than either Davis or Thorndike has done in analyzing Davis's data. Both
Davis and Thorndike based their factor analyses on the correlation matrices
between eight variables for the full sample (N = 988), tHe two matrices
being formed from the upper and lower triangular matrix respectively of
the Form C vs. Form D correlations. The employment of this procedure
makes it necessary to identify a factor representing a particular skill
on the basis of one high test loading only, or on the basis of the absence
of a significant loading for one test only. Davis's data, however, are
admirably suited to the application of standard factor analytic procedures,
in which a factor is defined by at least two and preferably three or more
measures of a hypothesized skill. For each of his eight hypothesized
skills, Davis prepared "an essentially equivalent pair" (DaVis, 1968,
p. 51'!) of twelve-item tests. Thus, skills which are experimentally
-6-
dist:i.nguishable should appear as factors on '~lhich an equivalent pair
of tests have the highest loadings, or in a simple structure solution,
the only significant loadings. For factor analysis purposes, then, the
basic correlation matrix becomes a 16 variable matrix of the form!
ClC2
c8DlD2
D8
Cl C2 ••••.•••••. c8 Dl D2 •••••••••••• DB
j.nstead of an eight variable matrix. Each pupil has a score on each of
16 tests. The fact that all of the tests are not taken on the same day
is not regarded as a limiting feature in factor analytic studies, ',{hich are
qui. te often based on measurements gathered over a number of days. 'dhile
a 16 variable correlation matrix based on 988 cases could have been factored,
there 'vrere cross-vallclati.on advantages in analyzing the two 16 variable
con elation matrices representing Sample 1 and Sample 2 respectively,
each based on 4911. cases.
'rhe design of Davis I s study lent itself pa.rticularly 'dell to the
use of the maxiJl11.lln likelihood factor analytic procedures developed by
J~reskog and others (1968, 1969, 1970, 1971) since it provided a clear
basis for specifying a theoretical factor model, and since the maximwa
likelihood procedures allO\ol a statistical test of the goodness of fit of
the obtained data to the model. 'rhe application of these proceduTes in the
form proposed puts the unique variance of the skills i.nto the common
-7-
factor space, and allows such variance to emerge as a factor if the factorial
solution is acceptable at the required level of statistical significance.
The test data were originally collected to test the hypothesis that
the eight reading skills '..rere in fact all differentiable, even thou[.,h they
might share a considerable amount of' common variance. Thin tlypott.es is can
be represented in oblique simple structure form, as follows:
HyPothesized Structure 1
FactorsTest I II III ..................... 'lIn
Cl (Recalling word meanings) x 0 0 0Dl ( 11 It It ) X 0 0 0
C2 0 x 0 0D2 0 x 0 0
C3 0 0 x 0D3 0 0 x 0
c8 (Following structure)D8 (" ,,)
oo
oo
oo
xx
The structure was hypothesized to be oblique as the tests all involved reading
comprehension and positive correlations among factors might therefore be expected.
In orthogonal and near-orthogonal solutions, skills may be differentiable in
terms of their specific pattern of loadings on the various factors, but if they
measure some unique aspect of reading comprehension, it should be possible to
isolate them by allowing the factors to become more highly correlated.
A restricted maximum likelihood factor analysis was made of the 16 variable
correlation matrix for Sample 1, using ACOVSF, a general computer program for
analysis of covariance structures (J8reskog, Gruvaeus, & van Thillo, 1970).
The following constraints were imposed:
L Each factor was required to have equal loadings on the two designated
tests (as they were equivalent by definition) and zero loadings on all other
tests.
-8-
2. Diagonal cells in the correlation matrix among factors \-rere
set at unity.
3. Unique variances '...rere required to be identical for each test
within an equivalent pair.
Except for these restrictions, the matrices of factor loadings, factor
correlations, and uniquenesses '...rere free to take any value required by the
minimization procedures.
The program produced maximum likelihood estimates of factor loadings
in the spec ified form, but given that the hypothesized structure '...rere true,
the fit for the obtained data was very poor indeed (X2 =158.48, df = 92,
£ ~ .001). It was similarly poor for Sample 2 (x2 = 157.92, df = 92,
£ <, .001). 1,lhen the program was reapplied to both Sample 1 and Sample 2
correlation matrices without placing equality constraints on paired factor
loadings and paired unique variances, a slightly improved fit was obtained
for Sample 1 (-1 = 99.43, df = 76, 12. = .037) and for Sample 2 (-1 = 104.38,
df = 76, £ = .017), but £ values were still below .05, the predesignated level
for an acceptable degree of fit. Hypothesized structure 1 \.;as accordingly
rejected, indicating that the eight hypothesized skills could not be
experimentally identified as separate skills.
With this hypot.~esis rejected, there appeared to be no clear a priori
or theoretical basis for postulating how many separate skills were involved in
the sixteen tests, or which tests would be associated with which skills.
An unrestricted maximwn likelihood procedure developed for exploratory
factor analysis (JBreskog & van Thillo, 1971) was therefore applied to
the data to determine the minimum number of factors needed to account
for the observed correlations. This program sho-wed that at least three
factors were needed to account for the correlations in Sample 1 (X2 = 9l.41,
df = 75, £ = .096), and at least four factors in Sample 2 (X2 = 64.43,
df = 62, £ = .392). Varimax-rotated factor loadings for the three-factor ana
-9-
four-factor solution for Samples 1 and 2 are ~resented in Table 2.
Insert Table 2 about here
Inspection of the loadings in Table 2 suggests that Factor I in all
four solutions might be tentatively labelled as reasonir.g in readi.ng on
account of the high loadings of Skills 5, 1+, and 3 and the subs tant ial
loadings of all other skills except vocabulary (recalling word meanings).
Factor II in all solutions is clearly a vocabulary factor. Except for the
three-factor solution for Sample 2, Factor III is largely a composite
of Skills 6, and 7, recognizing a writer's purpose, attitude, tone, mood,
and techniques. Factor IV is probably best interpreted as the ability
to follow the structure of a passage, a description which also applies
to Factor III in the three-factor solution for Sample 2.
In the strict application of the maximum likelihood procedures, this explor-
atory factor analysis would be undertaken with one sample only, and the results
used to generate an hypothesized structure matrix to be tested by the restricted
maximum likelihood procedures on the other sample. As the object of the present
study was to identity as many reading comprehension skills as possible, the
four-factor solution as '",ell as the three-factor solution in Smnple 1 was used
to assist in the generation of an hypothesized structure matrix. This decision
was supported by the results obtained when the unrestricted maximum likelihood
procedure was applied to the data for the combined samples, i.e., Sample 3,
N = 988. In this sample, three factors were insufficient to account for the
observed correlations (X 2 = 119.87, df = 75, p = .001), whereas four factors- -were adequate (x2
= 74.09, df = 62, £ = .140).
The exploratory factor analyses indicated that only the skills of reasoning
in reading, vocabulary knowledge, recognizing a writer's purpose, attitude,
mood, and technique, and following the structure of a passage, were likely to
be differentiable. On this basis, hypothesized structure matrix 2 was developed:
-10-
Hypothesized Structure 2 Factors
Test I II III IV
Cl 0 x 0 0Dl 0 x 0 0
C2 x 0 0 0D2 x 0 0 0C3 x 0 0 0D3 x 0 0 0C4 x 0 0 0D4 x 0 0 0C5 x 0 0 0D5 x 0 0 0
c6 0 0 x 0D6 0 0 x 0C7 0 0 x 0D7 0 0 x 0
c8 0 0 0 xD8 0 0 0 x
The hypothesized matrix was again set up as an oblique simple structure
matrix, on the grounds that any overlap in the skills measured by the different
tests could be represented by the correlations among factors.
Thi.s structure was tested by applying the ACOVSF restricted maximum likelihood
program to the ::.6 variable correlation matrix. I-lhen the paired test
loadings within each factor were required to be of equal value, and a similar
restriction was applied to the unique variances of each test within a pair,
the hypothesized structure was rejected for Sample 1 (X2= 208.53, df = 114,
:£ < .001) and also for Sample 2 (i i:::: 212.75, df = 114, :£ < .001). When no
equal value constraints were imposed, the hypothesized structure matrix was
againrejected both for Samplel(Jf = 153.35, df = 98, :£ < .001) and for
Sample 2 (X2 = 155.02, df = 98, :£ < .001). Thus, even when correlations
between factors were virtually unity in some instances, it was not possible to
achieve an acceptable degree of fit to these hypothesized structure matrices.
-11-
It was then argued that if the four skills identified in the exploratory
factor analys is i,rere S epm.'atel;)' dis tingui.shable, they should appear as
separate factors in an analysis based on the relevant tests only. For this
analysis, the Skill 5 tests ·vrere tal<::en to be the best measures of reasoning
in read.ing, and Skill 6 tests as the best measures of recognizing a ''''riter's
purpose, attitude, tone, and mood, as they generally had the highest
loadings on the relevant factors in the exploratory factor solutions. 'TIms,
hJ~othesized structure 3 took tbis form, again conceived as an oblique
simple structure:
Hypothesized Structure 3
Test
Cl (Recalling i·rord meanings)Dl ( " II " )
Ix;.c
Factorsn IIIo 0o 0
IVoo
C5 (Dray,{ing inferences from content)D5 (" II I! ")
c6 (Recognizing "rriter I s purpose,attitude, tone, and mood)
D6 ( II II " )
c8 (Following structure of passage)DB (" " II I! )
oo
oo
oo
xx
oo
oo
oo
xx
oo
oo
oo
xx
This st;i:'ucture w'as tested by applying the ACOVSF restricted. maximum
likelihood program to the appropriate eigl:Jt variable correlation matrix.
'Vlhen the restriction of equality ims imposed on the paired test loadings
",ithin each :factor and the unique variances of equivalent tes ts, the
;>hJ~othesized struct1n'e was rejected for Sample 2 (X- = 44.34, df = 22,
12. = .003): the structure 'was first tested in Sample 2, as the exploratory
factor analysis indicated that 13.t least four factors ',{ere required to account
for the correl? tiol'l's in this sample. 11hen no equal val.ue constraints '..rere
imposed, the l~~othesized structure was accepted for Sample 2 (X2 = 22.93,
-J 2-
Qf = 14, E = .061) and also for Sample 1 (X2
= 18.35, £f = 14, E = .191).
The factor loadings and factor correlation matrices for these two solutions
are presented in Table 3.
Insert Table 3 about here
It is clear from 'l'able 3 that the four skills of recalling word
meanir~s, draWing inferences from the content, recognizing a writer's
purpose, attitude, tone, and mood, and following the structure of a passage
are differentiable as separate skills. Except for Skill 3, finding answers
to questions answered explicitly or in paraphrase, this result is in general
agreement with Davis's findings for his uniqueness analysis. As a matter
of interest, the ACOVSF restricted maximum likelihood program was applied
to a 10 variable correlation matrix to test an extended version of hypothesized
structure 3, in which two measures of Skill 3 defined a fifth factor. If
five factors had been needed to account for the correlations, this extended
2structure would have been accepted for Sample 1 (x = 33.83, df = 25, E = .111),
but rejected for Sample 2 (X2= 39.00, df = 25, £ = .037). There was
thus insui'ficient evidence to support the claim that Skill 3 '"ras a separately
dist inguishable skill.
Tucker and Lewis (1970) have raised a number of questions about the
application of the likelihood ratio or significance test associated with
maximum likelihood factor analys is. They point out that in contrast with the
usual situation, rejection of the statistical hypothesis leads to rejection
rather than acceptance of the scientific hypothesis, in this case, the
specified factor model. They also assert that the statistical hypothesis
"lill almost certainly be rejected for very large samples at any usual level
of significance. In addition, they present empirical evidence to show that
the likelihood ratio procedure sometimes yields more factors than can be
-13-
meaningfully interpreted.. They propose a form of reliability coefficient to
indicate how' well a specified factor model represents the relationships
among the attributes being investigated, and suggest that if this coefficient
has a high value, the meaningfulness of the factor results might be used as
a guide in determining the number of factors to be accepted.
There was little doubt from tbe present study that four factors could
be readily interpreted. To allo\ol greater account to be tal,en of the criterion
of meaningfulness, hO'flever, and to checJ, further the possibility that Skill 3
might be identifiable as a separatel;}' distinguishable Skill, direct oblimin
rotational procedt~es were applied to the matrices of urrrotated factor
loadings produced by the unrestricted maximU>ll 1ikelihoocl analyses for four-
factor and five-factor solutions for ScU"nples 1, 2, and 3. For each of these
solutions, Tucker's reliability coefficient was .995 or above. As a reasonably
higb. level o.f correlations could be expected among factors, the oblimin
