J.
ED 104 905
AUTHORTITLE
O
ItipTimpTipx
DOCUMENT RESUME
95
Richards, James A., Jr.
TM 004 354
A Simulation Study of the Use, of Change Measures.to;Compare Educational Programs. Report No. 163.
Johns Hopkins Univ., Baltimore, Md. Center for the.
Stud.), of Social Organization-Df Schools.
SP WS_AGENCY National Inst.D.C.
of Educ ion (DREW), Washington,
`RE ORT'NO tS0S-R-163PU DATE .Oct 74CONTRACT-NOTE
BDRS'PRICEDESCRIPTORS
NE-C-00-3-011425p.
MF-$0.76 HC-$1.58 PLUS POTIGT*Academic Achievement; Achievement Rating; *Behavior
Chinge;/ Computers; Educational Asiessment;.
Measurement Techniques; *Program Effectiveness;*Public Schools; *Simulttion
ABSTRACTArtificial data_weri-nted to assess the-correlation--
between Several'estimites of average student change=in various,schools and the'iltrueiit imPaOt of those schools. Results indidate that-
allestimates involving pretest-posttest differences measure school
impact with'reaponable-accuracy. It is important to Measure change
oVer-_-the entire courSe af.learning, however, And-not-just over the
liter-stages of learning. le correlations between -change scores and
r-othe school_characteristic reflect with reasonable accuracy the-
relationships beiweenthoseicharacteristics and impact, but will be
large only -when tht 4pderlyp.ng relationships are substantial. Simple
gain scores-measvie_tlhe true situation About as accurately as other
change estimates, are easier to compute, and probably_are,more-meaningful to nonrese rchert.lAuthor) ;
O
,
L
A'SIMUSTION STUDY 011.'THE USE' OF CHANGE MEASURES
70-COMPARE EDUCATIONAL PROGRAMS
Contract No. NE-C-00-3-0114
Work Unit No. 2A
James M. Richards, Jr.
Report No. 183
October 1974
U.S. DEPARTMENT OF HEALTH.EDUCATION &WELFARENATIONAL INSTITUTE OF
' -EDUCATION-'THIS DOCUMENT HAS BEEN REPRODUCED EXACTLY A$ RECEIVED FROMTHE PERSON OR ORGANIZATION ORIGIN.
.....KriZIG.LT-POINTS-OF.VIEW OR OPINIONS-_STATED DO NOT-NECESSARILY REPRE.SENT OFFICIAL NATIONAL INSTITUTE OF'EDUCATION POSITION OR POLICY.
Published by the Center for Social Organization, of Schools,-
supported-inpart as a research and development center by
C\ funds -from the United States National Instittiteof'Education,
Department of Health,. Education and Welfare.- The opinions
expressed in this publication do not necessarily reflect'
the position or policy of the National Institute of Education,_
_and no 'official endorsement by the Institute should be inferred.
The Johns. Hopkins University
Baltimore, Maryland
STAFF
John L. Holland, Director
James--M.-McPartland, Assiitint Director
'Karl Alexander
Denise C. Daiger
David L. De Vries
Joyce L. Epstein_
Ann Forthuber
Staphadie G. Freestan
Gary 1;,-Gottairedson,= _
Ellen Greenberger
Edward J:'Hitrsch
Rosemary Hollici
John H. Holiifield
JosselSon
Nancy_L. KarWett
Hazel C.
Marie eliiikurath
Daniel
1
D;-:-Mc_ComOehie
Donna H.
Edwird-McDil2
James W. Michaels__
:Mate
Susan L.
Julian C. 'Stanley
0.4,411c
O
-
:Introductory Statement
-.The Center, for Social Organization of Schools has --two primary
to develop a scientific knowledge of how schools afect
their students, and to use this knowledge to.develop better school
Objectives:
-.practices and organization.
/
The'Center works through threeiprogams to achieve its objectives.,
The Schools and'Ilaturity program ia-su yidg,the effects of school,
family, and peer group experiences on he development'of,aititudes
consistent with psychosocial maturity The objectives are to formu-
late, assess, and research importan
traditional academic acWevement.
currently concerned_with authorit
educational soils other than.
School Orlianization program is://
control structures, task structures,
reward systems, and peer gtoup processes
-program (formerly Careers!
of career development, t hag' developed
rricula)
in schools. The careers/------ --
-,,,..
bases `its work upon/4.iheOry
/_ /i
a self-iadilnisteredrVocational
guidance device and.a elf-d rected career program to promote vocational
development and to foster satisfying curricular decisions for high
. /
/
,school, college, an adule populations.
This report, repar d by the School Orianization program, examines
effectiveness of schools and edutationalmethods of assessing
programs in prom ting educational growth of students.
ii
Abstract
'
Artificial data were used to assess the correlation between
- several estimates of average student change in- various schools and
the "true" impact of those schools. Results indicate t1u,t all
estimates involving pretest - posttest differences measure school---------
impact with-reasonable accuracy. It is important to measure change
over the entire course of learning, however, and not just over the,
later stages-of learning -ThV correlations-between change scores and
other school characteristics-reflect with reasonable` accuracy the
relationships-between those characteristics and impact,-_but will be
large only-when the underlying relationships-are substantial.
himple-gain scores measure the true situation about As- accurately as
-,other_change estimates, are easier to compute, and iprobably are more
,
meaningful-to non-researchers.
A
Introduction .
A basic purpose of education is to promote _desirable -change or
growth !in the educational attainment of students. It follows that-..
SchOols cr other educational ,programs should be evaluated largely' on
their effectiveness In promoting such change. There are many theoretical
--problems in estimating_ student change _from scores cin,standard tests of.
educatio al attainment, however, and these .problems are heightened in
the typica sit tion where the students entering various_ schools differ
2 systematically (Astin and Panos, 1971; Cronbach and Furby, 1970; Hatris; _
1963; _Herriott and Muse, 1973; Klittgard and Hall, 1973; O'Connor, 1972).
It has -been difficult. to assess the practical importanCe:of these
in, m------------77St.,
thearetical-problems-becau-se7true_ chatiFge: scores are unknown o
_,, ,.. _ _. _ -
-e-
computerlongitudinal research. Recently, a coMputer procedure waS developed to. _ ,,, -
. _ _ . _
._.. . .
provide a rtif If ia- l data -in- which these true change scores are knoWn-- _-_.t
(Richards, _ Karweit,_ and Prevaet, in press) ._When such artificial" ciata--
--4. _ , -. - .
_--- were used to_compare several statistical. techniques fOr -assessing change...,...-.
in individual students (Richards, 1374), the results indicated that
individual change-is meas- ured with -reasonable adcoracy by all techniques. . .
,. .
that involve the difference between the pretest and the posttest.- In
particular, the simple difference between the pretest and the posttest
is,about as_ accurate as other change estimates, such as regre-ssed gain. ,
.
-scores, and is'much -easier to compute than other estimates. These trend's,
, .
hold even when ;students are assigned nonrandomly to schOols that differ
In -their -impadt_ on Students.
----These results strongly suggest that the theoretical probleals-of.
change measures have limited practical -significance .for measurin6
individual growth, and it is important to deteriine whether this is also
the _case for measuring school impact. Accordingly, in this study artifi,6
,tial data were used-toassesS the correlation between several estimate-s
of average student change in various schools-and the. "true" impaLt- of- A
the. same schOols. This study-is. stated in'the-cohtext of education, but
D.
the procedures for gentrating,daia and measuring change- are abstract.
Therefore, the results should generalize to many situations -where one
- wishes to compare_ithe-impAct-_-of-varying social interventions.
MethodI
Simulation- Procedure._ Because it -seems desirable for artificial
data to resemble real data as cloSely-as posSikte, the computer procedure, .
IS
was designed (Richards, et at., in press) to reproduce selected aspects..
_
t
-of the ETS- Growth Study (Hilton, _Beaton, and Bower, 1971) arid of the
\Project TALENT study of high schools in the:United States (Flanagan,
_ \ -_. _,.
tt al ._,_1 962 . In the ETS GrOlth Study- students were assessed initially
with a measure academic potential (SCAT), and a measure of educational.
attainment (STEP). -Subject to the usual attrition in longitudinal' --4
research; the educati al ttainment_of these students was reassessed.
, :--/
. -
on three Subsequent occa ons. Project/TALENT provided' intercorrelations
4/
amOng a _variety of communit school, And student chatacteristica for/
.
/
a- representative sample -of U. S high schools.
t
.
2
4
. The computer procedure generates scores for ifidiVidhal students0 -
that strive2-to reproduce the means, standard deviations, and ittersorl-.:.,.________--. ______- __
-relations-obtat rhL7d-T-1-1----TS-----G-r-citithSttidy. The student '_s score on
academic potential is generated first and used to derive that student's-
score -on initial acadeMic attainment.. Theta gain scores 'are generated
and .added _to yield subsequent attainment TrUe standard scores
are generated initially, then the appropriate amount of .random erroz
added, to each score and the' scores are transformed to the metric of the
_ETS Growth Study_abserved, scores.r:;:rhis, simulation proceduie closely.
-reproduces the ETS Growth Study ressrts (Richards, 1974)..
The simulation procedure, permiie-the.investigator to assign students
to schools either randomly or nontandonily. When tudehts_are assigned.
nonrandomly, the prograi strives to reproduce the average -correlation
between commubily per,,Capitaincome and ayerage academic -potential of
s_ tudents - estimated---from'Ptoject results -(p= .54). The ratio
of between schools variance to total variance also simulates the,Projectt
.TALENT( ratio. ------------4_.=-------_,1
The simulation -proce-dore---a-Sifumes that commUnity pek capita income -TThe 4
determines school resources, and that school resources in turn determine., ii
'school impact. A review of Project-TALENT results sugge4te'd-an average
correlation of aPprOximately .25 between commuhitY'._incOme and those
on y assumed to facilitate student 'growth, so the-. 6
simulation procedure strives. to reproduce this re ationship between
income' and resources. Community- income is dra n randomly from a -normal
.
distribution, and it is assumedthat school resources and school impadt
-.also-ere-norMally distributed.
.There is little empirical basis for estimatifig either the correla--:
motion between resources and,t4pact or_the-extent-to-OcCh schobls vary
-in impact. Therefore, the simulation procedure allows th3 investigator;
to specify both the correlatiod between resources and impact and the_
standard deviation of the impact variable. This standard deviation is
.. ,
specified in the form of a number between 0 and ly When the standard
deviation is .10, the average growth values uded in senerating scores. .
are equal to the average growth scores obtained in the ETS study for a-,
school with average' impact, and 10% higher { than the ETS averaged for
a school one standard deviation above the mean on impact. (The siiulated
* - .
data appear to meet.the assumptions for this manipulation even if the
ETS,data do not.) -.
Gain scores for individuals are generated according to the following
principle:
=t_--
Gm d
I'-whete Gt t
is-total (true).- growth, G is average (or mean) -growth (i.e.,, -._ .
the parameter estimated from:the ERS data and Gd is-a deviation frOM/ ..
this aver' in ividual differences in true growth./ The
total gain score Is, added.tO,the pretest score to ,yield the pbstteSt
score, and the posttest score-then becomes the pretest-for the nekt
growth interval. For each giowth.interval., the e-pretest is one of the
/
elements entering a multiple regression formula used to generate the
.11
-
_Gdvalues. The correlationsbetween pretest and growth become increasingly
negative'lor'successive interval(Richardsi 1974).
Ifi-generating scores, the mean growth parameters for the three'------
t
z f
. . :
-.intervals are adjusted _for-school-impat,f, and no other changes are. made.. \
Consequently, the adjusted mean growth parSmeters_frequently-will-fiot be
i
------64fial.tos,theatained average true growth scores for a given school,___
_.
A school with above average'impact. ill,have higher than average-mean .-----,,
_growth-parameter and there ore higher than average true posttest ,scbres. ------
,-These become high r -than average truelmtest scores for SubSequent
\
learning-interva s, and these higher pretest scores make an inCreasingly-
--negative contrib,tion in the computation of subsequent true growth scores.
The averages of the obtained true growth _scores for that acboOl will tend
:to be lower than the adjusted mean growth_ parameters. Similarly, the
averages of the obtained true growth scores will tend to be-higher than
the adjusted-mean growth parameters for a school with below average imPact-;
.
Table 1. presents a simplified illustration of these trends for five
hypothetical schools that-are average in every_respect except for differing-.
__, - ., - . _n__- _ -_
\.
,inimpact. Because other parameters besides pretest score_are---involved 1
i--,
A
Insert Table I lAbout Here
Pia.5...Kg_tfig.Scores-(Richards; ITT , it is conceivable_that-a school-T
with below average impact (and therefore below average adjusted mean
growth parameters) will_have_ligher average Obtained_true growth-scot-ea=
than a' school with above average impact. his ts especially true when
students are assigned_to.schoolS n nrandmmly.
12, J
A
_,-
----'-- ., - -bata _Sets--Six -1-nde-pafden--t,sets O---f simulated data were generated,_ . . .
. _ . --,
____
._ , for the present study. Iii each set stndents_were-assigned.---to-100-schools,. .
or treatments: The number of students per 'school varied randomly with
mean = 150 and,-..tandard deviation ai".15. Therefore, the total number of
tlstudents in each of these sex sets* was _approximately 15,000..
,
In three,of these sets students were asdigned randixiily to schools'-,,._-:' .. z -- .
or treatments, nd in the other three sets students - were assigned`
-nonrandomly: U der each-typeo-Uassignnient, simulated data wtite generated
for three differ nt assumptions about the relationship-between tadsiitiOl.
resources and sc iinpaa., Specifically, it was assumed that school
esources account for-5%, 20%,'or 80% of t e iVariance in school impact
(corresponding to orrelatIons of .2236, 472, or .8944) .
Finally, in all six sets the standar deviatiOn of the impact variable
-1was\ set. at .10. \At approximately this rna nit_ ude two_ simulated schools
Mr
4 one standard deviation apait.on impact -(w th N's = no)' will differ
the ..05 level when compared- with- re'spec to educational growth between\
successive occasions. \.
.. ,_ ----
_:-__Change-Measures. "X Wide variety = of chdnge measures 'have been propoded
(Cronbach and Furby, 1970), but recent results suggest _that most ,of these
measures yield essentia10 e uivalestt_re-sult-s-(--Rfah-dcord-
ingly, this study used only four measures,of change, each representing
a different approach to est mating change._ These change estiinates
included:
1. Posttest score.
2.' Posttest score adjus ed for initial academic potential. This
change estimatefris.th= difference between posttest score and ,
-6 13
4 .
predicted posttest score, using initial academic potential as
the predictor. (The giediction equation for each data` set was
/ ,
--based on the Observed relationships in that set.) Thus,--this
technique, resembles analysis of covariance with academic poten-!
tial treated as e covaiinto.
A 3
3. Raw gain. This,chan score is the simile difference between
.-
pretest score and posttest score: /`,_ ".,,
)\\ I,
4. \Raw residual gain. This estimate is the difference b. eetf'\ -f 0 \
...
posttest score and predicAed posttest score, using pretiest--___
score as, the predictor
Results
To facilitate comparison with the earlier study of_individual change5
estimates (Richards,,1974)/the first step in the data analysis was to
compute the correlationsibetween average estimatectehangescores for
Various sehools and average.tr e.change scores for the same schools. -An-
\-/
unresolved question is whether it\is better\tb coMpute change -scores for'
individual students and then avers e-within als or to computo_change
scores fromschooI-meantr(Nir, Lint, and Patton, 1969), so both procedutes
were used to estimate change in this analysis. Table 2 summarizes the__
repUlts.'
Insert Table 2 About Here
These resultn seem quite consistent.with the resUats of the eatlier4
.study of individual change estimates (Richards, 1974). Chew is estimated
7
most accurately by techniques that.involie the difference between the
retest and the posttest, and these techniques seem equally accurate,
(i.e:, raw gain is just as accurate as residual gain). For the most .
part, there is little difference between change estimates based on
,individual students and change e'stima'tes based on school means. In a
few cases estimates based on school means have a clear advantage and
.
these' estimatesestimates are also easier,to compute, so subsequentianalyses in,
this paper-involve only estimates based on school means. .
The next analysis evaluated/the accuracy of these change estimates
as measures of School impact.
impact and various change\,
table al C.);suramArizes the
groWth scores.
Table 3.summarizet the correlations between
estimates. For .comparative purposes; this
correlations between impact and average true.
Insert Table 3 About Here
These results indicate that change estimates can be quite effective
in rank ordering schonls with respect to their impact even when studentsr,
. are assiwie to schoola,honrandomly. The simple gain score's again were .
just.as accurate s the residuifgain scores and, as Cronbach and FurFy
(1970),point out. postiest_sFore measures impact adequately when students
are assigned to treatments, randomly.
The results also indicate that it is imports to measure change
,over an appropriate interval. Adjusted potttest scores, :Amp in
acores, and regressed gain scores all. rank ordered schools accurately
15
1 \
\
i
-variations in impact. Such results are more typical of what would be
,ft
obtained in a'nreal"-longitUdinal study: 'Table 4 sumnarizes the relevant
.correlations betWeen resources and change. The- tagnitu es of these
correlations clearly follow the underlying relationship between resources
and impact, but are somewhat lower. The smaller magnitude of these
when they involved change from initial status, but none of th measures'
were particularly effective in rank ordering schools when they involved
growth in the'later stage's of the learning process. This ineffective-_
ness reflected the true situation, because it is also characteristic
of the true growth scores. The ETS data resemble otheriongitudinal or
learning data in-a number of respects (Richards,, 1974), so these findings
about when to measure changeihould have considerable generalizability.
The final question examined in\this study involves the relationships
am9ng these change Obasures and the school characteristics that cause
'Insert Table 4 About Here
t.
.
correlations p rhaps is partly-the,consequence of unreliability of the
Change score, but also appears to reflect the 'imperfect correspondence-m:'
. .
.
. .
between school impact and average true change. The results again indicate
-that raw gain is about-as accurate as any other change estimate, reempha-
size the importance of measuring changeover an appIopriate interval,
and suggest that the correlation between a school characteristic and
school impact must bereasonably substantial- before ani change score/
-'will reveal the relationship.
Discussion
Theoretical_ treatments of the issues considered in this paper have
emphasized the theoretical difficulties of using change scores in gener4
and of usi'hg.simple gain scoresin particular. The results of this study,
like those-of the earlier study of individual change (Richards,
suggest that the practical importance of these theoretical difficulties
may have been exaggerated. It appears that-change estimates over-an----71
appropriate interval (e.g., the entire course of_ilearning, not just the,\
.later stages) do measure-sthool impact with reasonable accuracy: The.
correlations, between change scores and other school characteristics
reflect with'reasonable accuracy the relationships between the same thar,
acteriatica and school impact, but consequently will be large:(or11Y'
ficant") only whin the underlying relationship is fairly substaniia
'These conclusions appear relatively unaffected by random vs. nonrandom0 .
assignment of. students (although this finding could change for more severe
nonrandamness), or,by-whether change:measures involve individual scores,
or, school means.1
Insensitivityto weak relationships almost certainly .is character-,
ffisticnot just of change scores; but of all static 1sp ca -procedures that
.
might be applied to these dataand simple,gain scores appear to reflect
the true situation about as accurately as'any other estimate of change
or impact; Simple gain scores also are easier to compute than most other
-- estimates and probably are more meaningful to non -researchers. Therefore,
f."
,the-resuits of this study suggest that it often may be quite appropriate.
'It should be.emphasized that these.conclusions'apply to true longitudinal,designs and this study should not be used to justify such procedures as
Measuring impact'by educational ittainment.adjusted for a test of academic"
potential administered at the same time.%
lo
5\
0
campfire educational programs on the basis of simple pretest-pOsttest
differences.
The, discrepancy between this study and. earlier, ,theoretical treat-s
mAnts may perhaps best be resolved in terms oedegree of concern about
"Type I" errors. That is, -theoretical treatments usually seem to assume:.
that educational treatments do not differ on impact and emphasize the
possibility that use of change scores, particularly simple gain scores,
will lead to the false conclusio that they do. differ. Certainly this
cannot be ignored, especially whenthe students assigned to
-various treatments differ considerably (Astin and Panos, 1971; Cranbath.
and Furby, 1970), and certainly if is,possible,0 propose hypothetital
situations where change scores could be mikeadiag or confusing, esPeciall
one has -a tastefor_paradoxes (Lord, 1967). (This study, on the other
hand, assumed that schools do differ on impact and asked how accurately
change Azores describe these differences. The answer to this question
appears much-more favorable to change scores. Indeed, the results
suggest that\when one uses change scoreasove an inappropriate:interval
_ ..\ ,
,
in a correlational study there may be a grA0er danger ofithe false_
.
.
P . ..
conclusion that schools do not differ with respect to impact than.of th.
r1
false conclusion that schools do differ.
Cronbach and Furby (1970) correctly point out that some of the
questions to which change scores might be applied could be an ered
directly with such techniques as partial correlation. The advantages of
such techniques are that they are more direct than change scores, howver,
-hot that they are more accurate, nor that they require less statistical
11'
sophistication
who prefers to
The results of this study lend support to the investigator
use change scores for reasons of convenience or ease of
understanding.
Finally, the results of this study again illustrate the-usefulness
of simulationtechniqueo.for investigations Of longitudinal methodology. ,
\ v 0
It would'be impossible to investigate the questions considered in this
study. with "real" _longitudinal data because the investigator would have
no way of knowing either the true individual growth scores or-the true
school impact scores: At best one could compute the intercorrelations
amongdifferent estimates of change (Dyer, et al., 1969). With simulated.
data it was easy to compute the correlations between true scores and-the
different estimated scores. It would also be easy to extend the simulation_
procedures to the situation where considerable' attrition of subjects-occurs,
s
to the-situation where one has only pseudo-longitudinal data (e:g.-, test
scores for Occasions 1 'nd 2 obtained from different groups of students'
in the same school), or to different models for growth: 'Ihu-s, 'simulation
/ .
technique's offer considerable promise for refining our knowledge about
. when various proceduresfor analyzing longitudinal data are appropriate.
rt,
12 19
c References
Astin, A.W. and lianos, R.J. I The evaluation of educational prosraim.
,_ .. . /
In/R.L. Thorndike (Ed .) Educational Measurement; - Washington:
AMerican Council on Education,- 1-971.
Tronbach, L.J. andiurby, -14--.- HoWahould-We measure change--or:shouldi - R
,.__..--
'we? PSychological Bulletin) 1070, 74,6840:/
Dyer, H.S.; Linn,' R.L.; n Patton, Ma._ A compariOn_of four methods
/ . ,: s_. H
. ,
,-
of obtaining discrepancy measures -based on observed-and predicted/
school system means oh.achieyement,tests. American Educational
Research Journal, 1969,- 6 591-605.
Flanagan, J.C.; Dailey, J.V.; Shaycoft, M.F.; ;,enctGoldberg,'-I.-
Studies`of the American_High school. Project_ TALENT Monograph No. '2,-;
-Pittsburgh: University_of Pittsburgh,-1962.
Harris, C.W. (Ed.) Problems in,Measuring-Change. Madison; Wiacon4in:.
=
University of WieconsinA2ress, 1963._
_Herriott, E.H. -and Muse,,D.N., Methodologicalissues in the studp.af-
school effeots._*n F.R. Kerlinget, (Ed.) -Review of Research in
- .
'Education. 1tasca, :Illinois: __Peacock,---1973.
Hilton, T.L.; Beaton, A.E.; andBoWer; C.P. Stability and-Instability-
in Academic Growth - -A Compilation of Longitudinal Data. Princeton,
New Jereey: Educational Testing.Service,;1971..
Klittgard, R.E. and Hall, G.R. A Statistical Search for Unusually
Effective -Schools. Santa Monica; California:. Rand
F.M. A paradox in the interpretations -of -group comparisons.
Ps cholo: 304-305.
0Connor, .Extending. classical test-theory to the measurement of
4
obange. Review of Educational Research _1972, 42 73-98.
ichards, J. M., Jr. A Simulation4Study Comparing Procedures for
Assessing individual 'Educational Growth. Research Report No.-182.
.
Baltimore, Maryland: Center for Social Orgagization of. Schools,, -
i,
, -,,,,.
.. : .
The Johns Hopkins University, 1974.
Richards, J. M., Jr.; Karweit, N.;, add Prevatt,T. W. FORTRAN program
-for --Simulating iducatiOriair growth. With 'varying achoOl impact ,
Educational and Psychological Measuremene, in prer.A.
I
,Comparison of Adjusted, NeanOain Parameters With
Average
Obtained Tide"Cain.ScoresforlipothetiOalSChoOls'
School
School
Impact
1L.20
21.1-0--
31.00
4.90
5.80
trarying in Impact -
,
Adjusted 'Mean Gain 'Parameter
-,For computingTrUe:Gain
Scores in
Or
of
'Average of Obtained True Gain
Scbres,in, Interval:
,i, Occasions' Agccasibps,
cas,
Ocions'
,04asions
1to
22 ''tr.',3
,-37 to,
*A. .to,',2
;.,'
1.0392
.9852
.7812
1.0392
i,
,
.9526 ':
.9031,
.7161
.9526
".,86 6o-
'.8210
,,
.,6510''
'
',,..:8660 ,,'.
.7794'
':7389'"H
,,
.'59,'' ,.,,,.
,7794
.6928
.6568-
.520S'
.6928
Occasions
Occasions
2.to
'3
3to 4:
.9277
.,8744'
.6672
.8210
.6510
.7676
.6348
.7143
.6186
NOTE':
Gain 'scores are exiiiessed inunits of the trde s'corestandard,dViation for
OCcaSiOn 1:
Iris assumesItheseschools, are
in ever*Y;respect except
H. for differipg' in impact
Values for"Schdoi 3 'are
identical with those
'.(
estimated'from the,ETS'Growth.Study.
Change' Estimate
posttest' Score
tO
CjPostteit Score
AdjUsted'For' Initial
'Academic 'Poteniiil
Raw 'Gain
'Residual Gain
Table
Correlations ,Between,Aveage'TrueGaih-in Diff rentSchodls
and Various'4timates of Average Change
(N's = 100)
',Correlation Between
'Reedurcei and Impact",
.2236
.4472,
.8944
NOTE:
.2236.
.4472
8944
.2236'
.4472
.8944
N.2236
-8944
.Assignment of Students
Noniandoli
Random
,
Average 'True Gain in Interval:
Average True Gain in
1to
22
to
33
to
41- to 2" 2
to
3
M
72
'77
70 86
'85
77;
'70
8087
85
"'
90
90
49, .89
88
88
'190
0,0
88
,88
,
Numbers: in,columns, labelled I :show correlations betweenaverage true gain
69,
80 H
.75
75''.42
'42'
42'
36
36
42
-46
4,
37
l3.7
49'
,,e4.9
72
72,,
86
.85
80
80
76
74
87
77 0
77
'80
43,
88'
71";
82
68'
71
71,
47,
67'
73
73'.
70
73'
658
73
75'
47
47
,S4
54
50
504
78
88
:xi,
90
82,
76 90
,91',.
91
'86
86
79
AV
+
85
83
80
80,
45
45
43
43
38
.38
7382
75
77
80 79
69 70
65'
6.9
81
76
77
80 79
70
.34
and
estiMates),Ased'On,indiVidUar,studenia Numbers' in columns labelled M show
Correlations, kor'estimate6 baSed'onaChoolHmeans.: Decimals are omitted.
Foi
'this and, subsequent, tabieil'" = .14,and,r '=-\18"
er:
Art
Interval
. 3
to .4
IM
0011
-03
0011
-03-
33
47
41
.4Q
:24-38
65
.70
'61
2235 19
65 70
61
'65
70sr
tt
IS8le
Co,:reIatiOns'o", Sctiool ImpdO,i,With4Ohailie',MeeSures and Estimites
),
'Change Measured 81,.
AVerSge'True' Gain
From OCasslon'A.
Avetagetrue
Gai
n,From Judtyreyious
Occasicin
)
'Change' Estimated
By:L
Posttest' Score
110
)
ToSttest Score Adjusted
'FOr Initial Academic
1-'7-Potential
Raw Gain Frbm-TOccasion 1
4
Raw Gain From Just
PreuioUt Occasion
"Residual Gain Trotti ()Cession
,rResidual'Gain From X*:
Previous Occasion,'
Correlations etween
R#sourcesend,Impact'
.8944
2236
.4472
8944
.2236
.4472
8944
.2236
.4472
.8944 '
.2236
q4472
:8944
. .4472
.8044 .7
.2236
.4472
.,8944
.2236
.4472
,.8944
'.87
,90
.89
!87
00
.644
.71
.63
.69
.83
.80
.90
.78
:85
:77
.81
.86
.75
.53
.78
.70
.81
.57
77-
.79,
.88
80
.85
.77
.31
.39
43
.80
:40
'Change to Occasion:
,3".
42
.92
05
.04
.71
.82
.70
77.
.83
.94'
.95
.18
.21
.35
'.77
.84
.80
.80
.90
.85
.88
.90
.12
.14
.18
.85
.96.
.30 7"
21
.30
4
.86
.p2
.94
.88
.93
.94
.83
.90
.92
.86
.71
.35
..88
.82
.46
:83
75
.33
.27
.34
.37
,31
.39
.43
.22
.31
.35
.73
.84
.87
.80
.87
.83
.87
.79
.85
.89
.82
.89
.92
-.71
.85
.89
.79
.54
.26
.82
.71
.66'
.30
.78
".85
.89
.82
.89
.92
.86
.88
.78
- .25"-
.82
.61
.34
.70
.63
ti
tOrXelations, Of
'Change Measured By:
Average' True Gain
".From Occasion' 1
AVerageTrUe Gain
From JusP':PreViOds
Occasion. °
ir
4't114ilge Estimated .13V:
7r
POsttest,ScOre
60[Foit:test-Score Adjusted
For Initial, Apademic
Potential
Rati Gain From Occasion 1
pol Resources with Change
Measureii4and Estimates
(N's = 100)
Correlations Between
ResOurcesiand Impact
.2z36
12236-,
L4472'
.8944
42236
.4472
.8'944
f .2236
t:484942
.2236
.4472
.8944
2236
.4472
;8944
282934641
.4472.
.2236
.447211
'1.8944
Raw Gain"FromJuit.
Previous Occasion
'ResIdUal Gaim
,*
FromOcciSiOn,1
ReidUalrein,Ftpm Just
PrevioutOCcasicin:
Assignment of Students
Random
'Nonrandom
Change to Occasion
Charige'tobccision
23
"4
23
4
.20
',
415
.13
,.22
.21
.18
.
-.38
,.42
,42
.50
'.52
.50
.78
483"
,83
-.74
.82
,.85
o.20
,.03
-.04 '
.22
.12
-.01
.38
.39
.09',
.50
.45
..16
.78
.69
.29
..74
.72
.31
.11
1,
.06
,,08
.22
.23
.22
.23
.33-
.34
,..36
.39
.41
.53
.66
.69'
',21
:29
.33
,1..,..-
.,07
:15
.(:)Ei
.15'
;14
.11
.
28'
437
.38
7.42
,.42
445
.65
.75
.75
,.66
.,,78
.80
.21
.12
.14
.21
.20
.17
.31
.39
.39
,.48,
.147
.50
.68 -
.76
.78
..66
.78
.80
.21
,'
-406
-.05,
.21
:10
'-402
.31
.34
.07
.48
,.28"
,.22
.
.68
.54
.12
.62
;63
.24
.05_,.... ..21
..12
,.17
.15
.12
'.31
',',.40
.39 :
.41,
,.41
.44
.68..
.76
.78
.62'
478
.79.
:OS
.13
-.02,
,.17
,.04
-.02
431
.26 °
.09 ..
.41
.21
-.th
0:
'.40%.
,.23
.62,
..59
,.32
P