A Brief History Of Leadership In Assessing Undergraduates’
LearningRichard J. Shavelson & Leta HuangStanford Education Assessment Lab
Website: http://www.stanford.edu/dept/SUSE/SEAL/
Invited TalkImproving Quality and Equity in Education: Inspiring a New Century of Excellence in Teaching
and AssessmentCarnegie Foundation for the Advancement of Teaching & ETS
June 1, 2006
Overview• The Foundation’s vision and role for “objective
testing”• Framework for comparing learning assessments
and locating vision• Brief history of college learning assessment
– Origins of objective testing: 1900-1933– Assessment of learning in graduate education—The
GRE: 1933-1947– Rise of the test providers: 1948-1978– Era of external accountability: 1979-present
• Concluding comments
The Foundation’s Vision And Role For “Objective
Testing” Of Learning
The Beginning Of Assessing Student Learning In Colleges
• Today’s call for campuses to account for their students’ learning appears almost as if there were no past…of course there was…
• The Foundation was arguably the ringleader under the leadership of its first president, Henry Pritchett
• Pritchett:– Was concerned with the quality of
higher education – Recognized the potential impact the
emergence of “objective testing” might have on monitoring quality, especially with the publication of E.L. Thorndike’s Mental and Social Measurements, and
– Had a Foundation staff member who agreed and avidly pursued the cause
Henry Pritchett The Foundation’s
1st President
William S. Learned:Instruments Of Testing Reform
• William S. Learned was that staff member who shared Pritchett’s passion for education quality and objective testing, became the instrument through which The Foundation transformed higher education learning assessment
• Learned along with Columbia College’s Ben D. Wood:– Led a large-scale assessment of college
learning in the state of Pennsylvania, – Parlayed this experience into the
development of the Graduate Record Examination, and
– Germinated the idea of a “National Examination Board”—a national testing agency—which 20 years later became the Educational Testing Service.
William S. Learned Foundation Staff
The Foundation’s Vision In Assessing Student Learning
Walter Jessup, The Foundation’s 3rd President, put their vision for learning assessment and higher-education improvement in words that sound remarkably familiar today:
“The central problems [in improving higher education] are three in number: first, the setting up of generally accepted standards of achievement; secondly, the devising of methods of measuring this achievement and holding pupils [now organization’s] to performance; and thirdly, the introduction of such flexibility in educational offerings that each individual [organization] may receive [deliver] the education from which he is able to derive the greatest benefit”
Walter JessupThe Foundation’s
3rd President
Framework for Comparing Learning Assessment
The Foundation’s Initial Approach
The Foundation’s Offspring at ETS
Current Claimed Approach—At Least Some Do
Source: Shavelson & Huang (2002)
Origins Of Objective Testing Of Undergraduates’ Learning:
1900-1933
Initial Objective Testing Of Undergraduates’ Learning: 1916-• Objective tests were seen as a scientific
improvement over previous oral and written examinations characteristic of learning assessment in earlier years
• Learned uses objective tests at the University Missouri’s Experimental School focusing on content knowledge in arithmetic, spelling, penmanship, reading, and English composition
• Thorndike’s studies MIT engineering students using objective tests in mathematics, English and physics
• Learned and Ben D. Wood conduct the Pennsylvania Study—the first large-scale objective testing of learning in higher education
E.L. ThorndikeEducationalPsychologist
Views Of College Learning: Prevalent Today As Well
• The Foundation’s Vision of Learning (Learned & Wood):– Accumulation of breadth and depth of content
knowledge – “Must be a relatively permanent and available
equipment of the student”– “Must be so familiar and so sharply defined
that it comes freely to mind when needed and can be depended upon as an effective cross-fertilizing element for producing fresh ideas”
• Competing Vision—Progressive Education Association (e.g. 8 Year Study)– Reasoning and problem solving– Life tasks– Moral, self and social development
Ben D. Wood Learned’s Collaborator
The Pennsylvania Study: Large-Scale Statewide Learning
Assessment• Objective testing of declarative and
procedural knowledge (achievement)of State’s college seniors (70% or 4,580) and high-school seniors (75% or 26,500) in 1928
• College learning of high-school seniors tested longitudinally in 1930 and again in 1932
• Testing Time: 12 hours in 1928; roughly half that later
• Number of Test Items: 3,200 in 1928; roughly half that later
Noteworthy Features Of The Pennsylvania Study
• Set clear conception of what was meant by learning in undergraduate education
• Distinguished learning and achievement
• Comprehensively covered content domain of general undergraduate education
• Developed testing technology linked to conception of learning including IBM machine scoring
Thomas J. Watson IBM’s 1st President
Assessment Of Learning In Graduate Education: 1933-1947
Motivation For Assessing Learning In Graduate Education
• As Pennsylvania study wound down in late 1930s, Learned and Wood parlayed this experience, and their staff, into doing the same for graduate schools
• Learned argued that the demand for graduate education had increased following the depression, that:– The AB degree had “ceased to draw the line between the fit and
the unfit” and – Something more than number of college (“Carnegie”) credits was
needed on which to base admission and graduate-student quality decisions
• Answer: The GRE– Initially tested students in graduate school– Rapidly moved to become a selection measure for graduate
school
GRE’s Rapid Growth• 1936-1938 Versions: 8 Profile content
knowledge tests—e.g., mathematics, physical sciences, social studies, literature and fine arts, foreign language [dropped in first revision], and a verbal factor
• 1939: Advanced tests in 16 academic major fields added
• 1946: General Education section (effectiveness of expression and general education index) added to Profile tests
• 1949: GRE Aptitude Tests—Q and V– Grew out of the Profile’s mathematics and
verbal factor tests – Offered as an alternative to the plethora
of tests available from the GRE• Beginning of shift away from measuring
content knowledge to measuring verbal and quantitative reasoning aptitudes
0
50
100
150
200
1937 1940 1945 1947Ten Years of Growth
(1937: Harvard, Yale, Columbia & Princeton)
Num
ber o
f Ins
titut
ions
Rise Of The Test Providers: 1948-1978
Motivation For Rise Of Test Providers
• End of WWII and GI Bill rapidly expands demand for higher education
• In response supply increases:– Number of colleges and universities
mushroom– Number of test providers
• ETS established in 1948• ACT established in 1959
Motivation For Creating ETS
• A response to The Foundation’s– Testing burden with the GRE program– Financial burdens created by its innovative
faculty retirement program (for which The Foundation was created)
• A response to demand after WWII• A means to preserve “meritocracy”
Origins Of ETS:Learned & Wood … Again!
• These events simply rekindled the notion of a “national testing service” that can be traced back to Learned and Wood who, as early as 1931 had envisioned a national college testing program, but the idea had “… languished, partly for lack of funds…”
• Their vision was a national testing service overseeing:– A neatly organized education pyramid from pre-college at
the base through college to graduate education at the apex– From base to apex, the pyramid increasingly relied on
“merit” as measured by scientifically sound “objective”tests, and
• According to Historian Ellen Lagemann, to blunt a challenge to objective testing launched by the Progressive Education Association (PEA) and the Eight Year Study beginning in 1933
Creation Of ETS• Notion of national testing service
languished between 1938-1945• Learned reopened the matter not only
with respect to the GRE but also “to the work of ‘other enterprises of a related nature’: the College Entrance Examination Board, the Co-operative Tests Service of the American Council on Education, and the Education Records Bureau”
• CEEB and Cooperative Tests Service joined the Foundation in creating ETS
• Carnegie Corporation provides $750,000 in start up for ETS
• The Foundation leaves the testing business…
Henry ChaunceyETS’ First President
Test Providers: Tests Of Undergraduates’ Learning
• Tests developed or ended up in The Foundation’s vision– ETS’ Undergraduate Assessment Program– ACT’s College Outcomes Measurement Project (COMP)
• Tests developed out of an alternative conception closer to the Progressive Education Association’s vision– Jonathan R. Warren’s study of “free-response questions” tapping
communication, analytic thinking, synthesizing ability, social/cultural awareness
– COMP began as performance assessment of effective functioning in adult life in social institutions, in using science and technology, and in the arts
– Tasks in Critical Thinking were performance based measures of critical thinking and communication skills
– Each program either died out (Warren, Tasks) or morphed into multiple-choice format (COMP) due to time, cost, and technology constraints
Era Of Accountability:1979-Present
Colleges’ Respond To Accountability Demands
Earlier this year, the American Association of Colleges & Universities recommends beating policy makers and politicians to the punch by campuses taking responsibility for assessing the value they add to learning:
“The best example of direct value-added assessment is the Collegiate Learning Assessment (CLA), an outgrowth of RAND’s Value Added Assessment Initiative (VAAI) that has been available to colleges and universities since spring 2004. The test goes beyond a multiple-choice format and poses real-world performance tasks that require students to analyze complex material and provide written responses (such as preparing a memo or policy recommendation).”
Déjà Vu All Over Again?
Technological Advances Make Alternative Visions Possible
• Assessments in The Foundation’s vision of objective tests with sophisticated testing and scaling technology– ETS’ Measure of Academic Proficiency and Progress
MAPP (Academic Profile renovated)– ACT’s Collegiate Assessment of Academic
Proficiency CAAP (new version of COMP)• Assessments closer to PEA’s vision with internet
technology for administration and scoring– Council for Aid to Education’s Collegiate Learning
Assessment (akin to Tasks and the early COMP)
Collegiate Learning AssessmentCharacteristic Attributes
Open-ended Tasks • Tap critical thinking, analytic reasoning, problem solving and written communication
• Realistic work samples• Engaging task as suggested by alluring titles such as “brain boost,”
“catfish,” “lakes to rivers”)• Applicable to different academic majors
Computer Technology
• Interactive internet platform• Paperless administration• Natural language processing software for scoring students written
communication• Online rater scoring and calibration of performance tasks• Report institution’s (and subdivision’s) performance (and
individual student performance confidentially to student)
Focus • Institution or divisions/programs within institutions• Not on individual student performance (although their
performance is reported to them confidentially)
Sampling • Samples students so that not all students perform all tasks• Samples tasks for random subsets of students• Creates scores at institution or subdivision/program level as
desired (depending on sample sizes)
Reporting • Controls for students’ ability so that “similarly situated”benchmark campuses can be compared
• Provides value added estimates—from freshman to senior year or with measures on a sample of freshmen and seniors
• Provides percentiles• Provides benchmark institutions
Written Communication
“Evaluate” An Argument*
A well-respected professional journal with a readership that includes elementary school principals recently published the results of a two-year study on childhood obesity. (Obese individuals are usually considered to be those who are 20 percent above their recommended weight for height and age.) This study sampled 50 schoolchildren, ages5-11, from Smith Elementary School. A fast food restaurant opened near the school just before the study began. After two years, students who remained in the sample group were more likely to be overweight–relative to the national average. Based on this study, the principal of Jones Elementary School decided to confront her school’s obesity problem by opposing any fast food restaurant openings near her school.
Performance Task
You are the assistant to Pat Williams, the president of DynaTech, a company that makes precision electronic instruments and navigational equipment. Sally Evans, a member of DynaTech's sales force, recommended that DynaTech buy a small private plane (a SwiftAir 235) that she and other members of the sales force could use to visit customers. Pat was about to approve the purchase when there was an accident involving a SwiftAir 235. You are provided with the following documentation:
1: Newspaper articles about the accident 2: Federal Accident Report on in-flight breakups in single engine planes 3: Pat's e-mail to you & Sally's e-mail to Pat 4: Charts on SwiftAir's performance characteristics 5: Amateur Pilot article comparing SwiftAir 235 to similar planes 6: Pictures and description of SwiftAir Models 180 and 235
Please prepare a memo that addresses several questions, including what data support or refute the claim that the type of wing on the SwiftAir 235 leads to more in-flight breakups, what other factors might have contributed to the accident and should be taken into account, and your overall recommendation about whether or not DynaTech should purchase the plane.
Value Added
Red= Seniors
Blue=Freshmen
Concluding Comments: 2006• Curiously, we may have come full circle with
the return of “written examinations” at the forefront of contemporary learning assessment reform rather than the “objective tests” pioneered by The Foundation in the last century
• What role in learning assessment might The Foundation take now?– With The Foundation’s role in policy and
improvement of teaching and learning, it might step in and fill the need for solid research to help us understand how assessment information from externally available sources (e.g., CAAP, MAAP, CLA) might be integrated into local campus efforts to gather assessment information to change and improve teaching and learning.
– This seems to be an under funded and under studied area with a long history contributed in the past by some of the researchers currently at The Foundation.
Lee Shulman
The Foundation’s Current President
