Date post: | 27-Dec-2015 |
Category: |
Documents |
Upload: | hillary-malone |
View: | 216 times |
Download: | 2 times |
1
Language, Learning, and Assessment: Improving
Validity
Eva L. Baker
UCLA Graduate School of Education & Information StudiesNational Center for Research on Evaluation,Standards, and Student Testing (CRESST)
26th Annual Language Testing Research ColloquiumOf the International Language Testing Association
Temecula, CA
March 26, 2004
2
Today’s Focus
Language and identity—a personal note
Quick assessment policy tour
Need for cognitive model-based assessment to link learning to external outcome measures
Examples
Research agenda involving language and assessment interplay
3
Assessment Policy Tour©
1965 F
an
tasy R
ecord
s
4
Assessment Policy Tour (Cont’d)
Accountability is pushing testing practice
Soon to have greater role in postsecondary
Assessment and testing high priority in K-12, language certification, and adult and professional learning
Students’ language background a major area
5
K-12 Accountability Requirements Goals (standards) to be linked to instruction,
classroom assessments
External examinations are to be lined up with goals and instruction
Few examples of alignment, unless goals are restricted
Lack procedures and measures to document level of practice
Early growth likely not sustainable
Transfer of learning a major issue
6
Description:Extra comfort for senior dogs. Our popular orthopedic pet bed, made extra thick for aging dogs. A full 4" of medical grade convoluted foam supports bones and joints, and the elevated headrest provides proper neck and spine alignment.
http://www.petdiscounters.com/dog/beds/cu_orthopedic.html
7
http://www.powerofyoga.com/
Joyful manifestation of the heart’s desire.
8
Cognitive Model-Based Assessment
Cross-curricular cognitive families define core constructs
Embed in content, like linguistic rules in AI language understanding systems
Derives from work in writing assessment and AI systems
Evidence that it works across subject matters, types of learning, and students
9
Intellectual Capital Cognitive Families
ContentUnderstanding
ProblemSolving
Teamwork andCollaboration
MetacognitionLearning to LearnCommunication
Learning
10
Cognitive Demands Examples
Factual knowledge Definitions, propositions (e.g., statements about events,
causes, relations; “math facts”; formulas), etc. Terms and symbols (beyond arithmetic)
Identify domain by letter: A for algebra, G for geometry, etc.
Conceptual knowledge Knowledge of “more complex, organized knowledge
forms,” including classification schemes, principles and generalizations, theories, models, and knowledge structures
Procedural knowledge Knowledge about how to do something, including
mathematical skills and algorithms, techniques and methods
Knowledge Categories
11
Cognitive Demands Examples (Cont.)
Recall information
Apply information
Interpret figures, charts, or tables
Transform representations Change or move from one representation to another
(e.g., language to diagram, text to equation, graph to equation)
Multi-step reasoning Requires “students to generate an intermediate image,
construct, or sub-problem before solving the original problem”
Process Categories
12
From Science to Models to Templates
DOMAIN-INDEPENDENT
COGNITIVEPRINCIPLES
CONTENT CONTENT CONTENT
CBA
TEMPLATE TEMPLATE TEMPLATE
MODEL
SCIENTIFICFINDINGS
COGNITIVE DEMANDS
SCIENTIFICFINDINGS
SUBJECT MATTERSPECIFIC MODELS
CONDITIONSTOPICS OF TASKSSPECIFIC SCORING FEATURES
13
From Templates to Tasks
CBA
TEMPLATE TEMPLATE TEMPLATE
TASK TASKTASK
TASKTASK
TASK
TASK
TASKTASK
TASK
TASK TASK
14
Templates for Model of Content Understanding
Primary source materials in each domain
Student required to integrate prior knowledge and principles to succeed
Scored by using expert model in subject matter
15
Content UnderstandingTemplate #1 Explanation
An array of primary source materials
A prompt that asks for an explanation in context
Constructed (written) answer
Evaluated by means of a scoring rubric
16
Excerpts from U.S. HistoryPrimary Source Documents
Judge Douglas made two points upon my recent speech at Springfield. Hesays they are to be the issues of this campaign. The first one of these pointshe bases upon the language in a speech which I delivered at Springfield, whichI believe I can quote correctly from memory. I said there that “we are now farinto the fifth year since a policy was instituted for the avowed object, and withthe confident promise, of putting an end to slavery agitation; under theoperation of that policy, that agitation had not only not ceased, but hadconstantly augmented.” “I believe it will not cease until a crisis shall havebeen reached and passed. ‘A house divided against itself cannot stand.’ Ibelieve this Government cannot endure permanently, half slave and half free.” “I do not expect the Union to be dissolved”—I am quoting from my speech—“I do not expect the house to fall, but I do expect it will cease to be divided.It will become all one thing or the other. Either the opponents of slavery willarrest the spread of it and place it where the public mind shall rest, in the belief that it is in the course of ultimate extinction, or its advocates will pushit forward until it shall become alike lawful in all the States, North as well asSouth.” [text continues]
ABRAHAM LINCOLN
17
U.S. History WritingAssignment: Civil War Era
Be sure to show the relationships among your ideas and facts.
Your essay should be based on two major sources:
1. The general concepts and specific facts you know about United States history, and especially what you know about the history of the Civil War.
2. What you have learned from the readings yesterday.
Imagine that it is 1858 and you are an educated citizen living in Illinois.Because you are interested in politics and always keep yourself well-informed, you make a special trip to hear Abraham Lincoln and Stephen A.Douglas debating during their campaigns for the Senate seat representingIllinois. After the debates you return home, where your cousin asks youabout some of the problems that are facing the nation at this time.
Write an essay in which you explain the most important ideas and issuesyour cousin should understand.
18
Excerpts from Hawaiian HistoryPrimary Source Documents
LILIUOKALANI
For many years our sovereigns had welcomed the advice of American residents who had established industries on the Islands. As they becamewealthy, their greed and their love of power increased. Although settledamong us, and drawing their wealth from resources, they were alien to usin their customs and ideas, and desired above all things to secure their own personal benefit.
Kalakaua valued the commercial and industrial prosperity of his kingdomhighly. He sought honestly to secure it for every class of people, alien ornative. Kalakaua’s highest desire was to be a true sovereign, the chiefservant of a happy, prosperous, and progressive people.
And now, without any provocation on the part of the king, having maturedtheir plans in secret, the men of foreign birth rose one day en masse, calleda public meeting, and forced the king to sign a constitution of their ownpreparation, a document which deprived [him] of all power and practically took away the franchise from the Hawaiian race.
19
Content Knowledge Prompt:Hawaiian History Writing
Assignment—BayonetConstitution
Be sure to show the relationships among your ideas and facts.
Your essay should be based on two major sources:
1. The general concepts and specific facts you know about Hawaiian history, and especially what you know about the period of the Bayonet Constitution.
2. What you have learned from the readings yesterday.
Imagine you are in a class that has been studying Hawaiian history. One ofyour friends, who is a new student in the class, has missed all the classes.Recently, your class began studying the Bayonet Constitution. Your friend isvery interested in this topic and asks you to explain everything that you havelearned about it.
Write an essay explaining the most important ideas you want your friend tounderstand. Include what you have already learned in class about Hawaiianhistory, and what you have learned from the texts you have just read. Whileyou write, think about what Thurston and Liliuokalani said about the BayonetConstitution, and what is shown in the other materials.
20
Content Knowledge Prompt (Cont’d)
*From Hawaii’s Story by Hawaii’s Queen, Liliuokalani (Boston: Lee and Shepard Publishers, 1898).
It may be asked, “Why did the king give them his signature?” I answerwithout hesitation, because he had discovered traitors among his mosttrusted friends and because the conspirators were ripe for revolution, andhad taken measures to have him assassinated if he refused.
It has been known ever since that day as “The Bayonet Constitution,” and the name is well-chosen; for the cruel treatment received by the king from the military companies. [text continues]
Explain to your friend who missed class the reasons and differences for the Queen and the Senator’s approach to Hawaii’s future.
Scoring Rubric •General impression (on task)•Principles and themes•Prior knowledge•Relevant concrete examples•Avoidance of misconceptions
21
History Template
22
Benefits
Architecture for design of classroom instruction, assessments, and external tests, not multipurpose test, but multipurpose architecture
Transfer of learning
Flexibility of teaching
Vertical coherence
Specific procedures
Evidence base
23
Measuring Transfer of Learning
Not teach x, measure y: but describe variations in conditions, task content, criteria that permit students to demonstrate their flexibility
Learning to learn measurement
Metacognition and self-regulation
Time to learn new domains or tasks
Safeguard against teaching to the test as a preferred intervention
24
How Language Testing Influences Achievement Testing
Rubrics and procedures for assessing open ended performance
Linguistic analysis as accommodations
Linguistic descriptors as key task dimension
Arguments about language, content, and context
25
Toward Cognitively Oriented Constructs for English Language
Development Exams (K-12) Ironies
Expedience and science
Social and academic constructs
Measure of beginning skill or transfer
Relationships to other external exams not clear
Characterizing text Readability
Lexiles
LEARNOME
26
Towards Common Language Across Domains
Genome: “Complete set of instructions”
LEARNOME: “Complete map of features in learning domain”
27
Preliminary Exploration of the LEARNOME
Fundamental and generalizable research on identification of components
Cognitive demands, e.g., problem solving
Linguistic requirements, i.e., discourse, syntax, lexicon
Task, e.g., context, constraints
Content (ontologies)
Descriptive language—granularity
Modeling and representation of results
Empirical verification
28
Research Interactions: Language Testing and Assessment
Model-based R&D
EFF Adult Literacy
Questions about differential validity and fairness
29
EFF A
ssessm
en
t Con
sortiu
m: S
RI In
tern
atio
nal &
Cen
ter fo
r Litera
cy
Stu
die
s. (20
04
, Jan
uary
). EFF sta
nd
ard
s an
d p
erfo
rman
ce le
vel
descrip
tors fo
r: Read
ing
, writin
g, m
ath
, speakin
g, a
nd
Listen
ing
. Wash
ing
ton
, DC
: Natio
nal In
stitute
for Lite
racy
.
30
EFF A
ssessm
en
t Con
sortiu
m: S
RI In
tern
atio
nal &
Cen
ter fo
r Litera
cy
Stu
die
s. (20
04
, Jan
uary
). EFF sta
nd
ard
s an
d p
erfo
rman
ce le
vel
descrip
tors fo
r: Read
ing
, writin
g, m
ath
, speakin
g, a
nd
Listen
ing
. Wash
ing
ton
, DC
: Natio
nal In
stitute
for Lite
racy
.
31
EFF Assessment Consortium: SRI International & Center for LiteracyStudies. (2004, January). EFF standards and performance levelDescriptors for: Reading, writing, math, speaking, andListening. Washington, DC: National Institute for Literacy.
32
EFF A
ssessm
en
t Con
sortiu
m: S
RI In
tern
atio
nal &
Cen
ter fo
r Litera
cy
Stu
die
s. (20
04
, Jan
uary
). EFF sta
nd
ard
s an
d p
erfo
rman
ce le
vel
descrip
tors fo
r: Read
ing
, writin
g, m
ath
, speakin
g, a
nd
Listen
ing
. Wash
ing
ton
, DC
: Natio
nal In
stitute
for Lite
racy
.
33
EFF A
ssessm
en
t Con
sortiu
m: S
RI In
tern
atio
nal &
Cen
ter fo
r Litera
cy
Stu
die
s. (20
04
, Jan
uary
). EFF sta
nd
ard
s an
d p
erfo
rman
ce le
vel
descrip
tors fo
r: Read
ing
, writin
g, m
ath
, speakin
g, a
nd
Listen
ing
. Wash
ing
ton
, DC
: Natio
nal In
stitute
for Lite
racy
.
34
EFF A
ssessm
en
t Con
sortiu
m: S
RI In
tern
atio
nal &
Cen
ter fo
r Litera
cy
Stu
die
s. (20
04
, Jan
uary
). EFF sta
nd
ard
s an
d p
erfo
rman
ce le
vel
descrip
tors fo
r: Read
ing
, writin
g, m
ath
, speakin
g, a
nd
Listen
ing
. Wash
ing
ton
, DC
: Natio
nal In
stitute
for Lite
racy
.
35
EFF A
ssessm
en
t Con
sortiu
m: S
RI In
tern
atio
nal &
Cen
ter fo
r Litera
cy
Stu
die
s. (20
04
, Jan
uary
). EFF sta
nd
ard
s an
d p
erfo
rman
ce le
vel
descrip
tors fo
r: Read
ing
, writin
g, m
ath
, speakin
g, a
nd
Listen
ing
. Wash
ing
ton
, DC
: Natio
nal In
stitute
for Lite
racy
.
36
EFF A
ssessm
en
t Con
sortiu
m: S
RI In
tern
atio
nal &
Cen
ter fo
r Litera
cy
Stu
die
s. (20
04
, Jan
uary
). EFF sta
nd
ard
s an
d p
erfo
rman
ce le
vel
descrip
tors fo
r: Read
ing
, writin
g, m
ath
, speakin
g, a
nd
Listen
ing
. Wash
ing
ton
, DC
: Natio
nal In
stitute
for Lite
racy
.
37
EFF A
ssessm
en
t Con
sortiu
m: S
RI In
tern
atio
nal &
Cen
ter fo
r Litera
cy
Stu
die
s. (20
04
, Jan
uary
). EFF sta
nd
ard
s an
d p
erfo
rman
ce le
vel
descrip
tors fo
r: Read
ing
, writin
g, m
ath
, speakin
g, a
nd
Listen
ing
. Wash
ing
ton
, DC
: Natio
nal In
stitute
for Lite
racy
.
38
EFF A
ssessm
en
t Con
sortiu
m: S
RI In
tern
atio
nal &
Cen
ter fo
r Litera
cy
Stu
die
s. (20
04
, Jan
uary
). EFF sta
nd
ard
s an
d p
erfo
rman
ce le
vel
descrip
tors fo
r: Read
ing
, writin
g, m
ath
, speakin
g, a
nd
Listen
ing
. Wash
ing
ton
, DC
: Natio
nal In
stitute
for Lite
racy
.
39
EFF A
ssessm
en
t Con
sortiu
m: S
RI In
tern
atio
nal &
Cen
ter fo
r Litera
cy
Stu
die
s. (20
04
, Jan
uary
). EFF sta
nd
ard
s an
d p
erfo
rman
ce le
vel
descrip
tors fo
r: Read
ing
, writin
g, m
ath
, speakin
g, a
nd
Listen
ing
. Wash
ing
ton
, DC
: Natio
nal In
stitute
for Lite
racy
.
40
Research Interactions: Language Testing and Assessment (Cont’d)
Technology
TBALL project—NSF education, linguistics, electrical engineering—automated testing of ELL children
Essay scoring—towards propositional rather than regression solutions
Authoring systems with smarts
Simulation architecture and narrative structures
41
42
Research Interactions: Language Testing and Assessment (Cont’d)
Validity inferences for different groups
Comparability among tasks, paths
Alternative to norms for comparisons among groups
43
Priorities
Linking context, purposes, and evidence to an appropriate grain-sized framework
Document growth patterns associated with different instructional paths
Explore validity interpretations for ELL groups
44
CRESST Web Site
http://www.cresst.org