EPSY 625LECTURE 3

COGNITIVE ASSESSMENT

COGNITIVEPROCESSORMEMORY

BEHAVIOR/RESPONSE

AURAL/VIS-UAL SYSTEMS

TASKDEMANDS

SPATIAL

REASONING

VERBAL: Written Spoken

VISUAL/SPATIAL

COGNITIVE SYSTEM FOR ASSESSMENT

MOTIVATION

WORKINGMEMORYCAPACITY

LONG-TERMMEMORY

MEMORY

AFFECT

TASK DEMANDS:STRUCTURING COGNITIVE TESTS

TYPES

• ARTIFICIAL

• ANALOG

• ACTUAL TESTS:

• 1. ACHIEVEMENT

• 2. INTELLIGENCE

TASK TYPES

ARTIFICIAL- intended to assess response to novel conditions not encountered before

ANALOG- intended to assess response to conditions not ethically or economically establishable

ACTUAL- intended to assess response in “real” setting

I. ACHIEVEMENT TESTS

A. Curriculum Guides/ Mandates: TEXAS ESSENTIAL KNOWLEDGE

AND SKILLS (TEKS) TAKS B. Textbook Surveys C. Political/Philosophical Selection (e.g..

Hirsch, “CULTURAL LITERACY) D. Myth/Nostalgic/History

“all _______ should know this

ACHIEVEMENT TESTS

D. Myth/Nostalgic/History“all _______ should know this”

E. Issue: content/pedagogical validity F. Content selection

• Random/Ordered

• Importance: how determined?

• Taxonomies

TAXONOMIES•a) Bloom et al.

•Evaluation

•Synthesis

• Analysis

• Application

• Comprehension

•Knowledge

b) Table of specifications

Topics TaxonomyK C Ap An

A 6 9 9 630%B 10 15 15 1050%C 4 6 6 420%

20% 30% 30% 20% 100%

SAMPLING OF BEHAVIORS

All tests sample a Universe, defined by a combination of all possible tasks, occasions, raters, and measurement methods

Domain refers to a content area to which the tasks refer

Sampling Variability of Performance AssessmentsRichard J. Shavelson, Gail P. Baxter, and Xiaohong Gao

Journal of Educational Measurement, Fall 1993, Vol. 30, No. 3, pp. 215-232

COGNITIVE RESEARCH1. Memory effects - STM/LTM or

Level: working-long term2. Processing effects

- spatial - analogical - reasoning - integrative/simultaneous

COGNITIVE RESEARCH

3. hot cognition/affect/ motivation

- Paivio’s “dual coding” theory4. Task structure

- VISUAL/SPATIAL

- SEMANTIC/VERBAL

- PROCEDURAL/ORDER

COGNITIVE RESEARCH

5. Knowledge structure

- Declarative (what)

- Procedural (how-strategy)

- Conditional (when-strategy)

CONTENT SELECTIONIMPLICATIONS

Situational nature of performance

Complexity in developmentLimitations in generalizability

II. INTELLIGENCE TESTS

A. THEORY BASIS:

1. “g” Construct- single factor

2. Limited # (2 or 3 factors)

3. Multiple intelligences

4. Limited scope for assessment (school)

1. “g” FACTOR

BINET-TERMAN: children’s mental functioning: STANFORD-BINET IQ

British psychology: SPEARMAN: factor analysis RAVEN: Progressive Matrices

2. Limited # of factors WECHSLER and adults: Verbal and non-

verbal IQ:• WAIS

• WAIS-R

Developmental downward extension:• WISC

• WISC-R

• WISC-III

2. Limited # of factors

KAUFMAN & KAUFMAN: KAB-C:• children’s IQ

• simultaneous & sequential (from Luria’s cognitive theory)

3. MULTIPLE INTELLIGENCES

MERCER’S SOMPA (System of Multicultural Pluralistic Assessment)

GARDNER’s Multiple Intelligences STERNBERG’s Analogical Reasoning

Theory Subtest use of SB, WISC-III, and KAB-C

4. LIMITED SCOPE

School intelligence (Publisher developed):• Differential Aptitude Tests

• Scholastic Achievement Tests: SAT, GRE, GMAT

• COGAT (Cognitive Abilities Test), etc.

B. SAMPLING OF PROCESSES

SPECIFIC ABILITIES/PROCESSES:• ANALOGICAL REASONING

• SPATIAL ABILITY

• MEMORY

• NONVERBAL: BLOCK DESIGN

TYPES OF RESPONSE

I. SUPPLY II. SELECT

I. SUPPLY RESPONSES

A. WRITTEN (ESSAY, SHORT ANSWER)

B. ORAL C. DRAWING/SKETCHING D. COMPUTATION E. PERFORMANCE

A. WRITTEN- ESSAY

Extended response allows greater sampling of knowledge domain

Dependent on writing (computer vs. pen) speed, legibility, strategic knowledge

Greater time to score- need for rubric, fatigue in scoring

Need to constrain topic, task

A. WRITTEN- SHORT ANSWER

Restrict topics:• Definitions or concepts

• Quick computations

Limit response length Establish protocol for scoring Establish scoring system

B. ORAL RESPONSE

Historical precedence (Greek, Roman, European Middle Ages, University system)

Performance aspect: knowledge and personal interaction

Typically faster response required- “thinking on one’s feet”

C. SKETCHING/DRAWING

Less commonly required Often task-specific (e.g.. Knowledge

maps) Incorporated into broader assessments

or tasks (e.g.. Part of physics or math problem)

D. COMPUTATION Most common to mathematics and

science fields Mental or written requirement Use of calculators or computers Verbal component may be important

(word problems) Spatial component may be important

(imageability)

PERFORMANCE

COMPLEX CONSTELLATION OF ACTIVITIES

SIMULATION REAL SITUATION ISSUES

PERFORMANCE ISSUES

COST TIME TO SET UP TIME TO SCORE- RELIABILITY # OF TASKS SAMPLED AUTHENTICITY VS. VALIDITY

II. SELECTION RESPONSE

MULTIPLE CHOICE (INCLUDING T-F) MATCHING

MULTIPLE CHOICE

EFFICIENT SAMPLING- Time, cost RELIABILITY- produces reliable

measures VALIDITY ISSUE: Does selection

represent same knowledge as supply?• Limitations overstated by critics

• current development does not take advantage of information available, new cognitive theory

MULTIPLE CHOICE

Livingston, Reynolds & Willson (2005) list for good item writing

# options- depends on good alternatives Options generated from incorrect

cognitive processes, become clues to knowledge structure (e.g.. BUGGY arithmetic program)

MATCHING

VARIANT ON MULTIPLE CHOICE LIMIT # OF MATCHES < 10 REQUIRE SINGLE CONCEPT REQUIRE MORE OPTIONS THAN

QUESTIONS