John M. Norris University of Hawai´i at Mānoa Using assessment For understanding and improving...

John M. Norris

University of Hawai´i at Mānoa

Using assessment

For understanding and improving

Language education

“Ich lerne sehen”

~R. M. Rilke

Dealing with change

In language education

Rethinking the value of language education

“English language learners (ELLs) are the fastest growing student population in America. Today, one out of every nine students is learning English as a second language…By 2025, English language learners will make up one out of every four students in our classrooms.”

Margaret Spellings (2005) – U.S. Secretary of Education


“Americans need to be open to the world; we need to be able to see the world through the eyes of others if we are going to understand how to resolve the complex problems we face.”

Daniel Akaka, U.S. Senator from HawaiiLanguage Crisis in

the U.S.


“The stakes are very high. The challenges college graduates face over the next 50, 60, and 70 years will intensify. They involve the understanding of different cultures, the balance of global power, the depletion of environmental resources, and the ability to continue to grow in a rapidly changing world. It is because the stakes are so high that I believe a focus on student learning is so critical and why FL programs have such an important role to play.” (Chase, 2006, p. 585).

Geoffrey Chase – Dean of Undergraduate Studies

San Diego State University


AAAL, ACTFL, MLA, Northeast Conference

Modeling Representation of Foreign Language Education at the Federal Level in the United

States

MLA White Paper

Foreign Languages and Higher Education: New Structures for a Changed World

Standards

Standards for foreign language learning in the 21st century

High-quality teachers

NCATE – TESOL/ACTFL Teacher Development Program Standards


LanguageEducation

How do we bridge the content vs. language divide?

Are we seeking communicative

competence or quality of mind?

How do we ensure advanced language

learning?

What’s the relationship

between L1, L2, and academic

achievement?

Should we adopt an

instrumental, aesthetic, ethical, or

moral justification for

FL learning?

Who trains teachers how?

Determining the value of language education

“What we assess is what we value”

--Lauren Resnick“How we choose to assess will determine what gets valued”

--Norris

Why worry about assessment?

Top 3 sources of pressure for assessment in college FL

programs:

1.University administration

2.The dean

3.Accreditation process

“As part of its re-accreditation, the

university has required all undergraduate

programs to create and implement outcomes-oriented assessment

plans.”

(survey respondent)

“Time-consuming. Takes away from the business of teaching. Many aspects of learning can’t be measured.”

(survey respondent)


NOCHILDLEFTBEHIND

COLLEGE STUDENTCollege Leaving Exam

“Does the Spellings Commission think about language education

at all?”

Michael Holquist (ADFL 2007 Summer Seminar West)


“thoughtless mastery”

ASSESSMENT

“bureaucratic nonsense”

“fails to improve teaching”

“duplicates existing efforts”

“reductionist”

“discourages teaching skills that are difficult to measure”

“Orwellian, punitive process”

“utilitarian technocracy”

Frequent teacher reactions to assessment:


capacity for change

faculties of problem-solving

FLs need to develo

p

Domna Stanton (2007, ADFL Summer Seminar West)

Perception of

assessment

managerial model of efficiency

not appropriate to academe

BUT…


Despite such problems, college FL survey respondents desired increased use of assessments and related processes for:

1. Understanding & improving program outcomes

2. Understanding & improving program functions

3. Improving FL education on the whole

4. Understanding & Improving the worth of the program

5. Raising awareness about FL programs


And some expressed a professional ethic to engage in outcomes assessment:

“We have a social and moral responsibility towards our students and towards society at large to state as clearly as we can what it is that we do for them and

why what we do is valuable.”

(Anonymous survey respondent)


TESOL•Language•Culture•Planning, managing•Assessment•Professionalism

ACTFL•Language, linguistics•Cultures, literatures•Acquisition, instruction•Integration of standards•Assessment•Professionalism

NCATE Teacher Preparation Program Standards

Assessment as change agent

Value of

LanguageEducation

KeyChallenge:

DefineDelimitDetractDismiss

Key Opportunity:

EnableEnhanceEngenderEmpower

Within this milieu, what is the role to be played by assessment?

Traditions, trends, and the status quo

In language education assessment

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

ADFL FLA LT MLJ UP

Ass

essm

ent

arti

cles

%Assess

%FL

%USCFL

%Proficiency

Percentage and type of FL assessment/evaluation articles in five journals, 1984-2002

0%

5%

10%

15%

20%

25%

30%

35%

ADFL 84-93 ADFL 94-02 FLA 84-93 FLA 94-02 LT 84-93 LT 94-02 MLJ 84-93 MLJ 94-02 UP 84-93 UP 94-02

US co

llege

FL

asse

ssm

ent

High, low, and average yearly percentage of articles on U.S. college FL assessment/evaluation, 1984-1993 and 1994-2002

Swender (2002), on FL teacher professional development in assessment:

“After all, if teachers do not know how to measure what students can do with language, how will they be able to determine whether their students are measuring up to the expectations of the 21st century”.

Received traditions of language assessment

Technocratic measurement problem

WebCAPE Foreign Language Placement Exam


Commercial assessment problem:

One size fits most purposes & settings

TSE – SPEAK – SLEP - TOEIC


Commercial assessment problem:

One size fits most purposes & settings

http://www.ets.org/portal/site/ets/menuitem.fab2360b1645a1de9b3a0779f1751509/?vgnextoid=69c0197a484f4010VgnVCM10000022f95190RCRD

http://www.ets.org/portal/site/ets/menuitem.fab2360b1645a1de9b3a0779f1751509/?vgnextoid=69c0197a484f4010VgnVCM10000022f95190RCRD


When we do address

assessment, in…

• FL teaching texts

• State of the art collections

• Professional organizations

• NCATE/ACTFL teaching standards

• Accountability requirements

We focus on…

• How to measure the four skills

• How to rate oral proficiency

• How to test cultural knowledge

• How to assign grades

• How to give feedback

Without much concern for…

• WHY are we assessing?

• What assessment methods fulfill what purposes?

• How do we contextualize it?

• What good is it doing?

Advanced vocabulary learning: individual vocabulary notebooks

Learners identify meaningful

words to studyBased on

individual needsor interests

Record 30/week in vocabulary

notebookRevisit, review, and acquire 30over the week

Instructional Intent

From: Moir & Nation (2002)

Advanced vocabulary learning: individual vocabulary notebooks

Learners identify meaningful

words to studyBased on

individual needsor interests

Record 30/week in vocabulary

notebookRevisit, review, and acquire 30over the week

Instructional Intent

Learners identifyunsuitable words

to studyCram the

night beforethe weekly test

Pass the recalltest and promptlyforget the words

Assessment Reality

The ‘assessment mindset’ in teacher practice:

“Although this student still has problems with grammar, the ideas are there. He is working through the choices the community has about their need for a better water supply system. Hmm, this is difficult. I just wish his grammar errors weren’t so bad then I could give him an ‘A’.”

Mohan example, TBLT 2007 Conference


What is the appropriate proficiency level to adopt as a student learning outcome for the 2-year language requirement?


INTERMEDIATE -

LOW ? ? ?

Outcomes embody the essential purpose of an educational program: developments in knowledge, skills, dispositions of learners

Requires rethinking of educational programs as something more than the delivery of experiences or the exposure of learners to information

Calls for articulation of curriculum and instruction in support of targeted outcomes, demands integrated thinking

Provides a clear statement of educational program value; answers the question “How do you know?” with evidence of educational effectiveness

Current trends in assessment

Student Learning Outcomes

Assessment


Process

•State outcomes

•Identify indicators

•Measure them

•Analyze the results

•Then what?

•(Let the chair/dean do it)

Student Learning Outcomes

Assessment

Large public institution Accreditation

pressures to assess learning

How about an electronic portfolio?

Huge expenditure, $$$, time, effort Thousands of student

portfolios created

BUT…

We have to assess our “liberal studies” core,

ASAP!


1. *!%#$!@*

…faculty didn’t understand it…students thought:

2. waste of time

Electronic Portfolio

…administrators wanted to do something with it but weren’t sure exactly what

NEVER GOT USED


Barrington (2003), on assessment in the liberal arts:

“To design and administer (intellectually honest) assessment plans that will measure such capabilities with a dozen or more standardized ‘learning objectives’ is next to impossible” leading to “pestilent repercussions” for the truly valued learning objectives that constitute the liberal arts, in that it “discourages teaching such skills because they are difficult to measure”.


Perception problem

Traditions, trends, and the status quo

1. Focus on doing…

2. Based on commercial testing, proficiency mvt.

3. Reactive v. proactive praxis

4. Driven by external impetuses

5. Technocratic measurement emphasis

6. Little scholarly investment

Traditions and trends1. Not useful—not used!

2. Potential negative washback, reductionism, waste

3. Not relevant to curriculum & instruction, program values

4. Not ours—mandated or purchased off the shelf

5. Not perceived as worth the effort by faculty

6. Minimal professional development

Status quo

in language education

Re-envisioning assessment

Resolving terminological confusion

Measurement is the consistent elicitation of quantifiable indicators of well-defined constructs via tests or related observation procedures; it emphasizes efficiency, objectivity, and technical aspects of construct validity.

Norris (2006) MLJ Perspectives


Assessment is the systematic gathering of information about student learning in support of teaching and learning…It may be direct or indirect, objective or subjective, formal or informal, standardized or idiosyncratic…It provides locally useful information on learners and learning to those individuals responsible for doing something about it.



Evaluation is the gathering of information about any of the variety of elements that constitute educational programs, for a variety of purposes that include primarily understanding, demonstrating, improving, and judging program value; evaluation brings evidence to bear on the problems of programs, but the nature of that evidence is not restricted to one particular methodology.


The nature of useful evaluations

Evaluation gets used when…

(a) intended users of evaluation participate;

(b) evaluation is pursued as a process, not an end-game;

(c) sufficient time and resources are allocated;

(d) evaluation produces interesting, credible, relevant findings;

(e) findings are reported in a timely fashion;

(f) Interpretations and recommendations are contextualized.

Evaluative assessment

Light (2001) on outcomes assessment:

“…a process of evaluating and improving current programs, encouraging innovations, and then evaluating each innovation’s effectiveness. The key step is systematic gathering of information for sustained improvement. And always with an eye toward helping faculty or students work more effectively.”

Richard Light (2001, p. 224)

PURPOSES

AccountabilityRevising

curriculum

ProgramDevelopment

Articulation

Diagnosis

Improving teaching

Raising Awareness

Motivating Learners

Certification

Justifying$ requests

Improvinglearning

Illumination

An evaluative approach to assessment

METHODS

TestsLanguageProfiles

Quizzes

Selfassessment

Observations

PeerAssessment

Interviews

Performanceassessment

Standardizedassessment

Journals

Meetings

Portfolios


Surveys

Focus groups

Approaching assessment: •The most important consideration is supposed to be the use for which it is intended (cf. Bachman & Palmer, 1996; Brown, 2005)

•BUT, in test development, what is the first question that gets asked? What are you going to measure?

“Decide on purpose of assessment:•What abilities are you assessing?

--What is your construct or model of these abilities?”

Coombe, Folse, Hubley (2007, p. 4))




PrimaryIntended Users

AssessmentAdvisors

Stakeholders &Audiences

Negotiate & specify:

• priority uses

• methods

• analyses

•reporting

• timelinesENABLING USE

EMPOWERING USERS


Products:

(a)public documents on the exact roles to be played by assessments in the FL program and the different forms that

those assessments take

(b) program policies on assessment practice at the individual, classroom, and program levels

(c) assessment methods that lead to local actions

(d) evaluative/evidentiary justification for assessments and their uses

(e) Identification of other factors in need of evaluation

Why bother?

Align assessment with curriculum, instruction, program values: foster ownership Raise awareness & buy-in

among students, faculty, others about roles for assessment and evaluation Increase the likelihood

that data will be used

Decrease frequency and number of (useless) assessments and evaluations

Support improvements in student learning

Why bother?

Received view of assessment:

What do we test? How do we test? What tests are available? Evaluative Vision:

What questions do we have about our learners, courses, and programs? How do we gather data appropriate to answering those questions? In what ways can we utilize those data to resolve the challenges we face?

What does it look like?

Using assessment to understand and improve language programs

Example: Needs assessment for curricular revision

Problem

•Perception of curricular ‘antiquation’

•Relevance to learners’ wants and needs?

•Shifting standards for college FL learning

•New expectation to pursue performance outcomes

•How to evolve? What values to prioritize?

Context

•Japanese at University of Hawaii

•2-yr. requirement (mostly non-majors)

•@1000 students per year

•@50 instructors

•Diversity of students, diverse uses of Japanese

Role of assessment

•Identify learners’ perceived L2 needs

•Identify teachers’ perceptions of same

•Identify gaps between the two

•Prioritize outcomes expectations for 2-yr. program

•Provide basis for revision of materials


Instruments and ProceduresQuestionnaire

s

L2 Use Domains

Social

Occupational Tourism

Likert scale ratings of priorities

Open-ended questions+

Teachers (N = 46)

Students (N = 688)

Academic


Identified Learner Needs

Teachers + Learners Prioritize

Learners Prioritize,

Teachers do notIn Hawaii…

•Academic routines

•Socializing

•Hosting guests

•Working retail

•Interacting w/tourists

•Using computers

In Japan…

•Academic routines

•Socializing

•Surviving as a tourist

•Participating in complex discussions

In Japan…

•Medical interactions

•Residing long-term (housing, etc.)

•Working abroad

•Formal ceremonies


Other Curricular Implications

•Instruction moves too rapidly

•Overemphasis on Kanji memorization

•Not enough attention to culture

•Not enough focus on situation-specific speaking tasks, performance practice


Using Assessment for Curricular Change

•Basis for review of texts, materials & pedagogy development

•Revised 2-yr expectations emphasize speaking, culture, performance

•Attention to Japanese use in Hawaii & long-term residence abroad (both authentic to learners)

•Development of performance-based outcomes assessments

•Revised balance/pace of instruction

Related Changes

•Dean commits to Department-level Assessment Specialist

•Evaluation utilized as campus-wide example of ‘closing the loop’ back to curriculum

•Increased enrollments, improved course evals

•Principal Investigator tenured…

See Iwai et al (1999)

Example: Realizing the curriculum via assessment

Problem

•New curriculum, new instruction, old assessment

•Accuracy of curricular learning trajectories?

•Students meeting expected outcomes?

•Feasible expectations for intensive and non?

•Relevance of external assessments?

Context

•German at Georgetown Univ.

•New curriculum: Developing Multiple Literacies

•Fully integrated Language & Content instruction

•Task- & genre-based

•Advanced L2 literacy target

Role of assessment

•Operationalize curricular expectations

•Provide evidence regarding learner development & outcomes

•Enable between-level, and intensive/non comparisons

•Fit seamlessly into teaching and learning processes!!!


Level performance profiles

Task assignment sheets

Sample student performances

TEACHERS

Development

Analysis

LEVEL

GROUPS

Deliberation

Revision

Prototypical performancewriting tasks


Task

Content

Language

Assignment 1Assignment 2

Semester…

Assignment 3Assignment 4

PrototypicalPerformance

WritingTask

Explicit performance

criteria

Curricular level expectations

Consistent assignment framework


“Assessment in this kind of a context is, I would almost say probably an indispensable

aspect in order to clarify any number of things. Because it is in the discourse about assessment and how we would do that that our knowledge became articulated or the

holes in that knowledge became clearer to ourselves. Or the cover-ups that we had engaged in were no longer possible if we wanted to be honest with ourselves about

it.”


Using Assessment for Curricular Change

•Forced the curriculum to become real

•Close specification of L2 progress within/across curricular levels

•Disambiguation of learning outcomes in terms of task, content, language

•Curricular ‘map’ for use by teachers and learners (what happens when?)

•Forged agreement between levels on what can and cannot be expected

See Byrnes (2002)

Example: Assessing and improving effectiveness

Problem

•Does it help?

•Does it get used?

•Do students and teachers understand how it is supposed to be used?

•Are conditions appropriate for benefiting from it?

•Should it be maintained?

Context

•LAUSD

•Kindergarten/First, ELLs with varying proficiencies (41%)

•Wide-spread reading ‘failure’

•Supplemental software introduced to enhance reading instruction

Role of assessment

•Evidence of reading development

•Enable comparisons between ages, proficiencies, treatment/control

•Capture the context of implementation

•Provide thorough interpretation of processes + outcomes


Instruments and Procedures

Standardized, mandated reading assessments

Extensive classroom observations

Teacher interviews

200 classrooms

(100 treatment, 100 control)

Enable estimation of added effect of new

software

Provide for meaningful interpretation of findings

in terms of classroom realities

Indicate English reading abilities among children


Findings

•No differences in reading ability between treatment and control classes or proficiency groupings

•BUT, Low implementation in classes and by individuals

•Lots of reasons: Lack of time, competition with standard curriculum, computer malfunction, lack of understanding by learners and teachers, lack of ‘on-task’ engagement

Uses

•Provide for professional development in software

•Articulate/align with standard curriculum

•Individualize for students with greatest need

•Shift school schedule to allow for implementation to occur

•Retain the software but continue assessing effectiveness with studentsSee Llosa & Slayton

(forthcoming)

Facing change:Learning to see assessment as a useful process

•There is more to useful assessment than just good measurement; measurement informs construct interpretations…assessment enables informed change

•Contextualization in classes, curricula, institutions is essential for assessment to be interpretable and meaningful

•Intentionality for specific purposes, by specific users, with specific consequences is essential for assessment to be used

•These processes should be built in from the beginning of assessment design; specification of intended use is one heuristic for doing so

•Educational assessment is only useful insofar as it does good; hold assessments accountable to that principle

A few resources…

University of Hawai‘i at Mānoa, Second Language Studies

http://www.nflrc.hawaii.edu/evaluation

http://www.nflrc.hawaii.edu/evaluation

Thank you!

Date post:	24-Dec-2015
Category:	Documents
Upload:	coleen-roberts
View:	216 times
Download:	0 times

John M. Norris University of Hawai´i at Mānoa Using assessment For understanding and improving...

Documents