Welcome to the UA Assessment Showcase 2012! Sponsored by the Office of the Provost, the Assessment...

Welcome to the UA Assessment Showcase 2012!

Sponsored by the Office of the Provost, the Assessment Coordinating Council (ACC), and the Office of Instruction and Assessment

Actionable AssessmentActionable Assessmentof Academic Programs:of Academic Programs:Principles and PracticesPrinciples and Practices

for Usable Resultsfor Usable Results

Jo BeldJo BeldProfessor of Political ScienceProfessor of Political ScienceDirector of Evaluation & AssessmentDirector of Evaluation & Assessment

Assessment ShowcaseAssessment ShowcaseApril 17, 2012April 17, 2012

AgendaAgenda

• Principles:Principles: Conceptual frameworks Conceptual frameworks

• Practices: Practices: Making assessment Making assessment usefuluseful

• Practices: Practices: Engaging facultyEngaging faculty

• Politics and policyPolitics and policy: The bigger : The bigger picturepicture

Conceptual frameworksConceptual frameworks

Utilization-focused assessmentUtilization-focused assessment

(Patton, 2008):(Patton, 2008):

Focus on Focus on intended usesintended uses

by by intended usersintended users


Backward designBackward design

(Wiggins & McTighe, 2005):(Wiggins & McTighe, 2005):

““Beginning with the end in mind”Beginning with the end in mind”


Traditional assessment design:Traditional assessment design:

Choose an assessment instrumentChoose an assessment instrument

Gather and summarize evidenceGather and summarize evidence

Send a report to someoneSend a report to someone


Backward assessment design:Backward assessment design:

Identify intended users and usesIdentify intended users and uses

Define and locate the learningDefine and locate the learning

Choose assessment Choose assessment approachapproach


Once you’ve Once you’ve defined your defined your outcomes, start outcomes, start planning your planning your assessment assessment project hereproject here

Making assessment usefulMaking assessment useful

Studio art majorStudio art major• Developed evaluation form for Developed evaluation form for

senior exhibit that doubles as senior exhibit that doubles as assessment instrumentassessment instrument

• Addressed disconnect between Addressed disconnect between student and faculty criteria for student and faculty criteria for artistic excellenceartistic excellence

• Revised requirements for the Revised requirements for the majormajor

• Refocused common Refocused common foundation-level coursesfoundation-level courses


Chemistry majorChemistry major• Using ACS exam as final in Using ACS exam as final in

Chem 371: Physical ChemChem 371: Physical Chem• Students outperform national Students outperform national

average and do well in average and do well in kinetics despite limited kinetics despite limited coverage in coursecoverage in course

• Chem 371 being retooled to Chem 371 being retooled to focus on thermodynamics focus on thermodynamics and quantum mechanicsand quantum mechanics


History majorHistory major % “exemplary” ability to…% “exemplary” ability to…

• Gathering evidence in 2011-12 (voluntarily!) to Gathering evidence in 2011-12 (voluntarily!) to examine sequencing in the majorexamine sequencing in the major

• Examining ability to understand and work with Examining ability to understand and work with historiography in new intermediate seminars for historiography in new intermediate seminars for majormajor

First-Year seminars

Senior seminars

Differentiate primary from secondary source

45% 91%

Use primary source to develop argument

19% 60%

Making assessment usefulMaking assessment usefulStatistics concentrationStatistics concentration

• Collaboratively designed final exam question and grading Collaboratively designed final exam question and grading rubric in Stats 270 to examine interpretation and rubric in Stats 270 to examine interpretation and communication of results; two faculty graded essayscommunication of results; two faculty graded essays

• Instructor adjusted teaching in response to findingsInstructor adjusted teaching in response to findings


Management Studies concentrationManagement Studies concentration

• Quiz scores:Quiz scores: Teams outperform best individual students Teams outperform best individual students

• Course evaluations: Course evaluations: Students believe they learned “much” Students believe they learned “much” or “exceptional amount” by working together in teams (73%)or “exceptional amount” by working together in teams (73%)

• Team-based learning being extended to other coursesTeam-based learning being extended to other courses

Mean Results – Management Studies 251 Course Quizzes

Highest individual score Team quiz score

Section A 4.36 4.79

Section B 4.39 4.74


Interdisciplinary programsInterdisciplinary programs

• Collaboratively developed Collaboratively developed assessment questionnaireassessment questionnaire

• Considering direct assessment Considering direct assessment of interdisciplinary proficiency of interdisciplinary proficiency using common rubric with using common rubric with program-level portfolio program-level portfolio

• Will consider whether all Will consider whether all programs should have programs should have capstone course or experiencecapstone course or experience


Benefits for individual courses:Benefits for individual courses:

•Setting priorities for content/instructionSetting priorities for content/instruction

•Revising/expanding assignmentsRevising/expanding assignments

•Clarifying expectations for studentsClarifying expectations for students

•Enhancing “scaffolding”Enhancing “scaffolding”

•Piloting or testing innovationsPiloting or testing innovations

•Affirming current practicesAffirming current practices


Benefits for the program as a whole:Benefits for the program as a whole:

•Strengthening program coherence Strengthening program coherence

•Sending consistent messages to studentsSending consistent messages to students

•Revising program requirementsRevising program requirements

•Extending productive pedagogiesExtending productive pedagogies

•Affirming current practicesAffirming current practices

Making assessment useful Making assessment useful More program benefitsMore program benefits::

•Telling the program’s story to graduate Telling the program’s story to graduate schools and employersschools and employers

•Enhancing visibility to disciplinary and inter-Enhancing visibility to disciplinary and inter-disciplinary associationsdisciplinary associations

•Supporting grant applicationsSupporting grant applications

•Meeting requirements for specialized Meeting requirements for specialized accreditationaccreditation

Making assessment useful Making assessment useful

Benefits for faculty members:Benefits for faculty members:

•Efficiencies in curriculum and instructionEfficiencies in curriculum and instruction

•Confidence that what you’re doing is Confidence that what you’re doing is workingworking

•Collaboration and collegiality within and Collaboration and collegiality within and across departmentsacross departments

•Professional development for early facultyProfessional development for early faculty

•Better integration of adjunct facultyBetter integration of adjunct faculty

Making assessment useful Making assessment useful

How might assessment be useful for an How might assessment be useful for an individual course, your program as a individual course, your program as a

whole, or your faculty colleagues?whole, or your faculty colleagues?

Engaging facultyEngaging faculty

• Consider your Consider your colleaguescolleagues

• De-mystify De-mystify assessmentassessment

• Reduce costs and Reduce costs and enhance rewardsenhance rewards


Consider your Consider your colleagues:colleagues:

Faculty roles, Faculty roles, commitments, and commitments, and disciplinary identities disciplinary identities offer both incentives offer both incentives and disincentives to and disincentives to engage assessmentengage assessment

Engaging facultyEngaging facultyYour colleagues as practitioners of Your colleagues as practitioners of

their disciplines: their disciplines: Studio ArtStudio Art

IncentivesIncentives DisincentivesDisincentives

Art is publicly Art is publicly displayed and displayed and evaluatedevaluated

Studio artists work Studio artists work alone rather than alone rather than collaborativelycollaboratively

Discipline values Discipline values creativity, risk-taking creativity, risk-taking and inventivenessand inventiveness

““Assessment” Assessment” connotes “counting” connotes “counting”


their disciplines: their disciplines: ChemistryChemistry


Habit of collecting and Habit of collecting and interpreting interpreting observable evidenceobservable evidence

More than two More than two variables is scaryvariables is scary

Collaborative inquiry Collaborative inquiry is practiced is practiced extensivelyextensively

Less experience with Less experience with qualitative inquiryqualitative inquiry


their disciplines: their disciplines: Political SciencePolitical Science


Embracing multiple Embracing multiple types of scholarshiptypes of scholarship

Awareness of limits of Awareness of limits of social scientific social scientific researchresearch

Embracing multiple Embracing multiple methods of inquirymethods of inquiry

View of assessment as View of assessment as a policy innovation a policy innovation that will eventually that will eventually “go away”“go away”


Demystifying assessment:Demystifying assessment:

•NotNot scholarship of teaching and learning scholarship of teaching and learning

•NotNot individual teaching evaluation individual teaching evaluation

•NotNot student satisfaction data student satisfaction data

•NotNot necessarily quantitative necessarily quantitative

•NotNot rocket science (unless that’s what you rocket science (unless that’s what you teach!)teach!)


““Direct” Assessment”Direct” Assessment”

“Direct” Assessment

Evidence of what students actually know, can do, or care about

“Indirect”Assessment

Evidence of learning-relatedexperiences or

perceptions


Common direct assessment “artifacts”Common direct assessment “artifacts”

Theses, papers, essays, abstractsTheses, papers, essays, abstracts

Presentations and postersPresentations and posters

Oral or written examination itemsOral or written examination items

Responses to survey or interview Responses to survey or interview questions that ask for examples of questions that ask for examples of knowledge, practice, or value knowledge, practice, or value


Common indirect assessment “artifacts”Common indirect assessment “artifacts”

Course mapping, course-taking patterns Course mapping, course-taking patterns or transcript analysisor transcript analysis

Responses to survey or interview Responses to survey or interview questions about experiences, questions about experiences, perceptions, self-reported progress, or perceptions, self-reported progress, or impact of program experiencesimpact of program experiences

Reflective journalsReflective journals


But wait!! Aren’t we But wait!! Aren’t we observing student observing student work all the time work all the time anyway? What’s the anyway? What’s the difference between difference between gradinggrading and and assessmentassessment??


GradingGradingsummarizessummarizes

many outcomesmany outcomes

forfor

one studentone student

AssessmentAssessmentsummarizessummarizes

one outcomeone outcome

forfor

many studentsmany students


The purpose of assessment is to The purpose of assessment is to provide provide systematic, summarized systematic, summarized

informationinformation about the extent to which about the extent to which a a group of students group of students has realized one has realized one or more intended learning outcomesor more intended learning outcomes


Reducing the costs of assessmentReducing the costs of assessment

• Use what you’ve already gotUse what you’ve already got• Borrow freelyBorrow freely• Integrate assessment into work you are Integrate assessment into work you are

already doingalready doing• Share the work broadlyShare the work broadly• Limit your agendaLimit your agenda


Reaping the rewards of assessmentReaping the rewards of assessment

• Address questions that matter to facultyAddress questions that matter to faculty• Build in collaborationBuild in collaboration• Pair direct with indirect methodsPair direct with indirect methods• Choose approaches that “multi-task”Choose approaches that “multi-task”• Dedicate time for discussion and applicationDedicate time for discussion and application


Plan intentionally for use of resultsPlan intentionally for use of results

•Borrow strategies from past successes in Borrow strategies from past successes in collective departmental actioncollective departmental action

•Focus reporting on planned actions, not on the Focus reporting on planned actions, not on the evidence itselfevidence itself

•Weight Watchers, not the Biggest LoserWeight Watchers, not the Biggest Loser

•Dedicate time and resources for actionDedicate time and resources for action


What can you do in your program to:What can you do in your program to:

Link assessment to faculty identities and Link assessment to faculty identities and incentivesincentives

De-mystify assessmentDe-mystify assessment

Reduce costsReduce costs

OROR

Enhance benefits?Enhance benefits?

The bigger pictureThe bigger picture


Accreditation by a federally-recognized accreditation agency is required for access to federal student aid

Recognition requires accreditors to evaluate whether an institution maintains clearly-specified educational objectives and is successful at meeting them.


Guiding values of new HLC criteria:“A commitment to assessment would mean assessment at the program level that proceeds from clear goals, involves faculty at all points in the process, and analyzes the assessment results; it would also mean that the institution improves its programs…on the basis of those analyses.”


Table Talk – Questions for Dr. BeldTable Facilitators (a.k.a. FLC members)

Paul BlowersChemical & Environmental Engineering

Eliud ChuffeSpanish & Portuguese

Faiz CurrimManagement Information Systems

Wendy DavisAnimal Science

Ryan FoorAgricultural Education

Herman GordonCellular & Molecular Medicine

Christopher JohnsonEducational Technology

Amy Kimme-HeaEnglish

Carl MaesCollege of Optical Sciences

Katrina MirandaChemistry & Biochemistry

John MurphyPharmacy Practice & Science

Teresa PolowyRussian & Slavic Studies

Claudia StanescuPhysiology

Hal TharpElectrical & Computer Engineering

Deb TomanekOffice of Instruction & Assessment

Assessment & Research in Student Affairs

Angela Baldasare, Ph.D.Director, Divisional Assessment & Research

[email protected]

Starting from Scratch:From Outcomes to Assessment Activities

Aurelie Sheehan, Ph.D.

Director, Creative Writing

[email protected]

Unlocking AssessmentLinking Findings to Outcomes

David Cuillier, Ph.D.Director, University of Arizona School of Journalism

[email protected]

Our Process1. Define learning outcomes

2. Measure at overall program level

3. Link findings specifically to outcomes

4. Make adjustments (report & faculty retreat)

5. Feedback loop – see if it worked

EXAMPLES…

Outcome #10: TechnologyMEASURE: 2009 survey of multimedia knowledge

On a scale of 0-9 Photoshop 6.24

Final Cut 2.59

Dreamweaver .76

Soundslides .47

Audacity .35

CSS .35

Flash .24

FINDING: Need more Soundslides/Audacity trainingADJUSTMENT: Created multimedia class in 2010FEEDBACK LOOP: To survey students again 2012

Outcome #9: WritingMEASURE: Survey of intern supervisors

FINDING: Positive trajectory on student writingADJUSTMENT: Keep doing what we’re doing

Lessons learned1. Faculty buy-in through program-level view

2. One person responsible

3. Model assessment plans (e.g., Elon)

4. Explicit reporting – state it clearly

5. Focus on findings, not methods

Program Assessment:Raw Data to Findings

Ingrid Novodvorsky, Ph.D.Director, College of Science Teacher Preparation Program

[email protected]

TEACHER PREPARATION

PROGRAMStudent Learning Outcomes—Core Understandings

(These describe attributes of a well-prepared science teacher.)

Core Understanding Course Where Assessed

Science content understanding Subject Methods Course, STCH 410, STCH 496a

Understanding of adolescent development

STCH 310, STCH 496a

Coherent curriculum decisions STCH 420, STCH 496a

Productive learning environment STCH 410, STCH 496a

Clear communication STCH 410, STCH 420, STCH 496a

Acknowledgement of teaching complexity

STCH 410, STCH 420, STCH 496a

Reflective practice STCH 410, STCH 420, STCH 496a

TEACHER PREPARATION






STCH 310, STCH 496a







TEACHER PREPARATION

PROGRAMSTCH 410

Mentor Teacher Evaluation

Interns are assessed on a three-point scale on five Core Understandings.

TEACHER PREPARATION

PROGRAMSTCH 410

Mentor Teacher Evaluation

TEACHER PREPARATION






STCH 310, STCH 496a







TEACHER PREPARATION

PROGRAMStudent Teaching

Exit Evaluation

University Supervisor/Mentor Teacher

Student Teachers are assessed on a four-point scale on each Core Understanding.

Program Assessment

Raw Data to Findings

Questions?

TEACHER PREPARATION

PROGRAM

Live Long & Assess!

Sponsored by the Office of the Provost, the Assessment Coordinating Council (ACC), and the Office of Instruction and Assessment

UseUse

ss

Date post:	27-Dec-2015
Category:	Documents
Upload:	jemimah-miller
View:	216 times
Download:	1 times

Welcome to the UA Assessment Showcase 2012! Sponsored by the Office of the Provost, the Assessment...

Documents