Effective Results and Actions for Course Assessment Plans

Kim AndersonASLO Subcommittee Chair

Eva BaggAssociate Dean, Institutional Effectiveness

John HettsResearch Consultant

Fall 2010

Why an Outcomes Assessment Process?

• Follows a cycle of planning, implementation, analysis and action to improve student learning.– Facilitates periodic assessment– Guides planning management guideline– Improves student learning, retention, & program

completion– Enables faculty to play central role– Encourages collaboration within and among

departments– Supports practice of professional educators

Course Outcomes Assessment Cycle

• Course Review Cycle - at least every 6 years• Starting up may require shortened cycle• The Three “Ms”:

– Meaningful– Measureable– Manageable

Outcomes Assessment Cycle Management

• Internal Management Structure– Who (SLO Officer, lead faculty)

– When analyze and report (noted in assessment plan)

– How (implement by course, not SLO)

• Examples– Gantt Chart/Excel spreadsheet– Web based tool – Open Source– TracDat

Outcomes Assessment Loop

Components of theOutcomes Assessment Cycle

1. State desired outcomes clearly2. Identify assessment tools3. Establish criteria and level of

expectation for success4. Gather and analyze results5. Take action to improve and re-

evaluate

Sample LayoutIntended Outcome Assessment Task Criteria /Expected

Level of Achievement Results of Assessment

Actions Taken

Write intended outcome in this column.• Student Learning Outcomes (SLOs) are identified in the course outline of record.

What: Describe assessment tool or task

How: Explain implementation of task

Who: Identify persons responsible for conducting assessment

When: Identify timeframe of assessment task; commencement and for how long (semesters/years)

Expected Achievement:Establish minimum expected benchmark for achieving outcome Success Level:Determine acceptable success standard based on identified assessment task

Students Included:Describe students to be included (e.g. successful completers, all queried, random sample)

Begin collection of evidence for this course at this point.

Results: Describe sample and summarize results on outcome measure

Key Findings:Highlight notable results, good and bad

Conclusions:Describe implications for curriculum development and assessment improvement

Action:Explain (in past tense) what you have done in response to results

Re-evaluation Date:Identify timeframe for reevaluation

Outcomes Assessment Plan Process Format

Intended Outcome Assessment Task Criteria/Expected Level of Achievement

Results of Assessment

Actions Taken

SLO WhatHowWhoWhen

Expected AchievementSuccess LevelStudents Included

Begin collection of evidence for this course at this point.

ResultsKey FindingsConclusions

ActionRe-evaluation Date

Assessment Plan

Ongoing Assessment

Assessment Results and ActionsOngoing Part of Outcomes Assessment Cycle

• Step 4: Results, Findings, and Conclusions– Meet and discuss assessment evidence collected– Analyze evidence making sure to:

• highlight key findings from assessment evidence• draw conclusions about needs/issues revealed• formulate actions to improve student learning

– identify decisions that people can live with/implement

• Step 5:– Take action based on conclusions

• (Note: this step requires immediate action)

– Record/document actions already taken

Assessment Results and Action:Things to Consider

• Time– balancing competing responsibilities and priorities

• Management– aligning with review cycle and resources available—needs planning

• Discussion– collaborating with stakeholders in various venues:

• staff development, department meetings, small groups

Assessment Results and Action:Things to Consider (cont.)

• Key questions to consider when examining results of assessment– Evaluation of the process

• How consistent is our evaluation?• How robust is our final assessment information?

– Application of the process• How can we use this information to improve student

learning?• How can we use this process to support and integrate with

the college’s planning process?• How effectively do our results and conclusions provide

guidance to our stakeholders?

Step 4. Gather results of assessment

• Gather and organize• Process protocol and who’s involved; internal deadlines; regular cycle in

conjunction with course review cycle dates • Discussions at various meetings – need time• Do not become a statistician • Aggregate (combined)

– Within the course/department/program– Office of Institutional Effectiveness– Protect identities and confidentiality

• Disaggregate– Determine more discrete information– Address components more effectively

Step 4: Simple strategies for analysis• Right now (where available/if applicable)

– Compare:• results to expected level of achievement • results to previous results• results to pretest• results across demographic categories• outcome to outcome• outcomes across sections

• In the future– Compare year to year– Evaluative effectiveness of actions taken

Step 4: Describing your results

• Describe your sample• Your sample is just the people who were assessed

– Indicate how many people were assessed• N: total number of students

– Indicate how they were selected from population• Population: entire group of people you are using the sample to

understand/draw inferences about• Clearly describe procedure used to identify and select participants

– Key issues:• Is sample (N) large enough?

– Challenging with small populations like courses• Is sample representative of population?

Sampling issues• So how big a sample should you aim

for?– For sample needed for estimate of

% meeting standard – see table• *for small populations, you need to

assess almost everyone

– In the absence of comprehensive assessment, sampling needs to be rigorous

• What about representativeness?– Selection biases

• You• Students

– Identify population and select randomly

• Must try to get everyone selected

Interval size

Pop. Size ±15 ±10 ±5

20 14 17 19

30 18 23 28

50 23 33 44

100 30 49 80

150 33 59 108

200 35 65 132

500 39 78 217

US Pop 43 96 384Calculations made using http://www.surveysystem.com/sscalc.htm

Step 4: Describing your results (cont)• Key distinction: is outcome categorical information or more

quantitative information– Categorical example

• Common educational rubric for outcome assessment– 4 - Advanced (98% and up) – exceed standard substantially on all criteria– 3 - Proficient (70-97%) – met standard on all criteria– 2 - Basic (50-70%) – met standard or close on all criteria– 1 - Below Basic (below 50%) – did not meet standard on multiple criteria– Categories are different “widths” – intervals are not equivalent, i.e., a “point” difference

doesn't always mean the same thing

– Provide frequency• Simple count of number of students in category

– Supplement with percentage• Ratio of frequency to total number (* 100)• 5 (frequency) out of 100 (N) would yield 5%

Step 4: Describing your results (cont)

• Describe your results – quantitative info– E.g., number of items answered correctly– Describe central tendency

• executive summary of your data• two key types

– Mean: add all scores then divide by N• Note: Be careful about summing (or averaging) across outcomes• Imagine two standardized assessments of outcomes in Astrology

– Number of astrological signs correctly identified– Estimate quality of astrological reading on a 1-10 scale

– Median: score at which ½ sample is below, ½ above• Use with mean when extreme scores make mean ambiguous• E.g., 10 item test with 10 scores: 0, 8,8,8,9,9,9,10,10,10• Mean = 8.1, Median = 9_

Sometimes things shouldn’t be combined (or averaged)

Step 4 - Analyze Key Findings• Analysis and evaluation

– Through discussion with vested individuals– Evaluate the validity of the assessment– Compare with goals, and, eventually, previous results– Discuss any discipline-specific issues– Engage data with critical eye to interpret and apply appropriately

• Highlight strong indicators– Significant patterns or trends in the data– Met, not met, or exceeded stated SLO’s expected level of

achievement

• Focus on accomplishments and examine difficulties or needed improvements

Step 4: Develop supportable conclusions

• Focus on actionable evidence– Keep information manageable– Identify what:

• worked well• could be improved

• Relate conclusions back to course SLO• Reflect faculty’s discussion• Make appropriate decisions for the data level• Note unexpected results, flaws in study, and faculty

observations based on the evidence

Step 5. Document Actions Taken• Based on Step 4, developed and implemented steps to

improve student learning and its assessment• Include what has been done to improve student learning• What still thinking about goes in “conclusion” in step 4

– Curriculum (content emphasis, sequencing, formative assignments, more time with a topic of concern)

– Pedagogy (varied delivery methods, technology, learner styles, tutors, Supplemental Instruction, prompt feedback)

– Assessment (SLO, assessment tool, internal/external factors, criteria, sampling)

Step 5. Document Actions Taken (cont)

• Key issues to consider when documenting actions taken

– Clarify extent of change: incidental modification or significant change– Specify definite actions taken and identify faculty responsible– Outline meaningful time frame for implementation and re-evaluation– Describe any follow-up process

• Teaching & Learning Center

ESL SampleIntended Outcome Assessment Task Criteria /Expected

Level of Achievement Results of Assessment

Actions Taken

Write afocused thesis statement and maintain unitythroughout an essay.

What: Write a compare/contrast essay basedon short readings of fiction and non-fiction.

How: The Test of Written English (TWE)scoring guide (rubric) items 1 and 4 ascriteria.

Who: Instructors of ESL 33 and33X.

When: Collect and report in Spring 2010

Expected Achievement: 70% ofstudents should achieve passing score.

Success Level: passing score is 4 or higher (on 6 pt scale)

Students Included: random sample of students enrolled in all sections of ESL 33and 33X.

5/31/10Results:: 55% (55 out of 100 total) of the randomlysample essays obtained a score of 4 or higher with 100% participation from all ESL 33 &ESL 33X•10% received a 6•15% received a 5•30% received a 4•25% received a 3•25% received a 2•5% received a 1

Key Findings:We were surprised by the low percentage, but noted a large number of students were close to meeting the expected level of achievement.

8/24/10Actions taken:1) Modified scoring guide to includesalient feature for each score ateach level. 2) Clarified task we are askingstudents to complete.3) Created guidelines for selection of readings.

Re-evaluation Date: Spring 2011.

ESL SampleIntended Outcome

Assessment Task Criteria /Expected Level of Achievement

Results of Assessment Actions Taken

Conclusions: Students were exposed to six writing modalities (twice as many as in ESL 34/34X) giving little time tounderstand the essay structure and apply skills to targeted writing tasks. Students were nearly able to achieve expected level of completion , but students' focus on the essay caused a drop in their use of standard academic English (see SLO #3). Also, the reading given to complete the task compared two groups as well as two time periods. Students struggled to organize their essays since they didn't know which task to complete: to contrast the groups or the time periods.

Questions?