Assessment:Developing

Learning Outcomes

Teaching Learning Assessment Cycle

• Establish Learning Goals

• Provide Learning Opportunities

• Assess Student Learning

• Use the Results

Establish Learning GoalsMeaningful Learning goals should lead to the improvement of teaching and learning

• Institution

• Program

• Course

Institution: UCCC’s Mission Statement:SUNY Ulster is a vibrant community of learners distinguished by academic excellence, collaboration, innovation, service and responsible use of resources. As a public, comprehensive two year institution dedicated to providing affordable accessible education, we work in an ever changing environment to:

-Prepare students to transfer to four institutions-Prepare students for success in college and in the workforce-Provide enrichment and life long learning opportunities-Augment learning through the integration and application of emerging technologies-Prepare students to live and work in a global society-Play an active role in economic development, and-Enhance the quality of life for residents in Ulster County

Benefits of Learning Outcomes

• Helps students understand the nature of the program or course and what skills or knowledge will be expected of them.

• Provides a guide to students so they are assured they have not missed any major content.

• Gives structure to the instructor in lecture development – what to teach

Gives structure to the instructor in exam or assignment development – link to goals. Insures the student learning outcomes are linked to the college mission. Enables the public and four year schools to understand what the program or course intends to accomplish. Most importantly – leads to improvement of teaching and curriculum development.

Developing Learning Goals:

• Meaningful learning goals will lead to improvement of teaching and learning.

• There should be an interconnectedness of goals in programs and courses.

Developing Learning Outcomes

• Learning outcomes should be developed collaboratively from research and reflection.

• Collaborate:• This ensures the integration of course

outcomes to program outcomes. As many faculty as possible should be involved.

Research• Internal Sources:–Mission statement–Current Syllabi–Previous program reviews and surveys – External Sources:–Associations–Advisory Councils–Other colleges Program descriptions (websites)

• External Sources:

Advisory CouncilsOther colleges Program descriptions (websites)

Associations

Reflection:

• Key word: WHY.

• Why do you want your students to know what you put down as a learning outcome.

Middle States Recommended Discussion Topics:

• What is this course or discipline all about?

• What does the department aim to achieve with this program?

• What do we value about our discipline?

• What are the most important things students learn in this discipline?

• Why are those things important?

• What does our disciplinary association think students should learn?

What do we want students to get out of this program?

How does this program relate to the college mission?

How does it relate to other programs?

What do our students do after graduation?

Why does the college offer this program?

Why is it important student study it?

How will this experience prepare them for post graduation?

What do we expect of students in this program?

How will students be different after completing the program?

What will the student be able to perform after the completing the program?

What skills will the students acquire after the completing the program?

What indicators will be used to measure the student’s performance?

Knowledge

Thinking

Attitudes

Knowledge• Duplication of procedure, learning definitions

summarizing characteristics, understanding components.

• Memorization

• Not as stressed today as in previous decades

Examples of Knowledge Goals

• Explain the three goals of a macro economy and how they are measured.• Summarize distinctive characteristics

of a various novelists• Describe the …scientific model

Thinking

• Application of knowledge in a new context• Analysis or the ability to understand the relationships

amongst the components of a complex concept• Evaluation or making informed judgments on new

material through research and problem solving• Synthesis or the ability to theorize, generalize, reflect

generate new hypothesis or ne ideas.• Critical Thinking or seeking truth and ability to

question

Examples of Thinking Goals• Explain a reaction• Explain the impact• Analyze errors• Compare and contrast• Explain why• Judge the effectiveness • Evaluate the validity

Research and identify

Choose the appropriate procedure

Write a poem using..

Theorize design

Recognize

Design and conduct….

Identify a problem…..

Attitudes • Dispositions and values• Appreciation of the subject• Enjoyment of learning• Metacognition• Formation of opinions and values • Development of a sense of ethics.

Example of Attitude Outcomes

• Be intellectually curious• Choose an ethical course of action• Appreciate the merits of …• Identify the strengths and weakness of others’ work• Work independently• Work in groups• Develop leadership skills

Writing Learning Outcomes

• Avoid broad nebulous terms:–Students will:

–Learn –Know–Understand–Become aware–Appreciate–Think critically

Action words help students understand what we want them to learn and it makes assessment easier

• Describe the solar the key characteristics of each planet in the solar system

• Make appropriate references and deductions from biological information

• Develop graphic spreadsheet to support a decision

• Design a chemical experiment to test a hypothesis

Systematically analyze and solve problems…

Defend one’s views

Present original interpretations…

Critically evaluate the effectiveness…

Apply basic problem solving skills…

Conduct research

Assessment of the Learning Outcomes

Two Categories of Measures

• Direct measures provide evidence of whether or not the student has command of the subject matter

• Indirect Measures provide data that is related to the act of learning. It implies the student has learned, i.e. how the student perceives whether or not he/she has learned. Questionnaires and surveys are indirect.

Direct vs Indirect

• Direct evidence should

answer the questions what did the student learn? What did he/she not learn? But it does not answer the question why did the student learn it? Why didn’t the student learn it?

• Indirect evidence focuses on the learning process. Comments about homework or lectures or group work.

Choosing Evaluation Instruments:

QUESTIONS TO ASK:

• 1. Is the evidence provided by the method linked to learning outcomes?

• 2. Is a standardized instrument appropriate for the learning goals of

the institution

• 3. Is the evaluation method comprehensive enough? • 4. Are important learning outcomes measured by multiple means?

5.Do the test questions test what is desired to be learned?

6. Are the questions clear?

7. Do the questions elicit information useful for making improvements?

8. Does everyone interpret the response the same way?

9. Do the results make sense? Are the results corroborated by evidence?

Tools and techniques:

TestsStandardized vs Home grown

Quote from Middle states:Although a test created locally may not have the same statistical validity and reliability as a standardized instrument, it’s relevance to the specific learning goals in question may make it a more appropriate and more effective instrument.

Test blueprint is an outline that matches test items to the intended learning outcomes.

Rubrics

• A set of criteria used to assess papers, projects, performances, field experiences.

• Four Types: •Checklist•Rating Scale•Descriptive•Holistic

Checklist Rubrics

• A list of yes or no or Did or Didn’t do

• Used primarily in lab or clinical settings

Rating Scale Rubric

• This is a checklist with a rating scale added to show the degree to which the things you’re looking of are present

• Uses terms like Outstanding, Very Good Adequate Or

• Strongly Agree, Somewhat Agree etc..

Descriptive Rubrics

• These give a description of what constitutes deficient adequate or exemplary

• Or unacceptable, Fair Proficient

Holistic Rubrics

• For the more massive assessments, rating scale and descriptive rubrics may not be feasible i.e reading 1000 essays for placement.

• Holistic rubrics have short narrative descriptions of the characteristics for outstanding work, acceptable work, and unacceptable work

Other Assessment Tools

• Self reflection

Engage in metacognition. Ask students to reflect on what they have learned

This gives insight into the learning process and it helps students integrate what they have learned

Ratings and comments from internship supervisors

Placement Rates

Ad Hoc Surveys

Focus Groups

Portfolios

What is Middle States Looking For?

• They will ask the fundamental questions: • • Is the Institution fulfilling it’s mission?• • Are institutional and program level goals clear to the public, students,

faculty and staff?• • Are Institutional programs and resources organized and coordinated to

achieve these goals?• • Is the Institution using assessment results to improve student learning and

advance the institution.•

Characteristics of Assessment Processes that meet Middle States Expectations

• Useful Lead to improvement• Cost Effective No need to start from scratch use what is there• Reasonably Accurate Is evidence clear and visable -both direct and indirect• Planned Assessment must be interrelated to strategic plan and academic plan• Organized, systemized and sustained Implement an ongoing system and review it.

What Should Be Documented?

• Learning outcomes in programs and courses . • An assessment Plan

• Assessment results

• Uses of results to show what is being changed towards improvement

Assessment: Using the Results

To

Improve Teaching and Learning

Three steps:

Choose a standard benchmark

Set a standard or benchmark for exemplary, adequate and inadequate

Setting targets for the collective class or program

Choosing a Benchmark

What is a Benchmark

• The process of comparing the results of your test, essay, paper or other assignment against something.

Local Standards:

Department members agree to what is a good grade on a specific exam. Having faculty involved in courses and exams leads to more attention paid to assessment. External Standards:

This is the use of disciplinary association standards. Usually there are tests that are required from the association. Certifications have required exams with established benchmarks i.e. NCLEX

Internal peer benchmarkThe bell curve – or looking at how Psychology

majors do on the economics test vs business majors. Or looking at all the students in the program. Which ones succeed and why?

External peer benchmarkWe compare Jane Doe’s 65 to other instructors’ or colleges’ 65.

Setting Standards:

• Choose the bench mark most appropriate for your course or program and set a specific appropriate performance standard.

• Ask what is an example of exemplary work?• How does this differ from adequate and

inadequate?

Setting Targets for Students’ Collective Performance

What do you want the class or students in a program as a whole achieve?

• Express targets as a percentage. Ex: 90 percent of students earn at least a minimally adequate rating

• Or 95 percent of the students will earn a 65 % on all exams.

Summarizing the Results

Five ways to summarize results:

Tallies

Percentages

Aggregates: Scores

Averages

Qualitative Summaries

Using The Results

Identify areas of strengths and weaknesses

Compare results with what you wanted to hear

Create strategy for improving the learning experience

Base changes on real data

Empower faculty to make those changes

Articulate exactly who is responsible for the changes

Share the results and actions with the whole institution