Developing an Effective Assessment Plan: From Mission to Outcomes

Assessment Plan WorkshopSUNY Oneonta

March 22, 2012

Presenter:

Patty FrancisAssociate Provost for

Institutional Assessment & Effectiveness

Background Development of College’s Action Plan for Planning

and Assessment (Spring 2008)◦ Endorsement by College Senate (12/2008)◦ Approval by President’s Cabinet (Spring 2009)

Formation of Institutional Assessment Committee (IAC) (Spring 2009)

Development of assessment guidelines by IAC and approval by President’s Cabinet (11/2009)

Distribution of guidelines in 12/2009, with first plans due June 1, 2010

Revision of guidelines by IAC in June 2011 Submission of first assessment reports in June

2011

Suggested Format for Unit Assessment Plans

All unit goals and objectives (i.e., Step 1) Results of activity mapping exercise (i.e.,

Step 2) {optional} Action plan summarizing those objectives

units intend to assess (and how) for the next year (i.e., Step 3)

Brief description of how unit will “close the loop” once assessment results are attained (i.e., Step 4)

Establishing congruence among institutional mission and goals, programmatic and unit objectives, unit activities, and assessments

Linking goals and objectives to outcomes through action plan

Assessment as an ongoing, iterative process Using a variety of meaningful measures,

both quantitative and qualitative, in search of convergence

Using existing data sources as much as possible

Some Assessment Basics

1. Setting goals and objectives: “What you aspire to do and what you do”

2. Objective mapping: “How you do what you say you do”

3. Assessment: “How you know you are doing what you say you do”

4. “Closing the loop”: “What you do next based on results”

Assessment’s Four Steps

Done correctly, assessment:

◦ Serves to align mission, goals, objectives, and assessments across all levels of the institution

◦ Initiates a “never-ending” dialogue among staff members regarding programmatic priorities, objectives and effectiveness

◦ Offers multiple, rich opportunities for professional interaction and development

◦ Provides (mostly) affirming data in support of existing programs and services

◦ Provides a systematic, focused direction for change and future activities

Most Important

Step I: Setting Objectives:

“What are you attempting to do?”

It All Starts With a Mission Mission statements:

◦Summarize the unit’s status and major functions and objectives at present

◦Should be congruent with existing higher-level mission(s) (e.g., at the institutional/divisional levels)

◦Are intended primarily for internal stakeholders

◦Are optional in IAC guidelines but strongly encouraged

“The Library’s mission is to provide comprehensive resources and services in support of the research, teaching, and learning needs of the University community.”

Sample Mission Statement (University of Chicago Library)

Sample Mission Statement(Creighton University Media Services)

“To serve Creighton University and the community by exploring, designing, supporting and facilitating learning, teaching and research opportunities through the effective and creative use of technology and media in accordance with the mission of Creighton University.”

Goals, Objectives, and Outcomes: What’s the Difference?

Goals ->Objectives ->Outcomes = More General (and Less Measurable)->More Specific (and More Measurable)

Goals:◦ Statements about general intentions/purposes that are

broad and more long-range in scope and not directly measurable

◦ May come directly out of unit mission statement◦ Usually developed at programmatic or divisional level and

often are in the form of a “process” statement (i.e., begin with verbs like “establish,” “provide,” “enhance”)

Goals, Objectives, and Outcomes: What’s the Difference? (cont.)

Objectives:◦ More specific than goals◦ Typically there are multiple objectives for each

goal◦ Usually developed at the unit level to reflect

“upper-level” goals

Goals, Objectives, and Outcomes: What’s the Difference? (cont.)

Outcomes:◦ Very specific statements that “translate” into

assessable measures◦ Process- vs. results-oriented statements◦ Two kinds, depending on assessment stage

“Expected outcome” refers to anticipated results of assessment – should include criterion to be used in determining success

“Actual outcome” refers to actual results of assessment

GoalEnhance student learning through civic, professional and research engagement.

Objective Students will learn to be responsible global citizens.

Expected Outcome80% of students will participate in a university-approved community service activity with an inter-cultural emphasis.

Student Affairs Example

GoalDevelop and implement processes that deliver value to the institution and campus community.

Objective To identify and carry out strategies that result in

the strategic allocation of resources.

Expected OutcomeDuring 2007-08, the total funds expended on institutional priorities will increase by 20%.

Finance and Administration Example

GoalEnhance public service and image throughout the region.

Objective Promote and facilitate faculty involvement in the

local public schools.

Expected OutcomeAt least 40% of full-time faculty will participate in activities sponsored by schools in the community.

Academic Affairs Example

GoalCreate a culture of communication that ensures positive, supportive recognition of the school throughout all relevant constituencies.

Objective Establish a research-based plan that defines

university marketing and communications needs.

Expected OutcomePlan is developed and approved by the President’s Cabinet.

Institutional Advancement Example

Establishing Goals and Objectives: Recommended

Actions Examine and review stated unit objectives for all

constituencies served Elicit and discuss staff members' perceptions of unit

objectives Analyze and compare unit objectives with

college/divisional Mission Statement and planning documents, those at comparable institutions, and criteria/standards of certification agencies or national associations as appropriate

Develop or revise unit Mission Statement that reflects objectives and that staff members understand, agree with, and support through their actions

Typically, Unit Objectives Reflect:

Institutional effectiveness performance indicators

Documentation of all services and programs offered

Tracking of use of services (and by whom) Constituent satisfaction with services/programs Direct impact of services/programs on

constituents (including student learning if appropriate)

Adequate resources to support planned activities

Consultation with other campus units (when joint activities are required)

And Remember:

Don’t try to do everything at once – typically, 5-7

objectives for each assessment plan are

plenty!

Step II: Evaluating Activities:

“How are you attempting to meet your objectives?”

What is the Purpose of This Step? This step is basically an “audit” of existing

programs, services, and resources, in order to determine if unit goals and objectives are realistic

Involves a comprehensive review, being sure to “map” objectives to existing activities

Possible results◦ Redundancy – same objective being addressed by

multiple programs◦ Gaps – specific objective not being addressed by any

program; could lead to: Identifying new\reallocating existing resources Abandoning the objective, at least for the present

Evaluating Activities: ExampleObjective/

UnitAlumni Affairs

Fundraising

Grants Community Relations

Objective #1 5 5 5 5

Objective #2 5 5 5 5

Objective #3 2 3 3 5

Objective #4 2 1 1 2

Objective #5 1 2 1 1

Note. Extent to which unit addresses objective is ranked on a scale of 1-5, with higher scores indicating more emphasis.

Evaluating Activities: Recommended Actions

Determine the extent to which unit objectives are reflected in unit activities, with the aim of assuring that all objectives can be met through those activities

Review and analyze coherence and inter-relatedness of unit activities in order to assure the most efficient use of resources

Identify resources that are necessary to support activities aimed at realizing all unit objectives and make sure those resources are available

Step III: Measuring Outcomes:

“How successfully are you meeting your objectives?”

What is the Purpose of This Step? To collect data that clearly demonstrate

whether or not the unit is meeting its objectives

Requires a priori identification of appropriate strategies\measures for each objective and statement of expected outcome (i.e., what is the unit aiming for?)◦ Utilization of services/programs (i.e., quantity)◦ Performance measures (i.e., quality)◦ Satisfaction surveys◦ Comparisons with other units (i.e., benchmarking)

Measuring Outcomes: Recommended Actions

Develop a detailed action plan linking objectives to outcomes, specifying strategies/actions intended to accomplish each objective and, for each action, a timeline, person/persons responsible, resources required, measures to be used, expected outcomes, and actual outcomes once assessments have been conducted

Use a wide variety of information sources, including existing data as much as possible

Develop and administer satisfaction surveys to internal and external constituent groups

Establish criteria for unit effectiveness through comparisons with information provided by similar units at other institutions or other relevant sources (e.g., certification agencies, national organizations)

Units whose functions are evaluated through SUNY-wide measures (e.g., the Student Opinion Survey, the National Survey of Student Engagement) should include these measures as performance indicators in their assessment plan

Examples: TV ServicesGoal

To provide high quality and comprehensive television broadcasting experiences to our student staff members

and interns.

Objectives and Measures1. To expand our formal internship program.

Number of students enrolled in internships2. To enhance technical aspects of the program’s facilities.

Equipment upgrades that meet standards of benchmark institutions3. To increase partnerships with academic programs, high

schools, and community organizations. Number of academic programs with co-curricular requirements

4. To establish minimal competencies for each of the program’s student learning outcomes. Percentage of students who meet minimal competency levels

Step IV: Using Outcomes to Plan:

“How can you improve what you’re doing?”

What is the Purpose of This Step? To feed back the information obtained from

the outcomes assessment into the unit’s objectives and activities

Will lead to the continuation of successful practices and to the discontinuation\revision of unsuccessful practices

May also lead to new objectives

Using Outcomes to Plan: Recommended Actions

Reach overall conclusions regarding unit effectiveness, based on comparisons between expected and actual outcomes and with special consideration given to the different constituent groups served by the unit

Identify major strengths and weaknesses of unit operations revealed through assessment

Make major recommendations for changes in unit activities based on assessment outcomes

Analyze relationship between available resources and unit/program success

Revise objectives and identify new outcome measures as appropriate for next assessment round

Linking Goals and Objectives to Outcomes

The Action Plan

Goals and Objectives for Unit Strategies or Actions Intended to

Accomplish Goals and Objectives For Each Action:

◦ Timeline◦ Person/Persons Responsible◦ Resources Required◦ Expected Outcomes◦ Actual Outcomes (Once Actions are Complete)

Essential Components of Action Plan

Objective Actions/Strategies Target Completion

Date

Resources Required

Person(s) Responsible

Expected Outcome Actual Outcome

1. Develop and implement a strong foundation of IT infrastructure and sound fiscal planning.

A. Build life-cycle replacement funding into planning at every level of investment in IT.

B. Budget standard amount each year per FTE to support replacement of faculty/staff desktop computers.

A. 01/01/10

B. 01/01/10

A. $150,000 annually

B. $175,000 annually

A. AVPIT, VPFM

B. AVPIT, VPFM

A. 20% of all IT replacement needs (except laptops) are funded every 5 years

B. 1/3 of faculty/staff desktops are replaced every 3 years

2. Provide faculty, staff, and students with reliable access to computing, information, and network services, both on- and off-campus.

A. Ensure that core production services are available and accessible during scheduled hours of operation.

B. Expand student residential network bandwidth.

C. Ensure that wireless network is reliable and accessible.

A. Ongoing

B. 01/01/10

C. 10/01/09

A. $150,000 annually

B. $100,000

C. $50,000 annually

A. CIO

B. CIO, VPSA

C. AVPIT, CIO

A. Average daily availability exceeds 99%

B. Bandwidth is increased to 64 mbs

C. Average uptime is 99% during campus on-hours and 95% during off-hours

Action Plan Example - ITGoal: Plan and deliver integrated information services to enable members of the campus community to access information when and where they need it.

Assessment Planning vs. Annual Reports:

What’s the Difference?

Assessment Planning Comprehensive process consisting of four steps

described earlier in presentation (i.e., mission to outcomes), following common guidelines

A unit’s first assessment plan:◦ typically takes a broader, longer view of the unit’s

functions and intentions◦ will provide information for each of the four steps◦ will delineate all goals and objectives for a unit, but action

plan will specify sub-set of those to be assessed each year In subsequent years, annual assessment plans will

highlight changes in assessment approach and specify those objectives to be assessed during the coming year

Annual Reports Required of administrative units at the

College for a number of years (although not in a consistent format across divisions)

Relatively short-term and even post hoc in nature, and not necessarily guided by unit mission and broader goals

Likely to include description of events and achievements not included in assessment plan

All That Said……. IAC intentionally developed assessment

planning process that, ultimately, would easily “translate” into annual reporting format

Much correspondence exists between “action plan” components and annual report grid

Eventually, once assessment planning becomes routine, there may be little distinction between that process and annual reporting◦ Reasonable to suggest that assessment plan could be

incorporated into the annual report