The Role of Faculty Institutes in Improving Student … · Web viewThe Role of Faculty Institutes...

The Role of Faculty Institutes in Improving Student

Learning: A Core Abilities Assessment Report from the

December 2011 InstituteSubmitted by Mirelle Cohen and Minerva Holk

February 2012

Table of Contents

Introduction ………………………………………………………... 1Planning for the Institute ……………………………………………1The Institute ………………………………………………………... 3Executive Summary … ……………………………………………. 5The Data ………………………………………………….……….. 6Feedback from Participants ……………………………………….. 7Recommendations …………………………………….…………… 8Appendix I Agenda ……………………………………………….. 11Appendix II Sample Collection Materials………………………… 13Appendix III Sample Rating Materials …………………………… 17Appendix IV Data Tables …………………………………….……. 21Appendix V Faculty Evaluation Tool …………………………….. 32Appendix VI Institute Evaluation Responses …………………… .. 33Appendix VII Post its Feedback …………………………………… 40

i | P a g e

Introduction

The Winter Institute held December 13 and December 14, 2011 continued the work of testing and

refining the Core Abilities. This time we worked with Thinking, Information Literacy and Technology,

and Communication. There were thirty-two participants (including the two Coordinators). Twenty-four

(75%) of the participants were full-time faculty and eight (25%) were adjunct faculty. Ten participants

(31%) had not attended a previous Institute. An overview of the process and the results attained will be

presented, followed by recommendations for future Institutes.

Planning for the Institute

Based on several comments from participants at the Summer 2010 Institute and a well-received

pilot workshop in Winter 2010, a pre-Institute workshop was scheduled for one hour at the start of the

first day. Attendance at this workshop was mandatory for all participants who had not attended an

Institute previously. All other participants who wished to refresh their memories of the goals and

procedures of the Institute were invited to attend. In fact, several returning faculty did attend. At this

workshop we reviewed the history of Core Abilities at Olympic College, reviewed the five Core Abilities,

and explained what the participants would be doing during the two-day workshop.

Another change to the planning process was in the method of soliciting samples from faculty and

in the content of the form used to solicit the samples. Addressing first the method of soliciting samples: It

was decided initially to solicit samples from all faculty across campus. In early September 2011an email

was sent to all faculty requesting samples from the institute. Instructions on what to submit, to whom,

and a deadline was included in the email. As well, the Institute Coordinators made a plea for samples at

the All Faculty Meeting at the beginning of Fall quarter, 2011. These two efforts resulted in zero samples

being submitted. Since rating samples is the raison d’être of the Winter Institute it became evident that

collecting samples from those who were going to be participating was critically important. Thus, it was

decide to bring forward the process of sign-up for the Institute and begin the process in late September.

The first email request for participants for the institute resulted in the speediest response rate we

have encountered thus far. The 33 spots were all claimed within 72 hours. Once the participant list was

finalized we began in earnest to solicit samples. In the first week after all the participants had submitted

their request to participate in the Institute, only on set of samples was received. The process of obtaining

samples from all participants then began in earnest. Follow up emails were sent, personally addressed, to

reach participant. Follow-up voicemails were also sent. By the first day of the Institute we had received

samples from approximately 27 (82%) of the participants. Nevertheless, some samples were received on

the day before the Institute, after the copying was completed, and one set were given on the first day of

the Institute. In general, sample collection and preparation is a monumental task both in terms of time

needed to follow-up with faculty, and time involved in readying the samples for copying. Based on the

experiences at this Institute, there are some recommended changes which will streamline the process for

the next samples rating Institute in December 201 2 (see Recommendations section which starts on p.7).

It was decided to use Olympic College catering to provide the lunches for the Institute this time.

In the past we have used an outside caterer and, although we have been very satisfied with the food in the

past, we were informed that Olympic College catering were able to compete on price with the outside

caterer, something that had not been possible in the past. Aside from lunch, however, and in the interest

of frugality, it was decided not to provide any other food, beverages or snacks other than coffee, which

was provided on both days. At previous institutes we have provided light breakfast fare and an afternoon

snack.

Name tags for participants, table, numbers, and packets containing several key documents were

prepared by April Rabe in the Office of Instruction. As well, April collected the lunch requests from

faculty and submitted the catering order to Olympic College catering.

The Institute

Upon arrival at the Institute participants were given packets of information relating to the Core

Abilities. The documents included an agenda (See Appendix I), the College’s Mission Statement and

Guidelines for Assessing Core Abilities, Outcomes and Rubrics for each of the Core Abilities, a Glossary

of Terms for Core Abilities Rubrics, and a copy of the Summer, 2010 Institute Report. 1 Each participant

was also given a name tag which included their name, division, and assigned table number.

There was representation at the Institute from each division at Olympic College (See Table 2

below). Participants were assigned to specific tables to provide a mix of those experienced with the

process and those who were attending their first Core Ability Institute. We also took into consideration

each participant’s discipline and divisions and whether they were full-time or part-time to ensure that we

achieved a diverse range of experience at each table. The participants had been given the opportunity to

request a specific Core Ability to work on at the Institute and we were able to accommodate most

people’s preferences. 2 Working in groups of four, with two groups of three, assignments and student

samples were rated using the rubrics developed by the Core Abilities’ Taskforce. The participants

engaged in the rating process with enthusiasm and diligence.

1 Note: For the first time we asked participants to return the packets to us at the end of the Institute so that they can be reused at the next Institute. Most participants did leave theirs behind and they were collected.2 On the RSVP form for the Institute, faculty were asked to indicate, which of the Outcomes: Thinking, Communication, or Information Literacy and Technology they would prefer to work on. Eleven chose Global Perspective, 13 chose Lifelong learning, 3 participants said they would like to work on both outcomes and 3 participants said they would like to work on either of the outcomes.

i | P a g e

The initial step in the process was a group norming session in which each participant read an

assignment and a sample of student work that was submitted for that assignment. The participants rated

both the assignment and the student sample according to the four levels of performance utilized in each

Core Ability: Emerging, Developing, Competent and Strong. The assignment and sample used for the

norming exercise were submitted for the Communication Core Ability. Individual evaluation was

followed by a collaborative discussion during which group members at each table collated their results

and created a table rating for the assignment and the sample. The table results were then reported to the

group at large. There was active participation in the discussion that followed resulting in clarification of

the purpose of the Institute which is to assess student learning using the Core Abilities. The Mission

Statement for Assessing Core Abilities states: Olympic College seeks to improve teaching and learning

by focusing inquiry to ensure that students are getting ample opportunities to develop Core Abilities; and

students are performing sufficiently on Core Abilities.

The tables below give the specific information according to teams arranged for the review and

discussion of assignments and samples each day.

Table 1: Institution Level Assessment of Core Abilities Winter Institute 2011Faculty Participant List by Core Ability Rating Group

Thinking Communication Information Literacy & Technology

Dianne MooreLawrence, Amy *#

Macias, Steve #Thompson, Jessica

Barbara ParkerDeitchman, Elaine *#

O’Neil, ElizabethScribner, Sara *#

Barbara ParkerO’Neil, ElizabethScribner, Sara *#

Jason HeinzeElisabeth Briggs

Krystal, Barbara *#Mathew, Philip #

Kandace MacKabenBright, Kathy

Bermea, NancyLand, Cynthia *#

Cami GeyerO’Shea, Constance *#

Robertson, DonaldPhayre, Alison

Herman AmyVisher, Peggy #Leetham, Colin

Salas, JoannFrederick. Chris

Hoover, Carmen #Hove, Theresa

Westlund, MarkCulbertson, JolenePatterson, Brian *#

Key:

*: Part time #: new Bolded name: Table leader

ii | P a g e

Table 2: Institutional Level Assessment of Core Abilities Winter Institute 2011Faculty Participant List by Division

Business and Technology Mathematics, Engineering,

Sciences and Health

Social Sciences and Humanities

Library

Nancy Bermea Elizabeth Briggs Carmen Hoover Amy HermanKathleen Bright Jolene Culbertson Theresa Hove Dianne Moore

Kandace MacKaben Christopher Frederick Barbara Krystal Sara ScribnerPhilip Mathew Cami Geyer Charlie MackallBarbara Parker Jason Heinze Constance O’SheaJoanne Salas Cynthia Land Brian Patterson

Mark Westlund Amy Lawrence Jessica ThompsonElizabeth O’Neil Colin LeethamBrian PattersonAlison Phayre

Donald RobertsonPeggy Visher

Executive Summary

The Core Abilities Winter Institute 2011 focused on three Core Abilities as identified in the Olympic

College master plan for core ability development and assessment.

Information Literacy and Technology

Outcome 1: Graduates use strategies to search for information that enhance the acquisition of knowledge.

Outcome 2: Graduates evaluate and appraise sources.Outcome 3: Graduates access and use information and/or technology ethically, legally and/or

responsibly.Outcome 4: Graduates use various inquiry tools and different formats of information e.g. media.Outcome 5: Graduates use technology and information appropriate to field or discipline,

synthesizing information to formulate insights and create knowledge. Thinking

Outcome 1: Graduates engage in critical analysis.Outcome 2: Graduates engage in creative problem solving.Outcome 3: Graduates engage in quantitative reasoning.

iii | P a g e

Communication

Outcome 1: Graduates understand and produce effective oral communication. Outcome 2: Graduates understand and produce effective written communication. Outcome 3: Graduates understand and use effective non-verbal communication skills. The Data

Each participant was given a stack of ratings sheets (See Appendix II). They were asked to rate

the assignment prompt and the set of samples that accompanied the assignment prompt as Emerging,

Developing, Competent or Strong. After each participant had rated the sample and the accompanying

assignments, the tables were asked to discuss both the assignment and the samples and to reach a

consensus rating for the assignment and for each sample. This determination was reached by referring to

the standards in the rubric that applied to the designated Core Ability. The data in Appendix IV on pages

26-36 below indicate:

1. Opportunity for Student Growth

The gap between the assignment ratings and the sample ratings in Tables 1 and 2 indicate faculty

agreement that all assignments, except for the assignment submitted for Global Perspective 5,

provide sufficient challenge so that students can learn, grow, and improve their comprehension of

the material being addressed in the academic unit being taught. However, it is interesting to note

that of the 22 assignments rated for the Global Perspective, only seven were rated at the

competent level and none of the student samples were rated at the competent level (see Tables 3-7

on pages 21-25). Similarly, of the nine assignments rated for Lifelong Learning, only one was

rated at the competent level and none of the student samples were rated at the competent level

(see Tables 8 - 10 on pages 26-28).

2. Opportunity to Discuss “Rigor” and What Constitutes a Competent Level assignment

In Tables 3-6 and 8-9 we see that there is general concordance between the rating of the

assignment by the faculty member who submitted it, and the ratings given that assignment by the

faculty at the Institute. It is worth noting, however, that samples ‘K’ in Tables 4 and 5 and ‘A’ in

Table 9 were rated as less challenging by the faculty at the Institute than by the faculty who

submitted them, and sample ‘H’ in Table 3 was rated more challenging by the faculty members at

the Institute than by the faculty member who submitted it.

iv | P a g e

3. Consistency

Table 11 indicates that there was consistency between; a) faculty at different tables who rated the

same sample and b) faculty ratings of their own assignments and the ratings given to those

assignments by their peers at the Institute.

The faculty members who attended the Institute were energized by the process and worked very

diligently. As a group they were also considerably more familiar with the Core Abilities in general than

was evident at previous Institutes. This increased awareness of the Core Abilities was particularly

evident among faculty who were attending for the first time. This reflects positively on the sustained

campus-wide efforts to increase awareness of, and buy-in for, Core Abilities at Olympic College. As a

result of their higher level of awareness and knowledge, we were able to assess more samples than had

been possible at the last Institute. Many participants expressed continued enthusiasm for integrating Core

Abilities into their classrooms and reviewing their assignments to improve their clarity and usefulness

vis-à-vis the integration of Core Abilities. With this enthusiasm and momentum in mind, curriculum

development will be the focus at the Summer 2012 Institute.

Feedback from Participants

An Institute Evaluation Form was distributed at the end of the second day. It was completed by

twenty-nine of the thirty-one participants. This evaluation tool and a complete list of the comments

received are included in Appendix V and VI. As well, Post-it Notes were placed on every table and

participants were encouraged to write suggestions and comments throughout the Institute. The Post-it

Notes were collected several times during each day and typed up at the end of the day by the two

Coordinators. For a full list of the Post-it Notes comments see Appendix VII.

The recommendations below are based on information gleaned from the Institute Evaluation

form, the Post-it Notes, observations made during the Institute by the two Coordinators, and data derived

from the rating sheets.

Recommendations

1. Make explicit the link between this work and Olympic College’s accreditation by the Northwest

Commission on College and Universities (NWCCU) and what accreditation really means to

Olympic College.

It is still evident that some faculty members do not see or understand the link between the work

being done on Outcomes and the stipulations placed on the college by our regional accrediting

commission. At the Winter 2012 Institute key passages from the most recent accreditation report,

v | P a g e

specifically those passages that pertain to the sample evaluation process should be included in the

packets3, to provide context for participants.

2. Create some Tegrity tutorials in order to streamline the work we do at the Institutes.

It was suggested in one of the feedback sessions during the Winter 2012 Institute that we convert

the Pre-Institute Workshop to a Tegrity lecture and require new participants to view the Tegrity

lecture before Day 1 of the Institute. This would allow all participants to start at the same time on

Day 1 and would enable new participants to view the material at their convenience and at their

own pace. It is therefore recommended that a Tegrity lecture is created during the Summer 2012

Institute. It was also suggested that a Tegrity lecture on the basics of Mapping would be helpful.

Key elements of this lecture might include: how mapping is done (a demonstration of the

database), why it is done, and some common errors and considerations

3. Streamline sample collection.

Submitting the samples takes a significant amount of time and effort on the part of the faculty

member. They must get student permission to use the work, make copies of the samples before

they grade them because graded samples cannot be used, select random samples, complete the

cover sheet, make a copy of the assignment prompt and then deliver the documents to the person

in charge of collecting them. Sample collection cannot occur at the beginning of the quarter

because faculty do not yet have any samples collected, and by mid-quarter they are already

sufficiently busy to collect samples If rating samples is going to remain a cornerstone of

Outcomes Assessment at Olympic College, and a broad range of samples is desired, other tactics

need to be employed to solicit samples. Although a stipend was suggested at the last Institute we

have learned that Institutional monies cannot be used for such incentives. Thus, in an effort to

improve the process these new strategies will be employed for sample collection in Fall, 2012:

i. Providing samples will become a condition of attendance at the Institute. Exceptions will be

made for faculty who teach in areas where providing samples may not be possible (library,

for example) and for new faculty who have never attended before.

ii. More detailed instructions will be provided to those submitting samples. This will make the

process of sample preparation and duplication more streamlined and efficient for the

facilitators. Specifically, participants will be asked to submit no more than 5 samples and

samples should be single-sided, paper-clipped rather than stapled, and all names must be

removed prior to submission.

3 See p.2 for a discussion of the packets, containing key documents, which are provided to each participant.

vi | P a g e

iii. An online drop box will be created so that faculty can submit electronically throughout the

quarter. A conversation has already been initiated with Kathy Bright to determine how best

to get this going – specifically which platform is utilized by the greatest number of faculty?

Angel, dropbox.com, Wiggio, Googledocs, or some other site?

iv. Faculty in online courses will be able to solicit student permission to use work via email. A

template for doing this has already been created and implemented by Elizabeth O’Neil in

math.

4. Consider having faculty who submit samples, rate their own samples.

Presently we ask faculty to rate the assignments they submit (see p.13-16 for sample submission

cover sheet). However, we do not ask them to rate the samples they submit. At the Winter 2012

Institute several faculty expressed interest in knowing how the faculty member who had

submitted an assignment had rated that assignment. Though it needs to be clarified for

participants that sharing this information is problematic, because samples are rated by multiple

tables and knowing how an assignment or sample is rated by the instructor before rating it oneself

defeats the purpose of the rating exercise, it would, nevertheless, be interesting and useful to

compare an instructor’s rating of the samples, with the ratings given by the Institute participants.

5. Explain to the campus community why stipends and free lunches are necessary for the

continuation of this work.

Faculty consistently report on the feedback survey that they would not continue participating in

the Institutes if a stipend were not offered. Yet, in these tough economic times, alternatives to

stipends must be considered. However, given the fact that the Institutes occur on non-contractual

days, that best practices stress that those involved evaluating student outcomes must participation

voluntarily (as opposed to, say, program evaluation by IPP which is mandatory) and also that the

time needed to conduct this work is extensive (two full workdays), it is not possible to undertake

it during the quarter while faculty are teaching and grading. Although one faculty member

suggests we cut the Institute to one day to save money, we believe that in order to thoroughly

evaluate the work we do, two full days of sample analysis is needed. Further, though it has also

been suggested that we cease providing lunch to save money or, instead, to pay higher stipends to

participants, the total cost per participant for lunch over two days including coffee and a beverage

with lunch is approximately $27. The risk in not providing lunch is that several faculty will

forget to bring food and will need to leave campus (because food services are not available during

vii | P a g e

breaks) to obtain food. This will be very disruptive to the whole group and will result in less

work being completed.

6. Reiterate to faculty that having their courses meet more or most of the Core Abilities is not

necessarily “better” in the eyes of the Administration or the Faculty Curriculum Team.

From the comments made at the Institute it is clear that some faculty members are still unclear

about the purpose of the mapping database and the implications of the entries made in the

database. As a result, there is still a widespread perception that the more Outcomes selected, the

more favorably the courses will appear. We need to explain, again, the importance to mapping

courses and that the mapping needs to be accurate.

7. Continue curriculum development as the central focus of the summer institutes.

In the feedback received at the Fall 2010 Institute there were numerous comments about the need

to vary the focus of the Institutes. Some faculty members reported feeling “burned out” on rating

samples or less than enthusiastic about being asked to do so at future Institutes. As a result it was

decided to focus on curriculum development at the Summer 2011 Institute. The response to that

Institute was overwhelmingly positive and, as a result, we have decided to devote future Summer

Institutes to curriculum development. There were several suggestions for the Summer 2012

Institute:

i. Collaborative assignment development – faculty members of the same division could work

on an assignment or assignment for a particular class such as English 101 or, alternatively,

faculty across several disciplines could work on a collaborative assignment.

ii. Developing grading rubrics – several participants said they would like to do this.

iii. Several faculty expressed an interest in learning how to collect data for Core Abilities

analysis in Angel

8. Feedback to those who submit samples merits ongoing monitoring.

Based on feedback from previous Institutes where an increasing number of faculty requested

feedback on their samples and assignments, we have been a lot more mindful about recording

which samples belong to which faculty member. As well, we added a “Suggestions for Faculty”

section at the bottom of each sample rating sheet (see the last section on p. 19-24 for examples).

As a result, we are now able to provide more detailed and specific feedback to those who submit

samples. We will continue to review and monitor the process of obtaining feedback and the kind

viii | P a g e

of feedback that faculty find most useful in order to improve the data we can provide to

participants who submit samples and are looking to improve their assignments.

9. Utilizing and sharing data collected at the Institutes.

On Day 1 of all Winter Institutes participants should be reminded of how data is collected at the

Institutes, how the data collected relates to the Core Abilities and the rubrics, and how such data

could be used to inform any decisions pertaining to significant changes to the curriculum. Also,

Institute Reports should be housed on the Core Abilities website not the X: drive as many faculty

cannot access the X: drive from home or find it very difficult to find documents on the X: drive

and thus prefer not to use it.

ix | P a g e

Appendix I

Agenda for Winter Institute 20119:00 9:50 AM: Orientation for New Attendees

10:00 to 16:00 PM Faculty Institute

Tuesday, December 13, 2011I. 0900-0950:

a. Overview history of the Assessment of Core Abilitiesb. Review of information contained in Institute Folders, mention x drivec. Core Abilities, definitions and explanations of rubricsd. Explanation and discussion of the goal(s) of the Institute

II. 1000: a. Welcome/Introductions b. Norming Sessions

i. Communicationii. Information Literacy and Technology

iii. Thinking

III. 1200 -1300 (60 minutes) Lunch

IV. Work with assigned tables to assess assignments and samples of student work.

V. 1500: Large Group Debrief (60 minutes)

Wednesday, December 14, 2011, 0900-1500I. Welcome & Review Schedule (5 minutes)

II. Respond to questions from previous Day (15 minutes)

III. Continue to work on assignments (2 hours 15 minutes)

a. 1030-1045: Break

IV. 1200: Lunch (45 minutes)

V. 1245 -1300, LibGuide: Sara Scribner

VI. 1300-1315, ANGEL© Data retrieval: Kathy Bright

VII. 1315-1415: Report back to group on accomplishments thus far

1 | P a g e

VIII. 1420-1450: Debrief

a. Discuss post-it notes

b. Discuss better ways to collect samples

IX. Evaluation and closing (10 Minutes)

2 | P a g e

Appendix II

DUE DECEMBER 1 2011

Contact Person (faculty name, please print): _____________________________(this name will be removed before samples are evaluated)

Course Number: _____________ Number of Samples4: ________ (example: SOC 190, MATH& 264) (Please submit 5-7 random samples)

If you would like feedback about what we learned after using your samples at the institute please indicate below:

Yes, I would like feedback No, I do not need any feedback

Please select one Core Ability that this work is most clearly demonstrating

Thinking Outcome 1: Graduates engage in critical analysis. What general level do you feel this assignment addresses (check only one box for each outcome)5:

Emerging Developing

Competent Strong

Thinking Outcome 2: Graduates engage in creative problem solving. What general level do you feel this assignment addresses (check only one box for each outcome):

Emerging Developing

Competent Strong Thinking Outcome 3: Graduates engage in quantitative reasoning. What general level do you feel this assignment addresses (check only one box for each outcome):

4 Please see the section “What NOT to Submit” on p.4 for some guidelines on the kinds of samples we can and cannot use.5 Please see any of the rubrics for a definitions of the standards expected at each of these four levels.

3 | P a g e

COVER SHEETFALL 2011 SAMPLES FOR ASSESSING CORE ABILITIES

Emerging Developing

Competent Strong

Communication 1: Graduates understand and produce effective oral communication. What general level do you feel this assignment addresses (check only one box for each outcome):

Emerging Developing

Competent Strong

Communication Outcome2: Graduates understand and produce effective written communication. What general level do you feel this assignment addresses (check only one box for each outcome):

Emerging Developing

Competent Strong

Communication Outcome 3: Graduates understand and use effective non-verbal communication skills. What general level do you feel this assignment addresses (check only one box for each outcome):

Emerging Developing

Competent Strong

4 | P a g e

Information Literacy and Technology 1: Graduates use strategies to search for information that enhance the acquisition of knowledge. What general level do you feel this assignment addresses (check only one box for each outcome):

Emerging Developing

Competent Strong

Information Literacy and Technology 2: Graduates evaluate and appraise sources. What general level do you feel this assignment addresses (check only one box for each outcome):

Emerging Developing

Competent Strong

Information Literacy and Technology 3: Graduates access and use information and/or technology ethically, legally and/or responsibly. What general level do you feel this assignment addresses (check only one box for each outcome):

Emerging Developing

Competent Strong

Information Literacy and Technology 4: Graduates use various inquiry tools and different formats of information e.g. media. What general level do you feel this assignment addresses (check only one box for each outcome):

Emerging Developing

Competent Strong

5 | P a g e

Information Literacy and Technology 5: Graduates use technology and information appropriate to field or discipline, synthesizing information to formulate insights and create knowledge. What general level do you feel this assignment addresses (check only one box for each outcome):

Emerging Developing

Competent Strong

Checklist for submissions:Please submit:

This coversheet completed in full 5 randomly selected samples 1 copy of the corresponding assignment The “Permission to Use Student Work” sheet on p.5 signed by the students

whose work you are submitting6

If you use a scoring rubric or grading criteria and/or checklist to evaluate student work, please include a copy of that evaluation tool as well.

Send submission is campus mail to:Mirelle Cohen [email protected]

What NOT to Submit

When selecting the 5 samples to submit please select random samples so that we can obtain a representative cross section of the student population. In short, we do not only need or want examples of the best work.

Do not submit graded work – only clean, unmarked copies of student work should be submitted.

Do not submit copies of multiple choice, True-False, short answer or matching tests.

Do not submit copies of entire student course or program portfolios or journals. Components of a portfolio or journal that demonstrate particular core ability may be submitted.

Do not submit samples in a foreign language or that cannot easily be understood by faculty outside your discipline (for example containing complex chemical or mathematical equations)

6 A simple way to do this is to have the whole class sign the Permission sheet and then you are free to randomly select from the whole class.

6 | P a g e

mailto:[email protected]

Materials will not be returned - do not submit original student work or work that you need returned for grading or other purposes.

Do not submit samples from developmental or ABE courses.

Do not submit samples without the corresponding assignment and Cover Sheet.

Thank you for your help with these efforts!

7 | P a g e

PERMISSION TO USE STUDENT WORKQuarter & Year: Instructor:

As indicated by my signature below, I authorize Olympic College to use my work as part of one or more Olympic College activities that are specific to campus-wide assessment for student learning outcomes. Your name and other identifying information will be removed.

PRINT NAME _________________________________________ SIGN NAME_____________________________ ________






8 | P a g e


Appendix III

TABLE #:_______

SAMPLE LETTER: _______

TABLE SUMMARY: _______

Please indicate outcome and the level of attainment the work is demonstrating (check all that apply):

Communication Outcome 1: Graduates understand and produce effective oral communication.

What general level does the team believe this assignment addresses (indicate the number of team members in box for each outcome level):

Emerging Developing Competent Strong

Communication Outcome 2: Graduates understand and produce effective written communication.



Communication Outcome 3: Graduates understand and use effective non-verbal communication skills.



9 | P a g e

EVALUATION TOOL-LIFELONG LEARNING STUDENT SAMPLE: WINTER 2011 EVALUATION TOOL-COMMUNICATION

ASSIGNMENT

Suggestions for faculty:

10 | P a g e

Table #:_______


TABLE SUMMARY: _______Please indicate outcome and the level of attainment the work is demonstrating (check all that apply):

Information Literacy and Technology Outcome 1: Graduates use strategies to search for information that enhances the acquisition of knowledge.



Information Literacy and Technology Outcome 2: Graduates evaluate and appraise sources.



Information Literacy and Technology Outcome 3: Graduates access and use information and/or technology ethically, legally and/or responsibly.



Information Literacy and Technology Outcome 4: Graduates use various inquiry tools and different formats of information e.g. media.



Information Literacy and Technology Outcome 5: Graduates use technology and information appropriate to field or discipline, synthesizing information to formulate insights and create knowledge.




11 | P a g e

EVALUATION TOOL-INFORMATION LIT & TECH ASSIGNMENT

WINTER 2011 ASSESSING CORE ABILITIES TABLE SUMMARY:

Table #: _______




Thinking Outcome 1: Graduates engage in critical analysis. What general level does the team believe this assignment addresses (indicate the number of team members in box for each outcome level):


Thinking Outcome 2: Graduates engage in creative problem solving.What general level does the team believe this assignment addresses (indicate the number of team members in box for each outcome level):


Thinking Outcome 3: Graduates engage in quantitative reasoning.What general level does the team believe this assignment addresses (indicate the number of team members in box for each outcome level):


12 | P a g e

EVALUATION TOOL-THINKING ASSIGNMENT WINTER 2011 ASSESSING CORE ABILITIES TABLE


EVALUATION TOOL-COMMUNICATION STUDENT SAMPLE

WINTER 2011 ASSESSING CORE ABILITIES TABLE SUMMARY

Table #: _______




Communication Outcome 1: Graduates understand and produce effective oral communication.



Communication Outcome 2: Graduates understand and produce effective written communication.



Communication Outcome 3: Graduates understand and use effective non-verbal communication skills.




13 | P a g e

EVALUATION TOOL-INFORMATION LIT & TECH STUDENT SAMPLE WINTER 2011 ASSESSING CORE ABILITIES TABLE SUMMARY:

Table #: _______




Information Literacy and Technology Outcome 1: Graduates use strategies to search for information that enhances the acquisition of knowledge.



Information Literacy and Technology Outcome 2: Graduates evaluate and appraise sources.



Information Literacy and Technology Outcome 3: Graduates access and use information and/or technology ethically, legally and/or responsibly.



Information Literacy and Technology Outcome 4: Graduates use various inquiry tools and different formats of information e.g. media.



Information Literacy and Technology Outcome 5: Graduates use technology and information appropriate to field or discipline, synthesizing information to formulate insights and create knowledge.




14 | P a g e

EVALUATION TOOL-THINKING STUDENT SAMPLE WINTER 2011 ASSESSING CORE ABILITIES TABLE

SUMMARYTable #: _______




Thinking Outcome 1: Graduates engage in critical analysis. What general level does the team believe this assignment addresses (indicate the number of team members in box for each outcome level):


Thinking Outcome 2: Graduates engage in creative problem solving.What general level does the team believe this assignment addresses (indicate the number of team members in box for each outcome level):


Thinking Outcome 3: Graduates engage in quantitative reasoning.What general level does the team believe this assignment addresses (indicate the number of team members in box for each outcome level):


15 | P a g e


Appendix IV

Core Abilities 2011 Winter Institute Data Table 1

Composite of Global Perspective Ratings:

Average Assignment Rating and Average Sample Rating(0=N/A, 1=Emerging, 2=Developing, 3=Competent, 4=Strong)

2.32.6

2.1 2.0

1.0

1.7 1.7 1.6 1.71.4

Global Perspective 1 Global Perspective 2 Global Perspective 3 Global Perspective 4 Global Perspective 5

5 Assignments 8 Assignments 4 Assignments 4 Assignments 1 Assignment

33 Samples 48 Samples 24 Samples 22 Samples 7 Samples

Assignment Ratings Sample Ratings

16 | P a g e


Composite of Lifelong Learning Ratings:


2.6

1.92.3

1.9 1.82.0

Lifelong Learning 1 Lifelong Learning 2 Lifelong Learning 3

3 Assignments 4 Assignments 2 Assignments

20 Samples 25 Samples 10 Samples


17 | P a g e


All Global Perspective 1 Assignments Rated:


2

33.3

1.4 1.3

1.7

2.4

1.41

3

2

1

N = 7 N = 7 N = 6 N = 6 N = 7

A G H I unknownNumber of Samples for Each Assignment and the Assignment Letter

Faculty Member's Rating (if included) Assignment Ratings Sample Ratings

18 | P a g e




2

3 3 3

2.5 2.5 2.5

1.3

1.9

1.01.3

1.5

2.21.8 1.7

3

2

33

2

N = 6 N = 7 N = 3 N = 7 N = 6 N = 6 N = 6 N = 7

C D E G H I K MNumber of Samples for Each Assignment and the Assignment Letter


19 | P a g e




3 3

1.71.4

2.2

1.61.4

2

3

2

N = 6 N = 6 N = 6 N = 6

H I J&L KNumber of Samples for Each Assignment and the Assignment Letter


20 | P a g e




3

2.4

1.0

2.3

1.3

2

1

2

3

N = 7 N = 3 N = 6 N = 6

B F I J&LNumber of Samples for Each Assignment and the Assignment Letter


21 | P a g e




1.41

N = 7

ANumber of Samples for Each Assignment and the Assignment Letter


22 | P a g e


All Lifelong Learning 1 Assignments Rated:


3

2.32.3

1.5 1.5

2

3

N = 7 N = 6 N = 7

A B CNumber of Samples for Each Assignment and the Assignment Letter


23 | P a g e




3

2.2 2.0

0.0

1.8

1

1.82

2.3

N = 7 N = 6 N = 7 N = 5

A B C DNumber of Samples for Each Assignment and the Assignment Letter


24 | P a g e




1.0

2.32.7

1

N = 5 N = 5

D (Multimedia Assignment)Number of Samples for Each Assignment and the Assignment Letter


25 | P a g e


CONSISTENCY OF RATINGS:Percentage with ratings differing by no more than one

Performance Level (for samples rated at more than one table)

82% 82%

67%62%

11 Assignments 53 Samples 6 Assignments 37 Samples

Global Perspective Lifelong Learning

ASSIGNMENTS AND SAMPLES RATED AT MORE THAN ONE TABLE

Additional Notes Regarding Consistency:

For one individual Lifelong Learning assignment, all seven samples were rated outside the consistency measure used above; that is, all seven samples had ratings differing by more than one Performance Level.

In the Winter Institute norming session, one assignment was rated by all eight groups. Faculty at six tables rated the assignment as Developing and two as between Developing and Competent. Following the rule of going with the lower value, all eight tables rated the assignment Developing, then.

Faculty members submitted their own ratings on 11 assignments. Of these, nine differed from the ratings at the Institute by no more than one Performance Level.

26 | P a g e

Appendix V

Winter Institute 2011 Evaluation Tool

1. Rate the likelihood that you will use the tools/information gained at the Institute, 1(not likely) to 5 (highly likely).

LibGuide 1. 2. 3. 4. 5.ANGEL E-Rubric 1. 2. 3. 4. 5.Core Abilities in syllabi

1. 2. 3. 4. 5.

2. Should students be required to acquire a “Level” of proficiency in a Core Ability to meet the graduation requirement? Check one: ____Yes ____No

3. If you answered yes to the previous question, circle the “Level” of proficiency that should be required.

a. Emerging b. Developing c. Competent d. Strong4. Should the core abilities be tied to a grade requirement?

a. No; would still need to meet current requirement of 0.7b. Yes, please specify ______c. Yes, 0.7 or higher, with average GPA of 2.0d. Other: ________________

5. Do you believe this event was a worthwhile endeavor? □ Strongly disagree □ Disagree □ Agree □ Strongly Agree □ No opinion

6. Would you participate in a similar event in the future?□ Strongly disagree □ Disagree □ Agree □ Strongly Agree □ No opinion

7. What additional training and/or information would help you better understand the Core Abilities and the process set up to assess them?

a. Course Mapping workshops b. OC Web site information and linksc. Other:

8. Do you have any suggestions that could help improve this event? (circle one)

9. What was the one most useful or meaningful thing you learned from participating in this event

10. Please share suggestions for improving the process of sample collection for the Faculty Institute:

27 | P a g e

Appendix VI

Winter Institute 2011 Evaluation Tool Responses

1. Rate the likelihood that you will use the tools/information gained at the Institute, 1(not likely) to 5 (highly likely).

1. 2. 3. 4. 5.

LibGuide 1 1 9 5 13

ANGEL E-Rubric

5 3 4 5 9

Core Abilities in syllabi

1 1 3 6 18

2. Should students be required to acquire a “Level” of proficiency in a Core Ability to meet the graduation requirement? Check one: __22__Yes __7__No

3. If you answered yes to the previous question, circle the “Level” of proficiency that should be required.

b. Emerging b. Developing c. Competent d. Strong1 9 10 0

1x depends on the outcome/level of class and student

4. Should the core abilities be tied to a grade requirement?e. No; would still need to meet current requirement of 0.7 4f. Yes, please specify ______ 0g. Yes, 0.7 or higher, with average GPA of 2.0 12h. Other: ________________ 2.0=6; 1.0=2, 2.7=1

5. Do you believe this event was a worthwhile endeavor? □ Strongly disagree □ Disagree □ Agree □ Strongly Agree □ No opinion

3 0 2 21 1

6. Would you participate in a similar event in the future?□ Strongly disagree □ Disagree □ Agree □ Strongly Agree □ No opinion

3 0 2 26

7. What additional training and/or information would help you better understand the Core Abilities and the process set up to assess them?

28 | P a g e

a. Course Mapping workshops 6b. OC Web site information and links 4 (suggestion: move everything off the X

drive)c. Other:

Examples of rated assignments and rated student work Learning to link them on Angel and more about grading rubrics. Videos that give the basic principles on mapping – for instance host a variety on the “how tos”

and include in a Lib Guide for viewing or an Angel classroom Developing grading rubrics. Collaborative assignment development. I would like to work with others on course/discipline/division descriptions of the outcomes. Grading rubric tied to core abilities. Examples of Core Abilities in a wide variety of syllabi. The OC website should have more links to rubrics etc. Overall accreditation process and how this all fits (MC note: do presentation on accreditation

recommendations next time).

8. Do you have any suggestions that could help improve this event? (circle one) Have assignments turned in and then devote an institute to evaluating each to see what Core

Abilities that the group feels are not covered and then share with the instructor. 1 day 8-5. Save money. Share instructor’s perspective on how they rated the assignment. More pay, less food. Less student samples per assignment, better organization of norming materials, and

agenda/schedule for both days. Focus on specific courses from list that meet a core ability requirement. More time with samples, deliberately include instructors’ own samples. Good to have instructors engaged in the discussions about their assignments.

9. What was the one most useful or meaningful thing you learned from participating in this event? Networking and seeing assignments from other disciplines. Working with other faculty and seeing their ideas and opinions. Stronger connection efforts among core abilities, assignment, and student work. The opportunity to meet other faculty was invaluable and the chance to see student work in other

disciplines. I could create curriculum that emphasizes core abilities The Info Lit and Technology rubric might need work The Info Lit and technology Core Ability needs to be split into two Always going more confidence in rating assignments and samples regardless of Core Ability

evaluated – helps me understand Core Abilities individually better It’s OK to vary perspective. Still feeling this all has to be black and white. Better understanding of how rubrics work across disciplines. Core ability Web Page Site. That there are many faculty, from many disciplines, willing to work together to improve OC Application of rubric to assignments. Collegial fellowship with other instructors was valuable. Difficulty of creating assignments which measure the core abilities you are interested in

assessing. I enjoyed seeing how various assignments were rated i.e. getting broader context for assignments. Evaluating the rubrics. What Core Abilities are.

29 | P a g e

Importance of having first time attendees. This was my second event and it made more sense this time. Overcoming confusion with terms/ideas of rubrics. Working with new faculty. We had only 3 people in our group which made it easier than when

I’ve worked in groups of 4. How to assess my own assignment for core abilities. Many things. So good just to think about my own assignment in relation to Core Abilities and

other instructors’ assignments.

10. Please share suggestions for improving the process of sample collection for the Faculty Institute: Start January 3rd – send out the checklist. I like the idea of holding over the samples we didn’t get to use at this session Get the Deans involved I would like to see an institute focus on a specific class such as Engl 101 or other classes An online drop box – that way we can submit assignment all quarter Start now for spring Have those who submit give some background/context for assignment Make sure those who submit give context of assignment. When in quarter given and teacher

prompts etc. Collect 1 quarter apart – collect in spring to be used in summer or later so no rush because you

could use material you have already. Have professors give some background as to what they communicate orally before giving the

assignment or other assignments previously to help them explain the background for the assignment

Limit to 6 samples, 8 was excessive, really 5 is enough. Shorter samples. Approach faculty who have “famous” or renowned assignments in a discipline and solicit

samples of there (flatter!) Comprehensive checklist for what to include including context for assignment Include contextual information from instructors about the assignments i.e. when assigned in the

quarter, rubrics used to evaluate the assignment Ongoing collection throughout the quarter; online submission guidelines; online submission of

samples Email specific instructors, stress that samples should be single sided, remove names and no

staples Give instructions EARLY and have faculty sign up EARLY. Have faculty sign up for one Core

Ability per assignment. Suggest an opt-in option for assignments Reminder early in the quarter Ask for electronic submissions, suggest including permission in syllabus Be firm about one page, no staples, no names. No reason why people cannot do this.

Appendix VII

30 | P a g e

Post-Its Day 1, December 13, 2011 Is the goal to create all assignments to give the opportunity to be at the "strong" level? Would clickers help with norming exercise? Create packages for the norming exercise so that there isn’t so much paper shuffling at the

beginning Thinking Assignment norming: The faculty whose assignment this is should include instructions

telling students to quote the source text 3x It would be useful to know what point in the quarter the assignment took place (context of

assignment) 3x Color code evaluation tools: table, assignment, samples Provide more context: when assignment was given, course pre-reqs, and other background info

that would help raters Instructors should provide clarification in terms of discipline specific words/phrases Remove Strong as a possible choice for Core Abilities – most classes never reach Strong Less student work to evaluate and more assignments please For each outcome bring in a student sample at each level. This is what “emerging” looks like;

this is what “developing” look like etc. Grading rubrics are really helpful if people could be encouraged to include them with samples Why are individual evaluations labeled as “Table summary”? Shouldn’t it be “Individual

Summary? Results of the Institute: It would be beneficial to participants to see what the “Product” of the

Institute’s 2 days of work looks like Record the Day 1 Orientation with Tegrity for new attendees to view before coming (or others to

review) – could save time on Day 1 Make N/A an actual option on the worksheet – writing N/A looks messy Literacy & Technology Outcome 2 “emerging”: determining whether sources are credible or

reliable should be at a higher level than emerging Could we have everyone complete a cover sheet that includes:

CourseMost relevant Core abilityBackground of assignmentRole of assignment in the course (introduce ideas/summarize ideas/synthesize lessons etc.)Time assignment is give (beginning or end of quarter)Additional Info (were there other/more requirements listed/discussed elsewhere?)Preparation of assignment (ties in with background – did students get additional/specific lessons like library instruction etc.)

31 | P a g e

Post-Its Day 2, December 14, 2011 After groups are summarized as to how they rated the assignment, could you share the actual

rating indicated by the instructor that submitted the samples? Including a checklist for what the assignment and samples should be like wouldn’t feel like a

burden to me. It would help to have targets and direct information: No staples, copy single sided, give context for assignment. It would feel like less work, not more.

Put final report on Google docs not X Drive and sent separately to Institute participants Where do assignment that focus on primary research fit (i.e. asking a student to observe a clinic

or interview an expert?). While there is an end product (written or oral report) the assignment is more about the process – creating questions, finding the subject to observe or interview (a valid source). We did not agree about whether it had to include technology to fit IL&T or not (See Outcome 2 about sources, and developing inquiry questions).

IL&T rubric: The rubric doesn’t address information/data that a student generates himself/herself, which seems to be a major gap.

If assignments include presentations with the potential for multi-media, use audio or Power points that include charts, graphs, images, etc. PDFs would be helpful to include with the assignments

Have instructors identify ONE or two outcomes that are most relevant or important – some of these felt like a stretch – e.g. doing the bibliography exercise for Thinking instead of IL&T

It would be good to know what outcomes on the Core Abilities the instructor intended for the assignment.

Summer Institute – if you want to split curriculum development with evaluation, instructors could evaluate student work/assignment then use the second day to revise the assignment.

Sara Scribner, Library: Core Ability Communication Outcome 2: “understand and produce effective written communication” – could this be expanded to include interpretation of things other than texts? E.g. Nurses observe clinical interaction and are required to identify information (also the OLRM example had students observe in the workplace), where would that type of communication come in?

32 | P a g e

Date post:	04-May-2020
Category:	Documents
Upload:	others
View:	2 times
Download:	0 times

The Role of Faculty Institutes in Improving Student … · Web viewThe Role of Faculty Institutes...

Documents