1

Assessing Faculty

By “Sitting Beside”

Larry A. Braskamp

Loyola University of Chicago

Lbraska@luc.edu

2

Assess ( e-ses’) v.t.

• …To take stock of; evaluate; to assess the situation

• … L assidere sit by (as a judge in court)…

• Source. Funk and Wagnalls New International Dictionary of the English Language, 1993

3

To Assess is to

• Understand

• Judge

• Act (decide, assist, help develop)

4

Challenge

• Design and implement a faculty assessment program that simultaneously fosters individual faculty development and fulfills the institutional mission

5

Assessing by “Sitting Beside”

“To SitBeside”

UsingEvidence

SettingExpectations

Collecting Evidence

•Source. Braskamp, Larry A. & Ory, John C. (1994) Assessing Faculty Work. San Fransisco: Jossey-Bass

6

“To Sit Beside” as an image

• To humanize the process• To understand as well as to judge• To enhance role of colleagues• To build community• To increase respect for diversity• To demonstrate individual accountability• To promote mutual and collective accountability

7

“To SitBeside”

UsingEvidence

SettingExpectations

Collecting Evidence

Assessing by “Sitting Beside”

8

Setting Expectations• Division of faculty work

– Teaching– Research and Creative Activity– Outreach/Professional Practice/Engagement– Citizenship

• Individual and Institutional• Responsibilities vs. quality• Quality/value/impact/influence

– Merit– Worth

9

Setting Expectations

• To expand the variety of faculty work (roles and responsibilities) to be recognized and rewarded (e.g., teaching; research and creative activity; outreach; citizenship)

10

The Work of Teaching

• Instructing

• Advising, Supervising, Guiding, and Mentoring Students

• Developing Learning Activities

• Developing as a Teacher

11

The Work of Research and Creative Activities

• Conducting Research

• Producing Creative Works

• Editing and Managing Creative Works

• Leading and Managing Funded Research and Creative Projects

12

The Work of Practice and Professional Service

• Conducting Applied Research and Evaluation

• Disseminating Knowledge• Developing New Products, Practices,

Clinical Procedures• Participating in Partnerships with Other

Agencies• Performing Clinical Service

13

Characteristics of Public Service

• They contribute to the public welfare or the common good

• They call upon faculty members’ academic and/or professional expertise

• They directly address or respond to real-world problems, issues, interests, or concerns

14

The Work of Citizenship

• Contributing to the Local Campus

• Contributing to Disciplinary and Professional Associations and Societies

• Contributing to civic, political, religious, and other communities

15

Setting Expectations

• To distinguish between expectations for (institutional) and expectations of (individual) faculty

16

Setting Expectations

• To distinguish between what one does and how well one does it, i.e., workload/

activities/ effort/ roles/ responsibilities

vs.

quality/ effectiveness/ influence/

impact/value/ merit/worth/excellence

17

Setting Expectations

• To distinguish between merit (“quality according to standards of the profession”) and worth (“value of work that is a benefit to the institution”) M. Scriven.

•Source. Scriven, Michael (1978) Avalue versus Merit. Evaluation News (8), 1-2

18

“To SitBeside”

UsingEvidence

SettingExpectations

Collecting Evidence

Assessing By “Sitting Beside”

19

Collecting and Organizing Evidence

• Multiple Perspectives

– Sources

– Methods

• Credibility of Evidence

• Trustworthiness of Evidence

– Validity

– Reliability

– Fairness

– Consequences

• Portrayal of Faculty Work

• Building a Case

20

Collecting and Organizing Evidence

• Everything counts, but not everything needs counting

• To think in terms of “building a case”

• To emphasize a multiple perspective approach to collecting evidence from multiple sources using multiple methods

21

Collecting and Organizing Evidence

• Evidence must be credible

• Evidence must be trustworthy (i.e., valid, reliable, fair, consequential)

• Faculty need to be able to portray their work for others to review, critique, and judge

22

Multiple Sources• Oneself• Faculty colleagues• Campus administrators• Faculty development professionals• Students• Parents• Participants• Alumni• Citizens and community groups• Public officials• Professional and disciplinary colleagues• Accreditation officials• Board members• Consultants• Experts• Customers

23

Multiple Methods

• Rating Scales

• Observations

• Interviews

• Written appraisals

• Measures of outcomes and achievements

• Documentation and records review

• Measures of eminence, quality, and impact

• Video and audio tapes

• Simulations

24

Types of Evidencefor Describing and Judging Teaching

• Descriptions of Teaching Activities– Summary of responsibilities and activities– Analyses of student learning and challenges– Audio and videotapes– Samples of teaching– Participation in improvement activities

• Outcomes– Student learning and achievements– Student development

25

Types of Evidence for Describing and Judging Teaching

• Judgments About Teaching– Ratings from various sources– Written appraisals from various sources

• Eminence Measures– Honors and awards– Invited presentations

• Self-Reflections and Appraisal– Personal journals and logs– Public self-appraisals

26

Types of Evidence for Describing and Judging Research

• Descriptions of Research and Creative Activity– Summary of responsibilities and activities– Analyses of research and creative problems– Participation in improvement activities

• Outcomes– Publications in journals– Papers presented at professional meetings– Books (authored and edited)– Chapters in books– Monographs– Grants and external funding– Unpublished papers and reports

27

Types of Evidence for Describing and Judging Research

• Judgments about Research

– Evaluations from faculty peers

– Evaluations from departmental chairs, deans

– Evaluations from experts (curators, critics)

• Eminence Measures

– Referee or editor of journal

– Honors and awards from profession

– Officer of national professional association

– Invited papers and guest lectures

– Invited exhibitions and performances

– Citation rate of published work

• Self-reflection and Appraisal

– Personal journals and logs

– Public self-appraisals

28

Types of Evidence for Describing and Judging Practice

• Descriptions of Practice Activities– Analyses of contemporary problems– Audio and videotapes– Samples of work– Participations in improvement activities

• Outcomes– Client feedback on progress– Client behavioral outcomes– Degree social problem addressed is understood– Policy changes linked to work of faculty– Influence on research and teaching within profession– Influence on teaching and research within institution– Inventions, improved clinical practices and procedures

29

Types of Evidence for Describing and Judging Practice

• Judgments about Practice

– Evaluations from participants, clients, patients

– Evaluations from sponsoring organizations

– Evaluations of and letters of appreciation

– Evaluations from faculty colleagues and experts

• Eminence Measures– Honors and awards from profession

– Officer of professional association

– Invited exhibitions and performances

• Self Reflection and Appraisal– Personal journals and logs

– Public self-appraisals

30

Types of Evidence for Describing and Judging Citizenship

• Descriptions of Activities– Attendance records of committee work– Representation at functions for institutional

advancement– Support of campus activities (cultural and sporting

events)– Degree of involvement in professional organizations– Degree of participation in religious/public/civic affairs

• Outcomes– Changes in policies in governance of campus and

professional associations

31

Types of Evidence for Describing and Judging Citizenship

• Judgments About Citizenship– Ratings of effectiveness by faculty peers and administrators– Evaluation by fellow committee members and chair– Modeling behavior as judged by colleagues and students– Evaluation from participants of community programs, public

officials• Eminence Measures

– Reappointment or reelection to public office– Reelection or reappointment to leadership positions

• Self-reflection and Appraisal– Personal journals and logs– Public self-appraisals

32

Administration Influencing Student Ratings

of the Instructor or Course

Student anonymity -- Signed ratings are more positive than anonymous ratings.

Instructor in classroom -- Ratings are more positive if the instructor remains in the room.

Directions -- Ratings are more positive if the stated use is for promotion.

Timing -- Ratings administered during final exam are generally lower than those given during class.

Midterm -- Ratings are less reliable if the student raters can be identified.

33

Nature of Course Influencing Student Ratings

Required/elective -- Ratings in elective courses are higher than in required courses.

Course level -- Ratings in higher-level courses tend to be higher that in lower-level courses.

Class size -- Smaller classes ten to receive higher ratings, yet low Correlations between class size and student ratings suggest class size is not a serious source of bias.

Discipline -- In descending order, lower ratings are given to courses in arts and humanities, biological and social sciences, business, computer science, math, engineering, and physical sciences.

34

Instructor Characteristics Influencing  Student Ratings

Rank -- Professors receive higher ratings than teaching assistants.Gender of instructor -- No significant relationship exists between gender of instructor and his or her overall evaluation, although ratings do slightly favor women instructors.Personality -- Warmth and enthusiasm are generally related to ratings of overall teaching competence.Years teaching -- Rank, age, and years of experience are generally unrelated to student ratings.Research productivity -- Research productivity is positively but minimally correlated with student ratings.

35

Student Characteristics Influencing Student Ratings

• Expected grade--Students expecting high grades in a course give higher ratings than do students expecting low grades.

• Prior interest in subject matter---Similar to elective courses, students with prior interest give somewhat higher ratings.

• Major or minor---Majors tend to rate instructors more positively than nonmajors.

• Gender---Gender of student and overall evaluations of instructors are not related although students tend to rate same sex instructors slightly higher.

• Personality characteristics---No meaningful and consistent relationships exist between the personality characteristics of the students and their ratings.

•  

36

Instrumentation Influencing Student Ratings

• Placement of items---Placing specific items before or after global items on the rating form has insignificant effect on the global ratings.

• Number of scale points--Using six-point scales yields slightly more varied responses and higher reliabilities than five-point scales.

• Negative wording of Items--Overall ratings of the course and instructor are not significantly influenced by the number of negatively worded items in the rating scale.

• Labeling all scale points versus only end-points---Labeling only end-points yields slightly higher average ratings

37

Assessing by “Sitting Beside”

“To SitBeside”

UsingEvidence

SettingExpectations

Collecting Evidence

38

Using Evidence

• To distinguish between individual and institutional uses of evidence

• “Assessment is everybody's business, but not everybody else's business” R. Stake

• To promote both individual and mutual and collective accountability

39

Using Evidence for Self Development

• Emphasize the informational rather than controlling use

• Design assessment feedback so that it is intrinsic to the task itself

• Rely on specific, diagnostic, descriptive information that focuses on faculty work

• Encourage feedback on work in progress• Develop mentoring relationships among faculty so

that discussions of work are encouraged

40

Using Evidence for Accountability

• Rely on various types of descriptive and judgmental evidence collected from multiple sources to develop a composite portrayal

• Interpret evidence in a way that is consistent with institutional goals

• Develop profiles of faculty over time• Closely link assessment with both faculty and

institutional development

41

A Portrayal of Faculty Work

• Statement of personal goals, roles and institutional expectations

• Teaching– Responsibilities and Activities– Assessment and its Use

• Research and Creative Activities• Practice• Citizenship• Honors and Recognition• Activities to Improve Faculty Work

42

Creating a Campus Culture of Assessment

• Do you collect evidence about your work solely for your own personal and professional use?

• Do you share this evidence with colleagues so that you can discuss your own effectiveness, enhance your own career development, and meet institutional expectations?

43

A Possible Reorganization

• Does your current organizational structure most effectively:

– Foster development of your faculty and staff?

– Fulfill the mission of your institution/college/school/department