1

The Power of Peer Evaluations: Why Don’t They Get More Respect?

Allen I. Kraut Professor Emeritus of Management Baruch College, CUNY

Presented at METRONew York, NYMay 8, 2013

2

3

4

Example of Peer Assessment In Industry

Reginald Jones, GE CEO in 1978, asked key executives:“Suppose you and I were killed in a plane crash this week. Who should be the next chairman of GE?”

Answers: “Jack Welch”… “Jack Welch “… “Jack Welch”

Recounted in J. Sonnenfeld’s The Hero’s Farewell (1991) from report in R. Vancil’s Passing the Baton (1987) .

5

Non-use of Peer Appraisal SystemsIs “ A Curious Paradox”*

“Long been known … Peer assessments… are among the most accurate assessments….predicting future performance….

Yet only a handful of organizations are using them.”

* From Latham &Associates (1993)

6

Three Recent Talent Management Books,With Not One Index Entry on Peer Evaluation

7

Research* Shows Peer Evaluations Useful in Predicting Success

Have High Reliabilities (in .80s) Even after short acquaintanceship Consistent across groups

Good Validities (.30s & 40s) For Training, Promotion, & Job Performance Unaffected by Friendship Patterns Same whether Research or Administrative Use Better than Staff/Observer ratings

*See Kane & Lawler, (1978), Downey, R. G., & Duffy, P. J. (1978).

8

“Modest” Correlations Have Powerful Practical EffectsExample for r=0.30 *

Predictor Outcome

Low High

High

Low

• From R. Rosenthal & R. L. Rosnow, “Essentials of Behavioral Research” (1984), on Binomial Effect – Size Display, pp.209-211

65%65%35%

35%65%

9

Peer Evaluations Have Higher Predictive Validity Than Most Other Measures

For Overall Job Performance

Top Measures (Partial list)* Validity (r)

General Mental Ability .51 Peer Ratings .49 Interview (Unstructured) .38 Assessment Center .37

* From Meta-Analytic study by F. L. Schmidt & J. E. Hunter (1998), Psych. Bulletin, on 19 predictors

10

Peer Evaluations: Three Types

Peer Nominations: Name Top People

(and Sometimes the Bottom Ones)

Peer Ratings: All Judged on a Common Scale

Peer Rankings: Listed from Highest to Lowest

11

Study #1

A Powerful and Simple Way to Predict Executive Success:  Results From a 25-Year Study of Peer Evaluations

12

Evaluation in Month-Long Training Program, At Integrated Fortune 100 Corporation

Middle Managers: (“Minors, Triple AAA”)

N = 184, 2nd & 3rd Level Managers Class size = 16

Executives: (“Major League”)

N = 99, higher level (titles like Director, V-P)

Class size = 12 From All Divisions and Functions Attended Program 1967 through 1968 Peer Ratings Done in Third Week

13

Form of Evaluation

Rating (Forced)

Mean= 2.93HI LO SD=

0.77

1 2 3 4 5

14

Participants’ Reactions to Peer Evaluation

Two Major Complaints:

1. Great deal of work to rate all course-mates on many attributes (11, plus 2 “Overall”)

2. Resent giving low ratings to fellow attendees

15

Two Overall Predictors

General Impression“Considering all of the (specific) factors, how would you rate this person?”

Capacity for Advancement“Indicate your judgment of the potential of this person for the position of general manager of a new (company) division or its corporate staff equivalent?”

Implied administrative set, but actually locked up for research

16

Development of Nomination Rating

What if we asked only for top nominations? Smaller number of ratings No “low” evaluations

To develop a “nomination” rating:- Count as “high” any rating of “1” or a “2” - For each person, divide number of

nominations by maximum possible

17

Two Forms of Evaluation

Rating

Mean= 2.93HI LO SD= 0.77

1 2 3 4 5

Nomination

Mean= 0.30SD=0.26

HI 1 0 LO 1 2 3 4 5

18

Peer Ratings and Peer Nominations Are Highly Correlated

Rating Vs.

Nomination

For Total Sample, N = 283

General Impression 0.92

Capacity for Advancement 0.93

19

Study # 1

Used Two Criteria of Success

After 25+ Years Highest Career Level Achieved? Became a Corporate Officer? (Less than One per 5,000 employees)

20

Peer Evaluations PredictHighest Career Levels Achieved

Correlation with Highest Level

Middle Managers Executives

Attribute Rating Nomination Rating Nomination

General Impression .43* .38* .29* .26*

Capacity for Advancement .42* .38* .26* .21

p < .05; Note: evaluations were not related to year of leaving company.

21

A Criterion of Really Long-Term Executive Success

Only 5.7% Became Corporate Officers (16 of 283)

These are the “Major Leaguers” Who Made it to The “Hall of Fame”

22

“High” Nominees More LikelyTo Become Corporate Officers

Correlation With

Attribute Rating Nomination

General Impression .22* .23*

Capacity for Advancement .25* .24*

* p < .05, N = 283

Note: Maximum possible r estimated at .49

23

Top Nominations on “Capacity for Advancement”Much More Likely to Become Corporate Officers

0 2 4 6 8 10 12 14 16

Bottom 30%

Middle 40%

Top 30%

Became Corporate OfficerPeer Nomination

PERCENT

Total (n=283) = 5.7%

14.1%

2.7%

1.1%

24

Another Look at Future Corporate Officers

6%

19%

75%

Their Peer Nominations Were

In Top 30%

In Middle 40%

In Bottom 30%

Of Those Who Became Officers (n=16)

25

An Army Study of Long-Term Success, Shows Peer Nominations Valid PredictorsR. J. Gerard, 2002

370 USMA Grads, Tracked 1976-2000 Peer Nominations of Leadership (top ¼; bottom ¼)

Better than Tactical Officer Ratings, Cadet Rank, or GPA

In Predicting Career Success Highest Rank Achieved (r = .33**) Top Service School Attended (r = .30**) Largest unit Commanded (r = .33**)** = <.01

26

Study # 2, Same Industry Population, Shorter Time, Differing Criteria

After Two Years Promotion (in Levels) Performance Appraisal

After Eight years Promotion (in Levels) Performance Appraisal Pay Increase (%)

27

Peer Ratingsa Predict Short-Term Success,But Vary By Criteria, and Population

a almost identical to peer nominations

Criterion Correlation with Criterion Attribute Middle Managers Executives

Levels Moved: M = 1.75, SD = 1.55 M = 1.00, SD = 1.75b

Promotions (2 Yrs.) General Impression .38* .10Capacity for Advancement .41* .12

Performance Appraisal (2 Yrs.)General Impression .12* .43*Capacity for Advancement .14* .37*

Pay % Increase (After 8 Yrs.)General Impression .41* .23*Capacity for Advancement .41* .24*

*p < .05; Note: evaluations were not related to year of leaving company. B Wide due to demotions

28

Peer Ratingsa Prediction of Managers’ Mid-Term Promotions Illusory?

a almost identical to peer nominations

Correlation with Promotions

After 2 yrs. After 8

Yrs. Attribute General Impression .38* .41*

Capacity for Advancement .41* .42*

After 8 Yrs.ControllingFor First 2 yrs.

.12

.17

Conclusion?•Peer Prediction most Valid in Fixed Short-Term

•Career Movement like a Tournament in Short-Term, But a Marathon over Long-Term

*p < .05

29

Overall Conclusions

Again, Peer Evaluations Predict Career Success Peer Nominations as Effective as Peer Ratings

Are Simpler Avoid Devaluing Colleagues

Proof of Effectiveness Varies with Situation Basis of Selection into Group Homogeneity of the Group Criteria Used, Restriction of Range

Peer Evaluations Yield Major Benefits in Near-Term Help Put Light on Promising Long-Term Candidates

A Simple, Easy Method Can Be Quite Powerful!

30

Some Limitations of This Research

Setting in a Large Stable Hi-Tech Firm Long Careers in Same Company All Were Male Some Data NA, Pay Grades Shift to “Bands”

Note: Data are “Old” is Not a Limitation

31

Today’s Organizational Conditions Favor Peer Evaluations For Selection

Greater Emphasis on Teams Group’s Opinion More Important Than in Past Peer Acceptance is Critical

Less Time to Develop Track Records Peers Can Make Relevant Judgments

Real Issue may be “Political,” not Scientific

32

Recommendation: Use Peer Nominations More Often

Include in Training Programs Ask Only For Top Nominations Do Not Confuse with “Feedback For Development”

Use in Formal Assessment Centers Use in 360 Evaluations (not Just Development) In Current Organizational Hierarchies (Gingerly) Include Data in Executive Succession Planning Word Evaluations for Specific Criteria Desired

(“to Head Division X, Start New Product Line, etc.) Respect Evaluation’s Short “Shelf Life” (2 to 4

Years)

33

An Example: Peer Ratings Are Used For Promotion Of State Troopers in Minnesota

Ratings on Seven Scales (on BARS) Also, "I Would Feel Fine Reporting to Him/Her?” Used together with Supervisor Ratings, and

Tests: Analytic Skills, Work Styles; Experience

Reactions? Peer Input seen As Valuable Candidates Comfortable with This

Source: Ron Page

34

35

If Peer Evaluations Are So Good, Why Aren’t They Used More??

Maybe They Are Being Used, and I Don’t Know?

Some Other Possibilities...?

36

Don’t Show Up in Top Journals, Because They Lack “Theory”

“In general, journals publish validity studies only when a case can be made for a contribution to scientific understanding.”

Paul Sackett P. 773, In J. C. Scott and D. H. Reynolds, 2010

37

Squeezed Out by Other TopicsIn Today’s Zeitgeist?

Some Examples:

Work-family balance Emotional Labor Workplace Incivility Employee Engagement Organizational Citizenship Behavior

38

Profitability to Con$ultant$ ??

Proven, but Simple, Powerful Method Vs. More Complex, Larger Efforts

Competency Modeling 360 Feedback Programs Talent Management Programs Et Cetera

39

‘Political’ Concerns Management Giving Up Their Power?

Friendship Biases, “Popularity” Contests?

Get Divisiveness, Peer Competitiveness?

Reaction to Changing Rules of Game?

Individualistic American Culture Hostile to Peer Inputs?

40

Your Thoughts??

…and Thank You!