Psychological Measurement in Psychological Measurement in IndustryIndustry

What Do Industrial-Organizational What Do Industrial-Organizational Psychologists Do?Psychologists Do?

Industrial-organizational psychologists study organizations and seek ways to improve the functioning and human benefits of business.

Psychology at WorkPsychology at Work

I/O Psychologist Improve Organizational Functioning by:

• Recruiting people that best fit your organization• Selecting the best people• Retaining the best people• Developing fair, legal, and efficient hiring procedures • Improving the skills of the people • Creating a diverse, qualified workforce

Selection ProcessSelection Process• Measure applicants’ qualifications

• Select the best applicant to hire

• For each selection method:

–Describe the selection method

–Rate the validity of the selection method:» Poor: validity coefficient = r ≈ .00

» Moderate: validity coefficient = r ≈ .25

» Good: validity coefficient = r ≈ .50

» Great: validity coefficient = r ≈ .75

• Evaluation of process

Workplace Testing SettingsWorkplace Testing Settings

• The government & the military

–90% tested for federal jobs

–80% tested for state, county, local government

–Largest amount of testing in military

–Over 1 mil. per year take ASVAB

–Military: emphasis on placement not selection

Workplace testing settings (cont.)Workplace testing settings (cont.)

• Testing for professional licensing

–Over 2,000 occupations require licensing

– In some states 25% of workforce is licensed

–Require written & oral examinations

–Tests prepared by licensing boards (local & national)

–National exams better than local

–“State of near chaos”

–For the same job – diff’t exams, diff’t requirements

–Little overlap between states

Workplace testing settings (cont.)Workplace testing settings (cont.)

• Private organizations–Frequency of testing varies

–Growing interest in testing w/ ease of web-based testing

»Applicants are administered tests online

»Info goes directly to hiring manager, no feedback to test-taker

»E.g., Target, Blockbuster

Approach for Examining Selection Methods

• Describe the selection device and the information that can be collected about applicants

• Describe how to develop and use the device appropriately

• Describe the frequency of use, reliability, validity, costs, adverse impact, and face validity (applicant reactions) of each device

• Depth and breadth of selection plan

–Costs, who is involved, time frame, which jobs

–Internal vs. external selection

Selection MethodsSelection Methods

• Ability Tests

• Job Knowledge Tests

• Performance Tests and Work Samples

• Personality Tests

• Integrity Tests

• Structured Interviews

• Assessment Center

Application Blanks & RApplication Blanks & Résumésésumés

• Routinely used

• Focuses on basic factual information:

–Education, training, work history, skills, accomplishments, etc.

• Used to screen out applicants who don’t meet minimum qualifications in terms of education, experience, etc.

Application BlanksApplication Blanks

• Validity: poor (typically r < .20)

–Why?

• Problems:

–Lack of agreement what to look for

–Possible discrimination

–Need to cross validate

–Keys are not stable over time, need to update

Biodata QuestionnairesBiodata Questionnaires

• Questionnaire on applicant’s life experiences

–Example questions:

»Did you ever build a model airplane that flew?

»When you were a child, did you collect stamps?

»Do you ever repair mechanical things in your home?

–Answers are scored according to a scoring key

• Validity: moderate (r ≈ .30)

Experience & Accomplishments Experience & Accomplishments QuestionnairesQuestionnaires

• Questionnaire focuses on applicant’s job-related experiences & accomplishments

–Example questions:

»For an Information Systems Analyst position:

• Describe the types of IT systems problems you have encountered.

• Describe your experience in testing hardware, software, or systems.

–Validity: moderate (typically content validity)

Employment InterviewsEmployment Interviews• Universal selection procedure

• Strong effect on selection decisions

• Preferred by managers

• Psychometric problems:

–No consistency of questions

–Questions unrelated to jobs

–No objective scoring system

–No interviewer training

• Overall, the more standardized the interview, the better

INTERVIEWS CAN.....INTERVIEWS CAN.....

• Assess Certain Characteristics

• Assess Organisational & Team Fit

• Satisfy Social Exchange Function

• High Face Validly

Employment InterviewEmployment Interview• Validity as typically done: poor (r < .20)

• Types of employment interviews:

–Unstructured: few (if any) pre-planned questions; commonly used; poor validity

–Semi-structured: some pre-planned questions, but with flexibility to pursue lots of follow-ups; moderate validity

–Structured: all questions are pre-planned; every applicant gets the same interview; some follow-up probes; answers are evaluated using numeric rating scales; good validity

Structured InterviewsStructured Interviews

• Standardized method of asking same job related questions of all applicants

• Carefully planned and constructed based on job analysis

• Responses are numerically evaluated

• Detailed notes are taken

Example of Structured InterviewsExample of Structured Interviews

– Situational:

» Hypothetical scenarios

» A sign of how they will behave

» How would the interviewee behave in a critical situation?

– Behavioral:

» Past incidents

» A sample of work behavior – better predictor

» How did the interviewee behave in a specific job situation in the past?

– Multiple raters

– Composite ratings used to make decisions

PROBLEMS WITH INTERVIEWSPROBLEMS WITH INTERVIEWS

• Unstructured & Unplanned

• Untrained & Biased Interviewers

• Same Sex Bias

• Structured Means Standardised or Artificial & Inflexible

Why Interviews are Often not Valid Why Interviews are Often not Valid AssessmentsAssessments

• Poor wording of questions

–No systematic scoring system used by interviewers—very subjective

–Applicants have been trained to give the appropriate responses to such open-ended questions

– Interviewer has no way to verify this information in short period of time of interview

Attempts to Improve the InterviewAttempts to Improve the Interview

• Training Interviewers

• Development of Appropriate Techniques:

–Situational Interview

–Behavior Description Interview

Central Issues in Interview Training Central Issues in Interview Training ProgramsPrograms

• Creating an open-communication atmosphere

• Delivering questions consistently

• Maintaining control of the interview

• Developing good speech behavior

• Learning listening skills

• Taking appropriate notes

• Keeping the conversation flowing and avoiding leading or intimidating the interviewee

• Interpreting, ignoring, or controlling the nonverbal cues of the interview

Evaluation of the InterviewEvaluation of the Interview• Unstructured interviews used frequently;

structured ones used less frequently

• More structured, more reliable and valid

• Structured interviews are highly correlated with cognitive ability tests

• Mixed adverse impact

• Structured interviews are costly to develop and use

• Might be appropriate for measuring person/organization fit

Ability TestsAbility Tests

• Measure what a person has learned up to that point in time (achievement)

• Measure one’s innate potential capacity (aptitude)

• Up to 50% of companies use some ability testing

Ability TestsAbility Tests

• Mental (Cognitive) Ability Tests

• Mechanical Ability Tests

• Clerical Ability Tests

• Physical Ability Tests

Cognitive Ability TestsCognitive Ability Tests

- Main purpose: to determine one’s level of “g” or aptitudes depending on setting

- Measure aptitudes relevant to the job

- short, group administration

- excellent predictor of job and training performance

Typical Cognitive AbilitiesTypical Cognitive Abilities

• Memory Span

• Numerical Fluency

• Verbal Comprehension

• Spatial Orientation

• Visualization

• Figural Identification

• Mechanical Ability

• Conceptual Classification

• Sematic Relations

• General Reasoning

• Intuitive Reasoning

• Logical Evaluation

• Ordering

Example of Ability TestsExample of Ability Tests

- Wonderlic Personnel Test (measures “g”)- 50 items, 12 minutes

- multiple choice

- Items cover verbal, math, pictorial, analytical material

- Highly reliable (alternate forms > .90)

- Correlated with job performance measures

- Correlated with WAIS

Examples of Other Frequently Used Examples of Other Frequently Used Mental Ability TestsMental Ability Tests

• Otis-Lennon Mental Ability Test

• General Aptitude Test Battery (GATB) used by the US Employment Service

• Employee Aptitude Survey (EAS)

Advantages of Cognitive Ability TestsAdvantages of Cognitive Ability Tests

• Efficient

• Useful across all jobs

• Excellent levels of reliability and validity (.40 - .50)

–Highest levels than any other tests

–Estimated validity:

» .58 for professional/managerial jobs

» .56 for technical jobs

» .40 for semi-skilled jobs

» .23 for unskilled jobs

–More complex job = higher validity

Disadvantages of Cognitive Ability TestsDisadvantages of Cognitive Ability Tests

• Lead to more adverse impact

• May lack face validity

–Questions aren’t necessarily related to job

• May predict short-term performance better than long-term

–can do vs. will do

Frequently Used General Frequently Used General Mechanical Ability TestsMechanical Ability Tests

• Bennett Mechanical Comprehension Tests

• MacQuarrie Test for Mechanical Ability

• What they generally measure:

–Spatial visualization

–Perceptual speed and accuracy

–Mechanical information

Tests of Mechanical ComprehensionTests of Mechanical Comprehension

- better than “g” for blue-collar jobs

- Good face validity

- Criterion validity w/ mech. job performance

- E.g., Bennett Mechanical Comprehension Test

- 68 items

- 30 minutes

- Principles of physics & mechanics

- Operations of common machines, tools, & vehicles

- High internal consistency

- Good criterion validity w/ job proficiency & training

Clerical Ability TestsClerical Ability Tests

Predominately measures perceptual speed and accuracy in processing verbal and numerical data

Examples: Minnesota Clerical Test Office Skills Test

Clerical TestsClerical Tests

- 2/3 of companies use written tests to hire & promote

- 60-80% of tests are clerical

- Specific vs. general

- E.g., Minnesota Clerical Test

- 2 subtests: number comparison & name comparison

- Long lists of pairs of numbers/names (decide if same)

- Strict time limit

- Reliable & valid for perceptual speed & accuracy

- Good face validity

Physical Ability TestsPhysical Ability Tests

• Most measure muscular strength, cardiovascular endurance, and movement quality

• Areas of concern:

–Female applicants

–Disabled applicants

–Reduction of

work-related injuries

Ability Tests and DiscriminationAbility Tests and Discrimination

• Differential Validity

–Are employment tests less valid for minority group members than non-minorities?

–Research has found that

differential validity

does not exist

Comparison of Mental Ability Tests Comparison of Mental Ability Tests and Other Selection Instrumentsand Other Selection Instruments

Mental ability tests have

high validity and low costs

compared to other methods

Biodata, structured interviews, trainability tests, work samples, and assessment

centers have equal validity, less adverse impact, and more fairness to the

applicant, but cost more

Work Sample TestsWork Sample Tests

• How do you perform job-relevant tasks?

• 2 characteristics:

–Puts applicant in a situation similar to a work situation – measures performance on tasks similar to real job tasks.

–Is it a test of maximal vs. typical performance?

• Range from simple to complex

Work Sample TestsWork Sample Tests

–Examples:

»For telephone sales job, have applicants make simulated cold calls

»For a construction job, have applicants locate errors in blueprints

Work Sample TestsWork Sample Tests• Advantages:

–Highest validity levels (r = .50s)

–High face validity

–Easy to demonstrate job-relatedness

• Disadvantages:

–Not appropriate for all jobs

–Time-consuming to set up and administer

–More predictive in short-term

–Cannot use if applicant is not expected to know job before being hired

Measuring PersonalityMeasuring Personality

- Early research showed no validity

- Recent research: 3 of Big 5 are predictive

- Criterion validity: .15 - .25 (less than “g”)

- Susceptible to faking – does not affect validity in predicting

- Useful when dependability, integrity, responsibility are determinants of job success

Myers-Briggs Type Indicator (MBTI)Myers-Briggs Type Indicator (MBTI)

• Dimensions of personality:» Introversion Extroversion: source of energy

» Intuition Sensation: innovation vs. practical

» Thinking Feeling: impersonal principles vs. personal relationships

» Judging Perceiving: closure vs. open options

–Validity: poor for selection; might be okay, if carefully used, to help a team work better together

The Big 5 Personality DimensionsThe Big 5 Personality Dimensions–Validity: typically moderate for selection (r ≈ .25

with measures of overall job performance)

–But, validity of personality inventories is hard to generalize

»Some dimensions of personality may correlate more strongly with particular aspects of a particular job

» Extraversion → success in sales

» High conscientiousness & high openness to experience → success in job training

» Low agreeableness, low conscientiousness, & low adjustment → more likely to engage in counterproductive work behaviors (e.g., abuse sick leave, break rules, drug abuse, workplace violence)

Advantages of Personality Advantages of Personality InventoriesInventories

• Intuitively appealing to managers (e.g., MBTI)

• No adverse impact

–Don’t show rates of differential selection

• Efficient

• Moderate reliability and validity

–Validity = .20 - .30

Disadvantages of Personality Disadvantages of Personality InventoriesInventories

• Response sets

–Lie or socially desirable responding

• All traits not equally valid for all jobs

Integrity TestingIntegrity Testing

• Why do it?

–Employee theft estimated between $15 and $50 billion in 1990’s

–Employee theft rate by industry: 5 to 58%

–2% to 5% of each sales dollar charged to customers to offset theft losses

Integrity TestingIntegrity Testing

– Purpose:- theft is expensive

- also want to avoid laziness, violence, gossip

- Honesty may not be a stable trait

- Honesty testing is controversial

- May depend on the situation (perceived unfairness)

- Viewed as coercive and inaccurate

- Honesty is a strong value in our society

Honesty & Integrity TestsHonesty & Integrity Tests

• Employee Polygraph Act (1988) prohibits (with some exceptions) the use of polygraph tests of applicants or employees

• Replaced by paper-and-pencil tests:

–Overt integrity tests: measures attitudes about dishonest behavior

»Question: “Everyone will steal if given the chance.”

»Examples:

•Pearson Reid London House: Personnel Selection Inventory (PSI)

Disadvantages of Honesty & Integrity Disadvantages of Honesty & Integrity TestsTests

• Fakable

• High rates of false positives

• Some states (e.g., Massachusetts) ban it

Evaluation of Integrity TestsEvaluation of Integrity Tests

• False positives: 40 to 70%, especially if cutoff scores are set high

• Validity: difficult to determine criteria to validate against (estimates ~.13 to .55)

• Usefulness depends on the base rate of theft occurring in particular industries (estimates range from 5% to 58%)

• Faking: not a major issue, but probably more so for overt tests

• Applicant reactions are usually negative

Assessment CenterAssessment Center

A procedure for measuring performance with a group of individuals (usually 12 to 24) that uses a series of devices, many of which are verbal performance tests

Behavioral Dimensions Frequently Behavioral Dimensions Frequently Measured in Assessment CentersMeasured in Assessment Centers

• Oral Communication

• Planning and Organizing

• Delegation

• Control and Monitoring

• Decisiveness

• Initiative

 

• Tolerance for stress

• Adaptability

• Tenacity

ASSESSMENT CENTRESASSESSMENT CENTRES

• Work-sample test for manager positions

–Measures: leadership, communication, decisiveness, organizing & planning, etc.

• Developed for WW2 Officer Selection Board

• Simulation Exercises Measure Competencies

• Various Techniques, Candidates, Assessors & Competencies

Frequently Used Performance Tests Frequently Used Performance Tests in Assessment Centersin Assessment Centers

• In-Basket

• Role Plays

• Leaderless Group Discussion

• Case Analysis

Evaluation of Assessment CentersEvaluation of Assessment Centers

• Adverse Impact: not much

• Validity: .36 and .54 (performance tests)

• Acceptance to applicants: High

• Costly to develop and run

DISADVANTAGES OF DISADVANTAGES OF ASSESSMENT CENTRESASSESSMENT CENTRES

• Poorly Defined Competencies & Exercises

• Poor Training & Selection of Assessors

• Poor Selection & Briefing of Candidates

360 Degree Instruments360 Degree Instruments

When a 360 makes sense

• It is the best method to measure external behaviors. Things that are best observed and judged by others.

Problems With 360 Degree Problems With 360 Degree InstrumentsInstruments

Problems with using a 360:–Honesty–Knowledge:

»observer’s biases and perceptions influence ratings

»abilities can be hidden

The Problem of Observer ReportsThe Problem of Observer Reports

THE IMPORTANCE OF VALIDITYTHE IMPORTANCE OF VALIDITY

• Face Validity

• The method looks plausible

• Content Validity

• The method looks plausible to experts

• Criterion Validity

• The method predicts performance

• Construct Validity

• The method measures something meaningful

.-.01 to .00AGEM

.00 to .10INTERESTS / VALUESL

.00 to .10EDUCATION / GPAK

.10 to .15SELF RATINGSJ

.13 to .15TRAINING RATINGSI

.14 to .26REFERENCE CHECKSH

.15 to .31PERSONALITY INVENTORIESG

.15 to .38UNSTRUCTURED INTERVIEWSF

.24 to .35WORK HISTORYE

.41 to .49PEER/SUPERVISORY RATINGSD

.41 to .50ASSESSMENT CENTERSC

.38 to .51INTELLIGENCE TESTSB

.38 to .54WORK SAMPLE TESTSA

AVERAGE VALIDITYPersonnel Selection: Which

are Most Predictive?

PREDICTIVE VALIDITY & PREDICTIVE VALIDITY & SELECTION TESTSSELECTION TESTS

–Combinations of methods»“g” + work samples = .63»“g” + structured interview = .63»“g” + integrity tests = .65