© PeopleFluent 2014 Testing the Test Presenter: Lisa Harpe, Ph.D. Peoplefluent Research Institute...

© PeopleFluent 2014

Testing the Test

Presenter: Lisa Harpe, Ph.D. Peoplefluent Research Institute (PRI)

2© 2014 PeopleFluent | Confidential Presentation Title | date

Agenda

• Regulatory Climate• What is a Test?• What is Validity?• Questions for a Third Party Test Vendor• Questions for a Test User• Documentation Checklists


Regulatory Climate• EEOC

• Experts on staff• Charges

• Some well known cases in the mid-2000s: Dial Corp, Whirlpool, Fed-Ex • Criminal Background Checks• ADA violations

• OFCCP• Experts on staff• Settlements (in last 2 years)

• Leprino Foods• GE• Lincoln Electric


UGESP• Uniform Guidelines on Employment Selection

Procedures (UGESP) (1978)

• Adverse impact (A.I)• No need to show validity if no A.I.• Justify, change, find alternative or eliminate if A.I.

• Justification• Validity evidence = job related• Business necessity


UGESP Citations• Section 60-3, Uniform Guidelines on Employee

Selection Procedure (1978); 43 FR 38295 (August 25, 1978).

• Equal Employment Opportunity Commission: 29 CFR Part 1607

• Department of Labor/ Office of Federal Contract Compliance Programs: 41 CFR Part 60-3

• Department of Justice: 28 CFR 50.14• Civil Service Commission: 5 CFR 300.103(c)


Resources• Uniform Guidelines on Employment Selection Procedures (UGESP, 1978)

• Department of Labor’s (DOL) Testing and Assessment: an Employer’s Guide to Good Practices (1999)

• American Psychological Association’s (APA) Standards for Educational and Psychological Testing (2014)

• Society of Industrial Organizational Psychologist’s (SIOP) Principles for the Validation and Use of Personnel Selection Procedures (4th edition, 2003)


What is a Test?• ‘Any instrument or procedure that samples behavior or

performance’ (DOL Guide)

• ‘Any device or procedure that samples behavior which is scored in a standard process’ (APA Standards)

• ‘Any procedure used singly or in combination to make personnel decisions’ (‘selection procedure, test, assessment, and predictor’ = interchangeable terms) (SIOP Principles)

• vs. basic qualification questionnaire


Types of Tests• Cognitive ability tests• Physical ability tests• Achievement tests• Personality tests• Job knowledge tests• Skills tests• Honesty and integrity tests• Biodata inventories• Work samples and simulations• Assessment centers


Types of Tests• Employment interviews• Education and experience requirements (including

licensing and certification)• Recommendations and reference checks• Criminal record check• Medical examinations• Drug and alcohol tests


What is Validity?

• Principles advocates the Standards’ definition as “the degree to which accumulated evidence and theory support specific interpretations of test scores entailed by proposed uses of a test” (AERA et al., 1999, p. 184)


Validity• Validity depends on the purpose of the test

• Measure of extraversion vs. predictor of job success vs. predictor of training success

• Selection vs. promotion vs. placement• Target population

• Validity depends on the use of the test results• Cut score (Pass/Fail)• Ranking• Part of assessment battery


Sources of Validity Evidence• Content validity

• Can you show a direct link between the test items and job behaviors, work outcomes or KSAOs?

• Criterion related validity• Can you show that the test is related to performance?

• Construct validity• Are there multiple lines of evidence that whatever is being measured by

the test is related to the job?

• Other• Response Process• Internal structure• Consequential


Content Validity

Test

Content Validity

Work behaviors, activities, outcomes

or KSAOs


Content Validity• Expert Judgment• SME

• Incumbents• Hiring manager• HR• Industrial Psychologists

• Define Important Work Behaviors, Activities and Outcomes

• Evaluate Test Content for Relevance


Content ValidityJob Representative

Work BehaviorTest Content

Administrative Assistant

Maintain supervisor’s calendar

Microsoft Outlook Skills Assessment

Administrative Assistant

Word processing Microsoft Word Skills Assessment

Truck Driver Drive truck Truck Driving Test


Criterion Related Validity• Criterion Measures

• Performance ratings• 6 month, annual• Supervisor, peer, customer / client

• Productivity• Sales revenue

• Turnover• Absenteeism

• Multi-dimensional



Criterion-related validity

• Predictive: Can test results tell us how well a person will perform in the job?

• Concurrent: Are test results related to job performance?


Criterion-related Validity

+1.0 Correlation Between Test Scores and Performance

40

50

60

70

80

90

100

40 60 80 100

Test Scores

Pe

rfo

rma

nc

e

Correlation Coefficient (r)


-1.0 Correlation Between Test Scores and Performance

40

50

60

70

80

90

100

40 60 80 100

Test Scores

Pe

rfo

rma

nc

e

Criterion-related ValidityCorrelation Coefficient (r)


Correlation Between Test Scores and Performance

40

50

60

70

80

90

100

40 60 80 100

Test Scores

Pe

rfo

rma

nc

e

Criterion-related ValidityCorrelation Coefficient (r)


• Question 1: Is the correlation statistically significant?

• Question 2: Is the correlation practically significant?




Correlation coefficient value Interpretation

above .35 very beneficial .21 - .35 likely to be useful .11 - .20 depends on circumstances below .11 unlikely to be useful

Guidelines for Interpreting Correlation Coefficients (DOL guide)


Test Job PerformanceR-sq = .75

Criterion-related ValidityCoefficient of Determination (R-squared)


Test

Job Performance

R-sq = .00

Criterion-related ValidityCoefficient of Determination (R-squared)


Test

Job Performance

R-sq = .2

Coefficient of Determination (R-squared)




What Influences the Validity Coefficient

• Validity of BOTH the test (predictor) and performance measure (criterion)

• Reliability of BOTH the test (predictor) and performance measure (criterion)

• Range restriction in the test (predictor) and / or performance measure (criterion)

• Sample size affects the significance of r


Construct validity• Construct = unobservable trait or

characteristic• Examples

• Extraversion• Customer service orientation• Intelligence• Oral presentation skills• Ability to trouble-shoot• Leadership


Construct ValidityPredictor Measure(e.g., CallCenterPro)

Criterion Construct Domain

(e.g., Interpersonal Interactions)

Criterion Measure(e.g., Supervisor Rating)

Predictor Construct Domain

(e.g., People Skills)


Other Types of Validity Evidence• Face Validity

• Response Process

• Internal Structure

• Consequential


Job Analysis• Why Do I Care About This?• Analysis of Work (SIOP)• 2 Purposes

• Develop predictors (tests / selection procedures)• Develop criteria (performance measures)

• Approaches• Questionnaires• Observations• Interviews• Documentation

• Currency


Reliability• Reliability = Consistency or Stability

• Why Do I Care About This?

• Types: • Test-retest• Interrater reliability• Internal consistency• Alternate form


ReliabilityGuidelines for Interpreting Reliability

Coefficients (DOL)

Reliability Coefficient Interpretation .90 and up excellent .80 - .89 good .70 - .79 adequate below .70 may have limited applicability


Validity Generalization• Meta-analysis• Synthetic Validity• Transportability

• Job similarity• Job context similarity• Criterion similarity• Test use similarity• Test taker similarity

• Test fairness


Third Party Tests• Sources of Validity Evidence

• Vendor manuals and tech reports• Independent reviews• Research literature

• Mental Measurements Yearbook (Buros Institute of Mental Measurements)

• Tests in Print (TIP) (Buros Institute of Mental Measurements)


Questions for a Test Vendor• What are your credentials and experience? Can you provide references?

• Do you have a technical report that describes the development and validation of the test? Will this report be updated over time? If so, how will you inform me of the updates?

• What costs are involved with using the test in my selection process?

• For which job(s) is the test valid? Can you show that the test is valid for my company and my job(s)?


Questions for a Test Vendor• Can you show that the test is reliable?

• Can you show that the test is not biased? Can you provide the results of adverse impact studies by race and gender? Can you conduct adverse impact studies for my job(s)?

• Can you provide information on test administration, scoring and interpretation?

• Will you evaluate the effectiveness of the test in my company? Will you make changes if warranted?


• Will the test by changed or updated? How will you inform me of any changes or updates?

• Can you document the validity of the test in accordance with the UGESP requirements?

• If the vendor can not provide an answer or does not know what you are talking about, run don’t walk!

What to ask a test vendor


Questions for the Test User• What employment tests are used in the selection process?

• For which job(s)?• Where in the process?

• Does the test have an adverse impact?

• Who developed this test? where did we get the test?

• Did we consider any alternatives?

• Where is the job analysis for this job?


• How are the test results used?

• Is the test reliable?

• Do we have evidence of the test’s validity? Does the evidence meet the UGESP and SIOP criteria for documentation?

Questions for the Test User


• Homemade Tests• See recommended process for work sample tests

• Tests found on the Internet

• Off the shelf tests ‘valid’ for a given industry

Common Issues


• Establish a ‘test’ panel of subject matter experts (individuals who know the job and what the job requires) – 1-2 supervisors, 2-3 incumbents (the more the better) with more than 1 year experience, HR rep.

• Is test related to job?• Identify and document what knowledge, skills or abilities (KSA) are

measured by the test.• Document that KSAs measured by test are required on the job

• Provide examples of tasks and duties representing the KSAs• Indicate frequency of performing tasks or duties requiring the KSAs• Indicate the importance of performing tasks or duties requiring the KSAs• Are the KSAs needed on the first day of the job?

RECOMMENDED PROCESS FOR WORK SAMPLE TEST


• How to score test?• Experts define criteria as specifically as possible• Panel of SMEs should score the test• Panel should agree on definition of success

• Conduct an annual review of the test• Relationship to performance (could include turnover)• Adverse impact



• Other• Use test results as input to facilitate the interview process and not as

a defining step unless you have strong evidence to support a cut score

• Use samples of work required on first day of job• Standardize administration and scoring and use of results• Maintain records

• Test taker name• Test taker race and gender• Requisition test taker is being considered for• Test date• Test results – component, overall and any decision resulting from the

test



Thank YouPlease contact me with questions or comments

[email protected](919) 645-2975

For more information www.peoplefluent.com

Corporate Headquarters

(877) 820-4400 (toll free) or (919) 645-2800

mailto:[email protected]

http://www.peopleclick.com/

Date post:	28-Dec-2015
Category:	Documents
Upload:	alice-paul
View:	214 times
Download:	1 times

© PeopleFluent 2014 Testing the Test Presenter: Lisa Harpe, Ph.D. Peoplefluent Research Institute...

Documents