1Home

Computer-Based Test Interpretation

James P. Sampson, Jr., M. Page Purgar, and Jonathan D. Shy

Florida State University

Copyright 2002 by James P. Sampson, Jr., All Rights Reserved

2Home

A Brief History of CBTI

• Over forty years old

• Most are computer versions of paper and pencil tests

• Unrestricted self-help tests do not require practitioner to score and interpret

• Well designed self-help tests do indicate when a practitioner is needed

• Restricted practitioner-assisted tests are designed for use in counseling settings

• Delivered via personal computers and the Internet

3Home

Organization

• Responsibility for Meeting Testing Standards

• The Nature of CBTI

• CBTI in Career Testing

• Establishing the Validity of CBTI

• Communicating the Validity of CBTI

• Need for Research

4Home

Meeting Testing Standards

• To what and to whom do standards apply?

• What is a test?

• What is an assessment?

• Who is responsible for test development?

5Home

Meeting Testing Standards

• Test:– Standardized, scored numeric measure– Measures a specific construct or behavior– Meaning is related to scores

• Assessment:– Inclusive and inferential description of a person– Utilizes tests as well as other sources of data such

as observations, interviews, rating scales, and projective techniques

– Considered within an ecological context

6Home

Meeting Testing Standards

• Tests vs. checklists

• Tests must meet standards

• Checklists are exempt

• If it looks like a duck, walks like a duck, then it’s a duck

7Home

Meeting Testing Standards

• Who is a test developer?

• Standards apply irrespective of the test developer

• Tests must meet standards

8Home

Organization

• Responsibility for Meeting Testing Standards

• The Nature of CBTI

• CBTI in Career Testing

• Establishing the Validity of CBTI

• Communicating the Validity of CBTI

• Need for Research

9Home

The Nature of CBTI

• Definition of CBTI• Consumers of CBTI• Categorizations of CBTI• Appropriate Use of CBTI• Sequence for Using CBTI• Qualifications for Competent Use of CBTI• Potential Benefits of CBTI• Potential Problems with CBTI

10Home

Definition of CBTI

• Uses information technology to integrate theory, practitioner judgment, and empirical evidence

• One source of data

• Help practitioners and clients better understand the meaning of test scores

• For the purpose of gaining insight and making decisions

11Home

Consumers of CBTI

• In practitioner-assisted assessment, the client and the practitioner are the consumers of CBTI– Some CBTI is intended to be used only by

the practitioner (such as the MMPI)

• In self-assessment, the individual taking the assessment is the CBTI consumer– The CBTI may also be used in counseling

12Home

Categorizations of CBTI

• Descriptive interpretations generate narrative statements based on test scores

• Clinician-modeled interpretations generate narrative statements and clinical hypotheses by simulating the judgment of a renowned clinician or a group of expert practitioners

• Clinical-actuarial interpretations generate narrative statements and clinical hypotheses based on empirical research

13Home

Appropriate Use of CBTI

• Should be used in a consulting role

• Should not be used to replace the practitioner

• Should not be used as a stand-alone psychological report

14Home

Sequence for Using CBTI

• Review the client’s career concerns

• Consider client data in light of theory

• Review the score profile

• Develop hypotheses the meaning of results based on test data and theory

15Home

Sequence for Using CBTI

• Review the CBTI to confirm or disconfirm hypotheses

• Use hypotheses to help the client understand the meaning of the results

• Use these insights to develop a service delivery plan to help resolve problems

• Create and review any assessment report

16Home

Qualifications for CBTI Use

• Completion of a degree or obtaining a credential is not sufficient for ethical use

• Competencies necessary– Understanding the construct or behavior being

measured – Understanding of the test– Understanding of the standard interpretation– Understanding of the CBTI– Supervised experience in using the CBTI

17Home

Potential Benefits of CBTI

• Capability to process a greater number of complex variables

• Access to an expanded knowledge base

• Capable of organizing and systematically accessing extensive and rapidly growing data bases

18Home

Potential Benefits of CBTI

• Less affected by subjectivity

• Less subject to interpreter bias

• More thorough and better documented

• More rapid access to interpretive reports

• More rapid updating of interpretations

19Home

Potential Problems of CBTI

• Developers may have no appropriate qualifications to develop CTBI

• Many Web sites do not list developer credentials

• Systematic interpretation bias

• Perception of infallible accuracy and validity

20Home

Potential Problems of CBTI

• Overworked, or inadequately trained practitioners may rely too heavily on CBTI

• Differences among CBTI reports for the same personality test generated by different software programs

• Some reports contain contradictory statements

21Home

Organization

• Responsibility for Meeting Testing Standards

• The Nature of CBTI

• CBTI in Career Testing

• Establishing the Validity of CBTI

• Communicating the Validity of CBTI

• Need for Research

22Home

CBTI in Career Testing

• Most CBTI in career testing is is descriptive and actuarial in nature

• Descriptive and actuarial statements are typically derived from theoretical constructs or research data from the professional manual for the original paper-and-pencil instrument

– Justifiable only if the the original measure and the computer version has been shown to be equivalent

• Few clinician-modeled CBTI are used in generating interpretive reports

23Home

CBTI in Career Testing

• Current examples:– Descriptive statements based on theory and

research are used to interpret scores on the Strong Interest Inventory

– Descriptive statements from research on the Career Factors Inventory is used to make recommendations on the need for counseling as part of career exploration

– Clinical-actuarial statements based on empirical data as the basis for interpretation of occupational scales on the Strong Interest Inventory

24Home

CBTI in Career Testing

• Possible applications:

– Clinician-modeled statements based on the judgment of expert practitioners

• on counseling strategies that are potentially appropriate for clients with specific Holland codes or levels of vocational identity

• on the interpretation of flat profiles on the Strong Interest Inventory

25Home

Organization

• Responsibility for Meeting Testing Standards

• The Nature of CBTI

• CBTI in Career Testing

• Establishing the Validity of CBTI

• Communicating the Validity of CBTI

• Need for Research

26Home

Establishing the Validity of CBTI

• Test validity concerns judging if theory and empirical evidence support test interpretations for a specific use of a test

• CBTI validity concerns judging if theory, practitioner judgment, and empirical evidence support test interpretations for a specific use

• Invalid interpretations are possible for valid tests

• Validity studies for CBTI are scarce

27Home

Establishing the Validity of CBTI

• Establishing validity for all types of CBTI– Identify the developer, including his or her

qualifications for creating the interpretation – Document the lack of systematic bias in interpretation– Describe the elimination of interpretations that are so

general as to be meaningless– Document the efforts to eliminate obviously

contradictory statements – Document the evidence that CBTI interpretations

compare with credentialed and experienced practitioners

28Home

Establishing the Validity of CBTI

• Establishing validity for descriptive CBTI– Assuming equivalency exists, document

how theory and empirical data from the original paper-and-pencil test are used in the CBTI

– Document that the links between test data and interpretive statements in the CBTI have been verified for accuracy

29Home

Establishing the Validity of CBTI

• Establishing validity for clinician-modeled CBTI

– Identify the credentials of any renowned practitioner that were used to generate the CBTI

– Identify the credentials of any group of experienced practitioners used to generate the CBTI

– Document the methods used to create a statistical model of multiple practitioner judgment for the CBTI

30Home

Establishing the Validity of CBTI

• Establishing validity for clinical-actuarial CBTI

– Document the empirical evidence for the decision rules that were used in developing actuarial predictions in the CBTI

– Document how clinician-modeled interpretive statements are used to augment a clinical-actuarial CBTI to avoid leaving a large number of test takers unclassified

31Home

Organization

• Responsibility for Meeting Testing Standards

• The Nature of CBTI

• CBTI in Career Testing

• Establishing the Validity of CBTI

• Communicating the Validity of CBTI

• Need for Research

32Home

Communicating the Validity of CBTI

• Establishing evidence of validity is meaningless if practitioners cannot access and evaluate this information

• Print materials traditional method for delivering technical information

• Internet delivery is a more cost-effective method of delivering information

33Home

Organization

• Responsibility for Meeting Testing Standards

• The Nature of CBTI

• CBTI in Career Testing

• Establishing the Validity of CBTI

• Communicating the Validity of CBTI

• Need for Research

34Home

Need for Research

• CTBI at a formative stage of development

• Many advanced applications of CTBI are possible, yet unavailable at present

• Integration of clinician-modeled and clinical-actuarial CBTI appear promising

• What types of CTBI is available for what categories of test?

• How is CTBI being used in practice?

35Home

Conclusion

• CTBI is a mixture of benefits and problems

• Despite lingering questions about design and use, CTBI is becoming increasingly popular

• Balance economic pressure to use CTBI to cut costs with the need to maintain quality CTBI design and use

Home

For Additional Information

www.career.fsu.edu/techcenter/

Thank You