Test AdministrationTest Administration

The Examiner & the SubjectThe Examiner & the Subject

Relationship between examiner & Relationship between examiner & subjectsubject• Feldman & Sullivan (1960)Feldman & Sullivan (1960)• WISC administered to children in two WISC administered to children in two

conditions: conditions: Enhanced rapportEnhanced rapport neutralneutral

Race of testerRace of tester• Little evidence that race of tester Little evidence that race of tester

significantly affects test performance (at significantly affects test performance (at least with black & white American least with black & white American children)children)

Training of testersTraining of testers• Patterson et al. (1995)Patterson et al. (1995)

Expectancy or Rosenthal effectsExpectancy or Rosenthal effects

Rosenthal & Jacobsen (1968)Rosenthal & Jacobsen (1968)

Classic study conducted at a public Classic study conducted at a public elementary school, described as “lower class”elementary school, described as “lower class”

School used a tracking system where children School used a tracking system where children were sorted into one of three tracks (fast, were sorted into one of three tracks (fast, medium & slow) based on reading medium & slow) based on reading performanceperformance

About 600 students in the schoolAbout 600 students in the school Students at the school described as “low Students at the school described as “low

achievers”; pre-test data showed an average achievers”; pre-test data showed an average IQ of 98 for boys and 99 for girlsIQ of 98 for boys and 99 for girls

In spring of 1964. students given the In spring of 1964. students given the “Harvard Test of Inflection”, which “Harvard Test of Inflection”, which supposedly predicted academic “blooming”supposedly predicted academic “blooming”

At beginning of following year, researchers At beginning of following year, researchers randomlyrandomly selected 20% of the student at selected 20% of the student at the school, and stated that these students the school, and stated that these students were “academic bloomers”were “academic bloomers”

At the end of the year, IQ scores of children At the end of the year, IQ scores of children labeled as “bloomers” were compared with labeled as “bloomers” were compared with the IQ scores of children who were not the IQ scores of children who were not given this labelgiven this label

ResultsResults Students labeled as “bloomers” gained Students labeled as “bloomers” gained

significantly more IQ points over the year, significantly more IQ points over the year, compared to control studentscompared to control students

Reinforcing responsesReinforcing responses• Sweet (1970)Sweet (1970)• Terrel et al. (1978)Terrel et al. (1978)

Administered WISC-R to lower-income Administered WISC-R to lower-income African-American 2African-American 2ndnd graders under one of 4 graders under one of 4 conditions:conditions:

• No feedbackNo feedback• Verbal praiseVerbal praise• CandyCandy• Culturally relevant verbal praise (“nice job, Culturally relevant verbal praise (“nice job,

blood”)blood”)

Standardized AdministrationStandardized Administration

Standardized testing procedures are Standardized testing procedures are so important that they are listed as so important that they are listed as an essential criterion for valid testing an essential criterion for valid testing in the in the Standards for Educational & Standards for Educational & Psychological TestingPsychological Testing

Requires that tester must be very Requires that tester must be very familiar with materials, requiring familiar with materials, requiring several test administrationsseveral test administrations

Background & Motivation of Background & Motivation of ExamineeExaminee

Test anxietyTest anxiety• When taking an important examination I When taking an important examination I

sweat a great dealsweat a great deal• I freeze up when I take intelligence tests I freeze up when I take intelligence tests

or school examsor school exams• I really don’t understand why some I really don’t understand why some

people get so upset about testspeople get so upset about tests• I dread courses in which the instructor I dread courses in which the instructor

likes to give “pop” quizzeslikes to give “pop” quizzes

Test Anxiety (cont’d)Test Anxiety (cont’d)

Test anxiety is negatively correlated Test anxiety is negatively correlated with school achievement, aptitude with school achievement, aptitude test scores, & measures of test scores, & measures of intelligenceintelligence

Test anxiety is exacerbated by tests Test anxiety is exacerbated by tests with time limitswith time limits

Siegman (1956)Siegman (1956)

Compared performance levels of high- and Compared performance levels of high- and low-anxious medical/psychiatric patients low-anxious medical/psychiatric patients on timed & untimed subtests from the on timed & untimed subtests from the WAISWAIS

9

9.5

10

10.5

11

11.5

12

12.5

untimed tests timed tests

low-anxioussubjects

high-anxioussubjects

Test-Smart or “Coached” Test-Smart or “Coached” IndividualsIndividuals

Powers & Swinton (1984)Powers & Swinton (1984)• Mailed test preparation materials, Mailed test preparation materials,

including extra practice tests, including extra practice tests, explanations to practice test questions, explanations to practice test questions, hints or strategies for answering hints or strategies for answering different item typesdifferent item types

• Special coaching, preparation yielded Special coaching, preparation yielded scores 53 points higher than scores of scores 53 points higher than scores of uncoached respondentsuncoached respondents

Fair Test Fact SheetFair Test Fact Sheet The GRE can be conquered with tricks The GRE can be conquered with tricks

having nothing to do with the knowledge, having nothing to do with the knowledge, persistence, thoughtfulness, and other persistence, thoughtfulness, and other qualities that are vital to graduate study qualities that are vital to graduate study and professional performance. and professional performance.

One coaching book advises: "Taking the One coaching book advises: "Taking the GRE is a game with its own rules, traps, GRE is a game with its own rules, traps, and measures of success…How you do on and measures of success…How you do on the GRE is an indication of how well you the GRE is an indication of how well you play the game, but it is not an indication of play the game, but it is not an indication of how 'intelligent' you are, or what kind of how 'intelligent' you are, or what kind of student you will make."student you will make."

Fair Test on Coaching (cont’d)Fair Test on Coaching (cont’d) The exam's susceptibility to coaching undermines The exam's susceptibility to coaching undermines

educational equity by advantaging students who can afford educational equity by advantaging students who can afford test prep materials - many of whom already score in the test prep materials - many of whom already score in the upper percentiles - over those who cannot. The most upper percentiles - over those who cannot. The most comprehensive coaching classes (which generally offer the comprehensive coaching classes (which generally offer the greatest score gains) cost upwards of $1,000 or more. One greatest score gains) cost upwards of $1,000 or more. One coaching company claims its students gain on average 212 coaching company claims its students gain on average 212 points on the GRE - a substantial advantage in the graduate points on the GRE - a substantial advantage in the graduate school application process. While ETS asserts that the GRE school application process. While ETS asserts that the GRE is not coachable, it promotes its own materials: test takers is not coachable, it promotes its own materials: test takers can purchase a diagnostic service for $15, Preparing to can purchase a diagnostic service for $15, Preparing to Take the General Test for $18, or can use the free Take the General Test for $18, or can use the free POWERPREP software package. While there are no POWERPREP software package. While there are no independent studies on coaching's impact on the GRE, independent studies on coaching's impact on the GRE, independent studies of coaching for the similar SAT exams independent studies of coaching for the similar SAT exams demonstrate that coaching can improve scores (see demonstrate that coaching can improve scores (see FairTest's The SAT Coaching Cover-Up).FairTest's The SAT Coaching Cover-Up).

Computer-Administered TestsComputer-Administered Tests Improves standardizationImproves standardization Can individually tailor administration of test itemsCan individually tailor administration of test items Allows for precise timing of questions & Allows for precise timing of questions &

responsesresponses Saves time & expenseSaves time & expense Allows as much time as necessary for responsesAllows as much time as necessary for responses Reduces bias in testingReduces bias in testing Examinees often disclose more in response to Examinees often disclose more in response to

computer test than human-administered testcomputer test than human-administered test

Computerized Adaptive TestingComputerized Adaptive Testing

An adaptive test is one in which the An adaptive test is one in which the questions are tailored specifically to the questions are tailored specifically to the individual being examinedindividual being examined

An adaptive test of mental ability, for An adaptive test of mental ability, for example, will include items that are example, will include items that are neither too easy nor too difficult for the neither too easy nor too difficult for the respondentrespondent

As a result, when two individuals of As a result, when two individuals of different ability take the same test, they different ability take the same test, they might respond to completely different might respond to completely different questionsquestions

Computerized Adaptive TestingComputerized Adaptive Testing

Ten items

Routing test

Difficult items (10)

Average items (10)

Easy items (10)

Most difficult items (10)

Less difficult items (10)

Easier items (10)

Easiest items (10)

Computerized Adaptive Testing Computerized Adaptive Testing Using Item Response TheoryUsing Item Response Theory

Theory allows for a calculation of the Theory allows for a calculation of the difficulty of each item, the discriminating difficulty of each item, the discriminating power of the item, and the probability of power of the item, and the probability of guessing the correct answerguessing the correct answer

It then provides procedures for It then provides procedures for • Estimating the respondent’s ability on the Estimating the respondent’s ability on the

basis of his or her response to each itembasis of his or her response to each item• Choosing the optimal test items on the basis of Choosing the optimal test items on the basis of

that estimatethat estimate• Revising that estimate on the basis of Revising that estimate on the basis of

responses to each new itemresponses to each new item

Advantages of Computerized Advantages of Computerized Adaptive TestingAdaptive Testing

Makes it possible to achieve high levels of Makes it possible to achieve high levels of accuracy using an extremely small set of accuracy using an extremely small set of test questions (reducing fatigue, boredom)test questions (reducing fatigue, boredom)

Improves item securityImproves item security Saves time & moneySaves time & money Research shows that CAT versions of test Research shows that CAT versions of test

produce scores that correlate highly produce scores that correlate highly (about .90) with paper-and-pencil versions (about .90) with paper-and-pencil versions of testsof tests

Behavioural Assessment Behavioural Assessment MethodologyMethodology

Research has shown that in certain Research has shown that in certain areas (e.g., estimating job aptitude), areas (e.g., estimating job aptitude), the best kind of test involves giving a the best kind of test involves giving a person some of the tasks he/she will person some of the tasks he/she will be required to perform on the job, be required to perform on the job, and then observing & rating that and then observing & rating that performanceperformance

Reliability of observers’ ratings is Reliability of observers’ ratings is critical in these kinds of measurescritical in these kinds of measures

Issues in Behavioural AssessmentIssues in Behavioural Assessment

ReactivityReactivity• Reliability and accuracy are highest when Reliability and accuracy are highest when

someone is monitoring the observers; someone is monitoring the observers; decreases when work is not monitoreddecreases when work is not monitored

DriftDrift• After training, observers tend to “drift” away After training, observers tend to “drift” away

from the standards or rules they followed in from the standards or rules they followed in trainingtraining

ExpectanciesExpectancies• Behavioural observers will pay more attention Behavioural observers will pay more attention

to & notice behaviour they expectto & notice behaviour they expect