Compass Math Test Validity March 2011

Mike Orkin, Jo Ann Phillips, Hui ZhangOffice of Institutional Research

Peralta Community College District

1

Compass Math Test• Peralta students take assessment tests for

placement into Math, ESL, and English courses. • Course placement based on test scores is

recommended not mandatory. • A student may take courses without taking

placement tests.

2

Compass Math Test • Peralta uses the ACT Compass test for math

placement.• When students take Compass, they advance

through “subtests” until they reach a “placement domain.”

• The higher the placement domain, the higher the difficulty level of the recommended courses.

3

Data• Students taking Compass-designated math courses

in F08, S09, F09, S10. • Took Compass test immediately preceding

semester when they took math course, or• Never took Compass test prior to semester when

they took math course.

4

Data Totals• 12,657 student enrollments. • 4,915 (39%) took Compass test.• 7,742 (61%) did not.

5

Data by Placement Domain Courses

Placement Domain Courses Compass

No Compass Total

% Take Compass

MPRAL 3,749 2,644 6,393 59%

MALGE 755 5,268 6,023 13%

MTRIG 179 643 822 22%

MCOAL 232 3,590 3,822 6%

6

Placement Domains and Courses• MPRAL – Prealgebra

Arithmetic – Math 250 (0-32)Pre-Algebra – Math 253 (33-39)Elementary Algebra – Math 201 (40-100)

7

Placement Domains (Subtests)• MALGE – Algebra.

Elementary Algebra – Math 201 (0-34)Intermediate Algebra – Math 203 (35-49)Statistics – Math 13 (50-100)

8

Placement Domains (Subtests)• MTRIG – Trigonometry.

Trigonometry – Math 50 (0-45)Pre Calculus – Math 1 (46-100)

9

Placement Domains (Subtests)• MCOAL – College Algebra.

Statistics – Math 13 (0-34)Pre Calculus – Math 1 (35-49)Calculus – Math 3A (50-100)

10

Compliance Within Placement Domain• Most complied with cutoff score assignment

within their placement domain.

11

Compliance Within Placement Domain

Placement Domain Compliance Total %MPRAL COMPLY 3,264 87%

NONCOMPLY 485 13%

MPRAL Total 3,749

MALGE COMPLY 682 90%

NONCOMPLY 73 10%

MALGE Total 755

MTRIG COMPLY 125 70%

NONCOMPLY 54 30%

MTRIG Total 179

MCOAL COMPLY 169 73%

NONCOMPLY 63 27%

MCOAL Total 232

Grand Total 4,915

12

Validity• A test is “valid” if it measures what it’s supposed

to measure. • We looked at this in two ways:

I. Are grades and success positively related to Compass scores?

II. Does Compass do “better” than students who don’t take Compass?

13

Validity• Are grades positively related to Compass scores?

On average, as grades increase, so do scores.

14

Validity ANOVA - Assessment test averages by grade.

Placement Domain A - Test ave B - Test ave C - Test ave D - Test ave F/W- Test ave Sig.

MPRAL 38.49 35.77 34.41 34.97 32.58 0.000

MALGE 48.77 44.73 43.33 41.36 44.51 0.014

MTRIG 39.55 39.03 38.62 37.45 36.56 0.401

MCOAL 71.87 69.6 65.23 64.43 64.77 0.037

ALL 44.12 39.35 37.25 36.54 34.53 0.000

15

Validity Grade Counts

Placement Domain A B C D F W Total

MPRAL 767 569 471 275 504 1,107 3,693

MALGE 239 139 113 47 70 144 752

MTRIG 69 39 26 11 9 25 179

MCOAL 127 43 22 7 10 21 230

ALL 1,202 790 632 340 593 1,297 4,854

16

Validity• Do assessment test scores predict a student’s

chance of success (C or better) in a course?

Logistic regression estimates the probability of success for a given assessment test score.

For all subtests, as scores get higher, the predicted chance of success increases, indicating predictive validity.

17

Validity

18

Validity

19

Validity

20

Validity

21

Validity

22

Validity• Does Compass do better than students who don’t

take the test? Assume that those who don’t take Compass

essentially select a class at random.

23

Validity• For non-Compass students, just by chance, some

strong students will take lowest-level courses (too easy) and some weak students will take higher level courses (too difficult).

• If Compass is properly assessing students, only weaker students will take lowest level courses and stronger students will take higher level courses.

24

Validity• Thus, students who take Compass should do worse

than those who don’t at lowest subtest and better than those who don’t at higher levels.

• This is exactly what we observe in the data.

25

ValidityChi Square - Success – Compass Test v. No Test

Placement Domain Courses

Take Test % Success

No Test % Success Sig.

MPRAL 48.2% 50.9% 0.020

MALGE 65.0% 57.5% 0.000

MTRIG 74.9% 56.9% 0.000

MCOAL 82.8% 60.8% 0.000

26

Validity• Similar result when comparing grades of students

who take Compass with those who don’t.

27

Validity

ANOVA - Test v. No Test Course GPA

Placement Domain Courses Test GPA No Test GPA Sig.

MPRAL 2.3 2.6 0.000

MALGE 2.7 2.6 0.245

MTRIG 3.0 2.8 0.083

MCOAL 3.3 2.8 0.000

28

Summary• All four methods of analysis yield statistically

significant results supporting Compass validity.

29

Summary• On average, as grades increase, so do Compass scores.• As Compass scores increase, the predicted chance of

success increases. • Students who take Compass do better than those

who don’t in the higher subtests and worse in the lowest subtest, in both course success and grades.

30