The effects of progress reports and goal orientation on effort and achievement of students using an online individualized instructional program
Roy B. Clariana, Penn State University & Ray Bernardi, Wyoming Area School District
2 of 26
Overview
This investigation examined the effects of advisement on 8th grade students’ (n = 194) effort and achievement. Advisement consisted of a progress report sent to parents of their child’s work on online computer-delivered lessons.
We asked, “Does advisement interact with achievement goal orientation and gender to affect effort (the number of online lessons completed) and achievement (PSSA performance)?”
3 of 26
Achievement goal orientation
Achievement Goal Theory (Pintrich & Schunk, 1996) proposes two broad classes of achievement goal orientations, performance or mastery (either approach or avoidance).
Learners with a performance goal orientation are ego-centered and need to validate or prove their ability through open performance. They may avoid difficult tasks when failure is possible.
In contrast learners with a mastery goal orientation seek to increase their knowledge and ability -- learning for the sake of learning.
4 of 26
Achievement goal orientation Achievement goal orientation may relate to
personal epistemological belief systems (Schommer-Aikins & Easter, 2006) and thus may be relatively stable.
It may be a state and/or a trait; but the learning context likely affects immediate achievement goal orientation.
Also, achievement goal orientation may shift over time based on experiences in the learning context (i.e., consider the likely effects of NCLB on achievement goal orientation).
5 of 26
Advisement
Advisement is not praise (Henderlong & Lepper, 2002).
Advisement is information that relates to required, actual, or anticipated lesson performance, and according to Smith (1988), addresses questions such as, "How am I doing in this lesson?"
Progress reports are one very common form of advisement (e.g., report cards, or a detailed report from the teacher showing every graded test and assignment within a date range).
Advisement is assumed to support effective instructional decision making, thus it directly relates to learners’ achievement goals.
6 of 26
Advisement in computer-mediated learning contexts Milheim and Azbell (1988) report that advisement in
computer-mediated settings improves posttest scores relative to no-advisement groups; increases the number of students that reach mastery; increases time on task; and thus increases instructional efficiency.
Clariana (1993) considered the effects of advisement on effort and achievement of at-risk 11th grade students working online. Progress reports were given to students at the end of their 1st and 3rd sessions (max. 5 sessions). Analysis of the attendance data showed a significant
difference between the advisement (M = 4.33 out of 5) and no-advisement (M = 3.44 out of 5) groups.
Analysis of the achievement data indicated that the advisement group did a little better (nsd) than the no- advisement group and advisement had a slightly larger effect on females’ mathematics achievement relative to males (p = .08).
Similar gender differences for achievement goal orientation have also been observed (Dunn & Shapiro, 1999).
7 of 26
Voluntary homework
Providing students with structured learning opportunities outside school is a way to extend the school day.
The online lessons used in this investigation run in a browser delivered from a server on the internet. This means that the online lessons are available not only in school but anywhere that students can access the internet.
Which students (if any) will choose to do the online lessons as voluntary ‘homework’?
8 of 26
Participants
8th grade students (total N = 218) in the Wyoming Area School District in northeastern Pennsylvania 17 excluded because they did not participate in
the online lessons. 7 excluded because they did not have 2006 PSSA
data. Thus the final sample used in the data analysis
consisted of n = 194 students. The students attended classes in a facility that
houses all 7th through 12th grade students (facility N = 1,315).
In this facility, 98% of students are Caucasian and 24% are on free or reduced lunch.
9 of 26
Setting and Treatment
Online IPs – CompassLearning lessons delivered over the internet that are aligned to state standards and examination anchors in mathematics and reading, and are unique for each student based on past performance measures.
Participants were randomly assigned to the Advisement or No-Advisement treatment that consisted of a progress report sent to parents in Week Three (in addition, gender and achievement goal orientation were two other independent variables).
10 of 26
Effort and achievement dependent measures
Effort – Number of online lessons completed
Achievement – PSSA Reading and Math scaled scores
Other descriptive data homework completion data Self-report survey measuring achievement
goal orientation and internet access.
11 of 26
Achievement goal orientation survey A ten item self-report survey to measure
achievement goal orientation (from Hannah’s dissertation) even numbered items – performance goal odd numbered items – mastery goal
The self-report survey was completed by 164 students (84 females and 80 males), 30 students (14 females and 16 males) did not take the survey
Breakdown: performance oriented (43%; n = 71 with 33
females and 38 males) mastery oriented (41%; n = 67 with 32 females
and 35 males) neutral achievement goal orientation (15%; n =
26 with 19 females and 7 males).
See the survey in Appendix A
12 of 26
Achievement goal orientation analysis
Principal components analysis with varimax rotation obtained two factors.
The ten items aligned as they should into mastery and performance categories.
Cronbach alpha for the 10-item achievement goal orientation: All 10, α = 0.866 Odd numbered (m), α =
0.789 Even numbered (p), α =
0.837coefficients sorted by size
.485.320Item14
.489.305Item10
.663.442Item18
.871.178Item16
.901.066Item12
.197.665Item19
.235.705Item11
.223.779Item13
.260.795Item17
.184.801Item15
21
Component
.485.320Item14
.489.305Item10
.663.442Item18
.871.178Item16
.901.066Item12
.197.665Item19
.235.705Item11
.223.779Item13
.260.795Item17
.184.801Item15
21
Component
mastery
performance
13 of 26
PSSA data availableJan 15 Feb 15 Mar 15
August
Mail online IP progress report letter to parents of randomly selected group
Students work on online IPs
Jan 27
Print all finalprogress reports
Mar 20
Student and teacher interviewsStudent self-report survey
Week of Mar 13
Jan 9 Mar 17Mar 21 Mar 31
Letter to 8th grade parents
Week of Dec 19
Timeline of milestones…
PSSA makeup
Changed to the Week of Mar 31
14 of 26
Achievement: PSSA results
Reading Mathematics
Adv
ance
d
Pro
fici
ent
Bas
ic
Bel
ow B
asic
Adv
ance
d
Pro
fici
ent
Bas
ic
Bel
ow B
asic
no report - mastery 17 8 1 2 15 9 1 3 - neutral 10 3 0 0 7 2 3 1 - performance 29 11 2 0 27 10 5 0 - unknown * 9 6 2 1 6 8 3 1 report - mastery 25 13 1 0 19 14 5 1 - neutral 10 3 0 0 4 8 1 0 - performance 19 7 2 1 17 8 4 0 - unknown * 8 3 0 1 7 4 1 0 (N = 194)
Cut score >1472 >1279 >1145 ≤1145 >1445 >1283 >1170 ≤1170 Total (%) 65% 28% 4% 3% 53% 32% 12% 3%
* unknown, are those students who did not complete the self-report survey
93% reading 85% math
15 of 26
PSSA Analysis
The 2006 reading and mathematics PSSA scaled score data were analyzed by a 3-between x 2-within mixed ANOVA.
The between-subjects factors were Treatment (report or no report), Gender (female or male), and Achievement Goal (mastery, neutral, performance, and unknown).
The within-subjects repeated measure was Content (reading and mathematics).
16 of 26
Mixed ANOVA
Source SS df MS F Sig. Intercept 660169193.941 1 660169193.941 7875.147 0.000 Treatment (T) 21913.967 1 21913.967 0.261 0.610 Gender (G) 42090.836 1 42090.836 0.502 0.480 Achievement Goals (A) 70509.520 3 23503.173 0.280 0.840 T * G 72418.436 1 72418.436 0.864 0.354 T * A 245969.187 3 81989.729 0.978 0.404 G * A 133613.943 3 44537.981 0.531 0.661 T * G * A 177922.931 3 59307.644 0.707 0.549 Error 14921641.530 178 83829.447 PSSA Reading and Math within subjects repeated measure Content (C) 547982.057 1 547982.057 32.686 0.000 C * T 657.237 1 657.237 0.039 0.843 C * G 137810.678 1 137810.678 8.220 0.005 C * A 6710.917 3 2236.972 0.133 0.940 C * T * G 81137.649 1 81137.649 4.840 0.029 C * T * A 23576.936 3 7858.979 0.469 0.704 C * G * A 30043.814 3 10014.605 0.597 0.618 C * T * G * A 47549.357 3 15849.786 0.945 0.420 Error(Content) 2984210.930 178 16765.230
17 of 26
PSSA: Content x Gender interaction
1400
1450
1500
1550
1600
1650
math reading
female
male
PSSA
Sca
led
Sco
re
1400
1450
1500
1550
1600
1650
math reading
female
male
PSSA
Sca
led
Sco
re
reading
math
18 of 26
PSSA: Content x Treatment x Gender interaction
1,520
1,540
1,560
1,580
1,600
1,620
Female PSSA Reading
Male PSSA Reading
no report report
treatment
1,440
1,460
1,480
1,500PSSA
Sca
led
Sco
res
Female PSSA Mathematics
Male PSSA Mathematics
1,520
1,540
1,560
1,580
1,600
1,620
Female PSSA Reading
Male PSSA Reading
no report report
treatment
1,440
1,460
1,480
1,500PSSA
Sca
led
Sco
res
Female PSSA Mathematics
Male PSSA Mathematics
19 of 26
Effort: Analysis of the number of lessons completed The CompassLearning online IP total lessons
completed data (combined reading and mathematics) were analyzed by a three factor ANOVA that included: Treatment (report or no report), Gender (female or male), and Achievement Goal (mastery, neutral, performance, and unknown).
Results: None of the main factors or interactions were significant except for the 3-way interaction of Treatment, Gender, and Achievement Goal (T * G * A), F(3,178) = 2.680, MSE = 394.182, p = .048
20 of 26
Lessons completed: Treatment x Gender x Achievement Goal interaction
Male, Mastery orientation
Male, Performance orientation
no report report
30
35
40
45
50
55
IEP
Les
sons
Com
plet
ed
Female, Mastery orientation
Female, Performance orientation
Treatment
(possible ceiling effect)
note
21 of 26
Did they do Homework?
The online lessons were not checked by teachers, and no daily grade was given for completion, so there was little or no pressure to do the online lessons outside school.
5.7% of the students worked on the online IPs outside of school time. Most were females (9 females and 2 males).
ANOVA indicates that there was apparently an interaction (see next slide).
22 of 26
Possible interaction of Treatment x Gender for minutes of homework completed
no report report
treatment
0
5
10
15female
male
Hom
ewor
k m
inut
es
Caution, small sample n=11
23 of 26
Findings
In general terms, sending advisement home to parents … relates to increased achievement for males in
reading and math, for females in math, but a decrease (sig.) for females in PSSA reading scores.
did not positively influence females to do more online IP lessons. In fact, the presence of reports seemed to depress females’ online IP lesson completion (especially those with a performance achievement goal orientation).
This is the opposite of the trend observed by Clariana (1993) that females may benefit more than males from progress reports.
24 of 26
Regarding praise
Deci (1975) proposed that males tend to be more sensitive to the informational aspect of praise and so perceive praise as an informational message that they are competent in the relevant task. This tends to enhance their intrinsic motivation for the task.
In stark contrast, he proposed that females tend to be more interpersonally sensitive and concerned with others’ evaluations of them and so perceive praise as a controlling intervention that diminishes their autonomy, thus decreasing their intrinsic motivation for the task.
Also Katz, Assor, Kanat-Maymon, and Bereby-Meyer (2006) extended Deci’s hypothesis to include the effects of gender differences in reaction to advisement.
25 of 26
Possible explanations…
Katz et al. (2006) In 7th grade students with a moderate task interest, advisement had a positive effect on males’ and a negative effect on females’ self-report of intrinsic motivation.
Katz’s et al. findings are in line with the results of the present investigation for females with a performance orientation (see Figure on slide 20).
Similar to Deci’s hypothesis that females tend to be more interpersonally sensitive and concerned with others’ evaluations of them, performance-oriented females in the present investigation perhaps resented advisement as an attempt to control or manipulate their actions (which, of course, was exactly the case).
26 of 26
What would you do next time?
If you are the principal?
If you are the teacher?
If you are the investigator?
What effect did the age of the students have on the study?
28 of 26
Appendix B: Correlation table
PS
SA
Rea
ding
PS
SA
Mat
h
IP L
esso
ns c
ompl
eted
IP L
esso
n sc
ore
IP H
omew
ork
min
utes
Item
10 (
P)
Item
12 (
P)
Item
14 (
P)
Item
16 (
P)
Item
18 (
P)
Item
11 (
M)
Item
13 (
M)
Item
15 (
M)
Item
17 (
M)
Item
19 (
M)
PSSA Reading 1 0.66 0.22 0.56 0.20 0.35 0.15 0.17
PSSA Math 0.66 1 0.26 0.51 0.28 0.21 0.15 0.19 0.18
IP Lessons completed 0.22 0.26 1 0.18 0.23
IP Lesson score 0.56 0.51 0.18 1 0.33 0.21 0.16 0.24 0.16 0.19 0.18 0.27 0.20
IP Homework minutes 0.20 0.23 1
Item10 (P) 0.35 0.28 0.33 1 0.33 0.32 0.34 0.37 0.35 0.27 0.29 0.30 0.26
Item12 (P) 0.21 0.21 0.33 1 0.33 0.77 0.56 0.25 0.33 0.25 0.29 0.25
Item14 (P) 0.16 0.32 0.33 1 0.35 0.38 0.23 0.32 0.34 0.27 0.36
Item16 (P) 0.15 0.24 0.34 0.77 0.35 1 0.57 0.36 0.35 0.35 0.43 0.23
Item18 (P) 0.37 0.56 0.38 0.57 1 0.45 0.43 0.42 0.53 0.44
Item11 (M) 0.16 0.35 0.25 0.23 0.36 0.45 1 0.46 0.54 0.61 0.33
Item13 (M) 0.15 0.19 0.19 0.27 0.33 0.32 0.35 0.43 0.46 1 0.61 0.62 0.50
Item15 (M) 0.18 0.29 0.25 0.34 0.35 0.42 0.54 0.61 1 0.60 0.42
Item17 (M) 0.17 0.18 0.27 0.30 0.29 0.27 0.43 0.53 0.61 0.62 0.60 1 0.44
Item19 (M) 0.20 0.26 0.25 0.36 0.23 0.44 0.33 0.50 0.42 0.44 1