The Connection Between Self-Regulated Learning and Student Success in a Hybrid Course
Kevin R. Guidry
Jorge Cubillos
Kathleen L. Pusecker
University of Delaware
Paper presented at the Association for Institutional Research Annual Forum
Long Beach, California
May 21, 2013
Author Note
Kevin R. Guidry, Center for Teaching & Assessment of Learning, University of
Delaware; Jorge Cubillos, Department of Foreign Languages and Literatures, University of
Delaware; Kathleen L. Pusecker, Center for Teaching & Assessment of Learning, University of
Delaware.
This research was supported in part by Pearson Education and a grant from the University
of Delaware Center for Teaching & Assessment of Learning. This paper was originally titled
Using Self-Regulated Learning to Predict Student Success in a Hybrid Course and appears under
that title in the AIR Forum program; helpful suggestions from reviewers resulted in this change.
Additionally, this version correct two errors made in the original conference paper: age was
miscoded in the original data set and the reference level for the dummy-coded prior Spanish
Self-Regulated Learning and Student Success in a Hybrid Course 2
learning experience was mistakenly included in the first multiple regression model. These errors
have been corrected and had little impact on the overall results of this study.
Correspondence concerning this article should be addressed to Kevin R. Guidry, Center
for Teaching & Assessment of Learning, 212 Gore Hall, University of Delaware, Newark, DE,
19716. E-mail: [email protected]
Self-Regulated Learning and Student Success in a Hybrid Course 3
The Connection Between Self-Regulated Learning and Student Success in a Hybrid Course
This paper describes preliminary results from a study at the University of Delaware
examining the relationship between self-regulated learning and student success in a hybrid
course using a modified version of the Motivated Strategies for Learning Questionnaire (MSLQ)
to determine if self-regulated learning is important to success in a hybrid course. By comparing
students’ performance in five sections of introductory Spanish, three face-to-face and two hybrid,
we examine how self-regulated learning behaviors and beliefs affect learning and student
success.
At the University of Delaware, there are currently 129 majors that have a foreign
language graduation requirement. In the face of increasing financial pressures, some colleges and
universities are turning to technology – specifically, online instruction – to preserve their
language offerings (Arnone, 2002). Indeed, current technological advances may make it
financially and pedagogically viable to transition away from the traditional classroom towards a
virtual one (Hokanson, 2000; Kinney & Robertson, 2003; Tunison & Noonan, 2001). Although
we are not yet experiencing direct political or financial pressure to reduce or streamline our
foreign language coursework, we anticipate that we may experience those pressures and we want
to ensure that we are prepared to meet them. Specifically, we want to ensure that (a) students in
hybrid courses exhibit similar or better gains in learning and (b) we understand some of the
characteristics of successful performance in hybrid courses so we can adequately advise students
who may have to select between hybrid and traditional face-to-face courses.
Self-Regulated Learning and Student Success in a Hybrid Course 4
Literature Review
Hybrid Courses
Although a move away from traditional forms of delivery of instruction is now possible
(and in fact, quite attractive from space requirement viewpoint), there is limited information
about the pedagogical impact of such a change for foreign language instruction. The body of
applicable research available has focused on the applications of technology-assisted instruction
to different aspects of language learning such as vocabulary, grammar, and reading
comprehension. Unfortunately, the information available on the comparative pedagogical
advantage of this online learning environment versus the traditional face-to-face model is very
limited (Young, 2002).
A slightly larger body of research exists that focuses on hybrid instruction, a combination
of traditional face-to-face with online teaching (Carroll, 2003; Hopper, 2003; Oblender, 2002;
Patterson, 2004). This option is being used in other disciplines such as history, economics, textile
science, nursing, and computer science with encouraging results (Brown & Liedholm, 2002;
Combs, 2004; Krawiec, Salter & Kay, 2005, Riffell & Sibley, 2005; Salamonson & Lantz,
2005). Despite the recent popularity of “flipped classes,” there is little high-quality research that
specifically addresses college-level language coursework taught in a hybrid format.
Our primary concern with the introduction of hybrid language learning courses is the
efficacy of the courses in meeting learning outcomes. The lack of research in this area forces us
to reluctantly turn to the larger body of literature addressing the efficacy of different media on
college-level coursework with much of it focusing specifically on distance education. The bulk
of this research can best be characterized as finding that the medium usually has no significant
difference on student learning (Clark, 2001; Russell, 1999). The most well-known recent meta-
Self-Regulated Learning and Student Success in a Hybrid Course 5
study concluded that students in hybrid courses performed better than students in online or face-
to-face courses; this finding, however, is clouded by the observation that such courses tended to
require students to spend more time-on-task (Means, et al., 2010).
Self-Regulated Learning
One of our primary beliefs is that students who practice appropriate behaviors and
beliefs, including adaptation to different circumstances, self-monitoring, and self-reflection, are
more successful. These ideas are nicely captured in the idea of self-regulation (Bembenutty,
2011; Weinstein, Meyer, Van Mater Stone, & McKeachie, 2006; Zimmerman, 2000). Broadly
construed, this concept includes areas such as time management, motivation, self-monitoring,
and management of anxiety. Alternatively, these ideas can be classified into cognitive,
metacognitive, and motivational strategies. Importantly, these are learned skills that we can help
students identify and improve (e.g. Weinstein, Husman, & Dierking, 2000).
Luckily, there is a well-tested and freely-available instrument to measure self-regulation:
the Motivated Strategies for Learning Questionnaire (MSLQ). This instrument was developed in
the 1980s with funding from the National Science Foundation and the Department of Education
(Duncan & McKeachie, 2005) and its psychometric properties have been thoroughly vetted
(Pintrich, Smith, Garcia, & McKeachie, 1993; Taylor, 2012). This vetting has also included
recommendations to modify the instrument to improve it such as those made by Dunn, Lo,
Mulvenon, and Sutcliffe (2012).
Research Questions
The following research questions guided this study:
RQ 1: What is the difference in student learning in face-to-face and hybrid sections of
this course?
Self-Regulated Learning and Student Success in a Hybrid Course 6
RQ 2: What is the relationship between self-regulated learning, as measured by a
modified version of the MSLQ, and student success when student success is defined as (a)
improved performance on a normed language proficiency test or (b) a higher average grade on
course assignments and tests?
Methodology
This study examines characteristics and measures of approximately 100 undergraduate
students enrolled in five sections of introductory Spanish in the spring of 2013. This is a four-
credit course that typically meets four times each week for one hour each day. Two sections,
however, were taught as hybrid courses where students met only twice a week and conducted
much of their learning online. Class time was designated to be used primarily to practice Spanish
with peers and the instructor.
We employed as many controls in this quasi-experiment as practicable. Students did not
know that two sections were going to be taught as hybrid courses when they signed up for the
course, reducing the impact of self-selection (although students were, of course, permitted to
change course sections or withdraw). Although the two sets of sections were taught by different
faculty, they were scheduled at similar days and times to minimize the impact of those variables.
Finally, the eight tests and major assessments – four tests, two in-class compositions, one oral
recording, and one oral exam – were identical across all five course sections.
Sources of Data
Data were collected from and about the students in four ways. First, some data were
extracted from the university’s student information system. This included demographic
information such as gender, race/ethnicity, and first generation status. Second, grades from
assignments and assessments were included in the study. These were common for all sections of
Self-Regulated Learning and Student Success in a Hybrid Course 7
the course and include four written tests, two in-class written compositions, an oral recording,
and an oral test.
A third source of data is a language proficiency exam administered at the beginning and
end of the semester. The STAMP 4S (STAndards-based Measurement of Proficiency – 4 Skills)
test was administered during class. Originally developed by language educators at the University
of Oregon’s Center for Applied Second Language Studies with a Fund for Improvement of Post-
Secondary Education (FIPSE) grant, STAMP 4S is a computer-adaptive test with four sections:
interpretative listening, interpretative reading, presentational writing, and presentational speaking
(Avant Assessment, 2013). Sixty-six students took both the pre- and post-test and completed
enough of the test to register scores in the listening and reading sections. These students were
beginning language learners, however, and too few completed the writing and speaking sections
to contribute meaningful data for this study. Avant Assessment, the company that has licensed
and administers the STAMP 4S, reports both general proficiency levels and scaled scores; in this
study, we use the scaled scores.
The fourth source of data is a survey that combined some demographic and demographic-
like questions with a modified version of the MSLQ. The first fourteen questions focused on
student characteristics, experiences, and preferences including several questions focused on
students’ use and access to Internet-connected devices. One critical question asked students how
long they had spent studying one or more foreign language courses. These questions were tested
in cognitive interviews with students in a Spanish course prior to finalizing the survey
instrument.
The remaining sixty questions are a modified version of the MSLQ. The MSLQ was
modified in three ways. First, two of the original subscales – Metacognitive Self-Regulation and
Self-Regulated Learning and Student Success in a Hybrid Course 8
Effort Regulation – were removed and replaced with General Strategies for Learning and
Clarification Strategies for Learning subscales as recommended by Dunn, Lo, Mulvenon, and
Sutcliffe (2012). In their study, the removed subscales did not exhibit satisfactory psychometric
properties but the new subscales emerged. Second, questions that were reverse-coded were
modified or deleted. Some of these questions were specifically deleted per the recommendations
of Dunn et al. Four questions not included in their study were also modified or deleted both to
keep the instrument consistent and because the arguments made by Dunn et al. against their use
are convincing. Third, questions were slightly modified to better fit the context of this study; one
question could not be modified and was deleted.
The modified MSLQ used in this study includes two scales: Motivation and Learning.
The motivation scale includes questions about control of learning beliefs, extrinsic goal
orientation, intrinsic goal orientation, self-efficacy for learning and performance, and task value.
The learning scale includes questions about clarification strategies for learning, elaboration,
general strategies for learning, help seeking, organization, peer learning, rehearsal, and time and
study environment.
Analysis Methods
This study focuses on the relationship between self-regulated learning, as measured by a
modified version of the MSLQ, and two measures of student success: Improved performance on
a normed language proficiency test and a higher average grade on course assignments and tests.
To answer our research questions, we created three multiple linear regression models. Model 1
uses the students’ average score on course tests and assessments as the dependent variable.
Models 2 and 3 uses improvement on the student scores on the language proficiency test as the
dependent variables; model 2 focuses on the improvement in reading score and model 3 focuses
Self-Regulated Learning and Student Success in a Hybrid Course 9
on the improvement in the listening score. The independent variables are listed below in Table 1.
Note that the models exploring improved performance on the language proficiency test do not
include prior Spanish language learning experience as an independent variable; we expected that
prior experience would have influenced performance on the pre-test so this is already accounted
for in those models.
Table 1: Independent Variables in Multiple Regression Models
Variable Models Variable type Age 1, 2, 3 Continuous First-generation status 1, 2, 3 Dichotomous (Non-first-gen as reference) Gender 1, 2, 3 Dichotomous (Male as reference) Hybrid course status 1, 2, 3 Dichotomous MSLQ learning scale 1, 2, 3 Continuous MSLQ motivation scale 1, 2, 3 Continuous Race/ethnicity 1, 2, 3 Dichotomous (White as reference) Spanish language learning experience
1 Dummy coded in three levels: less than two years (reference), at least two years but less than four, four or more years
Results
The first research question asks if there is a difference in student learning in face-to-face
and hybrid sections of this course. As discussed above, we employed three measures of student
learning in this study; means and standard deviations of those measures are shown in Table 2.
The differences between the means of the two groups are not statistically significant for any of
the measures.
Self-Regulated Learning and Student Success in a Hybrid Course 10
Table 2: Average Grades and Proficiency Scores
Mode of instruction
Number of students
Grades Average reading proficiency score
change
Average listening proficiency score
change Mean Standard
deviation Mean Standard
deviation Mean Standard
deviation Face-to-face
66 83.9 9.0 33.9 36.3 28.2 40.1
Hybrid 34 83.7 7.8 47.5 39.3 16.2 44.0
Moving beyond simple t-tests to allow us to employ controls, in model 1 we regressed
our selected independent variables against the students’ average test and assessment score. The
model is statistically significant (p = .027) and has modest predictive power (R2 = .287, F(9, 52)
= 2.329. As shown in Table 3, only two variables in the model (other than the constant) were
significant: the MSLQ motivation scale (B = .419, p =.001) and the students’ age (B = -.307,
p=.026).
Table 3: Multiple Regression Model 1
Variable B Standard Error of B p Constant 68.645 8.819 .000 Hybrid course status 1.983 2.260 .384 MSLQ learning scale -.167 .103 .109 MSLQ motivation scale .419 .113 .001 Race/ethnicity .415 2.699 .878 Age -.307 .134 .026 First-generation status -1.275 2.973 .670 Gender 3.488 1.912 .074 Previous Spanish experience: Some 0.219 2.393 .927 Previous Spanish experience: Most -0.232 2.847 .935
Self-Regulated Learning and Student Success in a Hybrid Course 11
Model 2 also produced interesting results. In this model we regressed our selected
independent variables against students’ gains in language proficiency reading scores. Like Model
1, this model has modest predictive power and is statistically significant (R2 = .400, F(7, 43) =
4.087, p = .002). Moreover, as shown in Table 4 two of independent factors had significant
loadings: the MSLQ learning scale (B = 1.537, p=.003) and the students’ age (B = 2.269,
p=.001). Model 3 was not significant nor did it have even modest predictive power (R2 = .029,
F(7, 42) = .182, p = .988). All three models satisfactorily satisfied tests of the assumptions of
multiple linear regression.
Table 4: Multiple Regression Model 2
Variable B Standard Error of B p Constant -28.334 48.378 .561 Hybrid course status 11.096 10.938 .316 MSLQ learning scale 1.537 0.492 .003 MSLQ motivation scale -1.173 0.605 .059 Race/ethnicity 6.565 12.723 .609 Age 2.269 0.646 .001 First-generation status 21.622 14.492 .143 Gender -22.449 10.779 .043
Discussion
First, we note that the regression models show that the mode of instruction did not
significantly affect students’ grades or language proficiency. This is consistent with the wider
body of research that has shown that the mode of instruction rarely has no significant effect on
learning. This reassures us that if we choose or are forced to implement more hybrid courses or
course sections they appear to be at least as effective as face-to-face ones that require more
classroom space and other campus resources.
Self-Regulated Learning and Student Success in a Hybrid Course 12
Second, in two of our three models we found significant independent variables. Age and
the MSLQ motivation scale were significant predictors of higher test and assessment grades.
Age and the MSLQ learning scale were significant predictors of improved proficiency in reading
Spanish. This is a muddled picture.
The role of age in predicting these two outcomes is unclear as age is a negative predictor
of higher test and assessment grades but a positive predictor of improved proficiency in reading
Spanish i.e., younger students earned higher grades but older students performed better on the
Spanish reading portion of the proficiency test. It is possible that older students have different
motivations with less focus on earning higher grades in the course and more focus on acquiring
skills and knowledge.
It is not clear why only the MSLQ motivation scale predicted higher test and assessment
grades, only the MSLQ learning scale predicted improved proficiency in reading Spanish, and
neither scale predicted improved proficiency in listening to Spanish. Our hypotheses were that
self-regulated learning – motivation and learning – would lead to higher grades and increased
learning. These muddled, fragmentary results do not seem to support those hypotheses since the
MSLQ scales only predict success in limited ways. Moreover, it is surprising that those scales
are independent of age; this is not what we expected based on our experience, observations of
these classrooms, and interviews with these students.
Further research should seek to expand the sample size so additional factors can be
explored and more powerful statistical models can be tested. For example, the MSLQ has several
robust subscales that make up the two primary scales and a larger sample size would enable us to
determine if some of those scales may be more useful or informative in determining correlates or
Self-Regulated Learning and Student Success in a Hybrid Course 13
predictors of student success. Further research should also expand beyond the first, introductory
Spanish course as some gains may not be apparent or even possible in only one semester.
Conclusion
In this study of 100 students enrolled in five sections of an introductory Spanish course,
we explored the impact of changing the mode of delivery to a hybrid mode for two sections. The
mode of instruction did not significant impact any of our learning outcomes, confirming the “no
significant difference” phenomenon widely reported in previous studies of online and hybrid
courses. Using a modified version of the Motivated Strategies for Learning Questionnaire, we
discovered a muddled picture that does not fully support our hypotheses that self-regulation has a
significant impact on our learning outcomes.
Self-Regulated Learning and Student Success in a Hybrid Course 14
References
Arnone, M. (2002). 3 Iowa Universities go online to preserve Eastern European language
courses. The Chronicle of Higher Education,48(45).
Avant Assessment. (2013). STAMP 4S (STAndards-based Measurement of Proficiency – 4 Skills)
Spanish Technical Report. Retrieved from
http://www.avantassessment.com/sites/default/files/Spanish%20STAMP%20technical%2
0document.pdf
Bembenutty, H. (Ed.) (2011). Self-Regulated Learning. New Directions for Teaching and
Learning, 126. San Francisco, CA: Jossey-Bass.
Brown, B. W., & Liedholm, C. E. (2002). Can web courses replace the classroom in Principles of
Microeconomics? The American Economic Review, 92(2), 444-448.
Carroll, B (2003). Going hybrid: Online course components increase flexibility of on-campus
courses. Online Classroom, February, 4-7.
Clark, R. E. (2001). Learning from media: Arguments, analysis and evidence. Greenwich, CT:
Information Age Publishers.
Combs, L. (2004). Science education in the web era. Journal of Computers in Mathematics and
Science Teaching, 23(2), 139-149.
Duncan, T. G., & McKeachie, W. J. (2005). The making of the Motivated Strategies for
Learning Questionnaire. Educational Psychologist, 40(2), 117-128
Hokanson, S. G. (2000). Distance education in foreign languages. Rocky Mountain Review of
Language and Literature, 54(2), 85-93.
Hopper, K. (2003). Reasons to go hybrid. Distance Education Report, 7(24), 7.
Self-Regulated Learning and Student Success in a Hybrid Course 15
Kinney, D. P., & Robertson, D. F (2003). Technology makes possible new models for delivering
developmental Mathematics instruction. Mathematics and Computer Education, 37(3),
315-28.
Krawiec, S., Salter, D., & Kay, E. J. (2005). A “hybrid” Bacteriology course: The professor’s
design and expectations; the students’ performance and assessment. Microbiology
Education, 6, 8-13.
Means, B., Toyama, Y., Murphy, R., Bakia, M., & Jones, K. (2010). Evaluation of evidence
based practices in online learning: A meta-analysis and review of online learning
studies. Washington, DC: U.S. Department of Education Office of Planning, Evaluation,
and Policy Development Policy and Program Studies Service.
Oblender, T. E. (2002). A hybrid course model: One solution to the high online drop-out rate.
Learning and Leading with Technology, 29(6), 42-46.
Patterson, J. (2004). For quality and cost-effectiveness build a hybrid program. Distance
Education Report, 8(21), 1-2.
Pintrich, P. R., Smith, D. A. F., Garcia, T., & McKeachie, W. J. (1993). Reliability and
predictive validity of the MSLQ. Educational and Psychological Measurement, 53, 801-
813.
Riffell, S., & Sibley, D. (2005). Using web-based instruction to improve large undergraduate
biology courses: An evaluation of a hybrid course format. Computers & Education, 44,
217-35.
Russell, T. L. (1999). The No Significant Difference Phenomenon As Reported in 355 Research
Reports, Summaries and Papers. Raleigh, NC: North Carolina State University.
Self-Regulated Learning and Student Success in a Hybrid Course 16
Salamonson, Y., & Lantz, J. (2005). Factors influencing nursing students’ preference for a
hybrid format delivery in a pathophysiology course. Nurse Education Today, 25, 9-16.
Taylor, R. T. (2012). Review of the Motivated Strategies for Learning Questionnaire (MSLQ)
Using Reliability. Unpublished doctoral dissertation, Auburn University.
Tunison M., & Noonan, B (2001). On-line learning: Secondary students’ first experience.
Canadian Journal of Education/Revue Canadienne de l’Education, 26(4), 495-511.
Weinstein, C. E., Husman, J., & Dierking, D. R. (2000). Interventions with a focus on learning
strategies. In M. Boekaets, P. R. Pintrich, and M. Zeidner (eds.), Handbook of Self-
Regulation (pp. 727-747). San Diego, CA: Academic Press.
Weinstein, C. E., Meyer, D. K., Van Mater Stone, G., & McKeachie, W. J. (2006). Teaching
students how to become more strategic and self-regulated learners. In W. J. McKeachie
and M. Svinicki (eds.) McKeachie’s Teaching Tips, 12th ed. Boston, MA: Houghton
Mifflin.
Young, J. R. (2002). ‘Hybrid’ teaching seeks to end the divide between traditional and online
instruction. The Chronicle of Higher Education, 48(28), A33-A34.
Zimmerman, B. J. (2000). Attaining self-regulation: A social cognitive perspective. In M.
Boekaets, P. R. Pintrich, and M. Zeidner (eds.), Handbook of Self-Regulation (pp. 13-39).
San Diego, CA: Academic Press.
Self-Regulated Learning and Student Success in a Hybrid Course 17
Appendix
Self-Regulated Learning and Student Success in a Hybrid Course 18
Self-Regulated Learning and Student Success in a Hybrid Course 19
Self-Regulated Learning and Student Success in a Hybrid Course 20
Self-Regulated Learning and Student Success in a Hybrid Course 21
Self-Regulated Learning and Student Success in a Hybrid Course 22