Paper ID #15770 Improving Students’ Learning in Statics Skills: Using Homework and Exam Wrappers to Strengthen Self-regulated Learning Kai Jun Chew, Stanford University Kai Jun (KJ) Chew is a Research Data Analyst in the Mechanical Engineering department at Stanford University. He is currently working closely with Dr. Sheri Sheppard on two fronts: introducing reflec- tive activities as part of the Consortium to Promote Reflection in Engineering Education (CPREE) and implementing the Continuous Improvement Program as part of the ABET evaluation. Born and raised in Malaysia, KJ received his Bachelor of Science in Mechanical Engineering at the University of South- ern California (USC) and his Master of Science in the same field at Stanford University. He is currently exploring the field of data science as his potential career path. Dr. Helen L. Chen, Stanford University Helen L. Chen is a research scientist in the Designing Education Lab in the Department of Mechanical Engineering and the Director of ePortfolio Initiatives in the Office of the Registrar at Stanford University. She is also a member of the research team in the National Center for Engineering Pathways to Innovation (Epicenter). Chen earned her undergraduate degree from UCLA and her Ph.D. in Communication with a minor in Psychology from Stanford University in 1998. Her current research interests include: 1) engineering and entrepreneurship education; 2) the pedagogy of ePortfolios and reflective practice in higher education; and 3) reimagining the traditional academic transcript. Beth Rieken, Stanford University Beth Rieken is a sixth year graduate student at Stanford University. She is currently working on her PhD in Mechanical Engineering with a focus on the relevance of mindfulness to engineers. Beth completed a BS in Aerospace Engineering from the University of Virginia in 2010 and a MS in Mechanical Engineering from Stanford in 2012. Autumn Turpin, Stanford University Autumn Turpin is a junior undergraduate studying Engineering, Product Design at Stanford University. She was born and raised in the Bay Area. She has been working with the Designing Education Lab since January ’14. Dr. Sheri Sheppard, Stanford University Sheri D. Sheppard, Ph.D., P.E., is professor of Mechanical Engineering at Stanford University. Besides teaching both undergraduate and graduate design and education related classes at Stanford University, she conducts research on engineering education and work-practices, and applied finite element analysis. From 1999-2008 she served as a Senior Scholar at the Carnegie Foundation for the Advancement of Teaching, leading the Foundation’s engineering study (as reported in Educating Engineers: Designing for the Future of the Field). In addition, in 2011 Dr. Sheppard was named as co-PI of a national NSF innovation center (Epicenter), and leads an NSF program at Stanford on summer research experiences for high school teachers. Her industry experiences includes engineering positions at Detroit’s ”Big Three:” Ford Motor Company, General Motors Corporation, and Chrysler Corporation. At Stanford she has served a chair of the faculty senate, and recently served as Associate Vice Provost for Graduate Education. c American Society for Engineering Education, 2016
Transcript
Paper ID #15770
Improving Students’ Learning in Statics Skills: Using Homework and ExamWrappers to Strengthen Self-regulated Learning
Kai Jun Chew, Stanford University
Kai Jun (KJ) Chew is a Research Data Analyst in the Mechanical Engineering department at StanfordUniversity. He is currently working closely with Dr. Sheri Sheppard on two fronts: introducing reflec-tive activities as part of the Consortium to Promote Reflection in Engineering Education (CPREE) andimplementing the Continuous Improvement Program as part of the ABET evaluation. Born and raisedin Malaysia, KJ received his Bachelor of Science in Mechanical Engineering at the University of South-ern California (USC) and his Master of Science in the same field at Stanford University. He is currentlyexploring the field of data science as his potential career path.
Dr. Helen L. Chen, Stanford University
Helen L. Chen is a research scientist in the Designing Education Lab in the Department of MechanicalEngineering and the Director of ePortfolio Initiatives in the Office of the Registrar at Stanford University.She is also a member of the research team in the National Center for Engineering Pathways to Innovation(Epicenter). Chen earned her undergraduate degree from UCLA and her Ph.D. in Communication witha minor in Psychology from Stanford University in 1998. Her current research interests include: 1)engineering and entrepreneurship education; 2) the pedagogy of ePortfolios and reflective practice inhigher education; and 3) reimagining the traditional academic transcript.
Beth Rieken, Stanford University
Beth Rieken is a sixth year graduate student at Stanford University. She is currently working on her PhD inMechanical Engineering with a focus on the relevance of mindfulness to engineers. Beth completed a BSin Aerospace Engineering from the University of Virginia in 2010 and a MS in Mechanical Engineeringfrom Stanford in 2012.
Autumn Turpin, Stanford University
Autumn Turpin is a junior undergraduate studying Engineering, Product Design at Stanford University.She was born and raised in the Bay Area. She has been working with the Designing Education Lab sinceJanuary ’14.
Dr. Sheri Sheppard, Stanford University
Sheri D. Sheppard, Ph.D., P.E., is professor of Mechanical Engineering at Stanford University. Besidesteaching both undergraduate and graduate design and education related classes at Stanford University,she conducts research on engineering education and work-practices, and applied finite element analysis.From 1999-2008 she served as a Senior Scholar at the Carnegie Foundation for the Advancement ofTeaching, leading the Foundation’s engineering study (as reported in Educating Engineers: Designingfor the Future of the Field). In addition, in 2011 Dr. Sheppard was named as co-PI of a national NSFinnovation center (Epicenter), and leads an NSF program at Stanford on summer research experiences forhigh school teachers. Her industry experiences includes engineering positions at Detroit’s ”Big Three:”Ford Motor Company, General Motors Corporation, and Chrysler Corporation.
At Stanford she has served a chair of the faculty senate, and recently served as Associate Vice Provost forGraduate Education.
Improving Students’ Learning in Statics Skills: Using Homework and Exam Wrappers to Strengthen Self-Regulated Learning
Abstract Statics is one of the fundamental courses required for engineering students, particularly for students in the mechanical, civil and aerospace fields. This course introduces students to modeling and solving real-world systems, including drawing Free Body Diagrams (FBD) and setting up equilibrium equations. These two skills are critical for bridging introductory courses to more advanced courses, such as Dynamics, Mechanics of Materials and others. The process of teaching these foundational skills typically involves giving students opportunities to hone their problem solving skills through homework assignments and exams. In this paper, the authors introduce reflection as a tool to gauge understanding, confidence and performance. This too is used to intervene in homework assignments and exams in order to enhance and improve students’ meta-cognitive awareness and self-regulated learning. Homework and exam wrappers are reflection activities that prompt students to review their graded assignments and exams, and encourage students to reconsider their study habits and preparations. In this course, four short self-assessments were designed to help students identify their strengths and weaknesses by reflecting on their performances, the mistakes they made, their confidence in certain concepts, and views on best strategies for completing homework assignments and preparing for take-home exams in the future. Two of these wrappers were implemented after the graded homework assignments were returned to the students, and the other two were part of the exam reflection. At the end of the course, the students completed an anonymous survey about the reflective activities. Analyses focused on several different aspects of the homework and exam wrappers: number of mistakes made, levels of confidence, study strategies and students’ satisfaction on their performance. The findings suggest wrappers can have an important impact on students’ learning in engineering statics. Quantitative findings highlight several bright spots demonstrating positive impact of wrappers while qualitative findings present a strong argument for the use of wrappers in teaching and learning. Wrappers are also proven to provide the teaching team with information about students’ understanding of content and level of skills so that appropriate measures and actions can be taken to help students who are struggling in the course. Future work will include devising reliable quantitative measures for metacognitive skills, gathering more data to obtain a larger sample size, and redesigning of wrappers to optimize mutual benefits for both students and the teaching team. 1. Introduction From Donald Schön’s emphasis on the “reflection in action” and “reflection on action” as being a key characteristic of professional practice1 to the reflective learning phase in Kolb’s Experiential Learning Cycle2, the value of reflection in teaching and learning is broadly recognized and documented. More recently, John Dewey’s criteria for defining reflection were updated by characterizing reflection as a meaning-making process, a systematic and rigorous application of scientific inquiry, and an activity that must happen in the context of a community, particularly one that values attitudes of personal and intellectual growth.3
Increasingly, the potential benefits and contributions of integration of reflection into the engineering curricula and engineering education more broadly are being explored more widely4,5. Reflection has been described as “intentionally making meaning of experiences in service of future action”.6 This definition of reflection relates to the concepts of self-awareness and self-assessment in metacognition: the ability for students to identify what they know, how well they know it, and subsequently make choices about future learning strategies. The relevance of these activities and approaches related to reflection and their potential applications to engineering education are particularly evident to the fields of self-regulated learning and metacognition. Self-regulation or self-regulated learning is defined as “an active, constructive process whereby learners set goals for their learning and then attempt to monitor, regulate, and control their cognition, motivation, and behavior, guided and constrained by their goals and the contextual features in the environment” (p. 453).7 In the context of academic achievement and student motivation, three key features of self-regulated learning are identified: 1) a recognition of how the metacognitive, behavioral, and/or motivation strategies are used to help self-regulated learners achieve their academic goals; 2) a cyclical feedback loop monitoring the effectiveness of learning methods on an ongoing basis; and 3) a motivation for why a specific strategy or response is understood, emphasizing the interdependence between student learning and motivation.8 A reflection activity that incorporates these features of self-regulated learning is the concept of homework and exam wrapper. Wrappers are defined as structured reflection activities that guide students in the practice of metacognition after they get back their graded exams or assignments.9 For example, the exam wrapper is designed to transform the use of exams, particularly in entry-level college courses, from being simply an “end” or summative learning assessment to a more formative, iterative process. In particular, exam wrappers (along with homework wrappers, lecture wrappers, etc.) promote more effective self-regulated learning in students by asking students to reflect on their learning strategies, compare their learning strategies to the learning outcome results, and finally to adjust their learning strategies appropriately.9,10 By reviewing their graded homework and exams, students are prompted to consider how they might change the way prepare and study for assignments and exams in the future. As part of work with the Consortium to Promote Reflection in Engineering Education (CPREE), a series of homework and exam wrappers were designed and implemented in a statics course. These wrapper reflection activities required minimal in-class time to complete, were tailored to specific homework assignment and exam contexts, and were implemented several times throughout the course. This paper explores the impact of homework and exam wrappers in helping students identify their common mistakes on assignments and their confidence in engineering statics concepts. Our research team was interested in how homework and exam wrappers could be introduced into an introductory statics course with a focus on two guiding research questions as followed: 1) What can be learned from students’ experiences with homework and exam wrappers with respect to the potential impact on students’ confidence, course satisfaction, type of mistakes made, and course performance (grades) in a statics course?
2) What insights into the student learning and the course experience can the teaching team gain from reflections captured in homework and exam wrappers? Do they provide educators with a window into their students’ learning experience? 2. Descriptions of Course and Subjects ENGR 14: Introduction to Solid Mechanics (E14), is a statics course required for engineering majors. The structure of E14 includes weekly lectures, six problem sets, three exams, and weekly group-based labs taken over a ten-week quarter term. In the current study, wrappers were incorporated into two homework sets and one take-home exam.
Figure 1: Chronology of the Implementation E14 Wrappers
Four in-class wrappers were implemented over a period of five weeks (see Appendices E-H for the wrapper assignments), as outlined in Figure 1. A wrapper was given after the third homework assignment (HW3), after the fourth homework assignment (HW4), before the second exam (Pre-Exam 2), and after the second exam (Post-Exam 2). Students completed the HW3, HW4 and Post-Exam 2 Wrappers immediately in-class after their graded assignments were returned. The Pre-Exam 2 Wrapper was conducted a week before Exam 2 was administered. The first exam (Exam 1) was an in-class exam that occurred a week before HW3 was returned to the students. Exam 2 was a take-home exam. The HW3, HW4 and Post-Exam 2 Wrapper questions addressed four common elements: (1) student satisfaction with their assignment performance, (2) types of mistakes made in the assignment or exam, (3) level of confidence in knowledge of course concepts, and (4) open-ended questions on study strategies. The Pre-Exam 2 Wrapper focused exclusively on open-
ended exam study strategy questions. In addition to Lovett’s original wrapper design that prescribes questions on assignment mistakes and study strategies, the researchers added questions on student confidence and satisfaction to help the teaching team gauge students’ performance in the class. In Fall 2015, 75 students were enrolled in E14. Of these students, 70 completed the three wrapper assignments (HW3, HW4 and Post-Exam 2). Over half (57 percent) of the students included in this study were male, and 39 percent were female. The majority of the students were sophomores and juniors (87 percent, n=61) with a small percentage of freshmen and seniors (under 5 percent each). The planned major for 89 percent (n=62) of the students was in School of Engineering. Underrepresented minority (URM) students made 31 percent (n=22) of the total population. 3. Methods 3.1 Wrapper Data Analysis
Figure 2: Analysis Overview
To explore the “impact” of wrappers on students’ learning, a qualitative analysis on open-ended responses, descriptive and correlation statistics of quantitative responses, and a linear regression model predicting exam performance improvement were undertaken, as detailed below. The independent and dependent variables are outlined in Figure 2 and Appendix A. A. Student Reaction to the Assignment The HW3 and HW4 Wrappers prompted the students to reflect on their homework completion strategies. The HW4, Pre-Exam 2, and Post-Exam 2 Wrappers prompted students to reflect on their exam preparation strategies. This follows Lovett’s wrapper design in that it gives the
students a chance to make a connection between their study strategies and their assignment performance. Appendix C provides further explanations on the codes used on these open-ended responses. Students were asked in the HW4 and Post-Exam 2 wrappers to comment on whether they implemented these study strategies they listed and if those strategies helped them in their learning. Additionally, an anonymous survey was conducted at the end of the quarter to ask students about the reflection practices conducted in the course. B. Data Set Overview Overall descriptive data for Type of Mistakes Identified per Student, Satisfaction, Confidence Levels and Grades are presented. This is done to provide an overall context of how the students performed on each of the three assignments associated with the wrapper exercises. C. Score Distribution Analysis The Exam 2 score was chosen as the dependent variable since it was the culminating assignment in the wrapper series. The Exam 2 scores are continuous and normally distributed. The scores were used to divide the students into two groups in order to understand the characteristics of students who performed Above Average (> 87/100) and Below Average (≤ 87/100). This analysis approach illustrates two things: 1) wrappers as a platform for the teaching team to understand how the students are learning and performing in the course; and 2) wrapper engagement based on student exam performance. D. Improvement Model To quantify the “impact” of wrappers as a direct measure, the Improvement Model was created to determine whether the wrapper variables help predict students’ improvement. In this model, “Improvement” is defined as the score difference between Exam 1 and Exam 2 for each student. The model explored how different variables, particularly the wrapper variables (listed in Appendix A), were correlated with the improvement outcome. Statistical Tools Several statistical tools were employed for the analysis. These include dependent and independent t-tests, Analysis of Variance (ANOVA), Cronbach’s Alpha, correlation and linear regression analysis. ( = 0.05 for hypothesis testing for the studies. 3.2 Variable Explanations The following presents a detailed explanation of the variables in the data set. A table of variables with descriptive statistics is attached as Appendix A. Type of Mistakes Identified per Student “Types of Mistakes” were measured using a chart; the students were instructed to check off which categories of mistakes they made, regardless of how often they made them. The students recorded their general mistakes (e.g. math errors, not showing work, not doing checks, sign errors and significant errors) on the HW3, HW4 and Post-Exam 2 Wrappers. In addition, on the HW4 and Post-Exam 2 Wrappers, the students recorded conceptual errors (e.g. mistakes in forming moment equations) and non-conceptual errors (e.g. missing coordinate axes). Appendix
B provides additional explanations of mistake categories. The total “Types of Mistakes Identified per Student” variables are represented by the average total type of mistakes identified in each category (general, conceptual, non-conceptual) per student for each assignment. Satisfaction Students were asked how satisfied they were with their homework and exam performances on the wrappers. Assignment satisfaction was measured on a 5-point Likert scale ranging from “not satisfied” to “extremely satisfied”, with 3 as “moderately satisfied.” Confidence Levels Concept confidence levels were measured using a 5-point Likert scale ranging from “not confident” to “extremely confident”, with 3 as “moderately confident.” The HW3 and Post-Exam 2 Wrappers asked about confidence levels on concepts related to “setting up and solving equilibrium equations.” The HW 4 and Post-Exam 2 Wrappers asked about confidence levels on concepts related to “drawing Free Body Diagrams.” Each concept was divided into smaller subgroups, with some subgroups consisting of several aspects (shown in the Wrappers included in Appendices E-H). For the final data set, all the confidence data for each major concept (equilibrium equations and FBD) were averaged. Cronbach’s Alpha tests show high inter-consistency between confidence data for both concepts on all three assignments. The concept “setting up and solving equilibrium equations” for both HW3 and Exam 2 has Cronbach’s Alpha of 0.90 and 0.94 respectively while for “Drawing FBDs”, Cronbach’s Alpha of 0.89 and 0.92 for HW4 and Exam 2 respectively. 4. Results 4.1 Results – Student Reaction to Wrappers The qualitative data collected on the wrapper sheets themselves and through the CPREE end-of-the-quarter survey provide preliminary insights into how the wrappers may be helping students identify concepts that they are challenged by, as well as better study strategies. E14 students agreed that wrappers were enjoyable, worth the time, a good tool for learning key ideas in the class, and a good tool for doing well in their education more generally (median score of 4=agreement, on 5-point scale, CPREE Assessment report, n=69). In open-ended responses on this same survey, students offered that the wrappers allowed for “comparing what I've improved upon since my last reflection and what work I still need to do for the course,” helped “me assess how I may look at my previous success and move forward to improve” and allowed them to “realize my strengths and weaknesses in doing an assignment and then fixing the weaknesses for the next one.” The clear iterative nature of learning and reflection come out in these comments. By looking at students’ open-ended wrapper responses about how they might tackle their future assignments (in light of reviewing their just graded assignment) and how well such planned strategies might have actually worked, we see two general categories of reflections. One category is students affirming specific things on which they made fewer mistakes. For example, on the HW3 and HW4 wrappers students offered such comments as: “Signs were not a problem anymore. Success!” and “I was organized with solutions, I answered specifically what homework asked, my signs were on point.” Similarly, a student on her post-Exam 2 wrapper
offered: “Yes, I made no mistakes in regards to external vs. internal loads, or pin joints specifically.” The second category of open-ended wrapper responses is on identifying and implementing strategies for successfully completing homework and for preparing for Exam 2. Examples of such strategies are: being deliberate and writing more in presenting a solution, reviewing past homework/solutions, starting preparation early enough, focusing study-time on hard (for that student) concepts, attending office hours, and studying with peers. When the 70 open-ended responses on the HW4 wrapper to the prompt “Looking back at your HW3 Review Sheet, comment on whether you implemented the 1-3 things you listed in Question 4 when you were completing HW4” were coded, over 80 percent of them affirmed implementing the strategies they had identified in the HW3 wrapper. Consistent with this, nearly 70 percent of students reported in the post-Exam 2 wrapper having implemented their planned study strategies for Exam 2. 4.2 Results – Data Set Overview An overview of the data is presented to provide a quantitative context to the student reaction to wrappers.
Table 1: Overall Confidence Levels of Both Concepts (from associated wrapper) (n=70) HW3 HW4 EXAM 2
Note: Confidence levels were measured in the 5-point Likert Scale, with 1 being “Not Confident”, 3 being “Moderately Confident” and 5 being “Extremely Confident”. Table 1 shows the students’ self-reported confidence levels for setting up and solving equilibrium equations and drawing FBDs. The confidence levels are all around 4, which represents “Very Confident” in this research. The mean confidence levels increase between the homework and exam for both concepts and the increase is statistically significant, albeit with low effect size.
Table 2: Type of Mistakes Identified per Student (n=70) General Mistakes Conceptual
Mistakes Non-Conceptual
Mistakes Assignment M SD M SD M SD HW3 1.23 1.07 -- -- -- -- HW4 0.76 0.84 0.37 0.54 0.64 0.68 EXAM2 0.87 0.90 0.93 0.80 0.40 0.60
Table 2 shows multiple trends in type of mistakes made across the three assignments. General mistakes dropped from 1.23 per student to 0.76 per student after HW3 wrapper (which is the first of the wrapper series). A dependent t-test was conducted and the drop of general mistakes between HW3 and HW4 is statistically significant (t = 3.22, p = 0.00). However, there is a insignificant increase in general mistakes (t = 1.02, p = 0.31) and a significant increase of conceptual mistakes between HW4 and EXAM 2 (t = 4.87, p = 0.00). This could be attributed to the increased conceptual difficulty of the exam. Non-conceptual mistakes post a statistically significant drop of 0.24 type of mistake per student (t = 2.58, p =0.01).
Table 3: Satisfaction Levels on Assignment Performances (n=70) Satisfaction Level Assignment n M SD HW3 69 3.15 1.09 HW4 67 2.79 1.08 EXAM2 67 2.96 1.09
Note: Satisfaction was measured in the 5-point Likert Scale, with 1 being “Not Satisfied”, 3 being “Moderately Satisfied” and 5 being “Extremely Satisfied”. Satisfaction levels drop by 0.36 between HW3 and HW4. This could be attributed to the material in HW4 which asked the students to work on drawing FBDs for the first time. Students made many FBD related mistakes (Conceptual and Non-Conceptual) that resulted in point loss in the assignment. It must be noted that the satisfaction levels hover within the confine of “Slightly Satisfied” and “Moderately Satisfied”. The missing data as shown in the different sample sizes were due to some students forgetting to answer the question. 4.3 Results – Score Distribution Analysis The following presentation shows the means of several independent variables (demographics, grades, satisfaction, confidence, types of mistakes) grouped by performance on Exam 2 (the dependent variable), with students whose Exam 2 score was above average (an 87/100) being the “Above Average” group, and everyone else being in the “Below Average” group. Demographics
Table 4: Demographics based on Score Distribution Groupings Overall (N=70) Above Average
(n=39) Below Average
(n=31)
Characteristic n % n % based on N n % based
on N Gender
Male 40 57 20 50 20 50 Female 27 39 16 59 11 41
Ethnicity URM 22 31 10 45 12 55
Note: For the survey, under gender, the students were given three options: “Male”, “Female” and “I prefer not to answer”. One student picked “I prefer not to answer” and two students did not answer the question. The researchers decided to remove the three students from the table for the students’ privacy due to the small sample number in this category. The demographics of the groups are presented in Table 4, and show that female students are
overrepresented in the Above Average group (59 percent of the female students in the class are in this group). Furthermore, 55 percent of the URM students in the class are overrepresented in the Below Average group. With regards to actual homework scores (see Table 5), the scores for each group are consistent across the assignments (the highest score means to the lowest score means); the students who did better in Exam 2 consistently performed better than those who did not, which may shed light on characteristics of students who populate these two groups (a point to be returned later in the paper).
Table 5: Assignment Grades based on Score Distribution Groupings Above Average (n=39) Below Average (n=31)
The satisfaction level for both groups has the same trend as compared to the overall class trend (lower satisfaction with HW4), as presented in Table 6. Students in the Above Average group were more satisfied than their Below Average counterparts, which suggests that they are generally happier with their performances in these assignments.
Table 6: Performance Satisfaction Levels on Performances Above Average (n=39) Below Average (n=31) Assignment n M SD n M SD HW3 39 3.35 1.08 30 2.90 1.05 HW4 37 2.89 1.07 30 2.67 1.09 EXAM2 36 3.58 0.78 31 2.24 0.96
Note: Satisfaction was measured in the 5-point Likert Scale, with 1 being “Not Satisfied”, 3 being “Moderately Satisfied” and 5 being “Extremely Satisfied”. We also consider the confidence levels of the two groups (Table 7). An overview of the results showed an increase in confidence levels of both concepts in both Above Average and Below Average groups. Notably, students populating the Below Average group have confidence levels below the “Very Confident” mark while students in the Above Average group have confidence levels above the “Very Confident” mark.
Table 7: Overall Confidence Levels of the Concepts based on Score Distribution Groupings Above Average
Note: Confidence levels were measured in the 5-point Likert Scale, with 1 being “Not Confident”, 3 being “Moderately Confident” and 5 being “Extremely Confident”. Finally, we consider students’ self-identified mistakes across the three assignments (Figure 3). The Above Average group outperformed the students in the other group consistently. Besides posting an overall lower number of type of mistakes identified per student as compared to the Lower Average group, the Above Average students also show decline in the number of type of mistakes identified across the three assignments (as opposed to the students in the Below Average group showing an increase of mistakes identified in Exam 2). A t-test shows that Above Average decline in mistakes identified between HW3 and HW4 is significant (t = 2.28, p = 0.03, d = 0.37) while the decline between HW4 and Exam 2 is not significant. As for the Below Average group, the decline between HW3 and HW4 is significant (t = 2.31, p = 0.03, d = 0.41) and the increase in mistakes identified between HW4 and Exam 2 is not significant. Both Conceptual and Non-Conceptual Mistakes results were not included as both do not provide any significant findings to the study.
Note: The bar charts are plotted chronologically from HW3 to Exam 2 for each group.
Figure 3: Type of General Mistake Identified per Student for All 3 Assignments Based on Score Distribution Groupings.
4.4 Results – Improvement Model Tests were conducted to explore the correlation between the important variables (see Appendix D). Two variables significantly correlate with the Improvement scores: HW4 General Mistakes Identified per Student (r = -0.32, p = 0.00) and HW4 Grades (r = 0.27, p = 0.03). Both variables have a moderately high predictive effects on Improvement. The negative correlation between Improvement Scores and HW4 General Mistakes is a sanity check as it makes sense for students to show improvement by reducing type of general mistakes made. It is also encouraging to see that improvement achieved by the students in HW4 could be reflected on students’ performance on Exam 2. To show impact of wrappers on students’ Improvement scores, a Wrapper Model that includes all Wrapper variables was created (Refer to Appendix A for categorization of each variable). Several steps were done to ensure the analysis is reliable:
1. The final sample size for the regression analysis is 67. Three observations were removed due to missing of data in Gender and Ethnicity.
2. Multiple Imputation using Chain Equations (MICE)11 was conducted to fill 0.78 percent of missing data due to randomness (students forgetting to answer those questions while filling out the wrappers).
3. Exam 2 related wrapper variables were not included in the analysis since the information was obtained only after the students completed the exam.
4. All the variables were scaled a Z-score to ensure a standardized field for the model. 5. Both confidence level measures are combined as an average of both named “HW
Confidence”. This was done to avoid multicollinearity as both are highly correlated with each other (r = 0.67, p = 0.00) and both have a Cronbach’s Alpha of 0.8.
Table 8: Linear Regression Analysis on Predicting Improvement Scores
Variable Model 1 (Wrapper Model) Model 2 , SE, t p , SE, t p
Adjusted ./ 0.095 0.04 0.113 0.07 Note: The variables are standardized, df = 61 for Model 1, df = 56 for Model 2
Model 1 (Wrapper Model) results (see Table 8) reveal that only “HW4 Type of General Mistakes Made per Student” plays a statistically significant role in predicting the Improvement scores.
Overall, the wrapper model only predicts 9.5 percent of the variation in Improvement scores, but is statistically significant. However, it is encouraging to note that the variable with the next highest impact is “HW Concepts Confidence”. This suggests that Concepts Confidence may play an important role in predicting variation in Improvement and it should be continued as part of future experiment designs. The Conceptual and Non-Conceptual Mistakes from HW4 wrappers were removed from the analysis as initial analysis presented both variables as non-factors in the model. To further explore other predictors, Model 2 was generated by including five additional variables: three students’ performance related variables (HW3 and HW4 Grades and self-reported Time Spent Working on Exam 2) and two demographic variables (gender and ethnicity). This model explains 11.3 percent of the variances, making it a better model than Model 1, even though it is not a statistically significant model. However, looking closely to each of the variables, HW4 General Mistakes is still the variable that contributes most significantly to the Improvement scores prediction. This is further supported by the HW4 Grades also shown as a significant contributor as a predictor. It must also be noted that students with lower satisfaction their HW4 performance show a higher increase in improvement. This could be attributed to students who were less satisfied with their HW4 performance working harder to improve their performance. Students who had higher grades in HW4 also show higher increase in improvement scores. 5. Discussion and Implications In considering our first research question on what can be learned from students’ experiences with homework and exam wrappers, with the exception of satisfaction levels, the Data Set Overview shows positive chronological changes during the course with respect to students’ confidence levels and type of mistake identified per student. This provides a context showing improvement occurred during the period in which wrappers were conducted. The low satisfaction levels of HW4 and Exam 2 could be attributed to students’ being tasked to set up the problem and draw FBDs. The results, albeit promising, do not show evidence of whether the wrappers played a direct role in the improvement. In the open-ended response analysis, however, there is compelling evidence demonstrating the direct role of wrappers in contributing to these improvements. A high percentage of students reported “Yes” when asked whether they implemented the future strategies they listed, which reflects students’ high engagement with the wrappers. Furthermore, the anecdotal quotations presented show the impact of wrappers on student learning and how the use of these strategies had a positive impact on student learning and course performance. A major insight gained from the score distribution analysis relates to students’ learning and performance obtained from the wrappers. The findings have proven to be helpful in understanding students’ performance in the class throughout the semester as patterns emerged from the data (e.g. the Above Average group consistently outperformed the Below Average group throughout the course). This shows that wrappers are useful tools for teaching teams to investigate how students learn in the class by extracting different information about the students’ performance in different assignments. Teaching teams can potentially utilize the information to identify students who have been consistently underperforming and implement tactical outreach
to these students, providing additional assistance in learning. This benefit provides a foundation for future studies of how wrappers can represent evidence of learning. The Improvement Model results show similar theme portrayed by the overall data overview and the score distribution analysis. The regression model shows that the wrapper variables from HW3 and HW4 wrappers only explains 10 percent of the variation in Improvement scores. However, several variables stand out as significant contributors to the model and these findings will help the team decision-making process in designing more optimized and effective wrappers. The overall pattern that emerged from this study echoes the findings of Craig’s paper on wrappers, which shows no evidence of wrappers affecting differential exam scores among students.10 However, both studies show qualitative evidence of students benefitting in terms of improved performance from utilizing the wrappers to reflect on their study strategies. 6. Conclusion and Future Work The findings presented in this paper suggest wrappers can have an important impact on students’ learning in engineering statics. The findings from the quantitative analyses identify some promising spots, such as the potential of wrappers to influence students’ improvement as seen in the increase in students’ confidence on statics concepts and a decrease in the type of mistakes identified in assignments. The qualitative findings make a stronger argument for the benefits of wrappers as students elaborated positively on how they used wrappers to study better and improve their performance in assignments. These findings reiterate the promise of wrappers in engineering education.
Following our second research question on what insights into the student learning and the course experience can the teaching team gain from reflections captured in homework and exam wrappers, this study demonstrated the usefulness of wrappers to obtain information about how students learn and study. The score distribution analysis demonstrated that information can be extracted from the wrappers to inform the teaching team about what students’ understanding of content and skill level. The teaching team could use this information and take appropriate measures and actions. Wrappers can be mutually beneficial to both students and the teaching team to improve and enhance the teaching and learning experience.
In future research, the researchers are planning to devise an approach for assessing metacognitive skills that incorporates triangulation among quantitative and qualitative self-reported data as well as independent measures in order to provide further evidence of the impact of wrappers. This will be done because this study used self-reports as proxy measures of metacognitive skill and the self-reported qualitative and quantitative data could be skewed since the students were asked to write their names on the wrappers, which might have prompted students to think twice before filling out the wrappers.
The researchers will also collect more data to obtain a larger sample for analysis as the size of the data set is small relative to those in other studies. The findings from this smaller sample may not be as representative or generalizable to a larger student population.
Lastly, the development of a control group or other comparison groups will also be explored as comparing responses and grades for students who were introduced to the wrappers and those who were not would provide a more nuanced understanding of the impact of wrappers. Previous studies on wrappers involved several groups for comparison; students were grouped based on how many courses they took that involved wrappers in Lovett’s paper and three randomized groups were given three different wrappers in Craig’s paper9,10. Also, the wrappers will be refined to optimize the benefits to student learning and performance. Several areas for improvement include reducing the number of questions and increasing the number of wrappers provided in the course. 7. Acknowledgements This research was supported by the Consortium to Promote Reflection in Engineering Education (CPREE) funded by the Helmsley Charitable Trust. We thank our colleagues in the Designing Education Lab who provided insight and expertise, particularly Dr. Mark Schar. We also thank the E14 Teaching Assistants, Jean-Claude Angles, Jeanny Wang, Kayla Powers, and Quinton Ford, for help and feedback in wrapper implementation and Anna Breed for help with data processing. Lastly, we also thank the students in E14 for participating in this study. References
1. Schön, D.A. (1984). The Reflective Practitioner. U.S.A.: Basic Books, Inc.
2. Kolb, D.A. (1984). Experiential Learning: Experience as the Source of Learning and Development. Englewood Cliffs, N.J.: Prentice Hall.
3. Rodgers, C. (2002). Defining reflection: Another look at John Dewey and reflective thinking. Teachers College Record, 104(4), 842-866. https://owl.english.purdue.edu/owl/resource/747/04/
4. Sepp, L.A., Orand, M., Turns, J.A., Thomas, L.D., Sattler, B., & Atman, C.J. (2015). On an upward trend: Reflection in engineering education. Proceedings from the ASEE Annual Conference, Seattle, WA.
5. Ambrose, S. A. (2013). Undergraduate engineering curriculum: The ultimate design challenge. The Bridge: Linking Engineering and Society, 43 (2), 16-23.
6. Turns, J., Sattler, B., Yasuhara, K., Borgford-Parnell, J. L., & Atman, C. J. (2014). Integrating reflection
into engineering education. Proceedings from the ASEE Annual Conference & Exposition, Indianapolis, IN.
7. Pintrich, P. R. (2000). The role of goal orientation in self-regulated learning. In M. Boekaerts, P. R.
Pintrich, & M. Zeidner (Eds.), Handbook of self-regulation (pp. 451–502). San Diego: Academic Press.
8. Zimmerman, B.J. (1990). Self-regulated learning and academic achievement: An overview. Educational Psychologist, 25:1, 3-17, DOI: 10.1207/s15326985ep2501_2
9. Lovett, Marsha C. (2013). Make exams worth more than the grade: Using exam wrappers to promote
metacognition. In M. Kaplan, N. Silver, D. LaVague-Manty, & D. Meizlish (Eds.), Using reflection and
metacognition to improve student learning: Across the disciplines, across the academy (pp. 18-52). Sterling, VA: Stylus.
10. Craig, M., Horton, D., Zingaro, D., & Heap, D. (2016). Introducing and evaluating exam wrappers in CS2.
Proceedings from SIGCSE’16, Memphis, TN. http://dx.doi.org/10.1145/2839509.2844561
11. Azur, M. J., Stuart, E. A., Frangakis, C., Leaf, P. J. (2011). Multiple Imputation by Chained Equations: What is it and how does it work? International Journal of Methods in Psychiatric Research, 20(1): 40-49, DOI: 10.1002/mpr.329.Multiple
Appendix A: Table of Variables Variable Type Category Range n Mean SD SE Median Min Max Range Skew
Appendix B: Table of Mistakes and Explanations Mistake Explanation
General (HW3, HW4, EXAM 2) Math/Calculation Error Made careless error during computation of equations. Sign Error Left out signs or obtained incorrect signs of final answers. Significant Figures Error Did not write final answer in expected significant figures. Lack of Work Presentation Did not show sufficient work to substantiate the final answer. Lack of Check Checks are alternative, brief solutions to validate the final
answer. Students are penalized for not providing checks. Conceptual (HW4, EXAM 2)
Mistakes on Moments Mistakes made while setting up the process of calculating moments.
Mistakes on Problem Setup Mistakes made while setting up the overall problem (identifying unknowns, drawing FBDs).
Mistakes on Forces Mistakes made while setting up the process of calculating forces.
Mistakes on Connections Mistakes made while identifying the different connections and drawing them onto the FBDs.
Non-Conceptual (HW4, EXAM 2) No Engineering Procedure E14 students are required to follow a standard problem solving
protocol known as the Engineering Procedure. Students are penalized for not utilizing the procedure.
No Coordinate Axes in FBD Coordinate axes are not included in any of the FBDs. No Dimensions in FBD Dimensions are not included in any of the FBDs. Force Not Drawn on Exact Point in FBD
The forces are not drawn on the exact location they exert on the system.
17. HW4 Satisfaction 0.11 0.34 0.05 -0.01 0.06 0.24 0.58 0.26 0.03 0.22 -0.25 0.33 0.06 0.26 -0.44 -0.30 Note: Bolded value of r signifies the correlation is statistically significant (p < 0.05).
Appendix E: E14: Homework 3 Review Name: __________________________ Pod: This activity is designed to give you a chance to evaluate your HW3 performance. Please skim through your graded HW3 and take 5 minutes to complete this activity sheet. Answer the questions as honestly as possible as you are only graded for completion. 1) How satisfied are you with your performance on HW3?
Not Satisfied
Slightly Satisfied
Moderately Satisfied
Very Satisfied
Extremely Satisfied
2) Skill Whole Homework Assignment.
Now, briefly skim the whole homework and check if the following mistakes resulted in point loss in HW3. (check all that apply)
Check Items Made math/calculation error Missed writing signs Did not follow significant figures convention Did no checks Did not include Free Body Diagrams/Drawings Did not show work Did not write on engineering paper Did not staple homework Others; Please describe:
HW3, Problem 3(a) for Question 3 (next page) Frame ABC, shown in Figure 3a, is in equilibrium (note that member AB is a two-force member). Using the free-body diagram given in Figure 3b answer the following questions:
a) What are the forces (in vector notation) acting at A and C? Include a check. [DRAW, FORMULATE & SOLVE, CHECK] hint1: if AB is a two-force member, how must FA be directed?
hint2: Fx∑ = 0; Fy∑ = 0; Mz@C∑ = 0;
).
Figure 3a. Situation
Figure 3b. Free-body Diagram
D"
1.2$m$
1000$N$$
0.6$m$
FCY$
FCX$
FAY$
FAX$
x
y D"
1.2$m$
1000$N$$
0.6$m$ 1
2
3) Review you Problem 3 work. After looking through the graded Problem 3(a) in your HW3, rate how confident you are with the following (please check one for each concept). The problem is included in the first page of this review sheet.
Items Extremely
Confident Very
Confident Moderately Confident
Slightly Confident
Not Confident
General Inspection of the Problem (a) Identifying the unknowns and the number of unknowns (b) Identifying the number of equations needed to solve the system
Moment Equations (a) Defining a moment center (b) Finding the position vector of each force (c) Writing the moment equation, and identifying the direction of
the moment
Equilibrium Equations (a) Writing the summation of forces in the x and y-direction
Solving the Equations (a) Applying a systematic method for solving the equations
Free Body Diagrams (FBD)/Drawing (a) Using FBD to find moment center and position vector (b) Using FBD to identify loads to include in equilibrium
equations
Others (a) Understanding the concept of two-force member (b) Understanding sign conventions for forces
4) List 1-3 things you could do differently in future homework.
Appendix F: E14: Homework 4 Review Name: __________________________ Pod: This activity is designed to give you a chance to evaluate your HW4 performance. Please skim through your graded HW4 and take 5 minutes to complete this activity sheet. Answer the questions as honestly as possible as you are only graded for completion. 3) How satisfied are you with your performance on HW4?
Not Satisfied
Slightly Satisfied
Moderately Satisfied
Very Satisfied
Extremely Satisfied
4) Skim Whole Homework Assignment.
Now, briefly skim the entire homework and check if the following mistakes resulted in point loss in HW4. (Mark all that apply)
Check Items Made math/calculation error(s) Missed writing signs Did not follow significant figures convention Did no checks Did not write in vector notations Did not show work Did not write on engineering paper Did not staple homework Did not follow engineering analysis procedure:
GOAL, GIVEN, ASSUME etc. Other reason(s) (please describe):
Mistakes related to Free Body Diagrams Did not include dimensions Did not include coordinate axes Did not draw forces at the exact point location (forces
drawn are “floating”)
HW4, Problem 3.2 and 3.3 for Question 3 (next page) (3.2) Consider the frame in Figure 2. Draw the free-body diagram for the frame and its parts, as listed below. Make sure to be consistent in drawing the direction of the loads on the different FBDs. (a) The entire frame as the system. (b) Member 1 as the system. (c) Member 2 as the system. (d) Pulley 1 as the system. (e) Pulley 2 as the system.
(3.3) Consider a frame ABC that is tethered at C by a cable. The end location of the cable (D) is at two different locations in Figure 3A (at 3,0,1)m and Figure 3B (at 3,0,0)m. Taking the frame as the system draw its free-body diagram (FBD) in each situation, noting whether it can be modeled as planar or non-planar; describe your reasoning.
Figure 3A (coordinate system the same as
in Figure 3b)
Figure 3B
Figure 2
1
2
3) Review your Problem 3 work. After looking through the graded Problem 3.2 and 3.3 in your HW4, rate your level of confidence with the following concepts (please mark one response per concept). The problem is included on the first page of this review sheet.
Items Extremely
Confident Very
Confident Moderately Confident
Slightly Confident
Not Confident
Overview of the Problems (c) Isolating the system as requested (d) Inspecting the system; concluding whether the system can be
drawn in planar
(e) Listing assumptions Identification of Connections
(d) Understanding what a pin is and how it works (e) Understanding what a fixed support is and how it works
Managing of the Loads (b) Identifying external, internal and not-in-system loads (c) Identifying the location of the loads (d) Labeling the loads
Understanding of Pulleys (a) Understanding what an ideal pulley means (b) Identifying loads that represent the mechanism of a pulley
Others (b) Defining the coordinate axes (c) Ensuring consistency between assumptions listed and free body
diagrams drawn
4) Looking back at your HW3 Review Sheet, comment on whether you implemented the 1-3 things you listed in Question 4 when you were completing HW4.
5) List 1-3 things you plan to focus on as you prepare for Exam 2 (i.e. general problem solving skills, specific sticky concepts, etc.).
Appendix G: Pre-Exam 2 Survey Name: ________________ Pod: You all were asked in the HW4 Review Sheet Question 5 on what you would focus for your preparation of Exam 2. The following is the compilation of your responses.
Exam Preparation Methods Problem Solving Process E14 Materials and Skills • Practice additional
problems/sample exams • Go through Examples from
course reader/lecture notes • Review past
homework/solutions • Start preparation earlier • Study with Peers • Have a good time management • Focus on Hard Concepts • Go to Office Hours
• Be deliberate; write more; explain clearly
• Do checks (intuition, additional calculations)
• Follow the Engineering Analysis Procedure
• Plan before tackling problems; be more efficient
• Rereading questions while tackling problems
• 2-force member • Connections • Coordinate Axes in FBDs • Loads in FBDs • Isolating System • Planar vs Non-Planar • Pulleys • Listing Assumptions • Safety Factor/Load Capacity • Stress-Strains • Trusses
Now, look at your HW4 Review Sheet Question 5, the HW3 and HW4 Review Sheets Analysis Report and the list above and answer the following questions. You are only graded for completion, so please answer this survey as honest as possible. 1) After briefly skimming through the documents listed above, would you like to change or alter your personal
response in Question 5?
2) Looking at the E14 Materials and Skills list and recalling briefly what you have learned so far in the course,
what topic would you focus on the most for Exam 2?
3) Have you started, in any way, preparation for Exam 2?
Is this your first time taking a take-home exam?
Yes No
Yes No
1
2
Appendix H: E14: Exam 2 Review Name: __________________________ Pod: This activity is designed to give you a chance to evaluate your Exam 2 performance. Please skim through your graded Exam 2 and take 10 minutes to complete this activity sheet. Answer the questions as honestly as possible as you are only graded for completion. 5) How satisfied are you with your performance on Exam 2?
Not Satisfied
Slightly Satisfied
Moderately Satisfied
Very Satisfied
Extremely Satisfied
6) Skim Whole Exam.
Now, briefly skim the entire exam and check if the following mistakes resulted in point loss in Exam 2. (Mark all that apply)
Check Items Made math/calculation error(s) Missed writing signs/Wrote the wrong signs Did not follow significant figures convention Did not write in vector notations Did not show work Did no check(s) Did not include dimensions in Free Body Diagrams (FBD) Did not include coordinate axes in FBD Did not draw forces at the exact point location (forces
drawn are “floating”) in FBD Did not follow engineering analysis procedure: GOAL,
GIVEN, ASSUME etc. Did not identify connections/supports correctly Included connections/supports drawing in FBD Made mistake in forming and calculate summation of
forces Made mistake in forming and calculate moment equations Made mistake in problem setup (did not understand the
problem etc.) If so, which problem? Problem Other reason(s) (please describe):
7) Looking back at your Pre-Exam 2 Survey, comment on whether you did the things you listed in Question 1 while preparing for this exam (If you wrote “No Change”, please look at the responds you gave in HW4 Review Question 5 provided).
8) Please rate the effectiveness of the each strategy you employed in
preparing for this exam with the following scales. Scales 1 2 3 4 5
9) If you did change your responses in Question 1 of Pre-Exam 2
Survey, do you think the changes help in preparing for the exam? Yes No
10) Review your Exam 2 Problem 1.1. After looking through the graded Problem 1.1 in your Exam 2, rate your level of confidence with the following concepts (please mark one response per concept).
Items Extremely Confident
Very Confident
Moderately Confident
Slightly Confident
Not Confident
Overview of the Problems (f) Isolating the system as requested (g) Listing assumptions
Identification of Connections (f) Understanding what a pin is and how it works
Managing of the Loads (e) Identifying external, internal and not-in-system loads (f) Identifying the location of the loads (g) Labeling the loads
Understanding of Pulleys (c) Understanding what an ideal pulley means (d) Identifying loads that cable and pin connection acting on a
pulley
Others (d) Defining the coordinate axes (e) Ensuring consistency between assumptions listed and free body
diagrams drawn
(f) Understanding the concept of two-force member (g) Understanding the concept of three-force member
11) Review your Exam 2 Problem 2.2. After looking through the graded Problem 2.2 in your Exam 2, rate your level of confidence with the following concepts (please mark one response per concept).
Items Extremely Confident
Very Confident
Moderately Confident
Slightly Confident
Not Confident
General Inspection of the Problem (h) Identifying the unknowns and the number of unknowns (i) Identifying the number of equations needed to solve the system
Moment Equations (g) Defining a moment center (h) Finding the position vector of each force (i) Writing the moment equation, and identifying the direction of
the moment
Equilibrium Equations (h) Writing the summation of forces in the x and y-direction
Solving the Equations (h) Applying a systematic method for solving the equations
Free Body Diagrams (FBD)/Drawing (c) Using FBD to find moment center and position vector (d) Using FBD to identify loads to include in equilibrium
equations
Others (c) Understanding sign conventions for forces
1
2
12) List an aspect of your exam performance that you are pleased with.
13) If you have other comments on your performance in Exam 2 (e.g. anxiety in taking the exam, heavy workload that affects your process of completing the
exam etc.), provide them here.
14) If you have comments on taking a TAKE-HOME EXAM, provide them here.
