eie.orgeie.org/sites/default/files/2011_IMD_Yrs_7-8_report_FINAL.pdf · E M B T G e v An ngineerin...

EMB

TGev

An

EngineerinMuseum ofBoston, MA

This materGrant No. 0expressed iviews of th

n Eval

ng is Elemef Science, A 02114

ial is based0454526. Ain this mathe National

Enginuation

entary 1 Science

d upon worAny opinioterial are thl Science F

neering of Yea

Cathy Christine

JenPree

JeremyRach

Ju

Park

rk supportons, findinhose of the Foundation

g is Elears 7 aP. LachapeM. Cunnin

nnifer Joczeya Phadniy Wertheimhell Arteag

uly, 2011

ed by the Ngs, and conauthor(s)

n.

ementaand 8 Felle ngham

is mer ga

National Snclusions oand do not

ary: Field Te

cience Fouor recommt necessari

Testing

undation unmendations ily reflect t

nder

the

Abstract Results are presented from formative and summative evaluation of five Engineering is Elementary (EiE) units that were field tested between autumn 2009 and spring 2011, the seventh and eighth years of the EiE project. Teachers submitted feedback forms as well as pre- and post-assessments completed by students in field test classrooms. Teachers rated the units highly, and most teachers indicated they would teach the unit they field tested again. In field test classrooms, students participated in both a related science and an EiE unit. EiE students performed significantly better on post-assessments of engineering and science knowledge than they did on pre-assessments. In addition, in almost all cases, students from all demographic groups tested (gender, race/ethnicity, Individualized Education Program (IEP) status, and others) improved as much as or almost as much as other students on the assessments after participating in EiE. Without a control group, we cannot claim that participation in EiE caused this growth; however the findings that teachers thought the units worthwhile to implement and students improved show promise for the efficacy of the units.

Table of Contents

1 INTRODUCTION............................................................................................................................. 5

2 METHODOLOGY ........................................................................................................................... 5 2.1 Study Design ...................................................................................................................................................................... 5

2.2 Assessment Instrument Development ................................................................................................................................ 7 2.2.1 Development of Teacher Feedback Forms ............................................................................................................... 7 2.2.2 Development of Student Assessments ...................................................................................................................... 7

2.3 Data Collection and Initial Cleaning .................................................................................................................................. 7 2.3.1 Data Collection of Teacher Feedback Forms ............................................................................................................ 7 2.3.2 Data Collection of Student Assessments .................................................................................................................. 7

2.4 Methods of Analysis .......................................................................................................................................................... 8 2.4.1 Reliability Analysis and Scale Construction for Student Assessments ..................................................................... 8 2.4.2 Analysis of Teacher Feedback Forms ....................................................................................................................... 8 2.4.3 Analysis of Student Assessments ............................................................................................................................. 9

3 RESULTS ........................................................................................................................................ 11 3.1 Results for the “Cleaning Up an Oil Spill” Unit Evaluation ............................................................................................ 11

3.1.1 Formative Evaluation: Cleaning up an Oil Spill ..................................................................................................... 11 3.1.2 Summative Evaluation: Cleaning Up an Oil Spill .................................................................................................. 19

3.2 Results for the “Replicating an Artifact” Unit Evaluation ............................................................................................... 34 3.2.1 Formative Evaluation: Replicating an Artifact ....................................................................................................... 34 3.2.2 Summative Evaluation: Replicating an Artifact ..................................................................................................... 44

3.3 Results for the “Designing Submersibles” Unit Evaluation ............................................................................................. 54 3.3.1 Formative Evaluation: Designing Submersibles ..................................................................................................... 55 3.3.2 Summative Evaluation: Designing Submersibles ................................................................................................... 62

3.4 Results for the “Designing Knee Braces” Unit Evaluation .............................................................................................. 74 3.4.1 Formative Evaluation: Designing Knee Braces ...................................................................................................... 74 3.4.2 Summative Evaluation: Designing Knee Braces .................................................................................................... 85

3.5 Results for the “Designing Lighting Systems” Unit Evaluation ...................................................................................... 95 3.5.1 Formative Evaluation: Designing Lighting Systems ............................................................................................... 95 3.5.2 Summative Evaluation: Designing Lighting Systems ........................................................................................... 105

4 CONCLUSIONS ........................................................................................................................... 127 4.1 Discussion ...................................................................................................................................................................... 127

4.1.1 Formative Assessment .......................................................................................................................................... 127 4.1.2 Summative Assessment ........................................................................................................................................ 129

4.2 Recommendations .......................................................................................................................................................... 131

5 REFERENCES .............................................................................................................................. 131 Table of Figures Figure 1. Timing of Assessments in Field Classrooms ................................................................................................................ 6 Figure 2. Cleaning Up an Oil Spill Questions 3-7 ..................................................................................................................... 21 Figure 3. Cleaning Up an Oil Spill Question 21 ........................................................................................................................ 21

Figure 4. Cleaning Up an Oil Spill Question 22 ........................................................................................................................ 22 Figure 5. Cleaning Up an Oil Spill PostEng Score – Conditional Model .................................................................................. 25 Figure 6. CO Change in Student Score: Engineering Scale ....................................................................................................... 26 Figure 7: CO PostEng Scores vs. Number of Minutes Spent Teaching the CO Unit ................................................................ 27 Figure 8. Cleaning Up an Oil Spill PostSci Score – Conditional Model ................................................................................... 29 Figure 9. CO Change in Student Scores: Science Scale ............................................................................................................ 30 Figure 10. Cleaning Up an Oil Spill PostPol Score – Conditional Model ................................................................................. 32 Figure 11. CO Change in Student Score: Pollution Scale .......................................................................................................... 33 Figure 12. Replicating an Artifact Question 5 ........................................................................................................................... 46 Figure 13. Replicating an Artifact Questions 14-16 ................................................................................................................. 46 Figure 14. Replicating an Artifact PostPM Score – Conditional Model .................................................................................... 49 Figure 15. RA Change in Student Score: PM Scale................................................................................................................... 50 Figure 16. Replicating an Artifact PostRocks Score – Conditional Model ................................................................................ 52 Figure 17. RA Change in Student Score: Rocks Scale .............................................................................................................. 54 Figure 18: Question 3 of the SB Assessment ............................................................................................................................. 64 Figure 19: Question 4 of the SB Assessment ............................................................................................................................. 64 Figure 20: Question 9 of the SB Assessment ............................................................................................................................. 64 Figure 21: Question 10 of the SB Assessment ........................................................................................................................... 65 Figure 22. Designing Submersibles PostEng Score – Conditional Model ................................................................................. 68 Figure 23: SB Change in Student Score: Engineering Scale ..................................................................................................... 69 Figure 24: SB PostEng Scores vs. Number of Minutes Spent Teaching the SB Unit ................................................................ 70 Figure 25. Designing Submersibles PostSci Score – Conditional Model .................................................................................. 71 Figure 26: SB Change in Student Score: Science Scale ............................................................................................................ 72 Figure 27: SB PostSci Scores vs. Number of Minutes Spent Teaching the SB Unit ................................................................. 73 Figure 28. Designing Knee Braces Question 9 .......................................................................................................................... 85 Figure 29. Designing Knee Braces Question 13 ........................................................................................................................ 85 Figure 30. Designing Knee Braces Question 15 ........................................................................................................................ 87 Figure 31. Designing Knee Braces Question 16 ........................................................................................................................ 87 Figure 32. Designing Knee Braces Question 17 ........................................................................................................................ 87 Figure 33. Designing Knee Braces Question 19 ........................................................................................................................ 87 Figure 34. Change in Score Pre to Post: KB All Scale .............................................................................................................. 90 Figure 35. Change in Score Pre to Post: KB Science Scale ....................................................................................................... 92 Figure 36. Change in Score Pre to Post: KB BME Scale ........................................................................................................... 93 Figure 37. Change in Score Pre to Post: KB Design-Models Scale ........................................................................................... 95 Figure 38. Designing Lighting Systems Questions 1 & 2 ......................................................................................................... 107 Figure 39. Designing Lighting Systems Question 3 ................................................................................................................. 107 Figure 40. Designing Lighting Systems Question 13 ............................................................................................................... 107 Figure 41. Designing Lighting Systems Question 15 ............................................................................................................... 107 Figure 42. Designing Lighting Systems Questions 20 & 21 ..................................................................................................... 108 Figure 43. Designing Lighting Systems Question 22 .............................................................................................................. 108 Figure 44. Designing Lighting Systems Question 25 .............................................................................................................. 108 Figure 45. Designing Lighting Systems PostOE Score – Conditional Model .......................................................................... 112 Figure 46. LS Change in Student Score: OE Scale .................................................................................................................. 113 Figure 47: LS PostOE Scores vs. Number of Minutes Spent Teaching the LS Unit ............................................................... 114 Figure 48. Designing Lighting Systems PostED Score – Conditional Model .......................................................................... 116 Figure 49. LS Change in Student Score: ED Scale .................................................................................................................. 117 Figure 50: LS PostED Scores vs. Number of Minutes Spent Teaching the LS Unit ............................................................... 118 Figure 51. Designing Lighting Systems PostPL Score – Conditional Model ........................................................................... 120 Figure 52. LS Change in Student Score: PL Scale .................................................................................................................. 121 Figure 53: LS PostPL Scores vs. Number of Minutes Spent Teaching the LS Unit ................................................................ 122 Figure 54. Designing Lighting Systems PostBML Score – Conditional Model ....................................................................... 124 Figure 55. LS Change in Student Score: BML Scale ............................................................................................................... 125 Figure 56: LS PostBML Scores vs. Number of Minutes Spent Teaching the LS Unit ............................................................ 126 Figure 57. Summary of Improvement: Reference Group Change in Score for Each Unit ....................................................... 130

1 Introduction

Engineering is Elementary (EiE) began curriculum development efforts in 2003. EiE has created 20 supplemental curriculum modules to teach elementary school children engineering in a hands-on manner, in the context of science (as taught by the most common hands-on science curricula). This report details evaluation findings (both formative and summative) for the field test (second draft) versions of the final 5 engineering units of 20 created by the EiE development team. Three units, Cleaning Up an Oil Spill, Replicating an Artifact, and Designing Submersibles, were created and piloted (first draft version) during the 2008-2009 school year, and field tested during the 2009-2010 school year. Two units, Designing Knee Braces and Designing Lighting Systems, were piloted during the 2009-2010 school year and field tested during the 2010-2011 school year.

The design methodology, theoretical framework, and development process for EiE have been extensively documented elsewhere (Cunningham & Hester, 2007; Lachapelle et al., 2011; Lachapelle, Cunningham, Lee-St. John, Cannady, & Keenan, 2010). We refer the reader to these documents for further information about the EiE project.

2 Methodology

This report details the results of pre- and post-assessments of units in field classrooms, as well as field test teacher feedback about implementation of the EiE units. Field test classrooms are those where students were taught an EiE unit and related science.

Results are to be considered indicative of the quality of the units in question and of EiE as a curriculum, but not authoritative, for several reasons: (1) units were tested while still subject to extensive revisions—the units evaluated here were still in a preliminary form, and have since been revised and improved; (2) the study is not a randomized controlled test—there were no control classrooms in which students were taught the related science, but not the EiE unit; (3) the assessments for the units were still under development at the time of testing, and the assessment versions used vary widely in internal reliability.

2.1 Study Design During the field test year for each EiE unit, 12 field test teachers were recruited from each of 5 states (Massachusetts, North Carolina, Colorado, Minnesota, and California). Field test teachers were recruited and trained by field site leaders who were EiE professional development providers local to each field test state. Massachusetts field test and control teachers were recruited directly by EiE staff. All field test teachers were provided with a professional development workshop at or near the beginning of the school year, during which the EiE unit they would test was introduced. Field test workshops addressed introductory topics in technology and engineering, including how to teach these concepts to students, as well as science and engineering background relevant to the unit. Teachers engaged in the hands-on activities from the unit, and discussed how they would introduce them to their students in an age-appropriate manner. They were provided with the materials needed for implementing an EiE field test unit (teacher guide, a classroom pack of storybooks, and a materials kit), and compensated with a small stipend after returning post-assessments and unit feedback.

Unfortunately, no control teachers could be recruited for this study. Because the original curriculum design grant did not include a significant evaluation component, there was not enough grant money to include control teachers. Therefore our results must be considered indicative and not conclusive, and findings will be presented as preliminary.

F

Gwktctasaul

W4Tgsiei

FFisw

Field test tea

• They

• They

• They

• They

• They

Grades 2-6 wwhile grade kindergartenthe teachers complete asstheir studentassistance wistudent who an “engineerunit-specific learning obje

Whenever po4 to 8 weeksThere are, hogiven to studstudents for nstructed to

engineering n the scienc

Field test teaFeedback fonstruction, w

suggestions fwere reviewe

achers were r

y must teach

y must teach

y must teach

y must compl

y must teach

were recruite2 students

n teachers paonly comp

sessments. Ats. They weith the assesparticipated

ring attitudeassessment

ectives of the

ossible, studs after compowever, varydents vary. Fa portion of

o administerunit or any e topic and e

achers complorms asked which studefor improvemed, compiled

recruited bas

appropriate

the science t

in a formal c

lete all surve

the EiE field

ed for both fcompleted

articipated inleted feedba

All teachers ere instructesments. In a

d in the fields” survey. Its each inclue unit and th

dents compleletion of theying circumsFor examplef the year. Nr pre-assessrelated scienengineering

Figure 1. T

leted feedbaspecific q

ent handoutsment of indid, and used t

sed on sever

grade levels

topic related

classroom se

eys and asse

d test unit an

field test anda shorter,

n EiE field teack forms—of grades 2-

ed to read tha given classd test programn addition, uded science

he target scie

eted pre-assee engineeringstances for de, some studNo matter wsments to thnce topic, anlessons (Fig

Timing of As

ack forms aftquestions abs were usedividual lessoto inform rev

ral criteria:

s.

d to the EiE u

etting (not, f

ssments prov

nd provide fe

d control. Gmodified as

esting, but th—because K-

-6 were askehe assessmeroom, each sm completedeach studene and engin

ence concept

essments in g and relateddifferent teacdents are tau

when teacherhe students nd the post-agure 1).

ssessments in

ter finishing bout implemd, and modins, and ratinvision of the

unit of intere

for example,

vided.

eedback abo

Grades 3-6 cossessment. Iheir students-1 students ed to collectents aloud istudent reced one genera

nt completedneering quests to be reinf

October or d science unchers, so theught by scienrs administer

before the assessments

n Field Clas

instruction mentation—tifications mngs of the une units.

est.

, in an after-s

out the exper

ompleted theIn addition,

s did not comcan not be t pre- and poif possible, eived the samal engineerin

d a unit-specstions relatedforced by the

November anits (in the se dates that pnce specialired their ass

students wafter they c

ssrooms

in the EiE utime spent

made by the nit as a whol

school progr

rience.

e regular ass, some gradmplete assess

expected toost-assessmebut not to

me assessmenng assessmecific assessmd to the enge unit.

and post-asssame academpre-assessmests who onlysessments, thwere exposecompleted in

unit they werin preparateacher—as

le. Teacher r

ram).

sessment, de 1 and sments—o reliably ents from give any nts. Each nt and/or

ment. The gineering

sessments mic year). ents were y see the hey were

ed to the nstruction

re testing. ation and s well as responses

2.2 Assessment Instrument Development 2.2.1 Development of Teacher Feedback Forms Teacher Feedback Forms were developed in the third year of the curriculum development project and revised yearly. For each unit, the forms asked teachers to rate and discuss aspects of the unit overall, as well as to give information about their teaching of each individual lesson. For each unit, teachers were asked to describe from their perspective what their students learned, and to provide general feedback. For individual lessons, they were asked how much time they spent prepping for and teaching each lesson, in both minutes and class periods—data that we used in our HLM models of student outcomes. Teachers were also asked to explain if and how they deviated from the instructions given in the teacher guide and why. Finally, we asked teachers to rate and provide comments on each lesson individually. Feedback Forms for each unit were customized with specific questions about aspects of the lessons the curriculum developers were most concerned about.

2.2.2 Development of Student Assessments Development of field test instruments for assessing students was a two-step process. During the pilot (initial) year of development for each unit, EiE staff brainstormed a number of questions pertaining to the learning objectives of each unit, as well as for related science objectives to be emphasized by the unit. Potential assessment items were also collected, when available, from standardized assessments such as the NAEP and state science assessments. The questions were reviewed for correctness, checked for ambiguity, and cross-referenced with the learning objectives of the unit. From these questions, a subset was chosen for the pilot assessment for the unit.

Validity evidence was then gathered for all candidate questions. EiE staff conducted cognitive interviews with children in grades 2-5 from Massachusetts classrooms, asking students to read each question aloud, explain what they thought the answer would be, read all answer choices aloud, and explain which answer they would choose and why. Any questions found to be unacceptably confusing for students were dropped or revised for clarity.

Each pilot assessment was then distributed to approximately 100 pilot classroom students (students in classrooms using pilot / initial versions of the EiE units) in grades 3-5 for testing. Based upon students’ responses, questions that were too easy (greater than 95% correct) were removed. Other questions were removed if they were no longer relevant, due to changes made to the EiE unit after pilot testing, or if they detracted from internal reliability. The final set of questions made up the field test (second draft) assessment, used for this evaluation, to collect data from field test classrooms—those testing the second drafts of the EiE units.

2.3 Data Collection and Initial Cleaning 2.3.1 Data Collection of Teacher Feedback Forms Teacher Feedback Forms were collected from field test teachers after they completed the EiE unit assigned. They were then entered by hand into Microsoft Access® (Microsoft Corporation) for initial coding. Researchers ran queries to check for duplicate forms and missing entries, which were then checked against the originals. With duplicates, the most complete form was retained.

2.3.2 Data Collection of Student Assessments Pre-assessments were mailed directly to participating teachers near the beginning of the school year. For a small subset of teachers who registered to field test mid-year, pre-assessments were sent immediately

after registration. In a given classroom, each student received the same assessments. Each student completed one general engineering assessment and/or an “engineering attitudes” survey. The results of the general engineering assessment and the engineering attitudes survey are not included in this report. In addition, each student completed a unit assessment. The unit-specific assessments included science and engineering questions related to the unit students had studied, and are the subject of this report, together with teacher feedback forms.

The timing of instruction and assessments in field test classrooms is shown in Figure 1 above. EiE students were tested at least twice—once before beginning the science curriculum and/or related Engineering is Elementary unit, and once after instruction was completed—allowing for a test-retest analysis.

Once all completed assessments were collected, they were digitized using an OMR scanner and then imported into a Microsoft Access database. This data was then exported to SPSS Statistics version 19.0, together with student demographic data, for initial analysis. Student responses to each question were recoded as correct (1) or incorrect (0), and a preliminary scale item (AllScore), which was calculated as the sum of all questions (equivalent to the sum of all correct questions), was also computed for each assessment. Data collected from all classrooms were checked for any transcription errors and unusual or missing data by inspecting the means from the AllScore variable for the pre-assessment and post-assessment separately. Data entry and transcription errors were corrected before continuing.

2.4 Methods of Analysis 2.4.1 Reliability Analysis and Scale Construction for Student Assessments Reliability analysis for each unit was conducted on the responses of all students who had returned both a pre- and a post-assessment. The sum of all corrected items for a given unit assessment was calculated to create a score, the AllScore, as described above.

Initial scores were checked for internal reliability and factorability by means of reliability analysis and principal component analysis with direct oblimin rotation in SPSS v19.0, on the pre- and the post-scores separately. We then undertook a principal component analysis to identify which questions grouped together. We anticipated that the items assessing science objectives would cluster in the same components, as would those assessing engineering outcomes. Subscales were then calculated based on the original learning objectives used to create questions, but taking into account the results of principal component analysis, and again checked for internal reliability.

2.4.2 Analysis of Teacher Feedback Forms Open-ended responses were coded by examining the first 100-200 responses and classifying these responses into mutually exclusive categories. Multiple codes could be applied to the same response if the teacher made comments that fit into more than one coding category. When codes were added or changed during the coding process, all responses were re-examined using this new coding scheme. Later, codes were examined for similar questions and standardized for these questions, with recoding done as necessary.

Open-ended responses, codes, and numeric answers for ratings and the amount of time spent teaching each unit were exported to Microsoft Excel® where the minimum, maximum, mean and standard deviation of ratings were calculated, and the most common codes were tabulated and representative responses marked for reporting. Teachers were able to report the amount of time they spent teaching in either minutes or class periods. If a teacher reported the amount of time teaching in periods but not in

minutes, the number of periods was multiplied by 45 minutes in order to determine an approximate number of minutes that the teacher spent teaching the lesson. Additionally, for lessons with multiple parts, the total number of minutes for the lesson was calculated by adding up the number of minutes the teacher reported spending on each part of the lesson. As long as a teacher reported either minutes or periods taught for at least one part of the lesson, they were counted as having taught the lesson and their overall time was calculated.

2.4.3 Analysis of Student Assessments Further data cleaning was completed after analysis of reliability of scores, but before beginning analysis. Any student missing either the pre- or post-assessment was dropped from the dataset. Students who had missed three or more questions on either the pre- or post-assessment were excluded from the dataset. All remaining missing responses were replaced with a zero – a coding of “incorrect”. 2.4.3.1 Analysis for Small Sample Sizes For the one unit (Designing Knee Braces) where the sample size was too small to conduct HLM analysis, backward stepwise multiple regression was carried out in SPSS Statistics version 19.0 using the post-assessment scores centered around the pre-assessment mean for each subscale as outcome variables and pre-assessment scores as covariates. This strategy results in an ANCOVA model, where the intercept can be interpreted as the improvement from pre- to post-assessment. Since we are also interested in evaluating whether effects of the curriculum are modified by gender, socio-economic status (as measured by student participation in the National Free and Reduced-Price Lunch (FRL) Program), race / ethnicity, status as an English language-learner with limited English Proficiency (LEP), and participation in an Individualized Education Program (IEP), these demographics were included as independent variables in the analysis.

In the backward stepwise method of multiple regression, all predictor variables are entered into an ordinary least squares model, and their coefficients are calculated. Each coefficient has a standard error associated with it, which is used to perform a t-test to determine the statistical significance of the coefficient. The least statistically significant predictor variable is removed, and the regression is recalculated. This process is repeated until only statistically significant predictors remain in the model. We then calculated Cohen's d for a given demographic by dividing the coefficient of the binary variable in question by the standard deviation of the residuals of the conditional model. This corresponds with the standard calculation of Cohen's d, in that the coefficient of a binary variable serves as the difference between the means of the two populations, while the standard deviation of the conditional model residuals serves as the pooled standard deviation of the two populations after controlling for any other factors.

2.4.3.2 Hierarchical Linear Modeling of Large Samples For units taught in 40 or more classrooms (which we deemed the minimum sample size suitable for HLM analysis), the data was cleaned further using SPSS Statistics v19.0 in a multistep process. It was determined how many students and classes were missing level-1 demographic information (one or more of gender, IEP status, LEP status, FRL status, and racial information). We found that the only problematic variable was Free or Reduced-Price Lunch (FRL), since that information is kept confidential in many districts, so FRL status was not considered in our analysis.

Our primary question of interest is whether participation in EiE affects student understanding of the engineering learning objectives and related science concepts, as measured by scores on the post-assessment. Curriculum effects for four of the five units (excepting Designing Knee Braces) were examined using a pre-assessment/post-assessment design using the HLM software version 6

(Raudenbush, Bryk, Cheong, Congdon, & du Toit, 2004). Post-assessment scores centered around the pre-assessment mean served as the outcome variable, while pre-assessment scores were entered as a covariate. This strategy again results in an ANCOVA model, where the intercept can be interpreted as the improvement from pre- to post-assessment. The ANCOVA model was implemented at the classroom level using hierarchical linear modeling (HLM) methodology that accounts for the clustering of students within classrooms. All HLM models were estimated using restricted maximum likelihood. Deviance statistics were used to test the relative model fit from competing conditional models.

One of the assumptions of the HLM analysis is that the within-class variance is homogeneous between different classes. Violating this assumption can lead to biased significance estimates and Type I error. Fortunately, the HLM software provides a module to easily test this assumption, and to model the level-1 variance with the model predictors. This allows us to understand what factors affect that variance, and allows HLM to take those factors into account when calculating significance estimates in the model. This leads to more accurate significance estimates, even when using the robust standard errors. The homogeneity of the level-1 variance was tested for each unit, and for those units where it was found to be heterogeneous (p<.05) the variance was logarithmically modeled with level-1 and level-2 predictors using the HLM software. In some instances, though not all, the predictors used were sufficient, and the level-1 variance was found to be homogeneous after controlling for those factors.

A two step modeling process was performed:

Step 1: Student level demographics were modeled at Level-1, the student level. The student level was modeled by first testing the significance of the fixed effects associated with student level demographics and a classroom-centered pre-assessment. Non-significant fixed effects were eliminated by order of least significance. Once only significant student level demographic fixed effects remained, their corresponding random effects were tested to examine if the fixed effects varied across classrooms. Where random variance was not found to be significant (p >= .05), we dropped the random coefficients from the model.

Step 2: Because the curriculum was delivered to intact classrooms, the student level intercept was modeled at the classroom level using an ANCOVA strategy where the classroom mean pre-assessment served as the covariate. Homogeneity of regression slopes (an ANCOVA assumption) was also tested here, and in all cases was confirmed.

Post-assessment scores served as the outcome variable and were centered on the grand mean of the pre-assessment, while pre-assessment scores were entered as a covariate and were group-mean centered. This method of centering was used (1) interpret the intercept as the predicted mean change in student score from pre-assessment to post-assessment and (2) determine if this change in score is significant. Continuous level-2 predictors (class size, number of years teaching, the classroom means of pre-assessment scores, the number of minutes spent teaching EiE) were grand-mean centered, so that their effect could be interpreted in the context of an average class, rather than in the context of a class with class size 0, classroom pre-assessment mean 0, a novice teacher (Bryk & Raudenbush, 1992). The only level-2 predictor that was not centered in any way was the science specialist dummy, as we wished to interpret our results in the context of a regular classroom teacher.

3 Results

3.1 Results for the “Cleaning Up an Oil Spill” Unit Evaluation In the EiE unit A Slick Solution: Cleaning Up an Oil Spill, students explore the field of environmental engineering as they design a process to clean an oil spill in a model river. They learn how pollution can negatively impact the organisms in the ecosystem and how engineers must consider both short and long-term effects of pollutants on the organisms in a given ecosystem. In Lesson 1, students read the storybook Tehya’s Pollution Solution, which introduces them to the field of environmental engineering, the connectedness of ecosystems and food webs, and how pollution, such as an oil spill, can have a profound impact on the environment. In Lesson 2, students work in environmental engineering teams to investigate possible sources of air and soil pollution in the fictional Greentown. During Lesson 3, students examine the ecosystem in which this model oil spill takes place: a river ecosystem in the Pacific Northwest United States. The class examines how the ecosystem might be impacted by an oil spill in the river. In part 2 of Lesson 3, students are introduced to the different materials, tools, and methods available to them for cleaning their oil spills and conduct an experiment to test the efficacy of each for containing the spread of the oil. Students undertake their design challenge in Lesson 4 which is to design a process to clean an oil spill in a model river so that the oil has the least impact on the surrounding ecosystem. They then implement, test, and analyze their designs to see how much oil remains in the river as well as how that remaining oil might impact the river ecosystem.

A Slick Solution: Cleaning Up an Oil Spill was field tested during the 2009-2010 school year.

3.1.1 Formative Evaluation: Cleaning up an Oil Spill Feedback forms for Cleaning Up an Oil Spill were completed by 53 teachers who taught the unit to their classes. Five states were represented in the sample: California, Colorado, Massachusetts, Minnesota, and North Carolina. Grades 2 through 6 are represented.

Table 1. Cleaning Up an Oil Spill – Classroom Grade and State Distribution

State Grade 2 Grade 3 Grade 4 Grade 5 Mixed Grade* Total

CA 8 1 5 14 CO 5 2 2 7 1 17 MA 3 1 1 11 16 MN 4 2 14 1 21 NC 1 1 15 17 Grand Total 13 13 5 52 2 85**

* CO had one mixed grades 3/4 classroom and MN one mixed grades 3-6 classroom. **The total number of classrooms is larger than the number of teachers responding

because 20 teachers taught multiple classrooms. 3.1.1.1 Overall Feedback for “Cleaning Up an Oil Spill” Unit Teachers were asked to rate the unit overall by answering questions on a scale from 1-7. Table 2 shows the number of the responses (N), the average response, the standard deviation (SD), and the minimum (Min) and maximum (Max) response to the overall unit, based on a seven point scale, with 1 representing “Not at all”, 3 representing “Slightly”, 5 representing “Moderate”, and 7 representing “Very”. Most questions were rated highly—in particular, teachers said that the unit furthered their objectives for engineering (mean=6.5, SD=0.82) and science (mean=6.0, SD=1.10); that it positively

affected students’ motivation (mean=6.4, SD=0.89); and that concepts (mean=6.1, SD=1.28) and activities (mean=6.1, SD=1.22) were age appropriate.

Table 2. Cleaning Up an Oil Spill Feedback – Unit Ratings

Question N Mean (1-7) SD Min Max

Did this unit further your objectives for science in your classroom? 53 6.0 1.10 3 7 Did this unit further your objectives for engineering? 53 6.5 0.82 3 7 Did this unit further your objectives for another content area in your classroom? 50 4.9 1.85 1 7

Did this unit positively affect your students' motivation? 53 6.4 0.89 4 7 Were the concepts presented in this unit age-appropriate for your classroom? 53 6.1 1.28 2 7

Were the materials and activities age-appropriate? 53 6.1 1.22 3 7 Did preparation for this unit require reasonable time, materials, and skill? 53 5.0 1.62 1 7

Were the lesson plans easy to understand? 53 6.0 1.13 3 7 Was this unit reasonably easy to implement and manage? 52 5.5 1.41 1 7 If you used a kit, were sufficient materials provided? 49 6.5 1.08 2 7

Teachers were less likely to agree that the unit furthered objectives for another content area (mean=4.9, SD=1.85). They were asked to specify which content area. In a separate question, teachers were also asked if they integrated the teaching of this unit with their teaching of other subjects, and if so, to explain how. Across these two questions, fifty-two teachers mentioned specific content areas that were enhanced by or integrated with the unit, with language arts being mentioned most often (52%, n=27). A number of teachers also reported integrating this unit with social studies (31%, n=16) and math (25%, n=13).

Teachers were asked a number of open-ended questions about the unit and lessons. Forty-eight of the 53 teachers responded to the question, “How did your students benefit, academically or otherwise, from taking part in this unit?” Coding for these responses is shown in Table 3.

Almost two-thirds of the respondents (64%, n=30) noted that the unit furthered student learning in STEM topics. Specifically, teachers noted that their students learned about environmental engineering, the engineering design process, and technology.

“My students benefitted by learning the engineering design process. They learned so much about engineering and technology through this unit.”–Minnesota teacher, grade 5

“They gained a better understanding of how scientists and engineers work together to solve problems. Use of academic language increased.”–California teacher, grade 5

“I think the students definitely had a clearer understanding of what environmental engineers do, as well as understanding the process of engineering.”–Minnesota teacher, grade 5

Table 3. CO – Categories of Teacher Responses to the Open-Ended Question, “How did your students benefit, academically or otherwise, from taking part in this unit?”

Coding Category Number of teachers

Percentage of respondents

(N=48) Students practiced discussion, communication skills, and teamwork 10 21.3 Students practiced problem solving and critical thinking skills 10 21.3 Students had opportunities to learn/apply STEM content and/or skills 30 63.8 Lesson makes non-STEM cross-disciplinary and multicultural connections 4 8.5 Students made connections to the real world 19 38.3 Students participated in hands-on activities and experiments 7 14.9 Students had fun; were engaged and motivated 9 19.2 Other 6 12.8 Total Number of Comments* 48 100.0

*The total number of comments is less than the sum of individual coding categories because some teachers provided comments that fit into more than one coding category.

As shown in Table 3, over one third of the teachers (39%, n =19) reported that, in addition to STEM concepts, their students also made connections to the real world. Many teachers mentioned that they taught this unit around the time of the destructive oil spill in the Gulf of Mexico in 2010, and that students were quick to make that connection.

“Coincidentally, the Gulf Oil spill happened just as I was beginning this unit. They got very interested in the current event/ethical side of this.” – California teacher, grades 3 and 5

“They loved the story that went with it, and they talked about it at home. Fortunately/unfortunately, the Gulf oil spill occurred simultaneously, which increased motivation and focus on the “real life” aspects.” – Minnesota teacher, grade 2

About one fifth of teachers (21%, n=10) listed as benefits that students practiced discussion, communication skills, and teamwork. One fifth (21%, n=10) reported that students practiced problem solving and critical thinking skills.

“They learned how to evaluate and improve their work.” – Colorado teacher, mixed grades 3 and 4, grade 5

“They used their knowledge of ecosystems, they had to work cooperatively, had to keep track of data” – Massachusetts teacher, grade 4

“I think that the students gained a good amount of information on how our environment works and what environmental engineers can be challenged with. It is also a great way for students to learn to work together and cooperate. Students were able to discuss opinions and designs with confidence.” – Massachusetts teacher, grade 5

Teachers were also asked, on a scale of 1-7, “What is the likelihood that you will choose to teach this unit again in your classroom?” Fifty of the 53 teachers provided a rating with the average score being 6.3 (SD=1.36) out of a possible 7 points. The teachers were asked to explain their rating. Thirty-six teachers responded to this question: the most common reasons given included that students were engaged and motivated (25%, n=9), that the unit fit well with the standards (22%, n=8), and that the students were able to improve in other subject areas.

“Student engagement was superior. Thinking and teamwork skills were enriched.” – North Carolina teacher, grade 5

“The students really enjoyed this lesson. They made connections immediately with the ecosystem lessons we did in science. The hands on activities were fantastic.” – North Carolina teacher, grade 5

“My students were very engaged and the unit drove home the challenges for environmental engineers. I also loved the many ways I could integrate this unit in other subject areas.” – Colorado teacher, grade 2

“There are too many positives not to teach this again as it dovetails so well with second grade science.” – Massachusetts teacher, grade 2

“Engaged students…but most importantly, state standard aligned.” – Colorado teacher, grade 5

Only 2 teachers said they would not teach the lesson again. One reported that the kit would in fact be used for another grade, but not in his or her present classroom, and the other felt he or she fell too far behind on other topics. 3.1.1.2 Lesson 1 of “Cleaning Up an Oil Spill” In Lesson 1, students read the storybook Tehya’s Pollution Solution. In the story, Tehya and her friend Sam discover an oil spill in a river located on the Olympic Peninsula of Washington State. As a member of the Lower Elwha Klallam tribe, Tehya feels an especially deep connection to the river. Working with their neighbor who is an environmental engineer, Tehya and Sam use their local knowledge of the river to aid in the clean-up efforts. The story introduces the foundational concepts for this unit, including the work of environmental engineers, the parts of ecosystems, and the connections between these parts.

All of the 53 responding teachers reported teaching this lesson in their classroom. As shown in Table 4, teachers spent an average of 115.5 minutes teaching this lesson. When asked to rate the quality of the lesson on a scale of 1-7, teachers gave Lesson 1 an average score of 6.1.

Table 4. Cleaning Up an Oil Spill Feedback – Lesson 1 Question N Mean SD Min Max

How much time did you spend teaching this lesson? (Minutes) 53 115.5 50.9 30 240 How would you rate the quality of this lesson, overall? (Scale: 1-7) 53 6.1 0.95 3 7

Table 5. CO – Categories of Teacher Responses to the Open-Ended Question, “Please explain your rating of Lesson 1”



(N=48)

Positive

Students made connections to the real world 3 6.3 Students had fun, were engaged and were motivated 19 39.6 Activities and / or supporting materials of high quality 22 45.8 Other positive comments 1 2.1 Total Positive Comments* 38 79.2

Negative

Students were not engaged/activities too easy or boring 3 6.3 Time constraints/takes too much time 8 16.7 Other negative comments 2 4.2 Total Negative Comments 13 27.1

*The total number of comments is less than the sums of individual and grouped coding categories because some teachers provided comments that fit into more than one coding category

Forty-eight of the 53 teachers who taught Lesson 1 provided an explanation for their rating of lesson quality. Table 5 shows coding categories used to code this question. Overall, teachers responded

positively to the unit, with the majority of the 48 teachers (79%, n=38) providing at least one positive comment, while only one-quarter (27%, n=13) gave a negative response. The most common criticism focused on time constraints (17%, n=8).

“I thought there is a lot that students can use in the upcoming lessons but I think the story is a little long.” – Colorado teacher, grade 5

“It took a long time to get through the story and do all of the worksheets. The students got bored with it.” – Colorado teacher, grade 3

However, almost half of the 49 respondents (46%, n=22) praised the quality of the materials. “The materials made the learning more comprehensible for the students.” – California teacher, grade 5

“Students interested; good prep for next lesson” – Minnesota teacher, grade 5

“Very good lesson, not too difficult. Gave the students the things they need to know.” – Massachusetts teacher, grade 5

Additionally, more than one-third (40%, n=19) of teachers reported that the students had fun and were motivated.

“Easy to stay engaged. Kids enjoyed it.” – Colorado Teacher, grade 2

“Tehya's Pollution Solution was a very enjoyable story. The kids enjoyed listening to it.” – Minnesota teacher, grade 5

3.1.1.3 Lesson 2 of “Cleaning Up an Oil Spill” In Lesson 2, students act as environmental engineers conducting an investigation of the soil and water in the fictional location Greentown. They are introduced to the concept of pH, the pH scale, and the idea that most organisms can only live in a specific pH range. Students are then presented with pieces of evidence that might indicate a problem affecting the ecosystem in Greentown. They perform pH tests on soil and water samples said to be taken from various locations in town to look for sources of pollution. The lesson concludes by having students create a presentation for the town to show the locations they think are most likely contributing to the pollution.

Forty-nine of the 53 teachers reported teaching Lesson 2. As shown in Table 6, teachers spent an average of 104 minutes teaching the lesson and gave it a mean overall rating of 6 on a scale of 1-7.

Forty-one teachers provided an explanation for their rating of lesson quality. Table 7 shows coding categories for this question. In general, teachers were positive about the unit with about three quarters of responding teachers (73%, n=41) providing at least one positive comment and slightly more than one third (39%, n=16) providing a negative comment.

As shown in Table 7, the two criticisms most mentioned were that of the supporting materials (15%, n=6), and time constraints (15%, n=6).

“The pH levels did not come out quite right so I would probably test prior and make corrections. I wonder if the soil (fertilized dirt) interfered with the pH level.” – Colorado teacher, grade 5

“Too time consuming.” – California teacher, grade 3

“Samples took a while to make.” – Massachusetts teacher, grade 4



Table 7. CO – Categories of Teacher Responses to the Open-Ended Question, “Please explain your rating of Lesson 2”

Coding Category Number

of teachers


(N=41)

Positive

Students had opportunities to learn/apply STEM content and/or skills 6 14.6 Students had fun, were engaged and were motivated 17 41.2 Activities and / or supporting materials of high quality 11 26.8 Other positive comments 9 21.3 Total Positive Comments* 30 73.2

Negative

Too difficult or confusing for students and/or teacher 3 7.3 Time constraints / takes too much time 6 14.6 Criticism of supporting materials / difficult to implement 6 14.6 Other negative comments 3 7.3 Total Negative Comments* 16 39.0


However, many respondents (41%, n=17) mentioned that this lesson was fun, engaging, and motivating.

“Students loved the ph testing.” – Colorado teacher, grade 2

“The students were engaged.” – Colorado teacher, grade 2

“The students liked being “detectives” to solve the problem of the pollution.” — Minnesota teacher, grade 2

Additionally, over a quarter of responding teachers (27%, n=11) praised the quality of the activities and supporting materials of Lesson 2.

“The last piece of the lesson that used the sand and food coloring was the real revelation for the class. Being able to see the spread of the pollution helped them better understand what could be happening in Greentown.” – Massachusetts teacher, grade 5

“All of the materials were provided! Set up was minimal. Duplication Masters were kid friendly and supported lesson objectives.” – Minnesota teacher, grade 3

“This lesson was great! Really helped students make pollution connections! The demonstration on pages 67-68 was wonderful. The teacher tips throughout this lesson gave great background knowledge.” –Massachusetts teacher, grade 5

3.1.1.4 Lesson 3 of “Cleaning Up an Oil Spill” In Lesson 3 of this unit, students are introduced to the design challenge: designing a process for Cleaning Up an Oil Spill to minimize the impact of the oil on a river ecosystem. In the first part, the class investigates an ecosystem as a whole, before pollution. They do so by creating a large-scale “web” which includes a food web that shows the flow of energy through the organisms in the ecosystem. Students then act out how the oil spill might affect this ecosystem. In Part 2, students test the efficacy of different materials and tools for containing an oil spill and for removing oil from the surface of the

water. Students are encouraged to think about which materials and tools they would and would not use in their designs for the next lesson (Lesson 4).

Fifty-two out of 53 teachers reported that they taught Lesson 3. Teachers spent an average of 134.6 minutes teaching this lesson and gave it an overall rating of 6.1 on a scale of 1-7 (see Table 8).



As shown in Table 9, 41 teachers commented on their rating of lesson quality. The coding categories used to code this question are shown in this table. In general, teachers were positive about the unit with the majority of respondents (71%, n=29) providing at least one positive comment, while about half (49%, n=20) gave at least one negative response. The most common criticism was that of the supporting materials (22%, n=9).

“The concepts were made clear to kids, but there are a lot of materials being passed out, handed in, etc.” – Minnesota teacher, grade 2

“They had fun doing this lesson. The materials made this lesson somewhat difficult” – Colorado teacher, grade 3

Table 9. CO – Categories of Teacher Responses for the Question, “Please explain your rating of Lesson 3”


of teachers


(N=41)

Positive

Students practiced discussion, communication, and teamwork skills 4 9.8 Students had opportunities to learn/apply STEM content and/or skills 10 24.4 Students participated in hands-on activities and experiments 8 19.5 Students had fun, were engaged and were motivated 12 29.3 Other positive comments 9 22.0 Total Positive Comments* 29 70.7

Negative

Students were not engaged/activities too easy or boring 3 7.3 Time constraints/takes too much time 6 14.6 Criticism of supporting materials/difficult to implement 9 22.0 Other negative comments 6 14.6 Total Negative Comments* 20 48.8


However, 29% (n=12) of teachers mentioned that their students enjoyed the lesson and were engaged. Also, about one quarter (24%, n=10) mentioned that students had the opportunity to enhance their STEM knowledge.

“My class really loved this lesson. As I circulated the room, their observations and comments to each other were amazing.” – California teacher, grade 3

“They loved the chain and web. They enjoyed trying different materials. They liked being surprised by the efficacy of certain materials.” – Minnesota teacher, grade 5

“Students were able to explain vocab [sic.] weeks after lesson was conducted — they got/met objective.” – California teacher, grade 3

3.1.1.5 Lesson 4 of “Cleaning Up an Oil Spill” In the final lesson, Lesson 4, students put all that they learned from the previous lessons into practice for the design challenge. They use examples of environmental engineering in the storybook to inspire their own designs. Students are also challenged to think about how the oil could further impact the ecosystem if it is allowed to reach the riverbank. As they design their oil spill cleaning process, students should keep in mind their results from testing materials in the previous lesson.

Forty-six of the 53 teachers reported that they taught this lesson in their classroom (see Table 10). On average, teachers spent 176.8 minutes teaching this lesson and gave it a rating of 6.6 on a scale of 1-7.



Table 11. CO – Categories of Teacher Responses to the Open-Ended Question,

“Please explain your rating of Lesson 4”


of teachers


(N=43)

Positive

Students practiced discussion, communication, and teamwork skills 4 9.3 Students practiced problem solving and critical thinking skills 3 7.0 Students had fun, were engaged and were motivated 24 55.8 Activities and / or supporting materials of high quality 11 25.6 Other positive comments 14 32.6 Total Positive Comments* 39 90.7

Negative

Time constraints/takes too much time 4 9.3 Too difficult or confusing for students and/or teacher 3 7.0 Criticism of supporting materials/difficult to implement 6 14.0 Other negative comments 8 18.6 Total Negative Comments* 15 34.9

*The total number of comments is less than the sums of individual and grouped coding categories because some teachers provided comments that fit into more than one coding category.

Forty-three teachers commented on their rating of lesson quality. Table 11 shows coding categories used to code this question. Overall, teachers responded positively with almost all (91%, n=39) providing at least one positive comment. A little over one-third of the teachers (35%, n=15) commented negatively.

As shown in Table 11, the most common criticism (14%, N=6) was about the supporting materials and difficulty implementing the lesson.

“I thought this was a fantastic lesson. I think there has to be a better material to use as on oil mixture. The black in the paint didn't look very black when mixed with the cooling oil.” –Minnesota teacher, grade 5

“I think it is a great culminating lesson, but to be honest, it is very messy. The students even complained about the greasiness of their materials.” –Minnesota teacher, grade 5

Nevertheless, over half of responding teachers (56%, n=24) mentioned that the lesson was fun, engaging, and motivating.

“Very engaging.” – Colorado teacher, grade 4

“It was a challenge to try to build a membrane that does exactly what needed to be done. It kept the students engaged the whole class period.” – California Teacher, grade 5

“From the beginning to the end the students loved it and they loved going through the EDP to solve their problem.” – Colorado teacher, grade 5

“Both homerooms were totally engaged. Our science labs are usually fairly noisy—fine with me if kids are on task. During this clean up lesson both homerooms were almost hushed as they worked. Very serious intent!” – Minnesota teacher, grade 5

“Kids loved it- great discussions!” – Minnesota teacher, grade 5

3.1.1.6 Summary: “Cleaning Up an Oil Spill” Formative Evaluation Many teachers expressed the belief that this unit provided good-quality opportunities for students to learn not only about science and engineering but other content areas such as language arts, social studies, and math. Given that the oil spill in the Gulf of Mexico occurred concurrently with field testing, many teachers also saw their students make real world connections between the unit and current events. Teachers also responded positively to each individual lesson, mentioned that their students enjoyed the lessons and were engaged, and praised the quality of the supporting materials. Forty-eight of 50 teachers reported that they would teach the unit again in their classrooms.

3.1.2 Summative Evaluation: Cleaning Up an Oil Spill 3.1.2.1 Assessment Design: “Cleaning Up an Oil Spill” The Cleaning Up an Oil Spill student assessment was designed and first tested during the 2008-2009 school year. Twenty-eight multiple-choice questions were chosen for the pilot assessment. In the spring of 2009, validity evidence was gathered for all questions from the Cleaning Up an Oil Spill assessment. EiE staff conducted cognitive interviews with children in grades 3 and 5 from Massachusetts classrooms, asking students to read each question aloud, explain what they thought the answer would be, and explain which answer they would choose and why.

During the summer of 2009, a number of questions were dropped or revised based on the results of the pilot study and validity testing to create a field test version of the assessment. In 2009-2010 it was used for field testing in five states (California, Colorado, Massachusetts, Minnesota, and North Carolina). The field test version included 22 multiple-choice questions.

On the identical pre- and post-assessments, students were asked engineering questions about what environmental engineers do for their job and how they use models. They were also asked science questions about ecosystems, food chains, food webs, and pollution. Table 12 describes the text for each question with the correct answer shown in brackets.

Table 12. Cleaning Up an Oil Spill Assessment Questions (Text) Q #

Learning Objective Scale Question Text

1 Food Webs Science A food web shows: [all of the above: what each animal eats in an ecosystem, how energy moves through an ecosystem, how nutrients move through an ecosystem]

2 Ecosystems Science What does a habitat provide for an animal or plant? [all of the above: air, food, shelter]

3 Food Webs Science Which are consumers? [everything but grass and the Sun] (see Figure 2)

4 Food Webs Science If the grass in the meadow stopped growing, what would probably happen? [all of the above: there would be fewer mice, there would be fewer hawks, there would be fewer robins] (see Figure 2)

5 Food Webs Science If a disease caused most of the mice to die, what would probably happen? [there would be more grass (see Figure 2)

6 Food Webs Science What does the arrow from the grasshopper to the robin in the food web diagram mean? [that robins get energy from grasshoppers] (see Figure 2)

7 Food Webs Science What do the arrows in the food web diagram show? [where each thing in the ecosystem gets its energy] (see Figure 2)

8 Environmental Engineering Engineering What would an environmental engineer MOST LIKELY do for his or her

job? [design a way to clean up an oil spill]

9 Food Chain Science Which of the following events involves a consumer and producer in a food chain? [a deer eats a leaf]

10 Environmental Engineering Engineering An environmental engineer works on problems related to: [soil]

11 Environmental Engineering Engineering An environmental engineer would MOST LIKELY design: [a way to clean

water]

12 Environmental Engineering Engineering

An environmental engineer thinks about: [all of the above: how to protect habitats, how to prevent pollution, how to make water safe for people to drink]

13 Pollution Pollution Which of these is a kind of pollution? [all of the above: oil in a stream, trash on the ground, chemicals and the ground]

14 Pollution Pollution A lake becomes polluted. What living things are affected by the dirty water in the lake? [all of the above: the plants around the lake, the fish that live in the lake, the underground water around the lake]*

15 Pollution Pollution

A tank outside of a factory has leaked, putting pollution into the soil. What plants and animals could become sick? [all of the above: plants next to the factory, animals that live and eat in a pond nearby, plants growing a few miles away from the factory]

16 Models Engineering

How could a model help an environmental engineer to do his or her job? [all of the above: models help engineers learn about how ecosystems work, models help engineers figure out what might happen to an ecosystem, models help engineers try out different designs to see how they work]

17 Pollution Pollution Which of the following are possible sources of water pollution? [all of the above: waste from animals, oil spilled by broken machines, fertilizer that people put on grass to make it grow]

18 Models Engineering & Pollution

Engineers are figuring out how to clean pollution from a lake. Using a model: [all of the above: could help them figure out where the pollution came from, could help them try different ideas to clean the pollution without hurting the lake, could help them to understand how the pollution could hurt the ecosystem in the lake]

Q #

LearObje

19 EnvironEngine

20 Pollu

21 Food C22 Mod

ning ctive

nmental eering Eng

ution Po

Chains Sdels Eng* The phrasin

Scale

gineering Wen

ollution

Bthhafl

Science Wgineering Wng of question

Figure 2

Figure 3

Which of the fngineer? [poll

Biologists havhe middle of aave caused thlowed through

Which of the fWhich of thesen 14 was unint

. Cleaning U

3. Cleaning

following woulute a stream e noticed thata forest, far awhe fish to get sh the ground ifollowing is a e things is a mentionally con

Up an Oil Sp

Up an Oil Sp

Question

uld NOT be pto test the eff

t the fish in a way from roasick? [yes, pointo the pond

a producer? [gmodel? [all ofnfusing as wat

pill Question

Spill Question

Text

part of the jobfects on the epond are sick

ads and buildiollution from d] grass] (see Figf these are moter is not a livi

ns 3-7

n 21

b of an environvironment] k. The pond iings. Could pfar away cou

gure 3) odels] (see Figing thing.

onmental s out in ollution ld have

gure 4)

3Tqp

WwmefcascsSswtat

3Cssad

3.1.2.2 The reliabiliquestion asspurpose of th

We initially was found tomajor learninengineers anfood webs ancorrect answas the “all osubscales baconsisting ofscience scaleScience scalescience unitswhich questithat questiona Cronbach’the pollution

Scale

All Environm

EngineeringScience (

Pollution (

3.1.2.3 Cleaning Upsample of dastudents (n=2a sample ofdropped due

Scale Coity sample cessment. Sixhis report, w

constructedo have a Crng objective

nd (2) the nand (4) pollu

wer choice, wof the abovased on the f the engineees: one addre as it contas about ecoions were us

n 18 was uses alpha of .7

n subscale ha

e St

mental g (Eng) Sci)

(Pol)

* Q

Sample:p an Oil Spilata collected298) were exf 1439 stude

to missing

Figure 4

onstruction:consisted ofx items were limit our a

d a scale whironbach’s alpes in mind: ature and usution. Due towe were unabe” questionlearning ob

ering questioressing the

ains questionsystems) ansed to created for both th701 (n=112)ad a Cronbac

TN of udents

#Ite

105 2

112 8

116 9

115 6

Question 18 is

: “Cleaning ll (CO) assesd from gradexcluded becents in 71 cdemographi

4. Cleaning

“Cleaning f 120 studenre dropped banalysis to th

ich was the pha of 0.85two enginees of model

o the large nble to use Ps grouped ibjectives meons about enscience topi

ns that test cnd one consie the subscahe Engineeri), the sciencech’s alpha of

Table 13. Su# ems Ques

22 1-

8 8,10-18*,

9 1-7,

6 13-15, 2

used in both t

Up an Oil Spssments werees 3-5 includause they wclassrooms. c informatio

Up an Oil Sp

Up an Oil Spnts in six pibefore field

he 22 questio

sum of all 1 (n=105). Tering learninls and two snumber of anrincipal Cominto the samentioned earnvironmentalic of food w

concepts simisting of quales and alsong and Pollue subscale hf .668 (n=11

ummary of C

stions

-22 12, 16, 19, 22 9, 21 17, 18*,

20 Wit

the Engineerin

Spill” e collected f

ded 1737 stuere missing Additionall

on. Nineteen

Spill Question

Spill” ilot classrootesting with

ons that were

correct answThe assessmng objective

science learnnswers for wmponent Anme componerlier. We fol engineers awebs and ec

milar to whatuestions abouo gives a deution subscahad a Cronb5).

CO Scales

Content

All quWork of en

engEcosystems

What is Pollutiinteract with

ng and Polluti

from studentudents in 72 either a pre-ly, 117 stud

n classes wer

n 22

oms who hah a revised e on the revi

wers on all qment had beees: (1) the jning objectivwhich “all ofnalysis (PCAent. We instormed one eand models. cosystems (wt is typicallyut pollution

escription of ale. The engiach’s alpha

t Assessed

uestions nvironmental ineers and food webion and how d the environmion subscale.

ts in grades classrooms.

- or post-assdents from re missing in

ad completeassessment

ised assessm

questions. Ten devised wob of enviroves: (3) ecosf the above”

A) to create stead chose engineering We also forwhich we c

y taught in trn. Table 13f each subscaineering subof .749 (n=

ReliaCronb

.8

.7

bs .7does ment .6

3 through 7.. However, sessment, res16 classroonformation o

ed the 28 . For the

ment.

This scale with four onmental systems / ” was the subscales to create subscale

rmed two called the raditional indicates ale. Note scale had

=116) and

ability: bach’s α 851

701

749

668

. The full 17.2% of sulting in

oms were on which

students received free or reduced-price lunch (FRL), so that variable was not considered in the analysis of this unit. Thirty-seven students answered fewer than 19 questions on either the pre-assessment or the post-assessment and were dropped from the dataset. We dropped one class of six students because they were missing specific student grade information and it was possible that the class contained some Grade 6 and Grade 7 students. We dropped three classrooms that reported not completing Lesson 4 because we consider this hands-on lesson critical to the EiE curriculum.

The final dataset used for analysis included 1216 students (Level-1 units) in 61 classrooms (Level-2 units), with an average classroom (cluster) size of 25.9 students and a standard deviation of 5.4 students. The classrooms were spread over 35 schools with 42 teachers total, including 4 science specialists. The teachers had an average of 13.2 years of experience with a standard deviation of 8.5 years and a range of 2 to 37 years.

The majority of the sample was in grade 5 with significant numbers from grades 3 and 4 (see Table 14). Girls made up slightly more than half of the sample (51.1%). Students with limited English proficiency (LEP) made up 5.0% of the sample, and students with an Individualized Educational Program (IEP) made up 7.2% of the sample (see Table 15). White students made up 62%, Asians 8%, Blacks 15%, Hispanics 11%, and other racial groups 4% of the sample (see Table 16).

Table 14. CO Grade Crosstabulation (N of Students and Classrooms)

Grade Total 3 4 5 N (students) 108 128 980 1216 N (classrooms) 6 7 48 61

Table 15. CO Percentages for Level-1 Dichotomous Variables

Gender (male) LEP IEP Proportion 48.9% 5.0% 7.2% N 595 61 88

Table 16. CO Percentages for Level-1 Variables – Race

Black Asian Hispanic White Other Total Proportion 15% 8% 11% 62% 4% 100% N 181 94 128 759 54 1216

3.1.2.4 Results: “Cleaning Up an Oil Spill” Three outcome variables were used for analysis. The engineering scale (Eng) was used to gauge students’ improvements in understanding models and what environmental engineers do for their jobs. The science scale (Sci) was used to examine students’ improvements in understanding ecosystems and food webs. Finally, the pollution scale (Pol), was used to measure students’ improvements in understanding the effects of pollution. The engineering scale had a range of 0 to 8, the science scale had a range of 0 to 9, and the pollution scale had a range of 0 to 6.

For this unit, we tested the student level (level-1) variables of IEP and LEP status, as well as gender and race (Asian, Black, Hispanic, and Other). We also included pre-assessment scores as a covariate. Next, we tested whether each level-1 variable was best modeled as a fixed or random coefficient, the default being a fixed coefficient, unless the random variance was found to be significant (p < .05). At the classroom level (level-2) we tested the classroom means of the pre-assessment scores, class size, the

number of minutes spent teaching the EiE unit (TotalMinutes), the number of EiE units the teacher had taught prior to that school year (PriorUnits), the number of units the teacher taught during that school year (CurrentUnits), and the number of years the teacher had been teaching overall (NumYearsTeaching) as covariates. We also tested the effects of grade and of teacher specialty (Science Specialist) here, as well as effects of other variables describing the overall classroom setting (the proportions of IEP students, LEP students, and male students in the classroom). The two-level final conditional models for each subscale of the CO assessment (see Figure 5, Figure 8, and Figure 10) include all of the variables and random variance coefficients which were found to be significant during the variable testing process described above.

CO: Engineering Scale Results. To test engineering outcomes, we created a model to predict the students’ post-assessment engineering scale score (PostEng Centered) centered on the grand mean of the pre-assessment. Scores were computed by adding together the number of correct answers for each of the engineering questions indicated in Table 12 and listed in Table 13. The Engineering scale had a possible range of 0 to 8. Mean pre- and post-assessment scores for the various demographic groups tested on the Engineering scale are shown in Table 17 below.

Table 17. CO Descriptive Statistics: Pre- and PostEngineering Scores by Demographic Groups

CO PreEngineering Score CO PostEngineering Score

Mean SD Min Max Mean SD Min Max Overall 4.26 2.00 0 8 5.84 1.85 0 8 Males 4.07 1.99 0 8 5.79 1.90 0 8 Females 4.44 1.99 0 8 5.89 1.79 0 8 IEP 3.41 1.88 0 8 4.77 2.12 0 8 LEP 2.79 1.72 0 7 4.77 1.95 0 8 White 4.60 1.95 0 8 6.21 1.71 0 8 Hispanic 3.30 1.85 0 7 4.67 1.92 0 8 Black 3.66 1.94 0 8 5.20 1.84 1 8 Asian 4.13 1.97 0 8 5.95 1.69 1 8 Other 4.02 2.09 0 8 5.37 2.05 0 8 Grade 3 2.68 1.53 0 7 4.58 1.87 1 8 Grade 4 3.88 1.93 0 8 5.18 2.06 0 8 Grade 5 4.49 1.97 0 8 6.07 1.74 0 8

For the engineering subscale, our model (Figure 5 and Table 18) shows that the baseline improvement between pre- and post-assessment was high, at 1.65 points (CI 1.47, 1.82) out of 8 total points (Intercept γ00, p < .001), meaning that the students not in groups explicitly listed in our model gained a better understanding of the work of an environmental engineer. Black students (1.03 points improvement; CI 0.63, 1.43), Hispanic students (0.98 points; CI 0.57, 1.38), students in the racial demographic category “Other” (0.96 points; CI 0.5, 1.42), and students with IEP’s (0.68 points improvement; CI 0.14, 1.22) showed less dramatic though still positive improvement (see Figure 6). Classrooms with instructors teaching more units of EiE this year and those teaching this unit for a longer time tended to show even more improvement. Our two-level model explains 87% (0.87=1−(0.108/0.859)) of the between-class variance.

Level-1 Model: 1 530 2 4PostEng (IEP) ( ) (Black) (Hispanic) (Other) r= β +β +β +β +β +β +PreEng

2Var(r) σ=

And:

2

0 1 2 3 4 5

6

ln (IEP) ( ) (Black) (Grade4) ( )(PriorUnits)

PreEngClassMeanσ α α α α α αα

= + + + + ++

PreEng

Level-2 Model: 0 00 01 02 03 0(Curren (t )Units) ( ) uPreEn Tg oCl taassM lMine ean ut sβ = γ + γ + γ + γ +

1 10β γ= 2 20β γ= 3 30β γ= 44 0β γ= 55 0β γ= Bold indicates group mean centered. Italicized indicates grand-mean centered. The outcome variable PostEng was centered around the PreEng mean.

Figure 5. Cleaning Up an Oil Spill PostEng Score – Conditional Model

Table 18. CO PostEng Score Conditional Model – Final Estimation of Fixed Effects (with robust standard errors)

Fixed Effect Coefficient Standard Error t-ratio Approx.

df P-value

For Intercept 1, β0 Intercept, γ00 1.647 0.088 18.711 57 <0.001 CurrentUnits, γ01 0.129 0.018 7.339 57 <0.001 PreEngClassMean, γ02 0.6191 0.061 10.183 57 <0.001 TotalMinutes, γ03 0.135 0.055 2.447 57 0.017 For IEP, β1 Intercept, γ10 −0.965 0.256 −3.770 1150 <0.001 For PreEng, β2 Intercept, γ20 0.344 0.027 12.708 1150 <0.001 For Black, β3 Intercept, γ30 −0.615 0.181 −3.405 1150 <0.001 For Hispanic, β4 Intercept, γ40 −0.671 0.181 −3.705 1150 <0.001 For Other, β5 Intercept, γ50 −0.688 0.214 −3.211 1150 0.001

Table 19. CO PostEng Score Conditional Model – Final Estimation of Variance Components

Random Effect Standard Deviation

Variance Component df Chi-square P-value

Intercept 1, u0 0.328 0.108 57 132.608 <0.001 Level-1, r 1.44 2.07

Table 20. CO PostEng Score – Model for Level-1 Variance

Parameter Coefficient Standard Error

Z-ratio

P-value

Intercept 1 ,α0 1.385 0.204 6.789 0.000 IEP ,α1 0.581 0.161 3.605 0.001 PreEng,α2 −0.086 0.024 −3.584 0.001 Black,α3 0.247 0.119 2.073 0.038 Grade 4,α4 0.330 0.139 2.381 0.017 PreEngClassMean,α5 −0.204 0.044 −4.594 0.000 Grade 5,α6 0.053 0.022 2.460 0.014

Table 21. CO PostEng Score Unconditional Model – Final Estimation of Variance Components



Intercept 1, u0 0.927 0.859 60 399.008 <0.001 level-1, r 1.64 2.68

Figure 6. CO Change in Student Score: Engineering Scale

The level-1 (within-class) variance σ2 was found to be heterogeneous (p=.006) and was modeled logarithmically (see Figure 5 and Table 20). The within-class variance was smaller among students with high pre-assessment scores (PreEng) and in classes with high pre-assessment means (PreEngClassMean—presumably due to ceiling effects). The within-class variance was larger among students with an IEP, Black students, and Grade 4 students, and slightly larger in classes with instructors who had taught more units of EiE in previous years (PriorUnits). Once modeled, σ2 was much less likely

ReferenceGroup

IEP

Black Hispanic Other

0

0.5

1

1.5

2

Student Change in Score: CO Eng scale with 95% confidence intervals

Change

in Score

(Pre to Po

st)

tv

Fuotfs

Tuwmm(P

to be heterovariance.

Figure 7 shounit (TotalMof the studenthe unit in thfit (PostEng significant in

The EiE currunit. The recwas 430 to 4minimum numaximum re(see Table 2PostEngineer

F

geneous (p=

ows the relatMinutes) and

nts have beeheir classes. = 0.135(Tot

n the analysi

riculum comcommended 490 minutes

umber of minecommended2). In generring scale.

Figure 7: CO

=.195), sugg

tionship betwtheir student

en graphed aThe solid li

talMinutes))s.

mes with recrange of mi

s, is shown nutes spent

d time, and tral, more tim

PostEng Sc

gesting that

ween the tots’ PostEng against the zine shows th), which also

ommendationutes to be sby two vertteaching the

the maximumme spent on

cores vs. Num

our model

tal number scores for C

z-score of thhe LOESS tro takes into a

ons as to howspent on thetical lines oe unit in ourm number othe unit tran

mber of Min

describes im

of minutes Cleaning Up he total numbrend; the dasaccount the

w much time entire Clean the graph

r data, the mf minutes spnslated into

nutes Spent T

mportant sou

that teacheran Oil Spillber of minushed line shodemographi

me should beaning Up an h. The labelsminimum recpent teachinghigher asses

Teaching the

urces of wit

s spent on tl. The PostEnutes of instruows the HLMic variables t

e spent teachOil Spill un

s show, in ocommended g the unit inssment score

e CO Unit

thin-class

the entire ng scores uction for M model that were

hing each nit, which order, the time, the

n our data es on the

Table 22. Notable Times Shown in Figure 7 Label (minutes) Description

265 Minimum number of minutes reported spent teaching EiE 430 Minimum recommended time 490 Maximum recommended time 855 Maximum number of minutes reported spent teaching EiE

CO: Science Scale Results. We modeled science outcomes for this unit as the outcome variable PostSciCentered. This scale had a possible range of 0 to 9. Mean pre- and post-assessment scores for the various demographic groups tested in modeling the Science scale are shown in Table 23 below.

Table 23. CO Descriptive Statistics: Pre- and PostScience Scores by Demographic Groups

CO PreScience Score CO PostScience Score Mean SD Min Max Mean SD Min Max

Overall 4.90 2.04 0 9 6.75 1.98 0 9 Boys 4.93 2.04 0 9 6.75 2.01 0 9 Girls 4.86 2.04 0 9 6.74 1.95 0 9 IEP 3.91 1.84 0 8 5.50 2.23 0 9 LEP 3.46 1.88 0 8 5.43 2.26 0 9 White 5.13 2.03 0 9 7.08 1.80 1 9 Hispanic 4.01 2.03 0 8 5.49 2.20 0 9 Black 4.50 1.91 0 9 6.21 2.13 0 9 Asian 5.21 1.92 1 9 6.97 1.77 2 9 Other 4.59 1.96 1 8 6.56 2.13 0 9 Grade 3 3.06 1.65 0 7 4.67 2.06 0 9 Grade 4 3.99 2.06 0 9 6.05 2.13 0 9 Grade 5 5.22 1.93 0 9 7.07 1.78 0 9

The two-level final conditional model for the Science scale was established in the same manner as that for the PostEng Centered outcome variable, except that the PreEng covariate and PreEng Classroom Mean were replaced by the variables PreSci and PreSci Classroom Mean during testing of variables. The final PostSci Centered model, shown in Figure 8 and Table 24 below, shows that there was an improvement of 2.00 points (CI 1.74, 2.27) between the pre- and post-assessment (Intercept γ00, p <.001), meaning that the students gained a significantly better understanding of ecosystems and food webs between the pre- and post-assessments. Black students (1.6 points improvement; CI 1.19, 2.02), Hispanic students (1.3 points; 0.85, 1.75), and students with IEP’s (0.98 points; 0.46, 1.5) showed less dramatic but still positive improvement.

As with the engineering scale, classrooms with an instructor teaching more units of EiE this year performed better. Grade 3 classrooms tended to show less improvement in science than the rest of the sample. Our two-level model explains 78% of the between-class variance (0.78=1− (0.265 /1.22)).

The level-1 (within-class) variance σ2 was found to be heterogeneous (p<.001) and was modeled logarithmically (see Figure 8 and Table 26). The within-class variance was smaller among students with high pre-assessment scores or in a class with a high pre-assessment mean (presumably due to ceiling effects). The within-class variance was also smaller in classrooms with a higher proportion of boys and slightly smaller in classrooms with an instructor currently teaching a greater number of EiE units. It was

larger among students having an IEP. Once modeled, σ2 was still heterogeneous (p=.037), suggesting that our model does not accurately describe some sources of different within-class variances.

Level-1 Model: 1 430 2PostSci (IEP) ( ) (Black) (Hispanic) r= β +β +β +β +β +PreSci

2Var(r) σ=

And:

2

0 1 2 3 4

5

ln (IEP) ( ) ( ) ( )(CurrentUnits)

ProportionMale PreSciClassMeanσ α α α α αα

= + + + ++

PreSci

Level-2 Model: 0 00 01 02 03 0(CurrentUnits) (Grade ) u( ) 3 PreSciClassMeanβ = γ + γ + γ + γ +

1 10β γ= 2 20β γ= 3 30β γ= 44 0β γ= Bold indicates group mean centered. Italicized indicates grand-mean centered. The outcome variable PostSci was centered around the PreSci mean.

Figure 8. Cleaning Up an Oil Spill PostSci Score – Conditional Model

Table 24. CO PostSci Score Conditional Model – Final Estimation of Fixed Effects (with robust standard errors)

Fixed Effect Coefficient Standard Error T-ratio Approx.

df P-value

For Intercept 1, β0 Intercept, γ00 2.003 0.133 15.042 57 <0.001 CurrentUnits, γ01 0.101 0.018 5.520 57 <0.001 Grade 3, γ02 −1.184 0.338 −3.499 57 <0.001 PreSciClassMean, γ03 0.487 0.085 5.709 57 <0.001 For IEP, β1 Intercept, γ10 −1.023 0.222 −4.609 1151 <0.001 For PreSci, β2 Intercept, γ20 0.360 0.028 12.890 1151 <0.001 For Black, β3 Intercept, γ30 −0.400 0.158 −2.539 1151 0.011 For Hispanic, β4 Intercept, γ40 −0.703 0.181 −3.879 1151 <0.001

Table 25. CO PostSci Score Conditional Model – Final Estimation of Variance Components

Table 26. CO PostSci Score – Model for Level-1 Variance Parameter Coefficient Standard Error Z-ratio P-value

Intercept 1 ,α0 2.501 0.259 9.659 0.000 IEP,α1 0.573 0.162 3.545 0.001 PreSci,α2 −0.203 0.025 −8.141 0.000 ProportionMale, α3 −1.327 0.476 −2.790 0.006 PreSciClassMean,α4 −0.253 0.038 −6.696 0.000 CurrentUnits,α5 −0.062 0.016 −3.781 0.000

Table 27. CO PostSci Score Unconditional Model – Final Estimation of Variance Components




Figure 9. CO Change in Student Scores: Science Scale

CO: Pollution (Pol) Scale Results. We modeled pollution outcomes for this unit as the outcome variable PostPol. This scale had a possible range of 0 to 6. Mean pre- and post-assessment scores for the various demographic groups tested on the Pollution scale are shown in Table 28 below.

ReferenceGroup

IEP

Black

Hispanic

0

0.5

1

1.5

2

2.5

Student Change in Score: CO Sci scale with 95% confidence intervals

Change

in Score

(Pre to Po

st)




Table 28. CO Descriptive Statistics: Pre- and PostPollution Scores by Demographic Groups

CO PrePollution Score CO PostPollution Score

Mean SD Min Max Mean SD Min Max Overall 3.17 1.77 0 6 4.19 1.69 0 6 Boys 2.96 1.73 0 6 4.09 1.77 0 6 Girls 3.37 1.78 0 6 4.29 1.61 0 6 IEP 2.66 1.70 0 6 3.47 1.72 0 6 LEP 2.20 1.62 0 6 3.48 1.71 0 6 White 3.40 1.76 0 6 4.45 1.61 0 6 Hispanic 2.45 1.63 0 6 3.42 1.68 0 6 Black 2.76 1.72 0 6 3.70 1.75 0 6 Asian 3.29 1.74 0 6 4.40 1.66 0 6 Other 2.80 1.64 0 6 3.74 1.75 0 6 Grade 3 1.64 1.34 0 6 2.78 1.93 0 6 Grade 4 2.75 1.66 0 6 3.57 1.61 1 6 Grade 5 3.39 1.73 0 6 4.43 1.57 0 6

The two-level final conditional model for the pollution scale was established in the same manner as those for the PostEng and PostSci outcome variables, except that the PreEng/PreSci covariate and PreEng Classroom Mean/PreSci Classroom Mean were replaced by the variables PrePol and PrePol Classroom Mean during testing of variables. The final PostPol model, shown in Figure 10 and Table 29 below, shows that there was an improvement of 1.15 points (CI 1.01, 1.39) between the pre- and post-assessment (Intercept γ00, p <.001), meaning that the students gained a better understanding of the characteristics of pollution and its effects on the environment between the pre- and post-assessments. Black students (0.83 points improvement; CI 0.47, 1.19) and students with IEP’s (0.40 points; CI 0.06, 0.75) also improved, though less dramatically. Confidence intervals for all demographic groups in the model did not include zero, indicating that it is at least 95% probable that students improved from pre- to post-assessment. Classrooms taught by a teacher with more years of experience (NumYearsTeaching) tended to show more improvement than other classrooms. Our two-level model explains 82% (0.82=1- (0.127/0.713)) of the between-class variance.

The level-1 (within-class) variance σ2 was found to be heterogeneous (p<.001) and was modeled logarithmically (see Figure 10 and Table 31). The within-class variance was smaller among students with high pre-assessment scores (PrePol) and in classrooms with a higher proportion of boys (ProportionMale) as well as in classrooms with high average pre-assessment scores (PrePolClassMean). It was also smaller in classrooms taught by instructors with a greater number of years of teaching experience (NumYearsTeaching). The within-class variance was larger among male students (Gender) and Black students. Once modeled, σ2 was more likely to be homogeneous (p=.123), suggesting that our new model more accurately describes sources of within-class variance.

Level-1 Model: 0 1 2 3PostPol (IEP) ( ) (Black) r= β +β +β +β +PrePol

2Var(r) σ=

And:

2

0 1 2 3 4

5 6

ln (Gender) ( ) (Black) ( )( ) ( )

NumYearsTeachingProportionMale PrePolClassMean

σ α α α α αα α

= + + + ++ +

PrePol

Level-2 Model: 0 00 01 03 0( ) ( ) uNumYearsTeachingPrePolClassMeanβ = γ + γ + γ +

1 10β γ= 2 20 2uβ γ= + 3 30β γ= Bold indicates group mean centered. Italicized indicates grand-mean centered. The outcome variable PostPol was centered around the PrePol mean.

Figure 10. Cleaning Up an Oil Spill PostPol Score – Conditional Model

Table 29. CO PostPol Score Conditional Model – Final Estimation of Fixed Effects (with robust standard errors)

Fixed Effect Coefficient StandardError T-ratio Approx.

df P-value

For Intercept 1, β0 Intercept, γ00 1.152 0.072 16.062 58 <0.001 PrePolClassMean, γ01 0.796 0.067 11.957 58 <0.001 NumYearsTeaching, γ02 0.012 0.004 2.778 58 0.007 For IEP slope, β1 Intercept, γ10 −0.747 0.156 −4.800 1092 <0.001 For PrePol slope, β2 Intercept, γ20 0.366 0.031 11.901 60 <0.001 For Black slope, β3 Intercept, γ30 −0.325 0.166 −1.964 1092 0.050

Table 30. CO PostPol Score Conditional Model – Final Estimation of Variance Components



Intercept 1, u0 0.357 0.127 58 161.639 <0.001 PrePol slope, u2 0.145 0.021 60 98.954 0.001 level-1, r 1.309 1.714

Table 31. CO PostPol Score – Model for Level-1 Variance Parameter Coefficient Standard Error Z-ratio P-value

Intercept 1 ,α0 2.034 0.239 8.518 0.000 Gender,α1 0.178 0.086 2.080 0.037 PrePol,α2 −0.140 0.028 −5.038 0.000 Black ,α3 0.312 0.120 2.606 0.010 NumYearsTeaching,α4 −0.011 0.005 −2.293 0.022 ProportionMale,α5 −1.347 0.463 −2.906 0.004 PrePolClassMean,α6 −0.320 0.052 −6.152 0.000

Table 32. CO PostPol Score Unconditional Model – Final Estimation of Variance Components




Figure 11. CO Change in Student Score: Pollution Scale

3.1.2.5 Summary: “Cleaning Up an Oil Spill” Summative Evaluation Our analysis shows that students who are taught the EiE Cleaning Up an Oil Spill unit, possibly together with a science unit on ecosystems and food webs, learn a significant amount about engineering, science, and pollution concepts. All demographic categories showed improvement on all three subscales; however, Black students and students with IEP’s consistently showed less improvement than their peers. In two out of three subscales, Hispanic students showed less improvement than peers as well. Interestingly, the years of experience of the teacher had no significant effects on the engineering and science subscales and only very weak positive effects on the pollution subscale. On the other hand, having a teacher who was teaching more units of EiE in the current school year correlated with better engineering and science outcomes for the class. This may be because these teachers were more recently familiar with the material and had more resources readily at hand.

ReferenceGroup

IEP

Black

0

0.5

1

1.5

Student Change in Score: CO Pol scale with 95% confidence intervals

Change

in Score

(Pre to Po

st)

The assessment itself had some flaws, the major one being that too many items had as the correct answer “all the above.” This led to correlation between questions covering different learning objectives. We are designing future assessments without “all the above” answers to limit these correlations and make it more difficult for students to use test taking strategies to improve their scores.

3.2 Results for the “Replicating an Artifact” Unit Evaluation In the EiE unit Solid as a Rock: Replicating an Artifact, students apply what they are learning about rocks and minerals in their science curriculum to the field of materials engineering. In Lesson 1, students read the storybook Galya and Natasha’s Rocky Adventure, set in Russia, which introduces them to the field of materials engineering and reviews science concepts such as the rock cycle and the processes that create igneous, sedimentary, and metamorphic rocks. In Lesson 2, students gain a broader understanding of the field of materials engineering by observing and testing two products (twill and terry fabrics) with different structures and properties that were created from the same raw material using different processes. In Lesson 3, students are introduced to their design challenge: designing a replica of a petroglyph for a museum. Students observe the materials available to them and investigate some of their properties in order to help them decide which materials would be good choices for their replica designs. In the final lesson, Lesson 4, students use the Engineering Design Process to design their own artifact replicas. Applying what they learned in Lesson 3, students create, evaluate and improve their replicas. They then write a letter to the museum commissioning the replicas explaining their replica designs.

Solid as a Rock: Replicating an Artifact was field tested during the 2009-2010 school year.

3.2.1 Formative Evaluation: Replicating an Artifact Feedback forms for Replicating an Artifact were completed by 47 teachers who taught the unit in their classroom. Six states were represented in the sample: California, Colorado, Massachusetts, Minnesota, New Hampshire, and North Carolina. Grades 2 through 6 are represented.

Table 33. Replicating an Artifact– Classroom Grade and State Distribution

State Grade 2 Grade 3 Grade 4 Grade 5 Grade Mixed* Total**

CA 2 1 3 6 CO 1 2 9 3 15

MA*** 1 10 4 8 23 MN 2 9 7 2 20 NC 1 15 16 NH 2 2

Total 2 17 38 23 2 92 * In MN there was 1 mixed Grade 4/5 classroom and 1 mixed Grade 3-6 classroom. **The total number of classrooms is larger than the number of teachers responding,

because 19 teachers taught more than one class and each class is counted individually. ***One MA teacher taught both a grade 2 class and a grade 3 class.

3.2.1.1 Overall Feedback for “Replicating an Artifact” Unit Teachers were asked to rate the unit overall by answering questions on a scale from 1-7. Table 34 shows the number of the responses (N), the average response, the standard deviation (SD), and the minimum (Min) and maximum (Max) response to the overall unit, based on a seven point scale, with 1 representing “Not at all” and 7 representing “Very”.

Table 34. Replicating an Artifact Feedback – Unit Ratings


Did this unit further your objectives for science in your classroom? 47 6.1 1.20 1 7

Did this unit further your objectives for engineering? 47 6.3 0.94 3 7 Did this unit further your objectives for another content area in your classroom? 42 4.3 2.03 1 7



Were the Lesson Plans easy to understand? 46 5.6 1.27 3 7 Was this unit reasonably easy to implement and manage? 47 5.5 1.23 3 7 If you used a kit, were sufficient materials provided? 46 6.4 1.20 1 7

Teachers were asked whether this unit furthered their objectives for another content area beyond science, and if so, to specify which content area. In a separate question, teachers were also asked if they integrated the teaching of this unit with their teaching of other subjects, and if so, to explain how. All 47 teachers responded to at least one of these questions. Nearly three-quarters of teachers (72%, n=34) mentioned specific content areas that were enhanced by the unit, with language arts being mentioned most often (79%, n=27). Approximately one-quarter of teachers who mentioned specific content areas beyond science also reported that the unit furthered their objectives in math (24%, n=8) and social studies (26%, n=9).

Teachers were asked a number of open-ended questions about the unit and lessons. Forty-six of the 47 teachers responded to the question, “How did your students benefit, academically or otherwise, from taking part in this unit?” The coding categories used to code this question are shown in Table 35. Nearly half of the respondents (46%, n=21) noted that the unit furthered their students’ understanding of STEM knowledge and skills. Specifically, teachers noted that their students learned about rocks and minerals, the work of engineers, and the engineering design process.

“We studied rocks and minerals at the same time and I feel the students understand the rock cycle and the 3 types of rocks with a greater sense of how and why they're different. Knowing their origins and properties helped.” – Massachusetts teacher, grade 3

“Students this semester learned much more about properties of various rocks than the second semester, without the unit. In addition MUCH more about the engineering process was learned.” – Minnesota teacher, grade 5

“The students have a better understanding of the three types of rocks. They also broadened their knowledge of types of engineers, archaeologists, artifacts, replicas, etc.” – North Carolina teacher, grade 4

Table 35. RA – Categories of Teacher Responses to the Open-Ended Question, “How did your students benefit, academically or otherwise, from taking part in this unit?”



(N=46) Students practiced discussion, communication, and teamwork skills 8 17.4 Students had opportunities to learn/apply STEM content and/or skills 21 45.7 Lesson makes non-STEM cross-disciplinary and multicultural connections 10 21.7 Students made connections to the real world 4 8.7 Students participated in hands-on activities and experiments 5 10.9 Students had fun, were engaged and were motivated 16 34.8 Other comments 12 26.1 Total Number of Comments* 46 100

*The total number of comments is less than the sum of individual coding categories because some teachers provided comments that fit into more than one coding category

Additionally, approximately one-third of teachers (35%, n=16) reported their students had fun and were engaged and motivated.

“Typically Rocks and Minerals is a low engagement content area with students. This kit brought high engagement to this content area.” – North Carolina teacher, grade 4

“They loved the unit and were very motivated to ‘solve the problem’!” – California teacher, grade 3

“Students loved being engineers! Engagement totally helped increase knowledge of rocks and minerals.” – North Carolina teacher, grade 4

“I cannot tell you how excited my kids were to “do science.” From this unit we ventured out into a geology project around our school and forged a partnership with our local watershed to “engineer” rain gardens to improve the drainage around our school. It has been a fabulous experience for all of us! We are also holding a mini science/engineering fair in December!” – Minnesota teacher, mixed grade class

Nearly one-quarter of the teachers (22%, n =10) reported that, in addition to science and engineering concepts, their students also had opportunities to improve in other areas such as math and language arts.

“Overall, I am a big fan of the EiE Kits. My students experienced increased motivation and comprehension of the science standards taught and I appreciate the integration of reading, math and writing.” – Massachusetts teacher, grade 3

“Literacy skill developed with a high interest reading” – California teacher, grade 4

“The kids enjoyed it, were engaged and were able to apply reading and math knowledge to the process.” – Massachusetts teacher, grade 3

About one-fifth of teachers (17%, n=8), noted that the unit gave their students the opportunity to practice other skills such as teamwork.

“Taking part in this unit really helped the students develop their group skills by participating together in the lab activities and the decision making. It was helpful that they had to defend their positions to try to convince other perspectives to change their minds and go with a different choice. This was a good activity to see that there's no ‘perfect’ choice. They had not been exposed to engineering and learning the processes of engineering was a good exercise.” – Minnesota teacher, grade 4

“It encouraged group work and cooperation across the grade levels. They learned the engineering process by DOING it. [It] created excitement for science and exploration inquiry.” – North Carolina teacher, grade 3

“The collaboration needed to complete the unit activities was a huge plus. The kids enjoyed it, were engaged and were able to apply reading and math knowledge and math to the process.” – Massachusetts teacher, grade 3

“I love the group work part of the process. They had to change their thinking when we got to the improve part.” – Massachusetts teacher, grade 5

A few teachers also specifically mentioned that the unit provided good hands-on experiences (11%, n=5) and allowed students to make real-life connections (9%, n=4).

“My students need a variety of background information built up due to lack of life experiences (for the most part). This offered them solid hands-on learning!” – North Carolina teacher, grade 4

“They were able to transfer their knowledge of rocks on our class trip to Raleigh. Their reflections told of metamorphic stairs and rock/mineral samples they saw in the Science Museum.” – North Carolina teacher, grade 3

Teachers were also asked, on a scale of 1-7, “What is the likelihood that you will choose to teach this unit again in your classroom?” All 47 teachers provided a rating, with the average score being 5.9 (SD=1.49) out of a possible 7 points. The teachers were asked to explain their rating. Thirty-six teachers responded to this question: the most common reasons for choosing to teach the unit again included that students were engaged and motivated (25%, n=9) and that the unit fit well with the standards (17%, n=6).

“I think it fit really well with our rocks and minerals unit and the kids got really excited about applying this knowledge. We are also a STEM school, so the engineering piece is vital.” – Minnesota teacher, grade 4

“Rocks and Minerals is an assigned unit in [the school district’s] Grade 3 science curriculum. The kids really enjoyed this unit!” – Massachusetts teacher, grade 3

“Students were highly motivated to work with the materials and also learned a lot about the rock types, cycle etc.” – Minnesota teacher, mixed grade class

Only three teachers (8%) commented that they would not teach the unit again. Of these three teachers, one was not teaching the same grade the following year, one stated that the unit did not fit with state standards, and the third mentioned that they would not have the materials needed to carry out the unit.

3.2.1.2 Lesson 1 of “Replicating an Artifact” In Lesson 1, students read the storybook Galya and Natasha’s Rocky Adventure. In the story, twin sisters from St. Petersburg, Russia spend the summer at an archaeological dig site where their mother is working. With the help of their mother’s friend Mila, a materials engineer, the girls learn about the work of materials engineers and are inspired to create replicas of the petroglyphs that their mother studies. Through the story, students learn about science concepts including the rock cycle and the processes that create igneous, sedimentary, and metamorphic rocks. They also learn about the work of materials engineers.

All 47 teachers reported teaching this lesson in their classroom. As shown in Table 36, teachers spent an average of 109 minutes teaching this lesson. When asked to rate the quality of the lesson on a scale of 1-7, teachers gave Lesson 1 an average score of 5.9.

Table 36. Replicating an Artifact Feedback – Lesson 1 Question N Mean SD Min Max


Table 37. RA – Categories of Teacher Responses to the Open-Ended Question, “Please explain your rating of Lesson 1”



(N=39)

Positive

Students made connections to the real world 5 12.8 Students had fun, were engaged and were motivated 15 38.5 Activities and / or supporting materials of high quality 5 12.8 Good foundation and/or preparation for future lessons 10 25.6 Other positive comments 12 30.8 Total Positive Comments* 26 66.7

Negative

Students were not engaged/activities too easy or boring 4 10.3 Time constraints/takes too much time 7 17.9 Other negative comments 0 0.0 Total Negative Comments 11 28.2


Thirty-nine of the 47 teachers who taught Lesson 1 provided an explanation for their rating of lesson quality. Table 37 shows coding categories used to code this question. Overall, teachers responded positively to the unit with two-thirds (67%, n=26) providing at least one positive comment. Eleven teachers (28%) gave a negative response.

As shown in Table 37, the most common criticism focused on time constraints. Although teachers found the storybook to be beneficial, nearly one-fifth of teachers (18%, n=7) mentioned that it was too long.

“The story helps get the concept going but takes too many lessons.” – Minnesota teacher, mixed grade class

“I think the reading is beneficial, but I just think that the stories take too long. I do like having them go through the engineering design process with the story.” – Minnesota teacher, grade 4

“Although the story is interesting, it takes several days to read. Because I see students twice a week, details had to be reviewed.” – North Carolina teacher, grade 4

Additionally, a few teachers (10%, n=4) reported that their students lost interest in the storybook, again indicating that this may be due to the length of the book.

“The story was great because it provided a context but it dragged and the kids lost interest.” – Massachusetts teacher, grade 3

“The book was very lengthy and many 3rd graders lost interest.” – Minnesota teacher, grade 3

However, many teachers (39%, n=15) stated that their students enjoyed the lesson and were engaged. “The story was engaging. The students enjoyed learning about Russia, and archeology.” – North Carolina teacher, grade 4

“Students enjoyed the story and it got them interested and curious about engineering. I loved the way it included subject matter about rocks that we had previously studied.” – Massachusetts teacher, grade 3

“The book is wonderful - students enjoyed it and were able to understand what a materials engineer would do. In addition they made connections to the rock cycle and content already learned in science class.” – Massachusetts teacher, grade 4

“The students enjoyed the story and finding where it took place on maps/globes.” – North Carolina teacher, grade 3

About one-quarter of teachers (26%, n=10) mentioned that the activities in this lesson provided a good foundation for future lessons.

“The kids liked the story. I liked how it was tied into the story (or vice versa). It was a great introduction to EDP.” – Massachusetts teacher, grade 5

“My kids just took off and so did I! It's a nice addition to FOSS. Books are a great way to begin!” – Minnesota teacher, mixed grade class

“Great introduction to the unit.” – Colorado teacher, grade 5

A smaller number of teachers also praised the quality of the activities and supporting materials (13%, n=5) and described how their students were able to make connections to the real world (13%, n=5).

“I think the students learned what we hoped from this kit. I like the guided reading questions!” – North Carolina teacher, grade 4

“I cook a lot and we discussed how every culture has a native food wrapped in a leaf. The list goes on. So much to talk about.” – Massachusetts teacher, grade 2 & 3

“It was a great review of the rock cycle and it helped students to ponder all of the ways that rocks can be used in their daily lives.” – California teacher, grade 4

3.2.1.3 Lesson 2 of “Replicating an Artifact” In Lesson 2, students are introduced to the field of materials engineering by examining and testing two products (twill and terry fabrics) that were created from the same raw material. Students use hand lenses to examine the structures of the fabrics, to learn more about how they were made, and to predict some properties of the materials. Working in groups, students perform controlled experiments to study the absorbency and durability of the materials. Based on the results of this testing, students decide which material is the best choice for designing a pair of pants and a bath towel.

Forty-five of the 47 teachers reported teaching Lesson 2. As shown in Table 38, teachers spent an average of 92 minutes teaching the lesson and gave it an overall rating of 6.3 on a scale of 1-7.


How much time did you spend teaching this lesson? (Minutes) 45 92.0 30.79 40 180 How would you rate the quality of this lesson, overall? (Scale: 1-7) 44 6.3 0.76 5 7 Thirty-six teachers provided an explanation for their rating of lesson quality. Table 39 shows coding categories used to code this question. In general, teachers were positive about the unit with nearly all of responding teachers (89%, n=32) providing at least one positive comment and only a few (14%, n=5) giving a negative comment.

As shown in Table 39, approximately one-third of responding teachers (36%, n=13) felt that their students found Lesson 2 to be fun, engaging, and motivating.

“[The lesson] created a motivation and curiosity for learning!” – Massachusetts teacher, grade 3

“Students were engaged. They were really into detailed diagrams, etc.” – North Carolina teacher, grade 4

“They enjoyed doing the experiments. They took them seriously” – Massachusetts teacher, grade 3

Table 39. RA – Categories of Teacher Responses to the Open-Ended Question, “Please explain your rating of Lesson 2”



(N=36)

Positive

Lesson made STEM cross-disciplinary connections 3 8.3 Students had opportunities to learn/apply STEM content and/or skills 5 13.9 Students participated in hands-on activities and experiments 10 27.8 Students were not engaged/activities too easy or boring 13 36.1 Activities and / or supporting materials of high quality 7 19.4 Total Positive Comments* 32 88.9

Negative Too difficult or confusing for students and/or teacher 3 8.3 Other negative comments 2 5.6 Total Negative Comments 5 13.9


About one-fifth of teachers (19%, n=7) praised the quality of the activities and supporting materials, while more than one-quarter of teachers (28%, n=10) appreciated the hands-on nature of the activities.

“This was a super lesson; kids got a lot from it.” – California teacher, grade 5

“The students were able to follow the directions well. Steps/expectations were clear.” – North Carolina teacher, grade 3

“Very hands on. Students were engaged. Nice way to explore the materials.” – Minnesota teacher, grade 3

“Students were able to grasp the concepts and liked the hands-on exploration. Easy to do and not too messy!” – Minnesota teacher, mixed grade class

Additionally, a few teachers mentioned that this lesson provided their students with opportunities to enhance their STEM knowledge and skills (14%, n=5) and make connections to STEM subjects (8%, n=3).

“I taught the FOSS fabric unit at another grade level and this was very complementary…students had great background knowledge to grow with.” – Minnesota teacher, grade 4

“It was enjoyable. I was able to introduce the concept of a controlled experiment.”– Massachusetts teacher, grade 2

“This lesson really helped stretch the thought process of all students regardless of their learning level. The hands-on tests they did made it so they could understand all the work that is put into forming things with layers and especially rocks.” – Colorado teacher, grade 3

The main criticism of this lesson was that it was too difficult or confusing; however, this was mentioned by less than 10% of teachers.

“Some lower level students struggled with finding the connections.” – North Carolina teacher, grade 4

“This lesson was suitable for 2nd/3rd grade, but the concepts were based more on upper elementary comprehension. I think they got the raw material, process, structure, property concept - but I'm not sure that they will retain it.” – Massachusetts teacher, grades 2 & 3

“Students had some difficulty associating this lesson with the theme of artifacts, but enjoyed the activities.” –North Carolina teacher, grade 4

3.2.1.4 Lesson 3 of “Replicating an Artifact” In Lesson 3, students are introduced to their design challenge: designing replicas of a petroglyph. They use hands-on exploration to examine the properties of different materials available to them for their replica designs in order to determine how well each material meets the specific criteria for the design challenge. Working in groups, students examine whether each material is natural or human-made, how it was formed, its hardness, and its durability. Based on their test results, students begin to think about which materials would be good choices for their replica designs.

Forty-five of the 47 teachers reported that they taught Lesson 3. Teachers spent an average of 149.6 minutes teaching this lesson and gave it an average overall rating of 6.3 on a scale of 1-7.



Table 41. RA – Categories of Teacher Responses for the Question,



of teachers


(N=39)

Positive

Students practiced discussion, communication, and teamwork skills 3 7.7 Lesson made STEM cross-disciplinary connections 3 7.7 Students participated in hands-on activities and experiments 14 36.0 Students had fun; were motivated and engaged 18 46.2 Activities and / or supporting materials of high quality 7 17.9 Other positive comments 5 12.8 Total Positive Comments* 31 79.5

Negative

Time constraints/takes too much time 5 12.8 Criticism of supporting materials/difficult to implement 4 10.3 Other negative comments 3 7.7 Total Negative Comments* 9 23.1


Thirty-nine teachers commented on their rating of lesson quality. The coding categories used to code this question are shown in Table 41. In general, teachers were positive about the unit with more than three-quarters (80%, n=31) providing at least one positive comment, while only about one-quarter (23%, n=9) gave at least one negative response.

As shown in Table 41, The two main criticisms of the lesson focused on the lengthy amount of time it took to teach or prepare for the lesson (10%, n=4) and issues that made the lesson difficult to implement (7%, n=3).

“Not as good as the first two lessons, but maybe because everything was taking longer and I was worried about getting everything in.” – Minnesota teacher, grade 5

“I see the merit in this lesson, it was just time consuming in prep time and the durability test was difficult for students.” – New Hampshire teacher, grade 5

“Really valuable, wonderful lesson—again the materials and preparation were very difficult, however.” –Massachusetts teacher, grade 5

“It was engaging for the students but it got long for them. Perhaps now that I have taught it though, it will go faster next time!” – Minnesota teacher, grade 4

However, nearly half of the teachers (46%, n=18) mentioned that their students enjoyed the lesson and were engaged and motivated.

“Engaging and thought provoking.” – Massachusetts teacher, grade 3

“Students were focused on what they were doing and took great care to follow directions, yet they had fun and made great observations.” – Minnesota teacher, grade 5

“Students enjoyed the ‘real memo’ from the museum. They also liked completing the tests to complete their chart.” – North Carolina teacher, grade 4

Additionally, slightly more than one-third of the teachers (36%, n=14) commented positively on the hands-on nature of the activities. A number of teachers (18%, n=7) also praised the quality of the activities and supporting materials.

“The students were excited to be allowed to use a nail and carve in a stone. It was a big deal to be actually handling all of the fabulous samples.” – Massachusetts teacher, grades 2 & 3

“[The] class loved being able to hold many different rocks that they normally don't see, touch, [and] certainly don't scratch!” – Colorado teacher, grade 3

“I liked how they had a design challenge from a ‘museum scientist’ and how they had to consider various properties of their materials before getting to create a petroglyph.” – California teacher, grade 4

“[The students] loved testing materials. [It was] so beneficial to match description cards to materials. – North Carolina teacher, grade 4

A few teachers also discussed how the lesson allowed their students to practice discussion and communication skills (8%, n=3) and make connections to STEM subjects (8%, n=3).

“The kids were so excited to get their hands on the materials. They were already predicting which ones they would use. Their conversations were very scientific. They had solid reasons with lots of background knowledge.” – Minnesota teacher, grade 4

“This was the favorite lesson of my class. Students really go to use their knowledge of the rock cycle and property testing of (materials) in this lesson. I enjoyed this one also.” – Massachusetts teacher, grade 3

“This lesson really made my students connect all of the learning.” – Minnesota teacher, grade 4

3.2.1.5 Lesson 4 of “Replicating an Artifact” In the final lesson, Lesson 4, students apply what they learned in previous lessons to the design of their replicas. Using the engineering design process, students create, evaluate, and improve their replicas based on the established design challenge criteria. Finally, students explain their replica designs in a letter written to the museum commissioning the replicas.

Forty-four of the 47 teachers reported that they taught this lesson in their classroom. On average, teachers spent 165.7 minutes teaching this lesson and gave it an overall rating of 6.2 on a scale of 1-7.



Table 43. RA – Categories of Teacher Responses to the Open-Ended Question,




(N=36)

Positive

Students practiced problem solving and critical thinking skills 4 11.1 Students had fun; were motivated and engaged 15 41.7 Students were challenged 4 11.1 Activities and / or supporting materials of high quality 9 25.0 Other positive comments 10 27.8 Total Positive Comments* 30 83.3

Negative

Too difficult or confusing for students and/or teacher 4 11.1 Criticism of supporting materials/difficult to implement 3 8.3 Other negative comments 7 19.4 Total Negative Comments* 12 33.3


Thirty-six teachers commented on their rating of lesson quality. Table 43 shows coding categories used to code this question. Most teachers responded positively about the lesson: 83% (n=30) provided at least one positive comment. One-third of teachers (33%, n=12) commented negatively.

As shown in Table 43, negative comments most often mentioned that the lesson was too difficult or confusing (11%, n=4) or criticized the supporting materials (8%, n=3).

“I wished we had more success with durability scoring. Many students' rubbings were showing no lines.” – Minnesota Teacher, grade 3

“A little too much for this age level. It is really difficult to find the perfect balance in a unit that covers so much material to make it adaptable in such a wide range of learning activities.” – Massachusetts teacher, grades 2 & 3

“Students really need more guidance to make choices for next time. I let them make the choices–some chose wax and foam.” – California teacher, grade 5

“I wish there were answer keys for the teacher.” – California teacher, grade 5

However, nearly half of teachers (42%, n=15) mentioned that the lesson was fun, engaging, and motivating. One-quarter (25%, n=9) also praised the quality of the activities and supporting materials.

“Kids were engaged and they had to work as a team to complete this project” – Minnesota teacher, grade 4

“Kids loved carving and saw how [the lessons] all tied together.” – Massachusetts Teacher, grade 3

“Students enjoyed this activity. Everyone remained engaged. The information they gathered was very helpful.” – Minnesota teacher, grade 4

“Kids enjoyed testing and having a chance to improve their choices.” – North Carolina teacher, grade 4

“The whole unit was a great experience for the kids.” – Massachusetts teacher, grade 5

Additionally, a few teachers mentioned that the lesson positively challenge their students (11%, n=4) and allowed them to practice problem solving and critical thinking skills (11%, n=4).

“Students’ thinking and rationale were superior. Engagement into a topic that previously didn't exist.” – North Carolina teacher, grade 4

“I loved that we discussed ‘why’ the petrolyglyphs wouldn't work according to the ‘letter’ I received. They had to re-evaluate their thinking and it was very helpful when teaching about the Scientific Method. –California Teacher, grade 5

“Challenging, but a high standard lesson.” – New Hampshire teacher, grade 5

“Good creative thinking challenges, communication, and collaboration.” – Massachusetts teacher, grade 3

3.2.1.6 Summary: “Replicating an Artifact” Formative Evaluation Overall, teachers found this unit to be fun, engaging and motivating for their students. They noted that their students had opportunities to learn about both science and engineering and to connect that knowledge to the real world. Teachers also mentioned that the unit provided their students with the chance to improve their communication, teamwork, problem solving, and critical thinking skills. Teachers also praised the quality of the lessons and supporting materials and appreciated the hands-on nature of the unit. In addition to integrating this unit with science, teachers mentioned that they also used this unit to further objectives in language arts, social studies and math. Despite the fact that some teachers found some of the lessons to be time consuming or difficult to implement, the majority provided positive comments and reported that they were highly likely teach the unit again.

3.2.2 Summative Evaluation: Replicating an Artifact 3.2.2.1 Assessment Design: “Replicating an Artifact” The Replicating an Artifact student assessment was designed and first tested during the 2008-2009 school year. Twenty multiple-choice questions and eight True/False questions were chosen for the pilot assessment. In the winter of 2008, validity evidence was gathered for all questions from the Replicating an Artifact assessment. EiE staff conducted cognitive interviews with children in grades 3-5 from three Massachusetts classrooms, asking students to read each question aloud, explain what they thought the answer would be, read all answer choices aloud, and explain which answer they would choose and why.

During the summer of 2009, a number of questions were dropped or revised based on the results of the pilot study and validity testing to create a new field test assessment. In 2009-2010 it was used for field testing in six states (California, Colorado, Massachusetts, Minnesota, New Hampshire, and North Carolina). The field test version included sixteen multiple-choice questions.

On the identical pre- and post-assessments students were asked science questions about different types of rocks, including their properties and how they are formed. They were also asked questions about the properties of materials. Table 44 describes the text for the questions with the correct answer shown in brackets.

Table 44. Replicating an Artifact Assessment Questions (Text) Q # Scale Question Text

1 Rocks New rocks are formed when: [all of the above: hot lava inside the Earth cools, rocks are squeezed deep underground, sand is squeezed together under deep water]

2 Rocks The main difference between sedimentary, metamorphic, and igneous rocks is: [how they are formed]

3 Properties of Materials

A boy has two rocks. Both are the same kind of rock. What property of his two rocks is MOST likely to be the same? [color]

4 Rocks One step in how many sedimentary rocks form is: [sand is squeezed together at the bottom of a lake]

5 Rocks A student wrote down some observations of four rocks samples she was studying. Based on her observations, which is MOST likely to be a sedimentary rock? [has many very small grains of sand in different layers] (see Figure 12)

6 Rocks A student has two blacks. He wants to know if they are both made of the same material. What information about the two blocks would help him to decide? [both blocks float in water]

7 Rocks Could a metamorphic rock become an igneous rock? [yes, by melting and then cooling]


A team of materials engineers invented a new material that can get very hot without melting. They are using this material in the Space Shuttle. How else might this material be used? [both A and B are possible: to design new baking pans, to design a new kind of oven]

9 Rocks Which type of rock is formed when hot lava cools? [igneous]


A materials engineer has been asked to help design a new running shoe. What might she do to design the shoe? [all of the above: ask a running coach what properties the running shoe should have, look at other running shoes to see what materials they are made of, design materials that have properties that will be important for the shoe]


Which of the following is a property of a brick? [all of the above: rough, made out of clay, can be used to make walls]

12 Rocks How could you test the hardness of a mineral? [scrape it with another mineral]


What information would BEST help someone to identify a kind of rock? [what the rock is made of]


Look at the pictures of four kinds of cloth viewed close-up. Which cloth would be BEST for making towels that soak up lots of water? [cloth C] (see Figure 13)


Look at the pictures of four kinds of cloth viewed close-up. Which cloth would be BEST to quickly separate a mixture of peas and water? [cloth D] (see Figure 13)


Look at the pictures of four kinds of cloth viewed close-up. Could the different kinds of cloth be made of the same raw material? [yes, they could all be made of the same raw material] (see Figure 13)

3Ta

IthedoPa

Dsahr

Figure 12.

3.2.2.2 The reliabiliassessment.

Initially, we that were on have a Cronbextracted foudivided rougof materials. Properties ofand .683, res

Due to the ssample. The assessment. have a Cronrotation extr

. Replicating

Scale Coity sample

constructedthe final fie

bach’s α of .ur compone

ghly into twoBased on th

f Materials (spectively.

small reliabisample con

Students’ ponbach’s alphracted four c

g an Artifact

onstruction:consisted o

d a raw scoreld test asses883 (N = 77

ents with eio componenthe results of PM) scale (s

ility sample nsisted of 13ost-assessmea of .734 (ncomponents

t Question 5

“Replicatinof 83 studen

e which wassment. This

7). Principal genvalues >ts about rockthe PCA, wesee Table 44

size, we als399 studentsent answers (n=1399). Pri

with eigenv

ng an Artifacnts in four

s the sum ofscore was teComponent

> 1.0, explaks and minere decided to 4), which we

so tested thes in grades 3(PostAll) weincipal Comvalues > 1.0

Figure 13Q

ct” pilot classr

f all correct ested for inte Analysis (P

aining 61.6%rals and two form two su

ere found to

e reliability 3-5 who hadere tested for

mponent Ana0, explaining

3. ReplicatingQuestions 14

rooms who

answers on ernal reliabil

PCA) with di% of the vacomponents

ubscales: thehave a Cron

of these scad returned br internal rel

alysis (PCA)g 42.1% of

g an Artifac4-16

had compl

all sixteen qlity and wasirect obliminariance. Thes about the pe Rocks scalnbach’s alph

ales on our both a pre- aliability and) with directf the varianc

t

leted the

questions found to n rotation ese items properties le and the ha of .895

field test and post-

d found to t oblimin ce. These

items divided roughly into one component about rocks and minerals and three components about the properties of materials. The two subscales devised earlier were also re-tested for internal reliability and found to have Cronbach’s α=.641 (PostRocks, n=1353) and α=.568 (PostPM, n=1334), respectively.

3.2.2.3 Sample: “Replicating an Artifact” Replicating an Artifact (RA) assessments were collected from students in grades 2 through 5. The grade 2 students received a different version of the assessment than grades 3-5; results of the grade 2 assessment are not described or analyzed in this report.

The full sample of data collected from grades 3-5 included 1589 students in 70 classrooms. However, 12.0% of the sample (n=190) was excluded because they were missing a pre-assessment or a post-assessment. Additionally, four classes were dropped due to missing demographic information. Twenty classes were missing information on which students received national free or reduced-price lunch, so that variable was not considered in the analysis of this unit. Fifty-seven students answered fewer than 13 questions on either the pre-assessment or the post-assessment and were dropped from the dataset. One additional classroom was dropped from the analysis because the teacher did not report the amount of time spent teaching the unit.

The final dataset used for analysis included 1240 students (Level-1 units) in 65 classrooms (Level-2 units), with an average classroom (cluster) size of 22.8 students per classroom, and a standard deviation of 4.38 students. The classrooms were spread over 38 schools with 45 teachers total. Two of the teachers were science specialists. The teachers had an average of 12.9 years of experience, with a standard deviation of 8.74 years, a minimum of 1 year and a maximum of 40.

The majority of the sample consisted of students in grade 4 (see Table 45). There were an equal proportion of males and females. Nine percent of the sample consisted of students with limited English proficiency (LEP); students in Individualized Education Programs (IEP) made up 11% of the sample (see Table 46). White students made up nearly two-thirds of the sample (64%) (see Table 47).

Table 45. RA Grade Distribution (N of Students and Classrooms)

Grade 3 4 5 Total

# N (students) 232 606 402 1240 #N (classrooms) 12 33 20 65

Table 46. RA Proportions for Level-1 Dichotomous Variables

Gender (male) LEP IEP Proportion .50 .09 .11 N 621 114 138

Table 47. RA Proportions for Level-1 Variables – Race

Black Asian Hispanic White Other Total Proportion .15 .06 .11 .64 .11 1.00 N 186 77 139 792 46 1240

3.2.2.4 Results: “Replicating an Artifact” Two outcome variables, one science and one engineering, were used for analysis. The Rocks scale (PostRocks) was used to gauge students’ improvements in understanding of science topic of rocks and

minerals. The Properties of Materials scale (PostPM), which measures engineering content, was used to determine whether students increased their understanding of material properties.

For this unit, we tested the student level (level-1) variables of IEP and LEP status, as well as gender and race (Asian, Black, Hispanic, and Other). We also included pre-assessment scores as a covariate. Next, we tested whether each level-1 variable was best modeled as a fixed or random coefficient, the default being a fixed coefficient, unless the random variance was found to be significant (p < .05). At the classroom level (level-2) we tested the effects of classroom-level variables on the outcome, including the classroom means on the pre-assessment scores, class size, the number of minutes spent teaching the EiE unit, the number of EiE units the teacher had taught prior to that school year, the number of units the teacher taught during that school year, whether or not the teacher taught Lesson 3 and Lesson 4 of the unit, and the number of years the teacher had been teaching overall as covariates. We also tested the effect of grade and of teacher specialty (SciencebgSpecialist) here, as well as the effect of the overall classroom setting (the proportions of IEP students, LEP students, and boys in the classroom).

RA: Properties of Materials Scale Results. To test engineering outcomes, the Properties of Materials scale (PostPM) was used. Scores were computed by adding together the number of correct answers for each of the engineering questions listed in Table 44. The PM scale has a possible range of 0 to 8. Mean pre- and post-assessment scores for the various demographic groups tested on the PM scale are shown in Table 48 below.

Table 48. RA Descriptive Statistics: Pre- and PostPM Scores by Demographic Groups

RA PrePropertiesMaterials Score RA PostPropertiesMaterials Score Mean SD Min Max Mean SD Min Max

Overall 4.06 1.83 0 8 5.21 1.90 0 8 Boys 4.05 1.82 0 8 5.12 1.94 0 8 Girls 4.07 1.85 0 8 5.29 1.86 0 8 IEP 3.62 1.85 0 8 4.57 1.99 1 8 LEP 3.38 1.61 0 7 4.24 1.82 0 8 White 4.36 1.80 0 8 5.59 1.78 0 8 Hispanic 3.50 1.72 0 8 4.47 1.85 0 8 Black 3.27 1.68 0 8 4.08 1.89 0 8 Asian 4.10 1.77 1 8 5.27 1.94 1 8 Other_race 3.65 2.06 0 8 5.33 1.61 1 8 Grade 3 3.53 1.70 0 8 5.16 1.94 0 8 Grade 4 3.86 1.79 0 8 4.92 1.91 0 8 Grade 5 4.66 1.81 0 8 5.67 1.76 1 8

The two-level final conditional model for the PM scale (see Table 49 and Figure 14) includes all of the variables and random variance coefficients which were found to be significant during the variable testing process described above. It shows that the baseline improvement between pre- and post-assessment was 1.20 points (CI 1.01, 1.39; Intercept γ00, p < .001), meaning that students gained a better understanding of what an optical engineer does. However, English language learners (LEP), students with IEP’s, and Black students improved to a lesser extent (see Figure 15: LEP 0.60 points improvement; CI 0.12, 1.08; IEP 0.61 points; CI 0.22, 1.00; Black 0.52 points; CI 0.16, 0.88). Classrooms with instructors teaching more units of EiE, either in the current year or in past years, tended to show even more improvement, while larger classrooms exhibited less improvement. Our two-level model explains 77% (0.77=1-(0.230/1.011)) of the between-class variance.

Level-1 Model: 0 1 2 43PostPM (LEP) (IEP) ( ) (Black) r= β +β +β +β +β +PrePM

2Var(r) σ=

And:

2

0 1 2 3 4

5 6

ln (CurrentUnits) (NotL3L4) (MA) ( )(ScienceSpecialist) ( )

PrePMClassMeanσ α α α α αα α

= + + + ++ + PrePM

Level-2 Model:

0 00 01 02 03

04 0

( ) (PriorUnits) (CurrentUnits)( ) u

ClassSizePrePMClassMean

β = γ + γ + γ + γ+γ +

1 10β γ= 2 20 2uβ γ= + 3 30β γ= 44 0β γ= Bold indicates group mean centered. Italicized indicates grand-mean centered. The outcome variable PostPM was centered around the PrePM mean.

Figure 14. Replicating an Artifact PostPM Score – Conditional Model

Table 49. RA PostPM Score Conditional Model – Final Estimation of Fixed Effects (with robust standard errors)


df P-value

For Intercept 1, β0 Intercept, γ00 1.196 0.095 12.535 60 <0.001 ClassSize, γ01 −0.033 0.015 −2.144 60 0.036 PriorUnits, γ02 0.041 0.017 2.369 60 0.021 CurrentUnits, γ03 0.082 0.030 2.690 60 0.010 PrePMClassMean, γ04 0.765 0.081 9.499 60 <0.001 For LEP slope, β1 Intercept, γ10 −0.596 0.222 −2.685 1231 0.008 For IEP slope, β2 Intercept, γ20 −0.586 0.168 −3.496 64 0.001 For PrePM slope, β3 Intercept, γ30 0.379 0.028 13.609 1231 <0.001 For Black slope, β4 Intercept, γ40 −0.677 0.153 −4.411 1231 <0.001

The level-1 (within-class) variance σ2 was found to be heterogeneous (p<.001) and was modeled logarithmically (see Table 51 and Figure 14). The within-class variance was smaller among students whose teacher has more experience teaching EiE in the current year (CurrentUnits), students in a class where the teacher did not teach Lesson 3 or Lesson 4 (NotL3L4), students in Massachusetts (MA), students with high pre-assessment scores (PrePM), and students in classes with high pre-assessment

means (PrePMClassMean). The within-class variance was larger among students taught by Science Specialists. Once modeled, σ2 became less significantly heterogeneous (p=.040), suggesting that our model accurately describes some, but not all of the sources of within-class variance.

Table 50. RA PostPM Score Conditional Model – Final Estimation of Variance Components

Intercept 1, u0 0.480 0.230 46 150.084 <0.001 IEP slope, u2 0.648 0.420 50 74.869 0.013 Level-1, r 1.446 2.090

Table 51. RA PostPM Score – Model for Level-1 Variance

Intercept 1, α0 1.551 0.212 7.304 <0.001 CurrentUnits, α1 −0.094 0.029 −3.283 0.001 NotL3L4, α2 −0.653 0.196 −3.354 0.001 MA ( Massachusetts), α3 −0.234 0.101 −2.313 0.021 PrePMClassMean, α4 −0.169 0.053 −3.186 0.002 ScienceSpecialist, α5 0.599 0.244 2.458 0.014 PrePM, α6 −0.103 0.026 −3.941 <0.001

Table 52. RA PostPM Score Unconditional Model – Final Estimation of Variance Components


Figure 15. RA Change in Student Score: PM Scale

Student Change in Score: RA PM scale with 95% confidence intervals

LEP IEPBlack

ReferenceGroup

0

0.5

1

1.5

Change in Score (Pre to Post)



Parameter Coefficient Standard Error Z-ratio P-value



RA: Rocks Scale Results. The science outcome for the Replicating an Artifact assessment was the Rocks scale. This scale had a possible range of 0 to 8. Mean pre- and post-assessment scores for the various demographic groups tested on the Rocks scale are shown in Table 53 below.

Table 53. RA Descriptive Statistics: Pre- and PostRocks Scores by Demographic Groups

RA PreRocks Score RA PostRocks Score Mean SD Min Max Mean SD Min Max


The two-level final conditional model for the Rocks scale (see Table 54 and Figure 16) includes all of the variables and random variance coefficients which were found to be significant during the variable testing process described above. It shows that the baseline improvement between pre- and post-assessment was 2.84 (CI 2.60, 3.08) out of 8 points (Intercept γ00, p < .001), meaning that students were better able to answer questions relating to rocks and minerals on the post-assessment. Although LEP (2.11 points improvement; CI (1.64, 2.58) students, IEP (1.93 points; CI 1.52, 2.34) students, and Black (2.22 points; CI 1.84, 2.6) students also improved (confidence intervals do not include zero), their scores increased to a lesser extent than other students’ (see Figure 17). Classrooms with instructors teaching more units of EiE in the current year (CurrentUnits) tended to show even more improvement, while classrooms taught by a Science Specialist tended to improve to a lesser extent. Our two-level model explains 49% (0.49=1-(0.637/1.246)) of the between-class variance.

The level-1 (within-class) variance σ2 was found to be heterogeneous (p<.001) and was modeled logarithmically (see Table 56 and Figure 16). The within-class variance was smaller among grade 5 students (Grade5), students in Massachusetts (MA) and Minnesota (MN), students taught by a science specialist (ScienceSpecialist), English language learners (LEP), students in classrooms where the teacher had taught many EiE units in the past (PriorUnits), and students from classes with high pre-assessment mean scores (PreRocksClassMean). The within-class variance was larger among students in New Hampshire (NH), students in classrooms with a high proportion of English language learners (ProportionLEP) and students with an IEP (IEP). Two minutes variables were also found to be significant in this model (TotalMinutes and TotalMinutesSquared), with the result being that the within-class variance was smaller among students in a class where the teacher taught the unit for a long period of time. Once modeled, σ2 became homogeneous (p=.450), suggesting that our model accurately describes important sources of within-class variance.

Level-1 Model: 0 1 2 43PostRocks (LEP) (IEP) ( ) (Black) r= β +β +β +β +β +PreRocks

2Var(r) σ=

And:

20 1 2 3 4

6 7 8

9 10 1

5

1 12

ln (PriorUnits) (Grade5) (MA) (MN) (NH)( ) (ScienceSpecialist) ( )( ) (LEP) (IEP) ( )ProportionLEP TotalMinutesTotalMinutesSquared PreRocksClassMean

σ α α α α α αα α αα α α α

= + + + + ++ + ++ + + +

Level-2 Model: 0 00 01 02 03 0(CurrentUnits) ( ) (ScienceSpecialist) uPreRocksClassMeanβ = γ + γ + γ + γ +

1 10 1uβ γ= + 2 20β γ= 3 30 3uβ γ= + 44 0β γ= Bold indicates group mean centered. Italicized indicates grand-mean centered. The outcome variable PostRocks was centered around the PreRocks mean.

Figure 16. Replicating an Artifact PostRocks Score – Conditional Model

Table 54. RA PostRocks Score Conditional Model – Final Estimation of Fixed Effects (with robust standard errors)


df P-value

For Intercept 1, β0 Intercept, γ00 2.839 0.120 23.598 61 <0.001 CurrentUnits, γ01 0.156 0.030 5.166 61 <0.001 PreRocksClassMean, γ02 0.513 0.069 7.418 61 <0.001 ScienceSpecialist, γ03 −1.486 0.250 −5.937 61 <0.001 For LEP slope, β1 Intercept, γ10 −0.731 0.203 −3.608 64 0.001 For IEP slope, β2 Intercept, γ20 −0.907 0.166 −5.472 1232 <0.001 For PreRocks slope, β3 Intercept, γ30 0.276 0.028 9.733 64 <0.001 For Black slope, β4 Intercept, γ40 −0.619 0.145 −4.266 1232 <0.001

Table 55. RA PostRocks Score Conditional Model – Final Estimation of Variance Components



Intercept 1, u0 0.798 0.637 26 130.543 <0.001 LEP slope, u1 0.854 0.729 29 53.690 0.004 PreRocks slope, u3 0.138 0.0192 29 47.908 0.015 Level-1, r 1.427 2.037

Table 56. RA PostRocks Score – Model for Level-1 Variance

Parameter Coefficient Standard Error Z-ratio P-value Intercept 1, α0 0.948 0.087 10.936 <0.001 PriorUnits, α1 −0.068 0.022 −3.077 0.003 Grade 5, α2 −0.576 0.103 −5.610 <0.001 MA (Massachusetts), α3 −1.038 0.127 −8.194 <0.001 MN (Minnesota), α4 −0.464 0.112 −4.151 <0.001 NH (New Hampshire), α5 1.301 0.328 4.273 <0.001 ProportionLEP, α6 1.189 0.406 2.928 0.004 ScienceSpecialist, α7 −1.110 0.155 −7.150 <0.001 TotalMinutes, α8 −0.226 0.053 −4.295 <0.001 TotalMinutesSquared, α9 0.247 0.043 5.705 <0.001 LEP, α10 −0.413 0.187 −2.211 0.027 IEP, α11 0.435 0.138 3.162 0.002 PreRocksClassMean, α12 −0.114 0.029 −3.888 <0.001

Table 57. RA PostRocks Score Unconditional Model – Final Estimation of Variance Components




Figure 17. RA Change in Student Score: Rocks Scale

3.2.2.5 Summary: “Replicating an Artifact” Summative Evaluation Our analysis shows that students who are taught the EiE Replicating an Artifact unit, together with a science unit on rocks, learn a significant amount about engineering and science concepts. While Black students, English language learners and students with an Individualized Education Program had a smaller improvement on the assessment, all demographic groups saw some increase in score from pre- to post-assessment for both the science (Rocks) and engineering (Properties of Materials) scales. The number of units the classroom instructor was teaching during the current school year was also associated with higher scores on both the science and engineering scales.

3.3 Results for the “Designing Submersibles” Unit Evaluation In the EiE unit Taking the Plunge: Designing Submersibles, students apply what they are learning about volume, mass, and density in their science curriculum to the field of ocean engineering. In Lesson 1, students read the storybook Despina Makes a Splash, set in Greece, which introduces them to the field of ocean engineering and reviews science concepts such as density and other ocean-related science facts. Lesson 2 introduces technologies that have been used to probe the depths of the ocean and allows students to explore the topography of a model ocean using a somewhat rudimentary tool called a “sounding pole.” Here students also learn about the more advanced technology of sonar, which has been developed and used by both ocean engineers and scientists to study the ocean. In Lesson 3 students begin using the Engineering Design Process as they “Ask” questions about how and why objects float or sink in water by experimenting with how an object’s mass and volume affect its ability to float in water. In the final lesson, Lesson 4, continue to use the Engineering Design Process as they “Imagine” several submersible designs and work with their team members to choose one to “Plan” and “Create.” Students then test and evaluate their submersibles and “Improve” their designs based on their analyses.

Taking the Plunge: Designing Submersibles was field tested during the 2009-2010 school year.

Student Change in Score: RA Rocks scale with 95% confidence intervals

LEPIEP

Black

ReferenceGroup

0

0.5

1

1.5

2

2.5

3

3.5

Change

in Score (P

re to Po

st)

3.3.1 Formative Evaluation: Designing Submersibles Feedback forms for Designing Submersibles were completed by 47 teachers who taught the unit in their classroom. However, 9 of these feedback forms were not included in the analyses because the unit was taught in Grade 1, which is outside of the range of grade levels this unit was created for. Six states were represented in the sample: California, Colorado, Massachusetts, Minnesota, New Hampshire, and North Carolina. Grades 2 through 6 are represented.

Table 58. Designing Submersibles – Classroom Grade and State Distribution

State Grade 2 Grade 3 Grade 4 Grade 5 Grade 6 Grade Mixed* Total**

CA 4 2 1 3 1 11 CO 4 6 10 MA 11 13 4 28 MN 5 2 3 3 13 NC 1 1

Total 9 7 14 25 3 5 63 * There was 1 mixed grades 4/5 class in CA.

There were 2 mixed grades 1/2 classrooms, 1 mixed grades 2-5 classroom, and 1 mixed grades 5/6 classroom in MA. **The total number of classrooms is larger than the number of teachers responding, because

15 teachers taught more than one class and each class is counted individually. 3.3.1.1 Overall Feedback for “Designing Submersibles” Unit Teachers were asked to rate the unit overall by answering questions using a 7-point Likert scale. Table 59 shows the number of the responses (N), the average response, the standard deviation (SD), and the minimum (Min) and maximum (Max) response to the overall unit, based on a seven point scale, with 1 representing “Not at all” and 7 representing “Very”.

Table 59. Designing Submersibles Feedback – Unit Ratings







Teachers were asked whether this unit furthered their objectives for another content area beyond science, and if so, to specify which content area. In a separate question, teachers were also asked if they integrated the teaching of this unit with their teaching of other subjects, and if so, to explain how. All 38

teachers responded to at least one of these questions and mentioned specific content areas other than science that were enhanced by the unit, with language arts being mentioned most often (50%, n=19). Over a quarter of teachers also reported that the unit furthered their objectives in math (26%, n=10) and almost a fifth reported social studies (16%, n=6).

Teachers were asked a number of open-ended questions about the unit and lessons. Thirty-seven of the 38 teachers responded to the question, “How did your students benefit, academically or otherwise, from taking part in this unit?” The coding categories used to code this question are shown in Table 60. Over half of the respondents (60%, n=22) noted that the unit helped to further their students’ understanding of STEM knowledge and skills. Specifically, teachers noted that their students learned about volume and density, the work of engineers, and the engineering design process.

“The story was a good introduction that maintains a unity for all lessons. Being able to apply to engineering processes with the vials, being able to manipulate, experiment and play scientifically with them was fun, motivating and a learning experience.” – California teacher, grade 2

“My students had a lot of fun while learning. I've taught the science of sink or float (mass volume and density) for several years now. I had often felt that some students were memorizing vocab but not really learning or understanding the material. These lessons gave the students a much better understanding of not only the vocab, but how mass and volume affect density. I really enjoyed circulating and listening to the conversations taking place. They stayed on task.” – Massachusetts teacher, grade 4

“This unit helped the students relate in-class science to real world applications. Kids who have struggled in the past really benefitted from the hands-on trial/error design process.” – Massachusetts teacher, grade 5

Table 60. SB – Categories of Teacher Responses to the Open-Ended Question, “How did your students benefit, academically or otherwise, from taking part in this unit?”



(N=37) Students practiced discussion, communication, and teamwork skills 6 16.2 Students practiced problem solving and critical thinking skills 5 13.5 Students had opportunities to learn/apply STEM content and/or skills 22 59.5 Students participated in hands-on activities and experiments 6 16.2 Students had fun, were engaged and were motivated 14 37.8 Students were challenged 5 13.5 Other comments 7 18.9 Total Number of Comments* 37 100.0


Additionally, over one-third of teachers (38%, n=14) reported their students had fun and were engaged and motivated.

“The story was a good introduction that maintained unity for all lessons. Being able to apply to engineering processes with the vials, being able to manipulate, experiment and play scientifically with them was fun, motivating and a learning experience.” – California teacher, grade 3

“They enjoyed engineering a submersible, they better understood terms of volume, mass and density. Great lesson!” – Massachusetts teacher, grade 5

About one-fifth of teachers (16%, n=6) noted that the unit gave their students the opportunity to practice other skills, such as teamwork, and that it provided good hands-on experiences (16%, n=6).

“They were readily able to apply the engineering design process - going beyond the theory and what they have been taught in class. They were also able to apply the inquiry process further. Socially, they enjoyed working in coop[erative] groups with their peers.” – Minnesota teacher, grade 5

“Students had to work together to accomplish the task. They recognize the importance of note taking and recording and how it can affect testing.” – Colorado teacher, grade 5

“The students amassed a wealth of information by exploring rather than me telling... All in all, they rose to work together, in turn create a design. Many went home and invented more designs to use more objects.” – California teacher, grade 2

Teachers were also asked, on a scale of 1-7, “What is the likelihood that you will choose to teach this unit again in your classroom?” All 38 teachers provided a rating with the average score being 5.9 (SD=1.50) out of a possible 7 points. Twenty-five teachers responded to this question. The most common reason for choosing to teach the unit again was that students were engaged and motivated (20%, n=5).

“Fun, engaging and students learned.” – Colorado teacher, grade 5

“My students were very interested in this unit.” – Massachusetts teacher, grade 4

“Engaged the students very well.” – Minnesota teacher, grade 6

Additionally, one-fifth of the teachers (20%, n=5) noted that they would teach the unit again with modifications.

“I will work out more of a challenge/extension [for] the high achievers.” – California teacher, grade 5

“I will use this unit again but attempt to extend the units with more background knowledge on the various concepts.” – Colorado teacher, grade 5

Only one teacher (4%) responded that she would not teach the unit again because it was not right for the age group (grade 3). 3.3.1.2 Lesson 1 of “Designing Submersibles” In Lesson 1, students read the storybook Despina Makes a Splash. The story introduces the field of ocean engineering and some of the key concepts for this unit, including sinking, floating, and density. The story is set on and around the island of Santorini in Greece, and features a young girl named Despina, who happens to use a wheelchair. The discoveries that she, her brother, and her cousin make in retrieving a sunken pair of goggles lead them down a road that teaches them about the work of ocean engineers and inspires them to design their own submersible.

Thirty-seven out of 38 teachers reported teaching this lesson in their classroom. As shown in Table 61, teachers spent an average of 109 minutes teaching this lesson. When asked to rate the quality of the lesson on a scale of 1-7, teachers gave Lesson 1 an average score of 5.9.

Table 61. Designing Submersibles Feedback – Lesson 1 Question N* Mean SD Min Max

How much time did you spend teaching this lesson? (Minutes) 36 142.5** 169.7** 15 950** How would you rate the quality of this lesson, overall? (Scale: 1-7) 37 5.6 1.02 3 7

*Two teachers reported having a colleague teach this lesson. Their data is included in this analysis. **The Mean and Standard Deviation include 2 outliers: 1 teacher who reported 520 minutes and another who

reported 950. Without these, the Mean is 111.9 minutes and the Standard Deviation is 78.76 minutes.

Table 62. SB – Categories of Teacher Responses to the Open-Ended Question, “Please explain your rating of Lesson 1”



(N=31)

Positive

Students had fun, were engaged and were motivated 11 35.5 Activities and / or supporting materials of high quality 9 29.0 Other positive comments 9 29.0 Total Positive Comments* 21 67.7

Negative Time constraints / takes too much time 6 19.4 Other negative comments 11 35.5 Total Negative Comments* 14 45.2


Thirty-one of the 38 teachers who taught Lesson 1 provided an explanation for their rating of lesson quality. Table 62 shows coding categories used to code this question. Overall, teachers responded positively to the unit with two thirds of teachers (68%, n=21) providing at least one positive comment. Less than half of the responding teachers (45%, n=14) gave a negative response.

As shown in Table 62, the most common criticism focused time constraints. Although teachers found the storybook to be beneficial, nearly one-fifth of teachers (19%, n=6) mentioned that it was too long.

“It was a story that helped to define some of the terms which provided some background knowledge for the unit. Some students felt the book was long.” – Massachusetts teacher, grade 5

“A little long but good.” – Colorado teacher, grade 5

However, many teachers (36%, n=11) stated that their students enjoyed the lesson and were engaged. “I enjoyed the book, and so did the kids.” – California teacher, grades 4 and 5

“The students were really into the story.” – California teacher, grade 5

“This story held the students’ interest. They read to themselves and then we discussed it as a group” – Minnesota teacher, grade 5

“Great fiction story and guiding questions” – Minnesota teacher, grade 5

Additionally, almost one-third of teachers (29%, n=9) praised the quality of the activities and supporting materials.

“Great lesson - gives good background for upcoming lessons.” – Massachusetts teacher, grade 5

“There is always time to work in the lessons throughout the year. It scaffolds in so many areas that need to be taught. I loved it! As I hung before and after charts during open visitations (Area Superintendents/ Principals visits) they were impressed with the quality of student learning posted!” – California teacher, grade 2

3.3.1.3 Lesson 2 of “Designing Submersibles” In Lesson 2, students learn more about the work of ocean engineers and design technologies that help solve problems in ocean environments, including designing instruments that allow scientists to gather information about the ocean. Students use one of these technologies – a sounding pole – to explore and map a model ocean floor. After students use this technology to gather depth data about the model ocean, they are asked to discuss what information they have gathered and what they believe is still missing.

Students are then introduced to sonar technology and observe that sonar is able to provide a much more comprehensive picture of an area than a sounding pole.

Thirty-seven of the 38 teachers reported teaching Lesson 2. As shown in Table 63 teachers spent an average of 92 minutes teaching the lesson and gave it an overall rating of 6.3 on a scale of 1-7.

Table 63. Designing Submersibles Feedback – Lesson 2 Question N Mean SD Min Max

How much time did you spend teaching this lesson? (Minutes) 37 130.5* 132.7* 45 820* How would you rate the quality of this lesson, overall? (Scale: 1-7) 36 5.8 1 4 7 *The Mean and Standard Deviation include 2 outliers: 1 teacher who reported 820 minutes and another who reported

360. Without these, the Mean is 74.9 and the Standard Deviation is 28.11 minutes.

Table 64. SB – Categories of Teacher Responses to the Open-Ended Question, “Please explain your rating of Lesson 2”


of teachers


(N=31)

Positive

Students had opportunities to learn/apply STEM content and/or skills 5 16.1% Students had fun, were engaged and were motivated 13 41.9% Other positive comments 10 22.6% Total Positive Comments* 21 67.7%

Negative Time constraints / takes too much time 8 25.8% Other negative comments 6 19.4% Total Negative Comments* 14 45.2%


Thirty-one teachers provided an explanation for their rating of lesson quality. Table 64 shows coding categories used to code this question. In general, teachers were positive about the unit with about two-thirds of responding teachers (68%, n=21) providing at least one positive comment and under half (45%, n=14) giving a negative comment.

As shown in Table 64, the main criticism of this lesson was that it was too long, a concern that was mentioned by one quarter (26%, n=8) of the teachers.

“All of it was great but it got long as we tested the depths as a class.” – Minnesota teacher, grade 3

“I thought it was too lengthy.” – Massachusetts teacher, grade 4

“This takes a LONG time with 35 students.” – California teacher, grade 5

However, about one-fifth of the responding teachers (42%, n=13) felt that their students found Lesson 2 to be fun, engaging, and motivating.

“It clearly got the idea of oceans and depth across - students had fun while learning. ALL were on task.” – Minnesota teacher, grade 3

“Students were actively engaged throughout [the] lesson.” – Minnesota teacher, grade 5

“The kids and I enjoyed ‘exploring’ the bottom of the ocean.” – California teacher, grade 2

About one-fifth of teachers (16%, n=5) reported that the unit provided opportunities to enhance STEM content.

“[Students] understood the limitations of the technology when we had finished the lesson.” – Minnesota teacher, grade 5

“My students loved this and it really helped them understand how the ocean floor is mapped.” – Colorado teacher, grade 5

“…students made great connections and checked out library books pertaining to submersibles sound, sonar, submarines, oceans, ocean life, etc.” – Minnesota teacher, grade 5

3.3.1.4 Lesson 3 of “Designing Submersibles” In Lesson 3, students gather data regarding the floating and sinking behavior of a series of vials filled with different materials. In Part 1, students observe the floating and sinking behavior of vials of the same volume. These vials are either half-filled or fully filled with beads, glass marbles, and sand. After placing a vial in water, students record whether or not and how it floated. They then use their data to generate statements and identify trends about the vials’ mass, volume, and density. In Part 2, students observe the floating and sinking behavior of larger vials. They observe that when mass stays the same and volume increases, density decreases. By reviewing all of the data they collect in this lesson, students have the opportunity to recognize that neither mass nor volume alone determine whether an object sinks or floats: density is the determining factor.

Thirty seven of the 38 teachers reported that they taught Lesson 3. Teachers spent an average of 131 minutes teaching this lesson and gave it an average overall rating of 6.2 on a scale of 1-7 (see Table 65).


How much time did you spend teaching this lesson? (Minutes) 35 131 65.46 30 360 How would you rate the quality of this lesson, overall? (Scale: 1-7) 37 6.2 0.82 5 7

Table 66. SB – Categories of Teacher Responses for the Question,




(N=31)

Positive

Students participated in hands-on activities and experiments 7 22.6 Students had fun, were engaged and were motivated 12 38.7 Activities and / or supporting materials of high quality 6 19.4 Other positive comments 12 38.7 Total Positive Comments* 28 90.3

Negative Total Negative Comments** 6 19.6 *The total number of comments is less than the sums of individual and grouped coding categories

because some teachers provided comments that fit into more than one coding category **Categories of negative comments are not listed as no more than two teachers provided a comment that fit into any

one coding category Thirty-one teachers commented on their rating of lesson quality. The coding categories used to code this question are shown in Table 66. Almost all of the teachers were positive about the unit (90%, n=28) providing at least one positive comment. One-fifth of teachers (19%, n=6) gave at least one negative response, and there was no trend to the negative comments: no more than two teachers reported the same criticism of the lesson.

Over one-third of the teachers (39%, n=12) mentioned that their students enjoyed the lesson and were engaged and motivated.

“Kids really enjoyed this lesson.” – Colorado teacher, grade 2

“Engaging, setting up the foundation for the actual problem to be solved.” – Colorado teacher, grade 5

“Students loved it - Interactive, [and] utilized concepts taught in story.” – Colorado teacher, grade 5

Additionally, almost a quarter of the teachers (23%, n=7) commented positively on the hands-on nature of the activities. A number of teachers (19%, n=6) also praised the quality of the activities and supporting materials.

“Loved the hands on.” – California teacher, mixed grade class (4/5)

“It is hands on and visual - students can 'see' flaws in their earlier thinking to help them correct misunderstandings.” – Massachusetts teacher, mixed grade class (1/2)

“This lesson really gave the students a clear picture of density and sinking/floating.” – Colorado teacher, grade 5

“Clearly presented, easy to follow and student friendly.” – California teacher, grade 2

3.3.1.5 Lesson 4 of “Designing Sumbersibles” In the final lesson, Lesson 4, students use the vials from Lesson 3 to assemble a submersible for their design challenge. Students design a vehicle that is able to float in water and pick up small “packages” from the bottom of the testing tank. These submersibles should be able to float while also carrying these items.

Thirty-four of the 38 teachers reported that they taught this lesson in their classroom (see Table 67). On average, teachers spent 165.7 minutes teaching this lesson; overall, the mean rating was 6.2 out of 7.



Table 68. SB – Categories of Teacher Responses to the Open-Ended Question,




(N=34)

Positive

Students practiced discussion, communication, and teamwork skills 3 8.8 Students had fun, were engaged and were motivated 15 44.1 Students were challenged 6 17.6 Activities and / or supporting materials of high quality 5 14.7 Other positive comments 12 35.3 Total Positive Comments* 23 67.6

Negative

Too difficult or confusing for students and/or teacher 6 17.7 Students were not engaged/activities too easy or boring 3 8.8 Criticism of supporting materials/difficult to implement 8 23.5 Total Negative Comments* 12 35.3


Thirty-four teachers commented on their rating of lesson quality. Table 68 shows coding categories used to code this question. Overall, teachers responded positively about the lesson with about two-thirds of teachers (68%, n=23) providing at least one positive comment. About one-third of teachers (35.29%, n=12) gave at least one negative comment.

As shown in Table 68, nearly half of teachers (44%, n=15) mentioned that the lesson was fun, engaging, and motivating. A few teachers (15%, n=5) also praised the quality of the activities and supporting materials.

“Kids love it, promotes teamwork, uses design process, creates some competition.” – California teacher, grade 4

“Kids were very invested in all aspects of the process/project and engaging their knowledge of density.” – Massachusetts teacher, grade 1/2

“The students were motivated and really enjoyed this. They used their gained knowledge and showed what they knew.” – Minnesota teacher, grade 3

“I felt this lesson was very good and met my needs as a teacher.” – Minnesota teacher, grade 4

“The materials were very workable and kid friendly. They loved to test and share results.” – Massachusetts teacher, grade 5

Negative comments most often mentioned criticism of supporting materials (24%, n=8) and that the lesson was too difficult or confusing (18%, n=6).

“I found that using the magnets did not improve the lesson for my kids. Just trying to change the materials to get it to sink down, stay for a second, then float the top was more than sufficient for 2nd graders. Tough too!” – North Carolina teacher, grade 2

“The sheets are too difficult and the lesson plans are very wordy.” – Massachusetts teacher, grade 4

“My students had a hard time with visualizing a real submersible and the vials.” – Colorado teacher, grade 5

3.3.1.6 Summary: “Designing Submersibles” Formative Evaluation Overall, teachers found this unit to be fun, engaging and motivating for their students. They noted that their students had opportunities to learn about both science and engineering and to connect that knowledge to the real world. Teachers also mentioned that the unit provided their students with the chance to improve their communication, teamwork, problem solving, and critical thinking skills. Teachers praised the quality of the lessons and supporting materials and appreciated the hands-on nature of the unit. In addition to integrating this unit with science, teachers mentioned that they also used this unit to further objectives in language arts, social studies and math. Despite the fact that a few teachers found some of the lessons to be time consuming or difficult to implement, the majority provided positive comments and reported that they were highly likely teach the unit again.

3.3.2 Summative Evaluation: Designing Submersibles 3.3.2.1 Assessment Design: “Designing Submersibles” The Designing Submersibles assessment was designed and first tested during the 2008-2009 school year. From a number of collected and brainstormed items, 17 multiple-choice questions were selected for the pilot assessment. In the spring of 2009, validity evidence was gathered for all questions from the Designing Submersibles assessment. EiE staff conducted cognitive interviews with children in grades 3 and 5 from Massachusetts classrooms, asking students to read each question aloud, explain what they

thought the answer would be, read all answer choices aloud, and explain which answer they would choose and why.

Table 69. Designing Submersibles Assessment Questions (Text) Question # Category Question Text

1 Engineering What might an ocean engineer do for his or her job? [improve instruments to collect data underwater]

2 Science To figure out how dense an object is, you need to know: [Its mass and volume.]

3 Science The black block is more dense than the white block. The white block sinks in a tub of water. What will the black block do? [sink to the bottom of the tub] (see Figure 18)

4 Science A student has two plastic bottles that are exactly the same. One is full of sand and the other is full of popped corn. Which is more dense? [the bottle filled with sand] (see Figure 19)

5 Science A jar of rocks sinks in a tub of water. What could you do to make the rocks float? [put the rocks in a larger jar]

6 Science A submersible is something that can float on the water and also carry things or people under the water. A submarine is a kind of submersible. The best way to get a submersible to float is to change its: [density]

7 Science Which objects always sink in water? Objects that are: [denser than water]

8 Engineering An ocean engineer would probably help to design: [all of the above: a cargo ship, a submarine, an underwater microphone]

9 Science

A student is dropping some bottles of marbles into a water tank to see if they sink or float. The large bottle has twice the volume of the small bottle. Both bottles are made of the same materials. The table shows what she has found so far. Which of the following is true, based on the data? [the more full a bottle is, the more likely it is to sink] (see Figure 20)

10 Science Some students are designing a submersible to take underwater pictures. Their first design didn’t work because it sank. What is the BEST thing they can do to improve it so it floats? [add some empty containers to the submersible] (see Figure 21)

11 Engineering What is an example of a technology used by scientists and ocean engineers to explore the ocean? [all of the above: SONAR, sounding, submersibles]

12 Engineering What might an ocean engineer do for his or her job? [all of the above: design an underwater vehicle to explore the ocean, design technologies that are used by marine biologists, figure out new ways to measure the depth of the ocean]

13 Engineering What might an ocean engineer do to help scientists learn about fish that live in deep water? [create a device that keeps track of where the fish swim]

14 Science What is the BEST way to figure out where a pond is exactly five feet deep? [use a stick to measure the depth in several different areas]

15 Engineering A scientist in a ship can use SONAR to figure out: [all of the above: how deep the ocean is under the ship, if the ocean bottom is sandy or rocky, and whether there is a sunken ship at the ocean bottom]

16 Science The ocean is largely unexplored because: [it is extremely large]

17 Engineering

Ocean engineers sometimes have to re-design a technology so it can be used in the ocean. This is because: [all of the above: salt water might damage the technology, the water in the ocean is always moving, the technology might harm ocean animals]

F

Figure 18: QQuestion 3 of

Figure

f the SB Asse

Figure

e 1

essment

e 20: Questio

Figu

on 9 of the S

ure 19: Ques

SB Assessme

stion 4 of the

ent

e SB Assessm

ment

DttD

Oeoeb

3SM

WtaacsiCa

3Dsd

Fc

During the sutesting to cretested in 4 sDesigning Su

On the preengineering. object floats explore the brackets.

3.3.2.2 Scales were Massachuset

We initially the final fielassessment hand flotationconsisted of science subsnternal relia

Cronbach’s aand 17, and h

3.3.2.3 Designing Sustudents recedescribed or

For the Desclassrooms

ummer of 20eate a final states (Califubmersibles

and post-asThese incluor sinks in ocean. Tabl

Scale Coconstructed tts.

constructed ld test assesshad been devn and a comp

questions 2scale due to ability. The alpha of .61had a Cronb

Sample:ubmersibles eived a diffeanalyzed in

signing Submin 4 states

Figure

009, questiofield test as

fornia, Colorunit assessm

ssessments suded science

water. Studele 69 descri

onstruction:from a relia

a scale whisment. This vised with tw

ponent on wh, 3, 4, 5, 6, its ambiguofinal scienc4 (n=62). Tach’s alpha

: “Designing(SB) assess

erent versionthis report.

mersibles fies (Californi

21: Questio

ns were revisessment. Inrado, Massa

ment include

students wequestions abents were alibes the tex

“Designingability sampl

ch was the sscale was fwo major cohat ocean en7, 9, 10, andous wordingce subscale, he engineeriof .655 (n=6

g Submersiblsments weren of the asse

eld test, datia, Colorad

n 10 of the S

iewed basedn 2009-2010achusetts, aned 17 multipl

re asked qubout what delso asked abt for each q

g Submersiblle of 64 stud

sum of all cfound to havomponents i

ngineers do fd 14. Questig, and quest

consisting ing subscale60).

les” collected fressment than

ta was colledo, Massach

SB Assessm

d on the resu0 the Designnd Minnesotle-choice qu

uestions aboensity is and

bout ocean equestion wit

les” dents in grad

orrect answeve a Cronbacin mind: a sfor their jobsion 7 was drtions 2 and of question

e consisted o

rom studentsn grades 3-5

ected from husetts, and

ment

ults of the pilning Submerta. The field

uestions.

out density, d how it affeengineers andth the corre

des 3 and 5 f

ers on all quch’s alpha oscience comps. The sciencropped from5 were drop

ns 3, 4, 6, 9of questions

s in grades 35; the grade

1000 studend Minnesot

lot study andrsibles unit wd test versio

flotation, ancts whetherd technologyect answer s

from 3 class

uestions thatof .729 (n = ponent abouce subscale o

m consideratipped to incr

9, 10, and 11, 8, 11, 12

3 through 5.e 2 assessme

nts in 44 gra) where E

d validity was field on of the

nd ocean or not an y used to shown in

srooms in

t were on 59). The

ut density originally ion in the rease the

14, had a 2, 13, 15,

. Grade 2 ent is not

rades 3-5 EiE was

implemented. One hundred and sixty-eight individual students, or 16.8% of the sample, were excluded due to missing pre- or post-assessments. Forty individual students were excluded due to missing demographic information. Ten classes were missing information on the income status of the student, as measured by whether the student received a free or reduced-price lunch, so that variable was not considered in the analysis. Sixteen individual students were excluded because they did not complete 11 or more of the questions on the assessment. Three classes were dropped because they did not complete either Lesson 3 or Lesson 4, which are the hands-on lessons that we consider critical to the EiE curriculum.

The final dataset for analysis included 741 students (level-1 units) in 41 classrooms (level-2 units), with an average classroom (cluster) size of 23 students per classroom, and a standard deviation of 6.36 students. The classrooms were spread over 24 schools with 27 teachers total, 5 of whom were science specialists. The teachers had an average of 14.2 years of experience, with a standard deviation of 11 years, a minimum of 1 year and a maximum of 43.

The sample overwhelmingly consisted of classrooms in grades 4 and 5 (see Table 70). Boys made up approximately half of the sample, while LEP and IEP students made up only 7% and 12% of the sample, respectively (see Table 71). The vast majority of the sample consisted of white students (76%), with all other races making up less than 10% of the sample (see Table 72).

Table 70. SB Grade Crosstabulation (N of Students and Classrooms)

Grade Total 3 4 5 N (students) 152 248 341 741 N (classrooms) 8 14 19 41

Table 71. SB Proportions for Level-1 Dichotomous Variables

Gender (male) LEP IEP Proportion .50 .07 .12 N 367 52 89

Table 72. SB Proportions for Level-1 Variables – Race


3.3.2.4 Results: “Designing Submersibles” Two outcome variables, the engineering scale (PostEng) and science/flotation scale (PostSci), were used to gauge students’ improvements in understanding what ocean engineers do and the concept of density, respectively. The engineering scale ranged from 0 to 7, and the science scale ranged from 0 to 6.

For this unit, we tested the student level (level-1) variables of IEP and LEP status, as well as gender and race (Asian, Black, Hispanic, and Other). We also included pre-assessment scores as a covariate. Next, we tested whether each level-1 variable was best modeled as a fixed or random coefficient, the default being a fixed coefficient, unless the random variance was found to be significant (p < .05). At the classroom level (level-2) we tested the pre-assessment classroom mean scores, class size, the number of minutes spent teaching the EiE unit, the number of EiE units the teacher had taught prior to that school year, the number of units the teacher taught during that school year, and the number of years the teacher

had been teaching overall. We also tested the effects of grade and of teacher specialty (Science Specialist) here, as well as the effect of the overall classroom setting (the proportions of IEP students, LEP students, and boys in the classroom).

SB: Engineering Scale Results. Engineering scale descriptive statistics for the demographic groups tested in the model are shown in Table 73. Questions making up the Engineering scale are shown in Table 69.

Table 73. SB Descriptive Statistics: Pre- and PostEngineering Scores by Demographic Groups

SB PreEngineering Score SB PostEngineering Score Mean SD Min Max Mean SD Min Max


The two-level final conditional model for the SB Engineering scale (see Figure 22 and Table 74) includes all of the variables and random variance coefficients which were found to be significant during the variable testing process described above. It shows that the baseline improvement between pre- and post-assessment was quite high, at 1.86 (CI 1.57, 2.15) points (Intercept γ00, p < .001): students gained a much better understanding of what an ocean engineer does, with students from classes that spent more time on the unit posting higher gains. However some students showed less improvement than others: Asian students (1.08 points improvement; CI 0.52, 1.63), students with an IEP (0.95 points improvement; CI 0.37, 1.53), and students in grade 3 (1.19 points; CI 0.59, 1.79). It is highly likely that all groups modeled showed improvement, since none of the confidence intervals includes zero points of improvement.

Our two-level model explains 79% (0.79=1-(0.124/0.583)) of the between-class variance. We found the level-1 (within-class) variance σ2 to be heterogeneous (p<.001) and so modeled it logarithmically (see Figure 22 and Table 75). The within-class variance was smaller among white students, Hispanic students, and students with high pre-assessment scores or in a class with a high pre-assessment mean (presumably due to ceiling effects). Once modeled, σ2 the test for heterogeneity was no longer significant at the .05 level (p=.095), suggesting that our model accurately describes some important sources of within-class variance.

Level-1 Model: 0 1 2 3PostEng (IEP) ( ) (Asian) rβ β β β= + + + +PreEng

2Var(r) σ=

And: 2

0 1 2 3 4ln ( ) (White) (Hispanic) ( )PreEngClassMeanσ α α α α α= + + + +PreEng Level-2 Model:

0 00 01 02 03

04 0

(Grade3) ( ) ( ))( u

PreEngClassMean TotalMinutesTotalMinutesSquared

β γ γ γ γγ

= + + ++ +

1 10 1uβ γ= + 2 20β γ= 3 30β γ= Bold indicates group mean centered. Italicized indicates grand-mean centered. The outcome variable PostEng was centered around the PreEng mean.

Figure 22. Designing Submersibles PostEng Score – Conditional Model

Table 74. SB PostEng Score Conditional Model – Final Estimation of Fixed Effects (with robust standard errors)


df P-value

For Intercept 1, β0 Intercept, γ00 1.864 0.145 12.876 36 <0.001 Grade3, γ01 −0.677 0.262 −2.580 36 0.014 PreEngClassMean, γ02 0.404 0.141 2.867 36 0.007 TotalMinutes, γ03 0.588 0.156 3.778 36 0.001 TotalMinutesSquared, γ04 −0.276 0.061 −4.498 36 <0.001 For IEP slope, β1 Intercept, γ10 −0.914 0.254 −3.593 40 0.001 For PreEng slope, β2 Intercept, γ20 0.479 0.032 15.020 733 <0.001 For Asian slope, β3 Intercept, γ30 −0.788 0.236 −3.336 733 0.001

Table 75. SB PostEng Score Conditional Model – Final Estimation of Variance Components



Intercept 1, u0 0.352 0.124 21 52.082 <0.001 IEP slope, u1 0.849 0.721 25 45.290 0.008 Level-1, r 1.592 2.536

Table 76. SB PostEng Score – Model for Level-1 Variance Parameter Coefficient Standard Error Z-ratio P-value

Intercept 1, α0 1.990 0.254 7.833 <0.001 PreEng, α1 −0.106 0.030 −3.533 0.001 White, α2 −0.311 0.133 −2.331 0.020 Hispanic, α3 −0.720 0.315 −2.282 0.022 PreEngClassMean, α4 −0.271 0.073 −3.698 <0.001

Table 77. SB PostEng Score Unconditional Model – Final Estimation of Variance Components




Figure 23: SB Change in Student Score: Engineering Scale

Figure 24 shows the relationship between the minutes spent on EiE instruction in a class and students’ PostEng scores for Designing Submersibles. The PostEng scores of the students in the test group have been graphed against the z-score of the number of minutes of instruction for each student’s class. The rough line shows the LOESS trend, while the smoothly curved line shows the HLM model fit ( 2PostEng 0.588 0.276( ) ( )TotalMinutes TotalMinutes−= ), which also takes into account the demographic variables that were significant in the analysis. The EiE curriculum comes with recommendations as to how much time should be spent teaching each unit. The recommended range of minutes for Designing Submersibles, which was between 325 and 400 minutes, is shown with the two vertical lines on the graph. The labels show, in order, the minimum number of minutes spent teaching EiE in our data, the minimum recommended time, the maximum recommended time, the number of minutes corresponding to the best scores according to the model fit, and the maximum number of minutes spent teaching EiE in our data. Note that the number of minutes corresponding to the maximum score as suggested by the model fit is higher than both the EiE recommended number of minutes and the

ReferenceGroup

Grade 3IEP Asian

0

0.5

1

1.5

2

2.5

Cha

nge

in S

core

(Pre

to P

ost)

Student Change in Score: SB Eng Scale with 95% confidence intervals

maximum score as shown by the LOESS trend line, suggesting that the model should not be used literally to predict the best number of minutes to spend on the unit.

Figure 24: SB PostEng Scores vs. Number of Minutes Spent Teaching the SB Unit

Table 78. Notable Times Shown in Figure 24

Label (minutes) Description 135 Minimum number of minutes reported spent teaching EiE 325 Minimum recommended time 400 Maximum recommended time 787 Number of minutes corresponding to the best scores according to our model

1480 Maximum number of minutes reported spent teaching EiE SB: Science Scale Results. The outcome variable PostSci was used to test for changes in the performance of students in grades 3 to 5 on the SB science questions. The science scores were computed by adding together the number of correct answers for each of the science questions listed in Table 69. This score had a possible range of 0 to 6.

Table 79. SB Descriptive Statistics: Pre- and PostScience Scores by Demographic Groups

SB PreScience Score SB PostScience Score Mean SD Min Max Mean SD Min Max


The two-level final conditional model for the Designing Submersibles science assessment was established in the same manner as that for the PostEng outcome variable, except that the PreEng covariate and PreEngClassMean were replaced by the variables PreSci and PreSciClassMean during testing of variables. The final PostSci model, shown in Figure 25 and Table 80 below, shows that there was an improvement of 1.27 points (CI 1.01, 1.53) between the pre- and post-assessment (Intercept γ00, p <.001), showing that the students gained a better understanding of density and flotation between the pre- and post-assessments. Male students improved significantly more than others (1.5 points improvement; CI 1.19, 1.81; Gender intercept γ10, p = .009), but IEP students improved significantly less (0.74 points; CI 0.29, 1.18; IEP intercept γ20, p = .006). Since none of the groups’ confidence intervals for the predicted improvement in score included zero, it is very likely that all groups improved. Our two-level model explains 55% (0.55=1-(0.226/0.503)) of the between-class variance. The level-1 (within-class) variance σ2 was tested and found to be homogeneous.

Level-1 Model: 0 1 2 3PostSci β β (Gender) β (IEP) β ( ) r= + + + +PreSci Level-2 Model: 0 00 01 02 03 0( ) ( ) ( ) uPreSciClassMean TotalMinutes TotalMinutesSquaredβ γ γ γ γ= + + + +

1 01β γ= 2 20 2uβ γ= + 3 30β γ= 4 40β γ= Bold indicates group mean centered. Italicized indicates grand mean centered. The outcome variable PostSci was centered around the PreSci mean.

Figure 25. Designing Submersibles PostSci Score – Conditional Model

Table 80. SB PostSci Score Conditional Model – Final Estimation of Fixed Effects (with Robust Standard Errors)


df P-value

For Intercept 1, β0 Intercept, γ00 1.271 0.129 9.880 37 <0.001 PreSciClassMean, γ01 0.757 0.179 4.222 37 <0.001 TotalMinutes, γ02 0.428 0.088 4.912 37 <0.001 TotalMinutesSquared, γ03 −0.097 0.034 −2.828 37 0.008 For Gender slope, β1 Intercept, γ10 0.232 0.087 2.652 734 0.009 For IEP slope, β2 Intercept, γ20 −0.534 0.182 −2.929 40 0.006 For PreSci slope, β3 Intercept, γ30 0.241 0.032 7.641 734 <0.001

Table 81. SB PostSci Score Conditional Model – Final Estimation of Variance Components



Intercept, u0 0.444 0.198 22 90.329 <0.001 IEP Slope, u2 0.588 0.346 25 43.258 0.013 Level-1, r 1.189 1.413

Table 82. SB PostScience Score Unconditional Model – Final Estimation of Variance Components



Intercept, u0 0.698 0.487 40 240.082 <0.001 level-1, r 1.253 1.570

Figure 26: SB Change in Student Score: Science Scale

ReferenceGroup

Gender (male)

IEP

0

0.5

1

1.5

2

Cha

nge

in S

core

(Pre

to P

ost)

Student Change in Score: SB Science Scale with 95% Confidence Intervals

Figure 27 shows the relationship between the time spent on EiE and students’ PostSci scores for Designing Submersibles. Students’ PostSci scores have been graphed against the z-score of the number of minutes of instruction per class. The black line shows the LOESS trend, while the red line shows the HLM model fit ( 2PostEng centered 0.428( ) 0 ( ).097Total minutes Total minutes−= ), which also takes into account the demographic variables that were significant in the analysis. The EiE curriculum comes with recommendations as to how much time should be spent teaching each unit. The recommended range of minutes for Designing Submersibles, which was between 325 and 400 minutes, is shown with the two vertical lines on the graph. The labels show, in order, the minimum number of minutes spent teaching EiE in our data, the minimum recommended time, the maximum recommended time, the number of minutes corresponding to the best scores according to the model fit, and the maximum number of minutes spent teaching EiE in our data. Note that the maximum score according to the HLM model fit occurs at a time higher than the recommended number of minutes and the highest score according to the LOESS trend line. The benefits for participation in EiE (for this scale score only) appear to level out at about 700 minutes, according to the LOESS trend line.

Figure 27: SB PostSci Scores vs. Number of Minutes Spent Teaching the SB Unit

Table 83. Notable Times Shown in Figure 27 Minutes Description

135 Minimum number of minutes reported spent teaching EiE 325 Minimum recommended time 400 Maximum recommended time

1144 Number of minutes corresponding to the best scores according to the HLM model fit 1480 Maximum number of minutes reported spent teaching EiE

3.3.2.5 Summary: “Designing Submersibles” Summative Evaluation Our analysis shows that students who are taught the EiE Designing Submersibles unit, together with a science unit on density and flotation, learn a significant amount about engineering and science concepts. Asian and grade 3 students did not improve as much as other demographic groups on the engineering scale, while boys improved more than girls on the science scale, and IEP students did not improve as much as other demographic groups on either scale. However, all students experienced a significant improvement in both scales, suggesting that they improved in their understanding of density and flotation and of what an ocean engineer does.

3.4 Results for the “Designing Knee Braces” Unit Evaluation In the EiE unit No Bones About It: Designing Knee Braces, students apply what they are learning about the human body in their science curriculum to the field of biomedical engineering. In Lesson 1, students read the storybook Erik’s Unexpected Twist, set in Germany, which introduces them to the field of biomedical engineering and reviews science concepts such as the various parts of the knee and relevant musculoskeletal vocabulary. In the story, Erik designs a temporary knee brace for a classmate who injures his knee on a hiking trip. Students follow Erik’s use of the Engineering Design Process (EDP), the same process they will use to create their own knee brace design during Lesson 4 of the unit. In Lesson 2, students gain a broader understanding of the field of biomedical engineering by gathering and analyzing data about the arches of their feet to aid in the design of new running shoes. In Lesson 3, students observe the range of motion of their knees and compare and contrast this motion with a model of an injured knee. Students also have the opportunity to examine materials and brainstorm how they might use them in the design of their knee braces. In the final lesson, Lesson 4, students use the Engineering Design Process to design their own knee braces. Applying what they learned in previous lessons, students create, test, evaluate, and improve knee braces for the model injured knee that they examined in Lesson 3. Students evaluate their designs based on how well their knee braces restore the normal, “healthy” range of motion to the model injured knee while also taking into consideration their knee braces’ durability and usability.

No Bones About It: Designing Knee Braces was field tested during the 2010-2011 school year.

3.4.1 Formative Evaluation: Designing Knee Braces Feedback forms for Designing Knee Braces were completed by 40 teachers who taught the unit in their classrooms; however, one teacher was a grade 1 teacher and since our curriculum is designed to be used with older elementary school students, this teacher was not included in our analysis. Thus our final sample consisted of 39 teachers who taught 56 grades 2-6 classrooms in six states: California, Colorado, Massachusetts, Minnesota, North Carolina, and New Hampshire (see Table 84).

Table 84. Designing Knee Braces– Classroom Grade and State Distribution

State Grade 2

Grade 3

Grade 4

Grade5

Grade6

Grade Mixed* Total

CA 1 1 7 9 CO 1 7 8 MA 1 1 11 2 15 MN 2 4 3 1 2 12 NC 7 7 NH 2 3 5

Total 1 9 9 21 12 4 56 *In MN there were was one mixed Grade 3-5 classroom and one Grade 3-6 classroom.

In MA there were 2 mixed Grade 5/6 classroom. **The total number of classrooms is larger than the number of teachers responding, because

11 teachers taught more than one class and each class is counted individually. 3.4.1.1 Overall Feedback for “Designing Knee Braces” Unit Teachers were asked to rate the unit overall by answering questions on a scale from 1-7. Table 85 shows the number of the responses (N), the average response, the standard deviation (SD), and the minimum (Min) and maximum (Max) response to the overall unit, based on a seven point scale, with 1 representing “Not at all” and 7 representing “Very”.

Table 85. Designing Knee Braces Feedback – Unit Ratings







Teachers were asked whether this unit furthered their objectives for another content area beyond science, and if so, to specify which content area. In a separate question, teachers were also asked if they integrated the teaching of this unit with their teaching of other subjects, and if so, to explain how. All 39 teachers responded to at least one of these questions. Approximately three-quarters of teachers (78%, n=30) mentioned a specific non-science content area that was enhanced by the unit, with language arts being mentioned most often (73%, n=22). A few teachers who mentioned specific content areas also reported that the unit furthered their objectives in math (20%, n=6) and social studies (10%, n=3).

Teachers were asked a number of open-ended questions about the unit and lessons. Thirty-seven of the 39 teachers responded to the question, “How did your students benefit, academically or otherwise, from

taking part in this unit?” The coding categories used to code this question are shown in Table 86. Slightly more than half of the respondents (57%, n=21) noted that the unit provided opportunities to further their students’ STEM knowledge and skills. Specifically, teachers noted that their students learned about the human body, including bones, muscles, and joints, the work of biomedical engineers, and the engineering design process.

“They were able to deepen their understanding of the human body and specific parts. They had a better understanding of joints, bones, and tendons and how they work together.” – North Carolina teacher, grade 3

“First, my students know what an engineer does. Some are even interested in becoming engineers. They also gained a much more sophisticated understanding of bones, muscles and joints. Lastly, they were motivated to participate in science!” – New Hampshire teacher, grade 4

“The design process reinforced what they have been learning in science about engineering. Students who participated in the unit have an advanced understanding of what the design process is and why it is useful.” – Minnesota teacher, mixed grades

“After the assessment, I sat down with the students and did an “I learned...” session. It was unbelievable what they could tell me about their learning. One thing that was very evident was that they all knew the steps in the engineering design process.” – Minnesota teacher, grade 3

“They learned about the Design Process earlier that I would have taught it. Students learned about bioengineers and the human body. When they get to middle school, they will know more than others who didn't take this unit.” – Massachusetts teacher, grade 4

Table 86. KB – Categories of Teacher Responses to the Open-Ended Question, “How did your students benefit, academically or otherwise, from taking part in this unit?”



(N=37) Students practiced discussion, communication, and teamwork skills 9 24.3 Students practiced problem solving and critical thinking skills 10 27.0 Students had opportunities to learn/apply STEM content and/or skills 21 56.6 Students made connections to the real world 3 8.1 Students had fun, were engaged and were motivated 13 35.1 Other comments 10 27.0

Additionally, approximately one-third of teachers (35%, n=13) reported their students had fun and were engaged and motivated.

“The students were motivated to take part in the science activities. They looked forward to reading the book and especially designing the knee brace.” – California teacher, grade 5

“Students were interested and enthusiastic. They read the book and instructions eagerly which was excellent reading practice.” – California teacher, grade 5

“This was a very motivating unit for the students that required thought and creativity. Students benefited from each activity. The footprints and designing/improving the brace work well to teach critical thinking and reasoning.” – Minnesota teacher, grade 4

“The hands-on of course is great. They loved the story and it did open their eyes to technology and its uses, and what “engineers” do. It was very engaging and they liked doing the project.” – California teacher, grade 5

Approximately one-quarter of the teachers reported that, in addition to learning science and engineering concepts, their students were also able to practice other skills such as discussion, communication and teamwork (24%, n=9) as well as problem solving and critical thinking (27%, n=10).

“They learned that there is not one answer or solution to a problem. They learned to collaborate, share ideas, and recognize the benefit of listening to others. They were beginning to understand the importance of taking the time to think about their design before building and creating.” – Massachusetts teacher, grade 6

“Huge benefits in collaborative work, engineering design process, and thinking skills.” – Minnesota teacher, grade 6

“My students benefited by problem solving, working in a team, collaborating, and working through a process from beginning to end.” – New Hampshire teacher, grade 4

“Working cooperatively, problem-solving, developing creativity, and expanding vocational awareness.” – California teacher, grade 5

A few teachers also specifically mentioned that the unit allowed students to make real-life connections (8%, n=3).

“[Students] learned about engineering design process. Used the design process for a personal project (one student is designing a Halloween costume for next year, and wrote on paper his design process). [Students] were creative.” – Minnesota teacher, grade 3-5

“Students benefit academically, and personally. Great application to real-life situations.” – Colorado teacher, grade 5

Teachers were also asked, on a scale of 1-7, “What is the likelihood that you will choose to teach this unit again in your classroom?” Thirty-eight of the 39 teachers provided a rating with the average score being 6.2 (SD=1.00) out of a possible 7 points, with the lowest score being a four out of seven. This indicates that teachers have a high interest in teaching the unit again. Teachers were also asked to explain their rating. Twenty-nine teachers responded to this question; the most common reasons for choosing to teach the unit again included that students were engaged and motivated (24%, n=7) and that the unit enhanced their science unit on the human body (14%, n=4).

“It was such a natural add-on to the Human Body unit and kids were genuinely excited by it.” – Minnesota teacher, grade 3

“This was a unit that both myself and my students enjoyed thoroughly. I really like having a medical unit; I have a strong background in medicine and this was a great teaching moment for myself and my students.” – Colorado teacher, grade 5

“I feel it was a great way to conclude my human body unit and see if they can apply knowledge.” – Colorado teacher, grade 5

“[The unit] had many positive benefits and inspired students – great introduction at the beginning of the year to motivate students.” – California teacher, grade 5

The most common reason given for why a teacher was less likely to teach the unit again dealt with the issue of time constraints.

“It takes a considerable amount of time to do it properly and we are ‘pressed’ with so many other district requirements that time is always an issue.” – California teacher, grade 5

“I'm conflicted about taking the time to do this in the future because it took significant time to complete the lessons (about five hours) and even that wasn't enough. Most students commented that they wanted more time in their feedback. With the pressure to stay on pacing charts and address standards (bones are not a fifth

grade standard). I'm not sure I should spend a week on this unit. On the other hand, this was a wonderful experience for my students and I. It was challenging and gave students an opportunity to work on a project as a group, something they rarely do. I loved the interaction of the groups when they made the Knee Brace. Interestingly, from their comments I see that group work was a challenge for many, which means we have to do it more often.” – California teacher, grade 5

“It was very motivating for my students, but it was also time consuming both in preparation work and teaching time. I'll have to decide when I see the needs of my next group of students.” – New Hampshire teacher, Grades 4

3.4.1.2 Lesson 1 of “Designing Knee Braces” In Lesson 1, students read the storybook Erik’s Unexpected Twist. In the story, Erik, a boy from Freiburg, Germany, goes on a camping trip. During a race back from the campsite, Erik’s friend, Matthias, injures his knee. Using knowledge he has obtained from his mother, a biomedical engineer, as well as his own experience injuring his knee while skateboarding, and the help of an adult doctor, Erik and the other campers are able to design a knee brace to help Matthias. Through the story, students learn about science concepts including some parts of the knee: muscles, bones ligaments, and tendons. They also learn about the field of biomedical engineering, including the different types of knee braces, technologies developed by biomedical engineers, and when they might be used.

Thirty-eight of the 39 teachers reported teaching this lesson in their classroom. As shown in Table 87, teachers spent an average of 104 minutes teaching Lesson 1. When asked to rate the quality of the lesson on a scale of 1-7, teachers gave Lesson 1 an average score of 6.3.

Table 87. Designing Knee Braces Feedback – Lesson 1 Question N Mean SD Min Max


Table 88. KB – Categories of Teacher Responses to the Open-Ended Question, “Please explain your rating of Lesson 1”



(N=35)

Positive

Students had fun, were engaged and were motivated 10 28.6 Activities and / or supporting materials of high quality 14 40.0 Good foundation and/or preparation for future lessons 7 20.0 Other positive comments 11 31.4 Total Positive Comments* 32 91.4

Negative Total Negative Comments** 5 14.3 *The total number of comments is less than the sums of individual and grouped coding categories

because some teachers provided comments that fit into more than one coding category **Categories of negative comments are not listed as no more than one teacher provided a comment that fit into any

one coding category Thirty-five of the 39 teachers who taught Lesson 1 provided an explanation for their rating of lesson quality. Table 88 shows coding categories used to code this question. Overall, comments were very positive with nearly all of the teachers (91%, n=32) providing at least one positive comment. Only 5 teachers (14%) gave a negative response; there was no trend to the negative comments as none of these teachers offered the same criticism of the lesson.

As shown in Table 88, 14 teachers (40%) praised the quality of the activities and supporting materials. “I thought all angles had been considered, covered, and great materials designed to support the content.” –California teacher, grade 5

“I love the book, DM’s (Duplication Masters), and ways to show other areas of world and vocabulary instruction.” – Colorado teacher, grade 5

“[My] students enjoyed the story. [The] sheets were good for review. [I] especially liked the knee coloring - helped students learn about the muscles, tendons, and ligaments.” – Minnesota teacher, grade 3-5

“I thought the unit was very informative, hands-on and fun for the students to learn about the leg and the country of Germany.” – Massachusetts teacher, grade 4

“The book was great! The author explained the EDP very clearly - my students were able to understand the Knee Brace problem and how the kids solved it.” – California teacher, grade 5

More than one-quarter (29%, n=10) also said that their students had fun and were engaged. “Students liked the story and were quite knowledgeable after reading it.” – California teacher, grade 5

“The book was enjoyable and the students had fun reading and acting out parts of the story.” – North Carolina teacher, grade 3

“I found that my students related very well to the story and to its main character with this, many had experiences with knee braces and injuries so their schema helped.” – Massachusetts teacher, grade 5

Additionally, 7 teachers (20%) commented that the activities in Lesson 1 provided a good foundation for future lessons.

“It was a great introduction to the following lessons. I felt the students were very interested and learned a lot of new vocabulary to help in their understanding later in the unit.” – North Carolina teacher, grade 3

“It introduced the concepts that we were to study and use in the lessons to follow.” – Massachusetts teacher, grade 5/6

“It was a good introduction to the Engineering Design Process.” – Massachusetts teacher, grade 6

3.4.1.3 Lesson 2 of “Designing Knee Braces” In Lesson 2, students are introduced to kinds of work done by biomedical engineers by helping the fictional Fast Feet Shoe Company design running shoe for customers with various foot types. Students examine their footprints and work together to sort them into different categories based on arch height in order to decide how many different running shoe models they should make so that every customer, regardless of their foot type, can find a supportive running shoe. The lesson concludes with students writing a report to Fast Feet, detailing the data they collected and their analyses of students’ arch heights. Based on their results, students make a recommendation to the shoe company as to how many running shoe models they should design and explain why it is important to use data in design decision-making.

Thirty-seven of the 39 teachers reported teaching Lesson 2. As shown in Table 89, teachers spent an average of 98 minutes teaching the lesson and gave it an overall rating of 6.2 on a scale of 1-7.



Table 90. KB – Categories of Teacher Responses to the Open-Ended Question, “Please explain your rating of Lesson 2”


of teachers


(N=33)

Positive

Students practiced discussion, communication, and teamwork skills 4 12.1 Lesson made STEM cross-disciplinary connections 9 27.3 Students made connections to the real world 4 12.1 Students participated in hands-on activities and experiments 4 12.1 Students had fun, were engaged and were motivated 12 36.4 Activities and / or supporting materials of high quality 7 21.2 Other positive comments 6 18.2 Total Positive Comments* 28 84.8

Negative Time constraints/takes too much time 4 12.1 Other negative comments 6 18.2 Total Negative Comments* 9 27.3


Thirty-three teachers provided an explanation for their rating of lesson quality. Table 90 shows coding categories used to code this question. In general, teachers were highly positive about the unit. Eighty-five percent of teachers (n=28) provided at least one positive comment while only 27% (n=9) gave a negative comment.

The most common criticism of the unit, given by 4 teachers (12%), focused on the amount of time needed to prepare for and teach the lesson (see Table 90).

“It takes a while to read through and understand all parts of the activity.” – California teacher, grade 5

“Loved the lesson – it taught the kids a lot. Lesson time wasn’t realistic. It took two sessions of 90 minutes each to complete.” – California teacher, grade 5

“I loved all the parts of the lesson – it was just a lot to fit in.” – North Carolina teacher, grade 3

Despite the time constraints, approximately one-quarter of teachers (21%, n=7) praised the quality of the activities and supporting materials. A few teachers (12%, n=4) specifically mentioned the hands-on nature of the activities.

“The lesson was great. The activity with the water and foot print, the kids loved and they really used sorting skills.” – Massachusetts teacher, grade 6

“The analysis of the footprints was great. The business letter was a nice connection to writing.” – Minnesota teacher, grade 4

“Good! Well written and well thought-out.” – Minnesota teacher, grade 5

“Excellent, authentic method to show data collection and interpretation.” – New Hampshire teacher, grade 5

Approximately one-third of responding teachers (36%, n=12) also felt that their students found Lesson 2 to be fun, engaging, and motivating.

“The kids had a lot of fun doing this and it gave them a different kind of insight about what biomedical engineers do.” – Minnesota teacher, grade 4

“The students were very engaged and excited. They loved comparing their arches and grouping them on the board.” – Minnesota teacher, grade 3-6

“The students really enjoyed this experiment. They were very excited to become biomedical engineers and categorize their arches.” – New Hampshire teacher, grade 4

Additionally, about one-quarter of teachers mentioned that this lesson allowed students to make connections to STEM subjects (27%, n=9).

“The students loved this activity and I liked the integration of math and language arts standards in this lesson.” – Minnesota teacher, grade 3

“Great way to model the process of collecting and analyzing data with something concrete and simple.” – California teacher, grade 5

“It was a great way to learn about collecting and analyzing data, and to see something else biomedical engineers do.” – Minnesota teacher, grade 3

A few teachers also mentioned that their students were able to make connections to the real world (12%, n=4) and that the unit enabled their students to practice discussion, communication, and teamwork skills (12%, n=4).

“The kids have a natural interest in footwear, so this activity tapped into their prior knowledge. Also, closely focusing on the mechanics of running gave them an appreciation for the importance of observations.” – Massachusetts teacher, grade 5/6

“I love how it was a real life example of how an engineer may be hired to work on a project.” – Minnesota teacher, grade 6

“Students were enthused, made real-life connections to a problem and found it interesting. Good team-building activity.” – California teacher, grade 5

“Students thought about their feet and were able to talk about it - most liked to see the footprint.” – Minnesota teacher, grade 3-5

3.4.1.4 Lesson 3 of “Designing Knee Braces” In Lesson 3, students begin preparing for their design challenge: designing a knee brace. In the first part of the lesson, students explore the range of motion of a healthy knee and of an injured knee model. In the second part of the lesson, students examine the materials available for designing their knee braces and identify the properties of these materials. Students also begin to think about how these materials could be used in their knee brace designs.

All 39 teachers reported that they taught Lesson 3 (see Table 91). Teachers spent an average of 141.8 minutes teaching this lesson and gave it an average overall rating of 6.3 on a scale of 1-7.



Table 92. KB – Categories of Teacher Responses for the Question, “Please explain your rating of Lesson 3”


of teachers


(N=33)

Positive

Students had opportunities to learn/apply STEM content and/or skills 5 15.2 Students had fun, were engaged and were motivated 6 18.2 Activities and / or supporting materials of high quality 5 15.2 Other positive comments 14 42.4 Total Positive Comments* 24 72.7

Negative

Activities/lessons too difficult or confusing for students and/or teacher 7 21.2 Criticism of supporting materials/difficult to implement 7 21.2 Other negative comments 3 9.1 Total Negative Comments* 14 42.4


Thirty-three teachers commented on their rating of lesson quality. The coding categories used to code this question are shown in Table 92. In general, teachers were positive about the unit with nearly three-quarters (73%, n=24) providing at least one positive comment; however, many teachers (42%, n=14) also gave at least one negative response.

Teachers were most likely to criticize the supporting materials (21%, n=7), mentioning that there were problems that made the lesson difficult to implement (see Table 92).

“Even though many of the students had to re-measure their knee, it was a good learning experience. When students help students the accuracy of measurement wasn’t there. An adult needed to help monitor the measurement with the goniometer.” – New Hampshire teacher, grade 4

“The goniometers were difficult for students to use to gather accurate data.” – North Carolina teacher, grade 3

“I really liked the lesson. The knees are pretty fragile and needed reinforcement to keep from bending in the ‘uninjured’ direction.” – California teacher, grade 5

Additionally, nearly one-quarter of teachers (21%, n=7) mentioned that the lesson activities were too difficult or confusing.

“It was difficult for them to understand the movements of the knees and their actual knee.” – North Carolina teacher, grade 3

“Students struggled with understanding that their knee only bends one way. They were confused because [of the] turning of hips. Third graders have a hard time evaluating resources.” – North Carolina teacher, grade 3

Despite these concerns, a number of teachers liked the activities and supporting materials (15%, n=5) and reported that lesson Students had opportunities to learn/apply STEM content and/or skills (15%, n=5).

“Students saw the differences between a healthy and an unhealthy knee.” – California teacher, grade 2

“Great idea giving the students a model injured knee to help them visualize what they needed to focus on.” – Minnesota teacher, grade 3-6

“Besides the construction of the injured knee model – the execution of both parts was easy and manageable. Students easily manipulated the goniometer to find readings on both their knees and the model (after my demonstration).” – Massachusetts teacher, grade 5

Additionally, about one-fifth of teachers (18%, n=6) commented that their students enjoyed the unit and were engaged and motivated.

“Students were engaged. The preview of the materials was helpful.” – Minnesota teacher, grade 4

“It was a fun activity and let the kids be aware of how it was important to know a healthy knee before they could fix an unhealthy one.” – Minnesota teacher, grade 4

“Kids really enjoyed it!” – Colorado teacher, grade 4

3.4.1.5 Lesson 4 of “Designing Knee Braces” In the final lesson, Lesson 4, students apply what they learned in previous lessons about the range of motion of a healthy knee, the materials available to them and their properties, and the field of biomedical engineering. Using the Engineering Design Process, students design, create, and test, evaluate, and improve a knee brace for the model injured knee they examined in Lesson 3. Their knee braces are evaluated on the degree to which their knee brace limits to normal the range of motion of the “injured” knee, the ease of putting on and taking off the knee brace, and the durability of the knee brace after repeated use.

Thirty-eight of the 39 teachers reported teaching this lesson (see Table 93). On average, teachers spent 179.3 minutes teaching this lesson and gave it an overall rating of 6.5 on a scale of 1-7.



Table 94. KB – Categories of Teacher Responses to the Open-Ended Question,



of teachers


(N=32)

Positive

Students had opportunities to learn/apply STEM content and/or skills 3 9.4 Students participated in hands-on activities and experiments 3 9.4 Students had fun, were engaged and were motivated 11 34.4 Activities and / or supporting materials of high quality 9 28.1 Other positive comments 17 53.1 Total Positive Comments* 29 90.6

Negative



Thirty-two teachers commented on their rating of lesson quality. Table 94 shows coding categories used to code this question. Overall, teachers responded positively with nearly all of teachers (91%, n=29)

providing at least one positive comment, while a little over one-third (38%, n=12) gave at least one negative comment.

As shown in Table 43, negative comments most often mentioned that the lesson was too difficult or confusing (13%, n=4) or criticized the supporting materials (9%, n=3).

“The worksheets were a little difficult to understand exactly what needed to be filled out.” – Minnesota teacher, grade 5

“Students followed the process, but have difficulty showing all their ideas on paper. They often veered away from their actual plan as they were creating.” – North Carolina teacher, grade 3

“The kids had a great time and were very creative, but there is a lot going on and they needed more clarification, explanation and time.” – New Hampshire teacher, grade 4

Despite these concerns, 9 teachers (28%) praised the quality of the lesson and supporting materials. A few teachers (9%, n=3) specifically mentioned that the lesson provided good hands-on experiences or that the lesson provided opportunities to improve their students’ STEM knowledge and skills (9%, n=3).

“The DM’s [Duplication Masters] were well organized and helped the students follow the process easily.” –Minnesota teacher, grade 4

“The materials were great! Kids were so motivated because they really understood the challenge.” –Minnesota teacher, grade 3

“It was great hands-on learning. They applied what they learned from Lessons 2-3 to finish Lesson 4.” –Massachusetts teacher, grade 4

“This was the best lesson. It required students to pull everything they knew together about biomedical engineering and apply it to their design. This experience gave them a practical use for what they had learned.” – Minnesota teacher, grade 3-6

“Very effective and creative lesson.” – Colorado teacher, grade 5

“This spelled out the EDP well.” – Massachusetts teacher, grade 6

Additionally, one-third of teachers (34%, n=11) mentioned that the lesson was fun, engaging, and motivating for their students.

“Kids loved designing and constructing and trying out their designs. We displayed the six braces at our school’s Portfolio Night.” – Massachusetts Teacher, grade 5/6

“The create part is always the students’ favorite.” – North Carolina teacher, grade 3

“The students loved to play the role of biomedical engineers. It was nice to see it all come full-circle and always relate back to Eric.” – Massachusetts teacher, grade 6

3.4.1.6 Summary: “Designing Knee Braces” Formative Evaluation Overall, teachers found the Designing Knee Braces EiE unit to consist of high quality supporting materials and hands-on activities that were fun, engaging and motivating for their students and that helped students enhance their STEM knowledge and skills. Additionally, teachers said that the unit helped students make connections to the real world and provided students with the opportunity to improve their communication, teamwork, problem solving and critical thinking skills. In addition to integrating this unit with science, teachers mentioned that they also used this unit to further objectives in language arts, math and social studies. Although a few teachers found some of the lessons to be time consuming or difficult to implement, the majority provided positive comments and reported that they were highly likely teach the unit again in their classroom.

33TssaMa

DptH

Oabt

3.4.2 3.4.2.1 The Designischool year.spring of 20assessment. Massachusetanswer woul

During the spilot study atesting in sHampshire).

On the identand functionbiomedical ethe questions

Figure 28.

SummatiAssessm

ing Knee Br Thirty-one 010, validityEiE staff c

tts classroomld be, read al

ummer of 2and validity tix states (CThe field te

tical pre- anns of bonesengineer ands with the co

Designing K

ive Evaluatment Design:races studen

multiple-chy evidence conducted c

ms, asking stll answer cho

010, a numbtesting to crCalifornia, Cest version in

nd post-asses, joints and

d the identifiorrect answer

Knee Braces

tion: Design “Designing

nt assessmenhoice questiwas gatherecognitive intudents to reoices aloud,

ber of questireate a new fColorado, M

ncluded 19 m

ssments studd muscles, aication and ur shown in b

s Question 9

ning Knee Bg Knee Bracent was desigons were ched for all qnterviews wead each quand explain

ions were drfield test assMassachusetmultiple-choi

dents were aas well as eusage of mobrackets.

Figu

Braces es” gned and firhosen for th

questions frowith 58 chiluestion aloudn which answ

ropped or resessment. Intts, Minnesoice question

asked sciencengineering odels. Table

ure 29. Desig

rst tested duhe pilot assom the Desldren in grad, explain wwer they wou

evised basedn 2010-2011 ota, North s.

ce questions questions a95 describes

gning Knee B

uring the 20essment. Du

signing Kneeades 3 and

what they thould choose a

d on the resuit was usedCarolina, a

about the pabout the ws the text fo

Braces Ques

009-2010 uring the e Braces

d 5 from ought the and why.

ults of the d for field and New

properties work of a or each of

stion 13

Table 95. Designing Knee Braces Assessment Questions (Text) Question # Scale Question Text

1 BME* What do all biomedical engineers need to know about? [how the human body works]

2 Science What part of your body do you use when you take a step forward? [bones, joints and muscles]

3 Design-Models What do models help engineers to do? [all of the above: learn about how things work, figure out what might happen, try out different designs to see how they work]

4 Science Which body parts do you need to bend your elbow? [all of the above: joints, muscles, bones]

5 BME A biomedical engineer is most likely to work on: [an artificial arm]

6 Science Why are humans able to bend their legs? [because they have joints in their knees]

7 BME A company is designing a new kind of shoe for people who have trouble walking. What would a biomedical engineer do to help? [study how people’s feet work when they have trouble walking]

8 Science Where in their bodies do people have muscles? [throughout their bodies]

9 Design-Models

You are designing a new shoe for soccer players. Which of the following is not important to do before you design your shoe? [all of these things are important to do: watch a soccer game to observe how soccer players run, learn how people’s feet are shaped so you can design a shoe that fits, interview soccer players to find out what they like and don’t like about soccer shoes] (see Figure 28)

10 Science Which of these statements is false? [joints and bones are the same thing]

11 Design-Models Someone is designing a new kind of brace to protect broken fingers. Which model would be most useful for her to use? [a model that bends like a real finger]

12 BME A biomedical engineer is most likely to do which of the following? [collect data about the human body]

13 Design-Models Someone needs to improve a knee brace design so that a knee does not move from side to side when the brace is on it. Which picture shows where he should add more support? [A, on both sides] (see Figure 29)

14 BME How might a biomedical engineer help people who hurt their legs? [design something to help their legs heal faster]

15 Science What part or parts of your body do you use to turn your wrist in a circle? [bones, joint, and muscles] (see Figure 30)

16 Design-Models Which of these things is a model? [all of these are models: a doll’s arm, map of hiking trails, a plastic skeleton] (see Figure 31)

17 Science The picture below shows the human skeleton. What does the skeleton do? [all of the above: helps the body move, supports the human body, protects the organs in the human body] (see Figure 32)

18 BME At work, what is a biomedical engineer most likely to do? [figure out new ways to do surgery]

19 Design-Models

Some people are figuring out how to design a brace for elbows. Using a model could help them: [all of the above: figure out how elbows work, test different ideas for how to make braces, figure out what shape they should make the braces] (see Figure 33)

*BME= Biomedical Engineering

Figure 30. D

Figure 32. D

Designing K

Designing K

Knee Braces

Knee Braces

Question 15

Question 17

5 Figu

7 Figu

ure 31. Desig

ure 33. Desig

gning Knee B

gning Knee B

Braces Ques

Braces Ques

stion 16

stion 19

3.4.2.2 Scale Construction: “Designing Knee Braces” Scales were constructed from a reliability sample of 148 students in seven classrooms in Massachusetts and New Hampshire who took the assessment in 2010 after completing the pilot version of the Designing Knee Braces unit.

Initially, we constructed a scale which was the sum of all correct answers on all questions that were on the final field test assessment. This scale was tested for internal reliability and was found to have a Cronbach’s alpha of .729 (n=135). We decided to form three sub-scales from the Designing Knee Braces questions: a science scale, a biomedical engineering scale, and a design-models scale. The science scale consists of all the questions that assess content common to Designing Knee Braces and most science curricula on the human body (see Table 95). This scale was used to test whether EiE students learned science content better after participating in the EiE Designing Knee Braces unit. The science scale has a Cronbach’s alpha of .552 (n=145). The biomedical engineer scale consists of all questions pertaining to the work of a biomedical engineer, and has a Cronbach’s alpha of .685 (n=144). The design-models scale consisted of the four questions about models and the two design questions, and was found to have a Cronbach’s alpha of .528 (n=138). See Table 96 below for a summary of scales and reliability.

Table 96: Summary of Reliability Results for Designing Knee Braces

Scale Questions Content Assessed Reliability (Cronbach’s α)

All 1-19 All science, engineering, design, and models questions .728

Science 2, 4, 6, 8, 10, 15, 17

Knowledge of joints, muscles and bones and their functions in the human body .552

Biomedical engineer (BME)

1, 5, 7, 12, 14, 18 The type of work that a biomedical engineer does .685

Design-Models 3, 9, 11, 13, 16, 19

Understanding of models and their uses, as well as design principles .528

3.4.2.3 Sample: “Designing Knee Braces” The full sample of collected data for Designing Knee Braces (KB) included 1036 students in 38 classrooms, grades 3-5. However, 23.4% of the sample (n=242) were excluded because they were missing a pre-assessment or a post-assessment. Additionally, 175 students were excluded due to missing demographic information and 11 students were dropped because they answered fewer than 16 questions on either the pre- or post-assessment. Thus the final dataset used for analysis included 608 students in 31 classrooms, grades 3-5.

The majority of the sample was in grade 5 (63%, n=384) (see Table 35). There were equivalent numbers of females and males in the dataset, and nearly half of the students (43%, n=264) received free or reduced-price lunch (FRL) (see Table 97). As shown in Table 98, the majority of the sample was either White (68%, n=412) or Black (12%, n=72).

Table 97. KB Proportions for Demographic Variables Gender (male) LEP FRL IEP

Proportion .50 .11 .43 .08 N 301 68 264 48

Table 98. KB Proportions for Demographic Variables – Race White Asian Black Hispanic Other Total

Proportion .68 .06 .12 .07 .07 1.0 N 412 36 72 43 45 608

Table 35. KB Proportions for Demographic Variables – Grade

Grade 3 Grade 4 Grade 5 Total Proportion .15 .22 .63 1.0 N 88 136 384 608

3.4.2.4 Results: “Designing Knee Braces” Due to the lack of a control group for the Designing Knee Braces EiE unit, we were limited to only assessing whether EiE students did significantly better on the post-assessment as compared to the pre-assessment. Additionally, we were unable to perform HLM analysis as our sample consisted of only 38 classrooms, and we deemed 40 classrooms to be the smallest sample size suitable for HLM analysis. Instead, backward stepwise multiple regression was carried out using the post-assessment scores for each subscale as outcome variables and pre-assessment scores as covariates. We centered both the pre- and the post- assessment scores around the pre-assessment grand mean, as this would allow us to (1) interpret the intercept as the average change in student score from pre-assessment to post-assessment and (2) determine if this change in score is significant.

Each of the scales described in Table 96 were used as outcome variables. These variables were computed by adding together the number of correct answers for all of the questions in each subscale.

KB: All Scale Results. The outcome variable PostAll, which measures students’ performance on the entire assessment, had a possible range of 0 to 19. Descriptive statistics for demographic groups are given in Table 99.

Table 99. KB Descriptive Statistics: Pre- and PostAll Scores by Demographic Groups

KB PreAll Score KB PostAll Score Mean SD Min Max Mean SD Min Max


Using the backward step-wise method, a significant model emerged: F (2,471) = 81.494, p < .001. The model explains 34.7% of the variance (Adjusted R2 = .347). Table 100 gives information about out the unstandardized (B) and standardized (β) regression coefficients along with the standard error (SE B), t-value, and p-value for each predictor variables that was included in the model.

Table 100. Summary of Variables Included in the KB PostAll Scale Model Variable B SE B β t-ratio p-value

(Constant) 3.457 0.129 26.864 <0.001 PreAll Score 0.448 0.030 0.517 14.825 <0.001 Hispanic -0.951 0.439 -0.072 -2.168 0.031 IEP -1.302 0.421 -0.103 -3.094 0.002 LEP -1.200 0.371 -0.111 -3.238 0.001

This model shows evidence that student scores on the assessment as a whole increase from pre-assessment to post-assessment by 3.46 points (CI 3.20, 3.72; p < .001). However, some demographic groups did not improve as much as other students: Hispanic (2.51; CI 1.59, 3.42), LEP (2.15; CI 0.88, 1.27) and students with IEP’s (2.26; CI 0.79, 1.47) all showed smaller though still positive improvement. (see Table 100 and Figure 34).

Figure 34. Change in Score Pre to Post: KB All Scale

KB: Science Scale Results. The outcome variable PostScience, which measures students’ understanding of the human body, had a possible range of 0 to 7. Questions comprising this scale can be found in Table 95. Table 101 gives descriptive statistics for demographic groups tested in our model for the Science scale.

ReferenceGroup

HispanicIEP LEP

0

0.5

1

1.5

2

2.5

3

3.5

4

Cha

nge

in S

core

(Pre

to P

ost)

Student Change in Score: KB All Scale with 95% confidence intervals

Table 101. KB Descriptive Statistics: Pre- and PostScience Scores by Demographic Groups

KB PreScience Score KB PostScience Score Mean SD Min Max Mean SD Min Max


Using the backward step-wise method, a significant model emerged: F (326) = 41.251, p < .001. The model explains 21.0% of the variance (Adjusted R2 = .210). Table 102 gives information about the unstandardized (B) and standardized (β) regression coefficients along with the standard error (SE B), t-value, and p-value for each predictor variables that was included in the model.

This model shows evidence that student scores on the Science scale increase from pre-assessment to post-assessment by 0.95 points (CI 0.81, 1.08; p < .001). Black students (0.54 points; CI 0.16, 0.93), Hispanic students (0.47 points; CI 0.00, 0.95), and those with IEP’s (0.42 points; CI 0.02, 0.82) had smaller improvements on the post-assessment than other demographic groups. All groups had confidence intervals that did not include zero or touched zero—meaning it is 95% likely that they improved significantly on this assessment after participating in EiE.

Table 102. Summary of Variables Included in the KB PostScience Scale Model Variable B SE B β t-ratio p-value

(Constant) 0.948 0.068 13.983 <0.001 PreScience 0.352 0.034 0.393 10.238 <0.001 Black -0.404 0.180 -0.083 -2.251 0.025 Hispanic -0.475 0.227 -0.077 -2.096 0.036 IEP -0.531 0.189 -0.106 -2.814 0.005

Figure 35. Change in Score Pre to Post: KB Science Scale

KB: BME Scale Results. The outcome variable PostBME, which assesses students’ understanding of the work of biomedical engineers, had a range of 0 to 6. As above, descriptive statistics for the demographic groups tested in our model can be found in Table 103.

Table 103. KB Descriptive Statistics: Pre- and PostBME Scores by Demographic Groups

KB PreBME Score KB PostBME Score Mean SD Min Max Mean SD Min Max


Using the backward step-wise method, a significant model emerged: F (183) = 55.407, p < .001. The model explains 21.2% of the variance (Adjusted R2 = .212). Table 104 shows the unstandardized (B) and standardized (β) regression coefficients along with the standard error (SE B), t-value, and p-value for each predictor variables that was included in the model.

This model shows evidence that student scores on the BME Scale increase substantially from pre-assessment to post-assessment (improvement of 1.50 points; CI 1.40, 1.59; p < .001). Again, IEP and

ReferenceGroup

Black Hispanic IEP

0

0.5

1

1.5

Cha

nge

in S

core

(Pre

to P

ost)

Student Change in Score: KB Sci Scale with 95% confidence intervals

LEP students had smaller improvements in their post-assessment scores than students in other demographic groups, although all students improved significantly (see Figure 36).

Table 104. Summary of Variables Included in the KB PostBME Scale Model Variable B SE B β t-ratio p-value

(Constant) 1.496 0.047 31.862 <0.001 PreBME Score 0.256 0.026 0.363 9.707 <0.001 IEP -0.647 0.160 -0.148 -4.050 <0.001 LEP -0.538 0.139 -0.143 -3.882 <0.001

Figure 36. Change in Score Pre to Post: KB BME Scale

KB: Design-Models Scale Results. The outcome variable PostModels, which assesses students’ understanding of design factors in biomedical engineering, as well as the use of models, had a range of 0 to 6—see Table 95 for a complete listing of questions for this scale. The means scores and other descriptive statistics are shown in Table 105 for all demographic groups tested during the modeling process.

ReferenceGroup

IEPLEP

0

0.5

1

1.5

2

Cha

nge

in S

core

(Pre

to P

ost)

Student Change in Score: KB BME Scale with 95% confidence intervals

Table 105. KB Descriptive Statistics Pre- and PostModels Scores by Demographic Groups

KB PreModels Score KB PostModels Score Mean SD Min Max Mean SD Min Max


Using the backward step-wise method, a significant model emerged: F (325) = 46.488, p < .001. The model explains 23.1% of the variance (Adjusted R2 = .231). Table 106 gives information about out the unstandardized (B) and standardized (β) regression coefficients along with the standard error (SE B), t-value, and p-value for each predictor variables that was included in the model.

This model shows evidence that student scores on the Design-Models Scale increase from pre-assessment to post-assessment of 1.09 points (CI 0.97, 1.22; p < .001). Again, Hispanic (0.57 points; CI 0.13, 1.01; p = .014), IEP (0.62 points; CI 0.20, 1.04; p = .019) and LEP (0.57 points; CI 0.20, 0.93; p = .003) students had smaller improvements in their post-assessment scores than students in other demographic groups (see Figure 37).

Table 106. Summary of Variables Included in the KB PostModels Scale Model Variable B SE B β t-ratio p-value

(Constant) 1.093 0.061 17.797 <0.001 PreModels Score 0.433 0.038 0.421 11.487 <0.001 IEP -0.475 0.201 -0.085 -2.359 0.019 LEP -0.525 0.173 -0.110 -3.030 0.003 Hispanic -0.519 0.211 -0.088 -2.465 0.014

Figure 37. Change in Score Pre to Post: KB Design-Models Scale

3.4.2.5 Summary: “Designing Knee Braces” Summative Evaluation Our analysis suggests that students learning through a combination of EiE’s Designing Knee Braces unit and their science unit on the human body have significantly improved scores on assessments of their science and biomedical engineering knowledge. The students especially improved in their knowledge of the work of a biomedical engineer, with a base improvement of nearly 1.5 points out of a possible 6. However, LEP students improved less than others on the BME and Design-Models scales, Hispanic students improved less on the Science and Design-Models scales, Black students improved less on the Science scale, and IEP students had consistently lower improvement on all of the scales. Still, all demographic groups did significantly better on the post-assessment than on the pre-assessment.

3.5 Results for the “Designing Lighting Systems” Unit Evaluation In the EiE unit Lighten Up: Designing Lighting Systems, students apply what they are learning about the behavior and properties of light in their science curriculum to the field of optical engineering. In Lesson 1, students read the storybook Omar’s Time to Shine, set in Egypt, which introduces them to the field of optical engineering, some properties of light, and the engineering design process. In Lesson 2, students gain a broader understanding of the field of optical engineering by investigating how light interacts with different materials. In Lesson 3, students ask questions about how light travels and is reflected. They deduce the law of reflection and learn that the intensity of light decreases with distance. They are also introduced to the Light Intensity Meter, a tool that they will use in the final lesson. In Lesson 4, students use the engineering design process to design and test a lighting system inside a model tomb. Applying what they learned in previous lessons, teams create, evaluate and improve their lighting systems to try to light up the greatest number of vulture hieroglyphs with the highest possible intensity of light.

Lighten Up: Designing Lighting Systems was field tested during the 2010-2011 school year.

3.5.1 Formative Evaluation: Designing Lighting Systems Feedback forms for Designing Lighting Systems were completed by 48 teachers who taught the unit in their classroom. Five states were represented in the sample: California, Colorado, Massachusetts, Minnesota and North Carolina. Grades 3 through 5 are represented along with one mixed grade classroom with some Grade 2 students.

ReferenceGroup

IEP LEP Hispanic

0

0.5

1

1.5

Cha

nge

in S

core

(Pre

to P

ost)

Student Change in Score: KB Design-Models Scale with 95% confidence intervals

Table 107. Designing Lighting Systems– Classroom Grade and State Distribution

State Grade 3 Grade 4 Grade 5 Grade Mixed* Total**

CA 2 7 2 11 CO 3 1 3 7 MA 4 3 12 2 21 MN 12 10 2 24 NC 8 8 Total 29 11 27 4 71

*In MA there was one mixed grade 3/4 class and one mixed grade 4/5 class. In MN there was one mixed grade 2-5 class and one mixed grade 3-5 class.

**The total number of classrooms is larger than the number of teachers responding, because some teachers taught multiple classrooms.

3.5.1.1 Overall Feedback for “Designing Lighting Systems” Unit Teachers were asked to rate the unit overall by answering questions on a scale from 1-7. Table 108 shows the number of the responses (N), the average response, the standard deviation (SD), and the minimum (Min) and maximum (Max) response to the overall unit, based on a seven point scale, with 1 representing “Not at all”, 3 representing “Slightly”, 5 representing “Moderately”, and 7 representing “Very”.

Table 108. Designing Lighting Systems– Unit Ratings


Did this unit further your objectives for science in your classroom? 48 5.8 1.79 1 7 Did this unit further your objectives for engineering? 47 6.1 1.23 3 7 Did this unit further your objectives for another content area in your classroom? 44 4.3 1.97 1 7



Were the Lesson Plans easy to understand? 48 5.3 1.52 1 7 Was this unit reasonably easy to implement and manage? 48 4.6 1.66 1 7 If you used a kit, were sufficient materials provided? 48 6.7 0.55 5 7 Teachers were asked whether this unit furthered their objectives for another content area beyond science, and if so, to specify which content area. In a separate question, teachers were also asked if they integrated the teaching of this unit with their teaching of other subjects, and if so, to explain how. All but one of the 48 teachers responded at least one these questions. Three-quarters of teachers (75%, n=36) mentioned specific non-science content areas that were enhanced by the unit, with language arts being mentioned most often (72%, n=26). Approximately one-third of teachers who mentioned specific content areas also reported that the unit furthered their objectives in math (33%, n=12) and almost one-quarter (22%, n=8) mentioned social studies.

Teachers were asked a number of open-ended questions about the unit and lessons. Forty of the forty-eight teachers responded to the question, “How did your students benefit, academically or otherwise,

from taking part in this unit?” The coding categories used to code this question are shown in Table 109. Nearly two-thirds of the respondents (62.5%, n=25) noted that the unit helped to further their students’ understanding of STEM knowledge and skills. Specifically, teachers noted that their students learned about light, technology, teamwork, and the engineering design process.

“This unit assisted in providing curriculum and motivation for both the topics of light and of engineering. They enjoyed the challenge of the final lesson and learning about the engineering design process.” – Minnesota teacher, grade 3

“We all learned together! Academically, there was a vast amount of growth in all my students. My third graders came to me with almost no science background. We spent a long time just a learning all the vocabulary for the unit. It was a learning experience they will remember.” – Massachusetts teacher, grade 3

“They learned so much about light AND technology (they knew very little about this), by investing time into a project of their own creation, than they did when I taught the unit on light - even though I had infused my teaching with hands-on labs.” – Massachusetts teacher, grade 5

Table 109. LS – Categories of Teacher Responses to the Open-Ended Question, “How did your students benefit, academically or otherwise, from taking part in this unit?”

Coding Category Number of Teachers


(N=40) Students practiced discussion, communication, and teamwork skills 10 25.0 Students practiced problem solving and critical thinking skills 4 10.0 Students had opportunities to learn/apply STEM content and/or skills 25 62.5 Lesson makes non-STEM cross-disciplinary and multicultural connections 4 10.0 Students participated in hands-on activities and experiments 3 7.5 Students had fun, were engaged and were motivated 10 25.0 Other Comments 11 27.5 Total Number of Comments* 47 100


Additionally, one-quarter of teachers (25%, n=10) reported their students had fun and were engaged and motivated.

“I felt students were very engaged in the entire unit. Learning takes on a practical application as the students are able to apply what they've learned in an engineering challenge.” – Minnesota teacher, grade 5

“Students were very engaged and used vocabulary that was not dumbed-down. They performed as a group very well and had to manage materials and other kids - they did this!” – Minnesota teacher, grade 3

“They loved it. This is a fourth grade Excel class and the students seemed to appreciate and love learning about light.” – California teacher, grade 4

One-quarter of the teachers (25%, n=10) reported that students practiced their discussion, communication and teamwork skills.

“They learned more about how to work together, plan and engineer a system that could reflect light. Students also learned a lot of the vocabulary for light.” – North Carolina teacher, grade 3

“We joined a third and fourth grade classroom to complete this unit - cooperation (social and academic benefits) working as teams - cooperation, listening skills, writing skills, science/engineering, math.” – Massachusetts teacher, grade 3

“This unit was a comprehensive, well-prepared unit; it was a great challenging project to finally synthesize their own “tomb” of ancient Egyptian hieroglyphics. The vocabulary of Omar's Time to Shine helps my students academically. Working in the three-person groups really built the teamwork skills, even with one group who historically butt heads a lot.” – California teacher, grade 4

A few teachers reported that students practiced problem solving and critical thinking skills (10%, n=4), had opportunities to improve in other areas (10%, n=4) or had good hands-on experiences (7.5%, n=3).

“My students were able to experience guided inquiry which allowed for extensive questioning and deductive thinking skills. Also, they were challenged with an appropriate design challenge. Students improved communication and collaboration skills.” – Minnesota teacher, grade 3

“Cooperative group work; Learned about the field of engineering; Learned the five steps of EDP; Problem-solved; Applied their knowledge; Connection between math and science.” – Massachusetts teacher, grade 5

“Yes. They learned science concepts in a hands-on, engaging way. (They were introduced to optical engineering for the first time.) I want to see if the students learned and retained the standards we were working on.” – Minnesota teacher, grade 3

Teachers were also asked, on a scale of 1-7, “What is the likelihood that you will choose to teach this unit again in your classroom?” Forty-six of 48 teachers provided a rating with the average score being 5.5 (SD=1.68) out of a possible 7 points. The teachers were also asked to explain their rating. Thirty-five teachers responded to this question and the most common reasons for choosing to teach the unit again included that it enhanced curriculum or correlated with other content areas (17%, n=6).

“I am required to teach a light unit. This engineering unit offers a way to meet state engineering standards that build on the light unit.” – Minnesota teacher, grade 3

“This unit worked great to enhance students' background knowledge in light.” – Colorado teacher, grade 5

Only four teachers (11%) commented that they would not teach the unit again. Of these four teachers, one has retired but still wants to use it as a basis for preparing future lessons. The other three said the unit was too advanced for their class and that they were limited by the amount of time they had. In addition, one of these teachers said they felt “overwhelmed” by the unit.

3.5.1.2 Lesson 1 of “Designing Lighting Systems” In lesson 1, students read the storybook Omar’s Time to Shine. In the story, Omar gets the chance to watch a film crew taping a documentary at the tomb of Ramses IX. While on location, Omar learns about the lighting system used by the crew, how light moves in straight lines until it hits another object or moves from one medium to another, and how light can be reflected by some materials and absorbed by others. Later, in a discussion with his brother Zane, an optical engineer, Omar also learns that the intensity of light is affected by the distance over which it travels. All of Omar’s knowledge about light is called upon when there is a brownout before the performance of a play at school. Through the story, students learn about science concepts including the properties of light such as reflection, absorption, and the change of intensity with distance. They also learn about the work of optical engineers.

Forty-seven of 48 teachers reported teaching this lesson, 44 of whom reported how much time they spent teaching it and rated the lesson quality. As shown in Table 110, the teachers spent an average of 101 minutes teaching this lesson. When asked to rate the quality of the lesson on a scale of 1-7, teachers gave Lesson 1 an average score of 5.8.

Table 110. Designing Lighting Systems Feedback – Lesson 1 Question N Mean SD Min Max


Table 111. LS – Categories of Teacher Responses to the Open-Ended Question,




(N=36)

Positive

Students made connections to the real world 4 11.1 Students had fun, were engaged and were motivated 21 58.3 Activities and / or supporting materials of high quality 4 11.1 Good foundation and/or preparation for future lessons 8 22.2 Other positive comments 6 16.7 Total Positive Comments* 33 91.7

Negative Students were not engaged/activities too easy or boring 3 8.3 Other negative comments 4 11.1 Total Negative Comments 7 19.4


Thirty-six of the 47 teachers who taught Lesson 1 provided an explanation for their rating of lesson quality. Table 111 shows coding categories used to code this question. Almost all teachers responded positively to the unit with ninety-two percent of teachers (92%, n=33) providing at least one positive comment. About one-fifth of the responding teachers (19%, n=7) gave a negative response.

As shown in Table 111, the most common criticism focused on the students finding the story boring or not engaging (8%, n=3).

“The interest of students was not very high. The story dealing with the dance didn't hold interest at the story's beginning.” – Colorado teacher, grade 5

“The reading drug on... hard for students to stay attentive to material.” – Minnesota teacher, grade 3

However, 58% of teachers (n=21) stated that their students enjoyed the lesson and were engaged. “The story was interesting and relevant to the students. They have all been in a school performance and were able to relate to Omar. They also had just finished doing Egyptian art during art class. The book showed the design process being used by an optical engineer AND by a child their age.” – Minnesota teacher, grade 5

“Students loved the book and were able to reference Omar and his experiences throughout the book to what they were trying to do within their own groups.” – Massachusetts teacher, grade 5

“Students enjoyed making connections between their knowledge of science concepts and the concepts in the story.” – Minnesota teacher, grade 3

Eight teachers (22%) mentioned that the activities in this lesson provided a good foundation for future lessons.

“I loved the story. What a great review of light concepts and a perfect introduction to the challenge ahead.” – Minnesota teacher, grade 3

“The book was a great tool to help the kids with background knowledge.” – North Carolina teacher, grade 3

A few teachers praised the quality of the activity or supporting materials (11%, n=4) and some indicated that they saw their students make connections to the real world (11%, n=4).

“I found the reading and the DM [Duplication Master] to be worthwhile and a great way to introduce the unit.” – Minnesota teacher, grade 3

“I felt the engineering process was addressed, vocabulary was excellent, hands-on activities were understandable and engaging.” – Minnesota teacher, grade 3

“The story was interesting. Kids could make connections between performance and relate to Omar. Also our Art Department had just completed a unit on Egyptian art.” – Minnesota teacher, grade 5

“Children in my school take part in dances/plays at “celebration” times and could easily identify with Omar and his unimportant role in the dance.” – Massachusetts teacher, grades 3-4

3.5.1.3 Lesson 2 of “Designing Lighting Systems” In Lesson 2, students are introduced to the field of optical engineering by observing what happens when they shine light on different materials. Working in groups, they collect data about whether (and how) light travels through the material, bounces off of the material, or is stopped by the material. All materials interact with light in more than one of these ways, so students may make multiple observations. Students are also formally introduced to the terms “transmit,” “reflect,” and “absorb,” as they relates to light. Students notice that they are able to observe many of the materials interacting with light in multiple ways.

Forty-six of 48 teachers reported on Lesson 2. As shown in Table 112, teachers spent an average of 99 minutes teaching the lesson and gave it an overall rating of 6.0 on a scale of 1-7.






of teachers


(N=38)

Positive

Students practiced discussion, communication, and teamwork skills 9 23.7 Lesson made STEM cross-disciplinary connections 4 10.5 Students had opportunities to learn/apply STEM content and/or skills 7 18.4 Students participated in hands-on activities and experiments 9 23.7 Students had fun, were engaged and were motivated 11 28.9 Activities and / or supporting materials of high quality 7 18.4 Other positive comments 5 13.2 Total Positive Comments* 31 81.6

Negative



Thirty-eight teachers provided an explanation for their rating of lesson quality. Table 113 shows coding categories used to code this question. In general, teachers were positive about the unit with most of the responding teachers (82%, n=31) providing at least one positive comment and less than half (42%, n=16) giving a negative comment.

The main criticisms of this lesson (see Table 113) were that it was difficult or set up or implement (13%, n=5) or that it was difficult or confusing to implement or understand (11%, n=4). Much of the criticism revolved around the difficulty of setting up the paper walls or the confusing nature of light.

“Students enjoyed and learned as they tested each material. It was very easy for them to see. The wall needs improvement though - they are too flimsy for young, wiggly students.” – Minnesota teacher, grade 3

“The testing station was hard to monitor with each group. I found it hard to do and ended up doing one station because it was easier to manage the student's behavior. (I have a rough group).” – North Carolina teacher, grades 3

“I think it is important to address the difference between observation/inference. Also, students struggled with identifying refracting light. I think it is also important to address the fact that all materials reflect some light. This is what they learned in FOSS, and felt constricted. Students also struggled to connect with Optical Engineering Examples. The engagement was low.” – North Carolina teacher, grade 4

“Content was excellent but needed lots of support for third graders to be successful.” – Minnesota teacher, grade 3

However, 29% of responding teachers (n=11) felt that their students found Lesson 2 to be fun, engaging, and motivating (again, see Table 113).

“Kids really enjoyed testing the materials. They were very engaged during lesson and at the end had lots of ideas how optical engineers might use this information.” – Minnesota teacher, grade 5

“Very engaging for the students, they came up with many ideas and were surprised by all of the designs.” – Minnesota teacher, grade 3

“This one was awesome, hands-on investigations for the students. They enjoyed being able to test different materials.” – Minnesota teacher, grade 3

Additionally, about one-quarter of teachers (24%, n=9) reported that the activity promoted discussion, communication, and teamwork while the same number (24%, n=9) appreciated the hands-on nature of the activities.

“Very good experiment for students to complete. Elicited very good discussions.” – Minnesota teacher, grade 2-5

“The activity was easy for children to see and it was great kids did not test all materials as they relied on their classmates to explain what they found out.” – Massachusetts teacher, grade 5

“The goal of the lesson was met through the hands-on investigation. We had students test their three materials and record the information. Then the students rotated to the other tables to experiment with all materials.” – Minnesota teacher, grade 5

“Great hands-on experience and opportunity for peer to peer discussion. The students were really excited to do this activity and share their findings. Also a great prep for the tasks in Lessons 3 and 4.” – California teacher, grade 4

Additionally, a few teachers mentioned that this lesson helped their students to enhance their STEM knowledge and skills (18%, n=7) and make connections to STEM subjects (11%, n=4).

“Students had the opportunity to see how light interacted with different materials and this helped them further see that some materials can react to light in multiple ways. It also helped cement the concepts of refraction, reflection, transmission, and absorption.” – Massachusetts teacher, grade 5

“The enjoyed seeing how the light would reflect, absorb, or refract. They did not have trouble describing how the light reacted to the materials.” – North Carolina teacher, grade 3

“Students mastered the concepts in two 45 minute sessions.” – California teacher, grade 3

3.5.1.4 Lesson 3 of “Designing Lighting Systems” In Lesson 3, students are introduced to their design challenge: lighting up hieroglyphs in a model tomb with mirrors. Groups of students use hands-on exploration to explore how light travels in straight lines and experiment with mirrors to determine the law of reflection. They also explore the relationship between the distance of an object from a light source and the intensity of light on the object and discuss how their findings might influence their lighting system design.

Forty-seven of the 48 teachers reported that they taught Lesson 3. Teachers spent an average of 150.1 minutes teaching this lesson and gave it an average overall rating of 5.2 on a scale of 1-7 (see Table 114).



Table 115. LS – Categories of Teacher Responses for the Question,


Lesson made STEM cross-disciplinary connections 2 5.6

Positive Students had fun, were engaged and were motivated 7 19.4 Other positive comments 13 36.1 Total Positive Comments* 19 52.8

Negative

Activities / lessons too difficult or confusing for students or teacher 13 36.1 Time constraints / takes too much time 7 19.4 Criticism of supporting materials / difficult to implement 10 27.8 Other negative comments 6 16.7 Total Negative Comments* 25 69.4


Thirty-six teachers commented on their rating of lesson quality. The coding categories used to code this question are shown in Table 115. While the numerical rating was positive, teachers comments were generally negative about the unit with more than two-thirds (69%, n=25) providing at least one negative comment, while half (53%, n=19) gave at least one positive response.

As shown in Table 115, over one third of teachers (36%, n=13) stated that the lesson was too difficult for their students. Many of the difficulties focused on measuring either the path or the intensity of light. Since flashlights were used instead of lasers (for safety reasons), the path of the light beam spread out


of teachers


(N=36)

and made angles hard to measure. Also, four teachers stated that the light intensity meters were hard to use. Some students were confused by the concept of “angle of incidence.”

“It was difficult for students to accurately draw the angle of incidence and reflection. It was hard for them to place the ruler on the beam of light for two reasons - the beams of light were thicker than the line made from the pencil which led to them having trouble tracing the line. When we were able to look at all the class data on the class data pad I was able to point out the similarities in the protractor measurements. I folded paper and held up to the window to help them see it better.” – Massachusetts teacher, grade 5

“Light intensity meters difficult to use. Not consistent readings between groups.” – Massachusetts teacher, grade 4

“The students had a difficult time understanding the two kinds of angles (Incidence and Reflection).” – North Carolina teacher, grade 3

Additionally, 28% of the teachers (n=10) had criticisms of the supporting materials or thought the lesson was difficult to implement. Again much of the criticism centered on the wide angle of light from the flashlights making it difficult to trace and measure angles.

“The light reflection was a good lesson but due to the type of flashlight used it was difficult for the children to see that both angles were the same. The light was too wide from the flashlights.” – Minnesota teacher, grade 3

“The lesson itself was good - but it was very confusing to set up. I also think it was much too long and needs to be simplified - or broken down into two lessons.” – Massachusets teacher, grade 5

“Time is of the essence. Next year, I'd definitely spend more time on preparation and prepare the Light Path cards. That took my third graders just about the entire science block. It shouldn't have, but it did. Again, because I taught this unit during August and October, I think it was a lot for some of them with such little knowledge in science concepts. They definitely need more demonstrations.” – Massachusetts teacher, grade 3

Almost one-fifth of teachers (19%, n=7) also mentioned that the lesson was too time consuming. “More than my students need to know for MCAS on light. Due to time constraints, didn't find angle of incidence overly valuable. Not sure kids got much out of it.” – Massachusetts teacher, grade 5

“SO MUCH in this lesson - a bit overwhelming to pack it all in - perhaps break it into three actual lessons, especially for lower grades: 1) light path, 2) angle of reflection, 3) light intensity.” – California teacher, grade 3

The main positive comment for this lesson was from 19% of teachers (n=7) who thought it was fun or engaging for their students.

“I felt that this activity was a little confusing. The angle of reflection went well. The kids really liked measuring the intensity of light.” – Minnesota teacher, grade 5

“The children were very highly motivated.” – Massachusetts teacher, grade 5

“Kids loved it!” – Colorado teacher, grade 4

3.5.1.5 Lesson 4 of “Designing Lighting Systems” In the final lesson, Lesson 4, students apply what they learned in previous lessons to the design of their lighting system. Using the engineering design process, students imagine, plan, create, evaluate, and improve their designs based on the established design challenge criteria. Finally, students prepare to be interviewed about their lighting systems by answering a series of interview questions.

Forty-six of the 48 teachers reported that they taught this lesson in their classroom (see Table 116). On average, teachers spent 159.6 minutes teaching this lesson and gave it an overall rating of 6.1 on a scale of 1-7.

Table 116. Designing Lighting Systems – Lesson 4 Question N Mean SD Min Max





of teachers


(N=33)

Positive

Students practiced discussion, communication, and teamwork skills 4 12.1 Students had fun, were engaged and were motivated 7 21.2 Students had opportunities to learn/apply STEM content and/or skills 8 24.2

Students were challenged 4 12.1 High quality activities / supporting materials. 11 33.3 Other positive comments 6 18.2 Total Positive Comments* 27 81.8

Negative

Activities/lessons too difficult or confusing for students and/or teacher 6 18.2

Time constraints / takes too much time 5 15.2 Other negative comments 3 9.1 Total Negative Comments* 12 36.4


Thirty-three teachers commented on their rating of lesson quality. Table 117 shows coding categories used to code this question. Overall, teachers responded positively about the lesson with the majority of teachers (82%, n=27) providing at least one positive comment. One-third of teachers (36%, n=12) gave at least one negative comment.

As shown in Table 117, negative comments most often mentioned that the lesson was too difficult or confusing (18%, n=6) and that it took too much time (15%, n=5).

“I felt it was a little too detail-oriented for third graders. Children in fourth grade initially wanted a lot of mirrors but cost put a cap on that impulse.” – Massachusetts teacher, grade 3/4

“Giving them their tomb and a baggie of what they could use made them want to start creating. I'm not sure they had enough practice angling mirrors and shining beams of light on objects - Only two of eight groups managed to get a beam of light to reflect off a mirror and hit vulture.” – California teacher, grade 3

“The experimentation and discovery portion of this lesson is excellent. My only issue/constraint is the time needed to really let the students learn through problem solving. I think this is very powerful, but sometimes finding the time in an otherwise packed schedule is difficult.” – California teacher, grade 4

“We ran out of time to reflect on their lighting systems. It was difficult for them to be accurate with intensity ratings. Children are still learning how to work cooperatively and some tears were shed in the process (not a problem of the lesson).” – Minnesota teacher, grades 3-5

However, one-third of teachers (33%, n=11) said that the activities and/or supporting materials were of high quality.

“Supplies are good and students were able to see exactly what they were to do and had enough background to make good choices.” – Minnesota teacher, grade 3-5

“The students loved this. They could have worked for days redesigning their system. It was unfortunate that I couldn’t give more time to the project. The holiday break needed to be the cut off for our unit.” – Minnesota teacher, grade 4

Around a quarter of the teachers commented on how fun and engaging it was for the students (21%, n=7) or how it enhanced their STEM knowledge and skills (24%, n=8).

“The students were engaged. We could see during their 5-step process that they were thinking about and implemented what had been taught during the previous lessons.” – Minnesota teacher, grade 5

“My students were very engaged during these three days. They worked cooperatively and discussed changes that they needed to make. This was their favorite part of the EiE lessons.” – North Carolina teacher, grade 3

“This design process was very thought provoking and required a lot of planning as well as incorporating many science concepts for success.” – Colorado teacher, grade 5

“Students actively involved in team work, improving their plan and understanding and justifying their reasons.” – Massachusetts teacher, grade 3

Additionally, a few teachers mentioned that the lesson positively challenged their students (12%, n=4) and allowed them practice with discussion, communication skills and teamwork (12%, n=4).

“Many possibilities to solve the design challenge allowed for creativity and pushed the students to reconsider. They never tired of improving which did not happen with the geothermal unit.” – Minnesota teacher, grade 5

“Students actively involved in team work, improving their plan and understanding and justifying their reasons.” – Massachusetts teacher, grade 3

“High motivation. Hands-on. Really great discussions an co-op work.” – Massachusetts teacher, grade 5

3.5.1.6 Summary: “Designing Lighting Systems” Formative Evaluation Overall, teachers found this unit to be fun, engaging and motivating for their students. Many commented that Designing Lighting Systems had high quality activities and supporting materials. Teachers also were pleased to see their students getting the opportunity to work on their discussion, communications and teamwork skills. On the other hand, some teachers found this unit too difficult or confusing for their students. This is not unexpected in a unit covering light since the science concepts of “angle of incidence” and the “change of intensity with distance” can be tricky even beyond elementary school. Teachers often commented that they felt like they needed more time to teach these lessons but many felt it was time well spent.

3.5.2 Summative Evaluation: Designing Lighting Systems 3.5.2.1 Assessment Design: “Designing Lighting Systems” The Designing Lighting Systems student assessment was designed and first tested during the 2009-2010 school year. Thirty-eight multiple-choice questions were chosen for the pilot assessment. In the spring of 2010, validity evidence was gathered for all questions from the Designing Lighting Systems assessment. EiE staff conducted cognitive interviews with children in grades 3-5 from five Massachusetts classrooms, asking students to read each question aloud, explain what they thought the answer would be, read all answer choices aloud, and explain which answer they would choose and why.

Table 118. Designing Lighting Systems Assessment Questions (Text)

Q# Scale* Question Text

1 BML On which of the objects will the light be BRIGHTEST? [the books] (see Figure 38) 2 BML On which of the objects will the light be LEAST BRIGHT? [the truck] (see )

3 BML In which diagram will light from the flashlight be BRIGHTEST in the corner marked by the X? [diagram B] (see Figure 39)

4 ED A student is trying to design a way to use a spotlight to light up two paintings. What should the student think about? [all of the above: the size of the paintings, the angle between the spotlight and the paintings, how far from the paintings she should put the spotlight]

5 EW At work an optical engineer is MOST LIKELY to: [improve lenses for a telescope]

6 BML A flashlight is turned on. What happens? [light moves from the flashlight in straight lines until it hits another object]

7 PL Which of the following objects reflects light? [all of the above: blue teapot, white wall, black wall]8 EW At work, an optical engineer might: [improve camera lenses]

9 ED A student is trying to use a spotlight to light up a flag. The light is not bright enough. What can she do to make the light on the flag brighter? [move the spotlight closer to the flag]

10 EW Which of the following would an optical engineer MOST LIKELY work on? [something hat helps you see things better]

11 PL Which of the following will transmit light? [glass window] 12 ED What would be BEST to use to get light to shine around a corner? [aluminum foil]

13 PL Which picture BEST shows what happens to light when it shines on a mirror? [picture C] (see Figure 40)

14 ED Someone is improving a lighting system for a playground. If he makes the streetlights taller, what will happen to the light on the playground? [the light on the playground will be less bright]

15 PL The picture below shows a girl standing in the sunlight. Which of the following best explains why she creates a shadow? [she absorbs light from the sun] (see Figure 41)

16 PL What happens if you shine light on a window? [some of the light goes through the window and some bounces off of the window]

17 BML What happens when someone turns on a light bulb in a room? [light travels in straight lines from the light bulb out in all directions]

18 EW What might an optical engineer thing about for his or her job? [how light gets from one place to another]

19 EW What is an optical engineer LEAST LIKELY to work on for her job? [lights on vehicles]

20 ED Where should you hold a flashlight so that it shines on the mirror and then on Position A? [position E] (see Figure 42)

21 PL If you shine a flashlight into the mirror from Position D, where would the reflected light appear? [position B] (see Figure 38)

22 ED

A student is designing a way to light a dark corner of her bedroom. The light is on the opposite side of the room from the dark corner. What could she do to light up the dark corner? [any of these ideas would work: cover the walls with shiny wallpaper, move the light closer to the middle of the room, paint the walls white to reflect more light to all parts of the room] (see Figure 43)

23 PL If an object reflects light: [it must also absorb light]

24 PL Which of these items will absorb light? [all of these items absorb some light: a mirror, a clear block of plastic, a sheet of metal painted black]

25 ED You use some mirrors and a flashlight to light up a painting in a room. Which of these set-ups will make the light on the painting the brightest? [set-up A] (see Figure 44)

*BML=Brightness & Movement of Light; ED=Engineering Design; EW=Engineering Work; PL=Properties of Light

Figure 40. DDesigning LQuestion

Figure 38. D

Figure 39

Lighting Syst13

Designing Li

9. Designing

tems

ighting Syste

g Lighting Sy

ems Question

ystems Quest

Figure 4

ns 1 & 2

tion 3

41. DesigninQuesti

ng Lighting Sion 15

Systems

Figure 4

F

43. DesigninQuesti

Figure 42. De

ng Lighting Sion 22

esigning Lig

Systems

ghting Systemms Questions

Figure 44. D

s 20 & 21

Designing LQuestion

Lighting Syst25

tems

During the summer of 2010, a number of questions were dropped or revised based on the results of the pilot study and validity testing to create a new field test assessment. In 2010-2011 it was used for field testing in six states (California, Colorado, Massachusetts, Minnesota, North Carolina, and New Hampshire). The field test version included 25 multiple-choice questions.

On the identical pre- and post-assessments students were asked science questions about properties of light including reflection, transmission, absorption, brightness and movement. They were also asked questions that posed engineering design challenges or asked about the work of an optical engineer. Table 118 describes the text for the questions with the correct answer shown in brackets.

3.5.2.2 Scale Construction: “Designing Lighting Systems” The reliability sample consisted of 178 students in eight pilot classrooms who had completed the assessment. Initially, we constructed a raw score which was the sum of all correct answers on all 25 questions on the final field test assessment. This score was tested for internal reliability and was found to have a Cronbach’s α of .827 (N = 178).

We decided to form four sub-scales from the Designing Lighting Systems assessment questions based on the content they addressed (see Table 119). The Optical Engineer (OE) scale consists of all of the questions that assess students’ understanding of the work of optical engineers. The Brightness and Movement of Light (BML) scale assesses content related to the basic motion of light as well as how distance is related to the brightness of light. The Properties of Light (PL) consists of questions related to properties of light such as reflection, transmission, and absorption. Finally, the Engineering Design (ED) scale consists of questions that ask students to solve an engineering problem or to improve an existing design.

Table 119. Summary of LS Scales

Scale N of Students

# Items Questions Content Assessed Reliability:

Cronbach’s α All 178 25 1-25 All questions .827

OE Optical Engineer (OE) 178 5 5, 8, 10, 18, 19 Work of optical engineers .625

Brightness and Movement of Light

(BML) 178 5 1, 2, 3, 6, 17 Motion and brightness of light .656

Properties of Light (PL) 178 8 7, 11, 13, 15,

16, 21, 23, 24 Properties of light: reflection,

transmission, absorption .584

Engineering Design (ED) 178 7 4, 9, 12, 14, 20,

22, 25 Solving an engineering problem,

improving an existing design .519

All subscales were analyzed using Principal Component Analysis with Oblimin rotation. Only one factor was extracted from the EW and BML scales. Three factors were extracted from both the PL and ED scales. As none of the factors for either consisted of more than three items, we did not attempt to divide these scales further.

3.5.2.3 Sample: “Designing Lighting Systems” Designing Lighting Systems (LS) assessments were collected from students in grades 3 through 5. The full sample of data collected from grades 3-5 included 1560 students in 65 classrooms. However, 14.2% of the sample (n=222) was excluded because they were missing a pre-assessment or a post-assessment. Additionally, one class was dropped due to missing demographic information. Thirty-four classes were

missing information on which students received national free or reduced-price lunch, so that variable was not considered in the analysis of this unit. Fifty-four students answered fewer than 22 questions on either the pre-assessment or the post-assessment and were dropped from the dataset. Four classrooms were dropped from the analysis because the teacher did not provide complete information about the time they spent teaching the unit.

The final dataset used for analysis included 1140 students (Level-1 units) in 60 classrooms (Level-2 units), with an average classroom (cluster) size of 23.3 students per classroom, and a standard deviation of 5.18 students. The classrooms were spread over 33 schools with 41 teachers total, 6 of which were science specialists. The teachers had an average of 12.2 years of experience, with a standard deviation of 7.15 years, a minimum of 0 years and a maximum of 34.

The majority of the sample consisted of students in grade 3 and grade 5 (see Table 120). There were roughly equal proportions of males and females. Ten percent of the sample consisted of students with limited English proficiency (LEP) and students in Individualized Education Programs (IEP) made up 8% of the sample (see Table 121). White students made more than two-thirds (69%) of the sample (see Table 122).

Table 120. LS Grade Distribution (N of Students and Classrooms)

Grade 3 4 5 Total

N (students) 421 230 489 1140 N (classrooms) 23 11 26 60

Table 121. LS Proportions for Level-1 Dichotomous Variables Gender (male) LEP IEP

Proportion .49 .10 .08 N 562 115 87

Table 122. LS Proportions for Level-1 Variables – Race


3.5.2.4 Results: “Designing Lighting Systems” Four outcome variables were used for analysis. Two engineering outcome variables, the Optical Engineering scale (PostOE) and Engineering Design scale (PostED) were used to gauge students’ improvements in understanding what optical engineers do and their ability to solve engineering problems, respectively. Two science outcome variables were the Properties of Light scale (PostPL), which examines students’ understanding of reflection, transmission, and absorption of light, and the Brightness and Movement of Light scale (PostBML), which looks at students’ understanding of how light moves and how distance can affect brightness.

For this unit, we tested the student level (level-1) variables of IEP and LEP status, as well as Gender and race (Asian, Black, Hispanic, and Other). We also included pre-assessment scores as a covariate. Next, we tested whether each level-1 variable was best modeled as a fixed or random coefficient, the default being a fixed coefficient, unless the random variance was found to be significant (p < .05). At the classroom level (level-2) we tested the effects of variables on the outcome, including the classroom

means on the pre-assessment scores, class size, the number of minutes spent teaching the EiE unit (TotalMinutes, Lesson4Minutes, etc.), the number of EiE units the teacher had taught prior to that school year (PriorUnits), the number of units the teacher taught during that school year (CurrentUnits), whether the teacher taught the corresponding science unit (TaughtScience), and the number of years the teacher had been teaching overall (NumYearsTeaching) as covariates. We also tested the effect of grade and of teacher specialty (ScienceSpecialist) here, as well as the effect of the overall classroom setting (the proportions of IEP students, LEP students, and male students in the classroom).

Engineering outcomes were tested using two outcome variables, PostOE and PostED, for students in grades 3 to 5. Scores for each outcome were computed by adding together the number of correct answers for each of the questions on each scale as listed in Table 118.

LS: Optical Engineering (OE) Scale Results. The OE scale had a possible range of 0 to 5. Mean pre- and post-assessment scores for the various demographic groups tested on the OE scale are shown in Table 123 below.

Table 123. LS Descriptive Statistics: Pre- and PostOE Scores by Demographic Groups

LS PreOpticalEngineering Score LS PostOpticalEngineering Score Mean SD Min Max Mean SD Min Max


The two-level final conditional model for the OE scale (see Figure 45 and Table 124) includes all of the variables and random variance coefficients which were found to be significant during the variable testing process described above. It shows that the baseline improvement between pre- and post-assessment was 0.76 points (CI 0.56, 0.96; Intercept γ00, p < .001), meaning that students gained a better understanding of what an optical engineer does. However, English language learners (LEP: 0.37 points improvement; CI 0.08, 0.65; p < .001), students with an IEP (0.44 points improvement; CI 0.17, 0.70; p < .001), and Black students (0.39 points; CI 0.08, 0.70; p = .002) improved to a lesser extent (see Figure 46). The performance of Hispanic students on this scale (0.28 points; CI ˗0.09, 0.64; p = .003) could not be distinguished from zero. Grade 5 classrooms (Grade5), classrooms in which the instructor had more teaching experience (NumYearsTeaching), students from classrooms with a higher class mean score on the pre-assessment (PreOEClassMean), and classrooms that spent more time on Lesson 2 (Lesson2Minutes) tended to improve to a greater extent. Our two-level model explains 49% (0.49=1-(0.174/0.342)) of the between-class variance. The level-1 (within-class) variance σ2 was found to be homogeneous (p>.500).

Level-1 Model: 0 51 2 3 4PostOE (LEP) ( ) (Black) (Hispanic) r(IEP)+= β +β +β β +β +β +PreOE Level-2 Model:

0 00 01 02 03

04 0

(Grade5) ( ) ( )( ) u

PreOEClassMean NumYearsTeachingLesson2Minutes

β = γ + γ + γ + γ+γ +

1 10β γ= 2 20β γ= 3 30β γ= 44 0β γ= 55 0 5uβ γ= + Bold indicates group mean centered. Italicized indicates grand-mean centered. The outcome variable PostOE was centered around the PreOE mean.

Figure 45. Designing Lighting Systems PostOE Score – Conditional Model

Table 124. LS PostOE Score Conditional Model – Final Estimation of Fixed Effects (with robust standard errors)


df P-value

For Intercept 1, β0 Intercept, γ00 0.755 0.100 7.527 55 <0.001 Grade5, γ01 0.245 0.125 1.959 55 0.055 PreOEClassMean, γ02 0.340 0.086 3.963 55 0.001 NumYearsTeaching, γ03 0.018 0.007 2.470 55 0.017 Lesson2Minutes, γ04 0.127 0.048 2.672 55 0.010 For LEP slope, β1 Intercept, γ10 −0.389 0.103 −3.796 1130 <0.001 For IEP slope, β2 Intercept, γ20 −0.320 0.084 −3.822 1130 <0.001 For PreOE slope, β3 Intercept, γ30 0.238 0.031 7.568 1130 <0.001 For Black slope, β4 Intercept, γ40 −0.364 0.117 −3.111 1130 0.002 For Hispanic slope, β5 Intercept, γ50 −0.48 0.152 −3.155 59 0.003

Table 125. LS PostOE Score Conditional Model – Final Estimation of Variance Components



Intercept 1, u0 0.418 0.174 15 64.156 <0.001 Hispanic slope, u1 0.567 0.322 19 37.482 0.007 Level-1, r 1.123 1.262

Table 126. LS PostOE Score Unconditional Model – Final Estimation of Variance Components




Figure 46. LS Change in Student Score: OE Scale

Figure 47 shows the relationship between the minutes of Lesson 2 instruction and the students’ PostOE scores for Designing Lighting Systems. As described above, in Lesson 2 students explore how light interacts with a variety of materials, then discuss how optical engineers would use this knowledge to design optical technologies. The PostOE scores of the students have been graphed against the z-score of the number of minutes of Lesson 2 instruction. The rough line shows the LOESS trend, while the smooth line shows the HLM model fit ( 0P .os 1tOE )27(Lesson2Minutes= ), which also takes into account the demographic variables that were significant in the analysis. The EiE curriculum comes with recommendations as to how much time should be spent teaching each unit. The recommended range of minutes for Lesson 2 of Designing Lighting Systems, which was between 75 and 90 minutes, is shown with the two vertical lines on the graph. The labels show, in order, the minimum number of minutes spent teaching Lesson 2 in our data, the minimum recommended time, the maximum recommended time, and the maximum number of minutes spent teaching Lesson 2 in our data.

Student Change in Score: LS OE scale with 95% confidence intervals

LEPIEP Black

Hispanic

ReferenceGroup

‐0.2

0

0.2

0.4

0.6

0.8

1

1.2

Change

in Score (P

re to Po

st)

LDt

F

LS: EngineeDesign (ED)the various d

Figure 47: LS

Label (m457590

360

ering Design) scale. This demographic

S PostOE Sc

Table 127. Nminutes)

M M M0 M

n (ED) Scalscale had a

c groups teste

cores vs. Num

Notable Les

Minimum numMinimum recoMaximum recoMaximum num

e Results. possible ran

ed on the ED

mber of Min

sson 1 times Des

mber of minuteommended timommended timmber of minut

The second nge of 0 to 7D scale are sh

nutes Spent T

shown in Fiscription es spent teach

me me tes spent teach

d engineering7. Mean prehown in Tab

Teaching the

igure 47

hing EiE in ou

hing EiE in o

g outcome we- and post-able 128 belo

e LS Unit

ur data

our data

was the Engassessment sw.

gineering scores for

Table 128. LS Descriptive Statistics: Pre- and PostED Scores by Demographic Groups

LS PreEngineeringDesign Score LS PostEngineeringDesign Score Mean SD Min Max Mean SD Min Max


The two-level final conditional model for the OE scale (see Figure 48 and Table 129) includes all of the variables and random variance coefficients which were found to be significant during the variable testing process described above. It shows that the baseline improvement between pre- and post-assessment was 0.38 points (CI 0.29, 0.47; Intercept γ00, p < .001), meaning that students were better able to solve engineering design problems. However, the performance of English language learners (LEP: 0.10 points improvement; CI ˗0.16, 0.37; Intercept γ10, p = .023) and Black students (˗0.13 points improvement; CI ˗0.35, 0.07; Intercept γ30, p < .001) could not be distinguished from zero (see Figure 49). Students from classes where the instructor had more experience teaching, both general experience (NumYearsTeaching) and experience teaching EiE (PriorUnits), tended to score higher on this scale. Students from classes where the mean score on the pre-assessment was higher (PreEDClassMean), and those from classes where more time was spent on Lesson 4 (Lesson4Minutes), also scored higher on this scale: Lesson 4 is particularly relevant to this scale, since it tests the skills (engineering design) practiced in lesson 4. Our two-level model explains 83% (0.83=1-(0.021/0.124)) of the between-class variance.

The level-1 (within-class) variance σ2 was found to be heterogeneous (p<.001) and was modeled logarithmically (see Figure 48 and Table 131). The within-class variance was smaller among students with high pre-assessment scores (PreEd), students from classes with a high mean score on the pre-assessment (PreEDClassMean), students from classes where the teacher had taught a large number of EiE units in the past (PriorUnits), and students in a class where the teacher spent a large amount of time teaching Lesson 3 (Lesson3Minutes). The within-class variance was larger among students in Colorado (CO). Once modeled, σ2 became much less significantly heterogeneous (p=.171), suggesting that our model accurately describes important sources of within-class variance.

Level-1 Model: 0 1 2 3PostED (LEP) ( ) (Black) r= β +β +β +β +PreED

2Var(r) σ=

And:

2

0 1 2 3 4

5

ln (Pr iorUnits) (CO) ( ) ( )( )

PreEDClassMean Lesson3Minutesσ α α α α αα

= + + + ++ PreED

Level-2 Model:

0 00 01 02 03

04 0

(PriorUnits) ( ) ( )( ) u

PreEDClassMean NumYearsTeachingLesson4MinutesSquared

β = γ + γ + γ + γ+γ +

1 10β γ= 2 20β γ= 3 30β γ= Bold indicates group mean centered. Italicized indicates grand-mean centered. The outcome variable PostED was centered around the PreED mean.

Figure 48. Designing Lighting Systems PostED Score – Conditional Model

Table 129. LS PostED Score Conditional Model – Final Estimation of Fixed Effects (with robust standard errors)


df P-value

For Intercept 1, β0 Intercept, γ00 0.383 0.045 8.556 55 <0.001 PriorUnits, γ01 0.025 0.010 2.608 55 0.012 PreEDClassMean, γ02 0.343 0.093 3.781 55 0.001 NumYearsTeaching, γ04 0.012 0.005 2.636 55 0.011 Lesson4MinutesSquared, γ05 0.024 0.006 3.874 55 <0.001 For LEP slope, β1 Intercept, γ10 −0.281 0.124 −2.268 1132 0.023 For PreED slope, β2 Intercept, γ20 0.242 0.032 7.612 1132 <0.001 For Black slope, β3 Intercept, γ30 −0.520 0.094 −5.553 1132 <0.001

Table 130. LS PostED Score Conditional Model – Final Estimation of Variance Components



Intercept 1, u0 0.146 0.021 55 85.743 0.005 Level-1, r 1.046 1.095

Table 131. LS PostED Score – Model for Level-1 Variance Parameter Coefficient Standard Error Z-ratio P-value

Intercept 1, α0 1.593 0.265 6.005 <0.001 PriorUnits, α1 −0.042 0.013 −3.211 0.002 CO (Colorado), α3 0.217 0.152 2.085 0.037 PreEDClassMean −0.532 0.095 −5.705 <0.001 Lesson3Minutes, α4 −0.151 0.044 −3.401 0.001 PreEDClassMean, α8 −0.147 0.039 −3.737 <0.001

Table 132. LS PostED Score Unconditional Model – Final Estimation of Variance Components




Figure 49. LS Change in Student Score: ED Scale

Figure 50 shows the relationship between the minutes of Lesson 4 instruction and the students’ PostED scores for Designing Lighting Systems. The PostED scores of the students have been graphed against the z-score of the number of minutes of L4 instruction. The black line shows the LOESS trend, while the red line shows the HLM model fit ( 20.0PostE )24D (Lesson4Minutes= ), which also takes into account the demographic variables that were significant in the analysis. The EiE curriculum comes with recommendations as to how much time should be spent teaching each unit. The recommended range of minutes for Lesson 4 of Designing Lighting Systems, which was between 145 and 180 minutes, is shown with the two vertical lines on the graph. The labels show, in order, the minimum number of minutes spent teaching Lesson 4 in our data, the minimum recommended time, the maximum recommended time, and the maximum number of minutes spent teaching Lesson 4 in our data.

Student Change in Score: LS ED scale with 95% confidence intervals

LEP

Black

ReferenceGroup

‐0.5

0

0.5

1

Change

in Score (P

re to Po

st)

LaPtd

F

LS: Propertiassess changPostPL outcothe science demographic

Figure 50: LS

Label (m60

145180490

ies of Light (ges in perforome variablequestions l

c groups test

S PostED Sc


M5 M0 M0 M

(PL) Scale Rrmance on te were compisted in Ta

ted on the PL

cores vs. Num

Notable Les


Results. The the science puted by add

able 118. ML scale are sh

mber of Min

sson 4 times Des


PL scale hacontent Pro

ding togetheMean pre- an

hown in Tab

nutes Spent T



ad a possibleoperties of Ler the numbend post-assble 134.

Teaching the

igure 50

hing EiE in ou

hing EiE in o

e range of 0 tLight. Scoreer of correctessment sco

e LS Unit

ur data

our data

to 8, and waes for the Prt answers foores for the

as used to rePL and

or each of e various

Table 134. LS Descriptive Statistics: Pre- and PostPL Scores by Demographic Groups LS PrePropertiesLight Score LS PostPropertiesLight Score

Mean SD Min Max Mean SD Min Max Overall 2.04 1.20 0 6 2.70 1.18 0 6 Boys 2.14 1.21 0 6 2.85 1.12 0 6 Girls 1.94 1.17 0 6 2.55 1.23 0 6 IEP 1.85 1.14 0 5 2.51 1.21 0 6 LEP 1.97 1.15 0 5 2.57 1.18 0 5 White 2.06 1.17 0 5 2.78 1.17 0 6 Hispanic 2.01 1.23 0 5 2.39 1.15 0 6 Black 1.93 1.39 0 6 2.26 1.26 0 5 Asian 2.18 1.08 0 4 2.92 1.09 0 5 Other 1.78 1.22 0 4 2.75 1.20 0 5 Grade 3 1.88 1.18 0 6 2.49 1.15 0 6 Grade 4 2.07 1.21 0 6 2.80 1.17 0 6 Grade 5 2.17 1.19 0 5 2.83 1.20 0 6

The two-level final conditional model for the PL scale (see Figure 51 and Table 135) includes all of the variables and random variance coefficients which were found to be significant during the variable testing process described above. It shows that the baseline improvement between pre- and post-assessment was 0.59 points (CI 0.47, 0.71; Intercept γ00, p < .001), meaning that students gained a better understanding of properties of light, including reflections, transmission and absorption. However, Hispanic students (0.30 points improvement; CI 0.05, 0.56; Intercept γ40, p = .011) improved to a lesser extent than others, while the performance of Black students (0.17 points improvement; CI ˗0.09, 0.42; Intercept γ30, p = .001) could not be distinguished from zero. Male students (0.87 points; CI 0.71, 1.04; Intercept γ10, p < .001) improved more than female students (see Figure 52). Classrooms in which the instructor had more teaching experience (NumYearsTeaching) and spent more time teaching Lesson 4 (Lesson4Minutes) tended to improve to a greater extent. Our two-level model explains 69% (0.69=1-(0.066/0.212)) of the between-class variance. The level-1 (within-class) variance σ2 was found to be homogeneous (p>.500).

The level-1 (within-class) variance σ2 was found to be heterogeneous (p=.025) and was modeled logarithmically (see Figure 51 and Table 137). The within-class variance was smaller among students with limited English proficiency (LEP) and students in a class where the teacher spent a large amount of time teaching Lesson 3 (Lesson3Minutes). The within-class variance was larger among students in a class with a high proportion of English language learners (ProportionLEP) and students in a class where the teacher spent a large amount of time teaching Lesson 1 (Lesson1Minutes). Once modeled, σ2 became much less significantly heterogeneous (p=.153), suggesting that our model accurately describes some important sources of within-class variance.

Level-1 Model: 0 3 41 2PostPL (Gender) ( ) (Black) (Hispanic) r= β +β +β +β +β +PrePL

2Var(r) σ=

And: 2

0 1 2 3 4ln ( ) ( ) ( ) (LEP)ProportionLEP Lesson1Minutes Lesson3Minutesσ α α α α α= + + + + Level-2 Model: 0 00 01 02 03 0( ) ( ) ( ) uPrePLClassMean NumYearsTeaching Lesson4Minutesβ = γ + γ + γ + γ +

1 10β γ= 2 20β γ= 3 30 3uβ γ= + 44 0β γ= Bold indicates group mean centered. Italicized indicates grand-mean centered. The outcome variable PostPL was centered around the PrePL mean.

Figure 51. Designing Lighting Systems PostPL Score – Conditional Model

Table 135. LS PostPL Score Conditional Model – Final Estimation of Fixed Effects (with robust standard errors)


df P-value

For Intercept 1, β0 Intercept, γ00 0.589 0.062 9.540 56 <0.001 PrePLClassMean, γ01 0.600 0.109 5.509 56 <0.001 NumYearsTeaching, γ02 0.027 0.008 3.281 56 0.002 Lesson4Minutes, γ03 0.197 0.040 4.913 56 <0.001 For Gender slope, β1 Intercept, γ10 0.285 0.055 5.187 1132 <0.001 For PrePL slope, β2 Intercept, γ20 0.146 0.028 5.192 1132 <0.001 For Black slope, β3 Intercept, γ30 −0.421 0.111 −3.785 59 0.001 For Hispanic slope, β4 Intercept, γ40 −0.287 0.112 −2.561 1132 0.011

Table 136. LS PostPL Score Conditional Model – Final Estimation of Variance Components



Intercept 1, u0 0.257 0.066 31 75.290 <0.001 Black slope, u1 0.078 0.006 34 49.815 0.039 Level-1, r 1.067 1.140

Table 137. LS PostPL Score – Model for Level-1 Variance Parameter Coefficient Standard Error Z-ratio P-value

Intercept 1, α0 0.051 0.050 1.008 0.314 ProportionLEP, α1 0.924 0.313 2.952 0.004 Lesson1Minutes, α2 0.178 0.068 2.608 0.009 Lesson3Minutes, α3 −0.174 0.067 −2.596 0.010 LEP, α4 −0.368 0.183 −2.013 0.044

Table 138. LS PostPL Score Unconditional Model – Final Estimation of Variance Components




Figure 52. LS Change in Student Score: PL Scale

Figure 53 shows the relationship between the minutes of Lesson 4 instruction and the students’ PostPL scores for Designing Lighting Systems. The PostPL scores of the students have been graphed against the z-score of the number of minutes of L4 instruction. The black line shows the LOESS trend, while the red line shows the HLM model fit ( 0.Po 19stPL )7(Lesson 4 minutes= ), which also takes into account the demographic variables that were significant in the analysis. The EiE curriculum comes with recommendations as to how much time should be spent teaching each unit. The recommended range of minutes for Lesson 4 of Designing Lighting Systems, which was between 145 and 180 minutes, is shown with the two vertical lines on the graph. The labels show, in order, the minimum number of minutes spent teaching Lesson 4 in our data, the minimum recommended time, the maximum recommended time, and the maximum number of minutes spent teaching Lesson 4 in our data.

Student Change in Score: RA PL scale with 95% confidence intervals

Male

BlackHispanic

ReferenceGroup

‐0.2

0

0.2

0.4

0.6

0.8

1

1.2

Change

in Score (P

re to Po

st)

LBct

F

LS: BrightneBrightness acomprising tthe various d

Figure 53: LS

Label (m60

145180490

ess and Movand Movemethis scale aredemographic

S PostPL Sc


M5 M0 M0 M

vement of Lent of Light e listed in T

c groups teste

cores vs. Num

Notable Les


Light (BML)(BML) scal

Table 118 aned on the BM

mber of Min

sson 4 times Des


Scale Resule. This scal

nd Table 119ML scale are

nutes Spent T



ults. The sele had a pos9. Mean pree shown in T

Teaching the

igure 53

hing EiE in ou

hing EiE in o

econd scienssible range - and post-a

Table 140 be

e LS Unit

ur data

our data

nce outcomeof 0 to 5; q

assessment selow.

e was the questions scores for

Table 140. LS Descriptive Statistics: Pre- and PostBML Scores by Demographic Groups

LS PreBrightnessMovementLight Score LS PreBrightnessMovementLight Score Mean SD Min Max Mean SD Min Max


The two-level final conditional model for the BML scale (see Table 141 and Figure 54) includes all of the variables and random variance coefficients which were found to be significant during the variable testing process described above. It shows that the baseline improvement between pre- and post-assessment was 0.91 points (CI 0.73, 1.10; Intercept γ00, p < .001), meaning that students gained an understanding of the basic movement of light and how distance can affect brightness. However, Black students (0.65 points improvement; CI 0.35, 0.94; Intercept γ20, p = .019) improved to a lesser extent (see Figure 55). Classrooms in which the instructor had more teaching experience, both in general (NumYearsTeaching) and with EiE (PriorUnits), and classrooms that were currently studying a larger number of EiE units (CurrentUnits) tended to improve to a greater extent, as did classrooms that spent more time on Lesson 4 (Lesson4Minutes) and classrooms with higher means on the pre-assessment (PreBMLClassMean). Classrooms with a larger proportion of English language learners (ProportionLEP) tended to improve to a lesser extent. Our two-level model explains 71% (0.71=1-(0.138/0.472)) of the between-class variance.

The level-1 (within-class) variance σ2 was found to be heterogeneous (p=.004) and was modeled logarithmically (see Table 143 and Figure 54). The within-class variance was smaller among grade 4 students, students in classrooms where the teacher had been teaching for a large number of years (NumYearsTeaching), students in classrooms where the teacher spent a large amount of time teaching lesson 4 (Lesson4Minutes), and students with high pre-assessment scores (PreBML). The within-class variance was larger among students in classrooms where the teacher spent a large amount of time teaching lesson 2 (Lesson2Minutes). Once modeled, σ2 became much less significantly heterogeneous (p=.197), suggesting that our model accurately describes some of the sources of different within-class variances.

Level-1 Model: 0 21PostBML ( ) (Black) r= β +β +β +PreBML

2Var(r) σ=

And:

2

0 1 2 3

4 5

ln (Grade4) ( ) ( )( ) ( )

NumYearsTeaching Lesson2MinutesLesson4Minutes

σ α α α αα α

= + + ++ + PreBML

Level-2 Model:

0 00 0

06

1 02 03 04

05 0

(CurrentUnits) (PriorUnits) ( ) ( )( ) ( ) u

PreBMLClassMean ProportionLEPNumYearsTeaching Lesson4Minutes

β = γ + γ + γ + γ + γ+γ + γ +

1 10 1uβ γ= + 2 20β γ= Bold indicates group mean centered. Italicized indicates grand-mean centered. The outcome variable PostBML was centered around the PreBML mean.

Figure 54. Designing Lighting Systems PostBML Score – Conditional Model

Table 141. LS PostBML Score Conditional Model – Final Estimation of Fixed Effects (with robust standard errors)


df P-value

For Intercept 1, β0 Intercept, γ00 0.914 0.091 10.060 53 <0.001 CurrentUnits, γ01 0.181 0.029 6.199 53 <0.001 PriorUnits, γ02 0.053 0.013 3.998 53 0.001 PreBMLClassMean, γ03 0.461 0.121 3.998 53 <0.001 ProportionLEP, γ04 −0.567 0.281 −2.020 53 0.048 NumYearsTeaching, γ05 0.247 0.044 5.604 53 0.033 Lesson4Minutes, γ06 0.247 0.044 5.604 53 <0.001 For PreBML slope, β1 Intercept, γ10 0.366 0.041 8.959 59 <0.001 For Black slope, β2 Intercept, γ20 −0.269 0.114 −2.352 1131 0.019

Table 142. LS PostBML Score Conditional Model – Final Estimation of Variance Components



Intercept 1, u0 0.372 0.138 53 183.472 <0.001 PreBML slope, u1 0.215 0.046 59 121.070 <0.001 Level-1, r 1.203 1.447

Table 143. LS PostBML Score – Model for Level-1 Variance Parameter Coefficient Standard Error Z-ratio P-value

Intercept 1, α0 0.594 0.093 6.380 <0.001 Grade4, α1 −0.273 0.111 −2.461 0.014 NumYearsTeaching, α2 −0.015 0.006 −2.340 0.019 Lesson2Minutes, α3 0.219 0.083 2.646 0.009 Lesson4Minutes −0.276 0.084 −3.270 0.001 PreBML −0.088 0.379 −2.335 0.020

Table 144. LS PostBML Score Unconditional Model – Final Estimation of Variance Components




Figure 55. LS Change in Student Score: BML Scale

Figure 56 shows the relationship between the time spent on Lesson 4 instruction (Lesson4Minutes) and the students’ PostBML scores for Designing Lighting Systems. The PostBML scores of the students have been graphed against the z-score of the number of minutes of Lesson 4 instruction. The solid line shows the LOESS trend, while the dashed line shows the HLM model fit (PostBML=0.269(Lesson4Minutes)), which also takes into account the demographic variables that were significant in the analysis. The EiE curriculum comes with recommendations as to how much time should be spent teaching each unit. The recommended range of minutes for Lesson 4 of Designing Lighting Systems, which was between 75 and 180 minutes, is shown with the two vertical lines on the graph. The labels show, in order, the minimum number of minutes spent teaching Lesson 4 in our data, the minimum recommended time, the maximum recommended time, and the maximum number of minutes spent teaching Lesson 4 in our data.

Student Change in Score: LS BML scale with 95% confidence intervals

Black

ReferenceGroup

0

0.2

0.4

0.6

0.8

1

1.2

Change

in Score (P

re to Po

st)

3OtWiEcAi

Fi

3.5.2.5 Our analysistogether withWhile studencluding Bl

Education Prcases their Additionallymprovemen

gure 56: LS

Label (mi60 75

180 490

Summars shows thath a science uents tended lack and Hisrogram werescores were

y, classroomnt on all scien

PostBML S

Table 145. Ninutes)

MiMiMaMa

ry: “Designit students wunit on light,to improve

spanic studee sometimese still likely

ms with instnce and engi

Scores vs. Nu

Notable Les

inimum numbinimum recomaximum recomaximum numb

ing Lighting who are taug, learn a signe on all scaents, Englishs associated y to show tructors witineering scal

umber of Mi

sson 4 times Des

ber of minutemmended timmmended timber of minute

Systems” Sught the EiE nificant amoales that wh language lwith a smala significan

th more teales.

inutes Spent

shown in Fiscription s spent teachi

me me es spent teach

ummative EvDesigning

ount about enwere analyze

earners and ller improvemnt increase

aching expe

t Teaching th

igure 56

ing EiE in ou

hing EiE in ou

valuation Lighting Sy

ngineering aed, certain d

students wiment on thefrom pre-

erience tend

he LS Unit

ur data

ur data

ystems unit, and science cdemographicith an Indiv

e assessmentto post-ass

ded to show

possibly concepts. c groups idualized t, in most sessment. w greater

4 Conclusions

This report details findings of formative and summative assessment for five Engineering is Elementary curricular units. The five units were early drafts, designed to teach elementary school students about engineering and technology while reinforcing science concepts. For each of the five field test units we have evaluated in this report, surveys of implementing teachers revealed that teachers, overall, found the units worthwhile and indicated that they would teach them again in the future. Teachers most often commented on (1) students’ enjoyment of the EiE unit and their motivation to participate, (2) the high quality of activities and materials, and (3) the value of opportunities designed into the EiE unit helping for students to learn science and other STEM content.

In our summative assessment, for all five field units we have found that students participating in EiE had higher scores on the post-assessment than they did on the pre-assessment; improvements were significantly large, and as the confidence intervals did not include zero, it is unlikely that improvements are due to chance. In many cases, demographic variables including whether a student has limited English proficiency (LEP), received free or reduced-price lunch (FRL), has an Individualized Education Program (IEP), or is from an underrepresented minority group (Black or Hispanic) were associated with weaker though still significant levels of improvement from the pre- to the post-assessments than that of other students. For a few scales, the performance of underrepresented minorities was indistinguishable from zero.

4.1 Discussion 4.1.1 Formative Assessment Formative assessment for the field test (second draft) versions of the EiE units was mostly comprised of written feedback from field test teachers. Teachers consistently reported high satisfaction with the units (see Table 146) with the mean “likelihood to teach again” rating ranging from 5.5 to 6.3 out of 7 (“Very Likely”) across all five units. All lessons but 2 were rated 5.8 out of 7 or higher; Designing Submersibles Lesson 1 was rated 5.6, and Designing Lighting Systems Lesson 3 was rated 5.1—this lesson frequently suffered from failure of materials.

Table 146. Summary of Mean Lesson Ratings and Likelihood to Teach Again on a Scale of 1-7 CO RA SB KB LS

Rating of Lesson 1 6.1 5.9 5.6 6.3 5.8Rating of Lesson 2 6.0 6.3 5.8 6.2 6.0Rating of Lesson 3 6.1 6.3 6.2 6.3 5.1Rating of Lesson 4 6.6 6.2 6.3 6.5 6.1Likelihood to Teach the Unit Again 6.3 5.9 5.9 6.2 5.5

A summary of positive teacher feedback about the units as a whole and the individual lessons is provided in Table 147. The most common feedback for each unit as a whole, as well as for each of the individual lessons, was that doing EiE was fun, engaging, and/or motivating for students. The percentage of teachers making this comment in response to open-ended questions ranged from 18% to 61% across all units and lessons. Teachers also consistently praised the quality of materials and activities: from 5% to 60% of all teacher responses were coded for praise of this kind. All units, and all lessons 2 and 3, provided good opportunities for students to engage in science and other STEM comment, according to teachers: 46% to 64% of teachers made this comment when asked about the benefits of the units they taught. Also, for all units, teachers consistently commented that their students

practiced discussion, communication, and/or teamwork, which they saw as giving important opportunities to improve their students’ skills. Sixteen percent to 25% of teachers made this comment.

Table 147. Summary of Coding of Feedback Form Responses—Positive Responses

CO RA SB KB LS # Units Overall # Units

Students practiced discussion, communication, and teamwork skills

Unit 21 17 16 24 25 5

5 L1 * * * * * 0 L2 * * * 12 24 2 L3 10 8 * * * 2 L4 9 * 9 * 12 3

Students practiced problem solving and critical thinking skills

Unit 21 * 14 27 10 4

5 L1 * * * * * 0 L2 * * * * * 0 L3 * * * * * 0 L4 7 11 * * * 2

Students had opportunities to learn / apply STEM content and / or skills

Unit 64 46 60 57 63 5

5 L1 * * * * * 0 L2 15 22 19 33 26 3 L3 24 10 7 18 5.6 2 L4 * * * 9 24 2

Lesson makes non-STEM cross-disciplinary and multicultural connections

Unit 9 22 * * 10 3

3 L1 * * * * * 0 L2 * * * * * 0 L3 * * * * * 0 L4 * * * * * 0

Students made connections to the real world

Unit 38 9 * 8 * 3

4 L1 6 13 * * 11 3 L2 * * * 12 * 1 L3 * * * * * 0 L4 * * * * * 0

Students participated in hands-on activities and experiments

Unit 15 11 16 * 8 4

5 L1 * * * * * 0 L2 * 28 * 12 24 3 L3 20 36 23 * * 3 L4 * * * 9 * 1

Students had fun, were engaged, motivated and / or challenged

Unit 19 40 38 41 28 5

5 L1 42 41 36 29 61 5 L2 41 39 45 36 31 5 L3 29 46 39 18 19 5 L4 56 53 50 38 33 5

Activities and / or supporting materials of high quality / provide good foundation for future lessons

Unit * * * * * 0

5 L1 46 51 29 60 31 4 L2 27 25 7 24 21 4 L3 5 18 19 18 6 3 L4 26 25 15 28 33 5

Teachers were asked to explain their rating of each lesson. A summary of negative teacher feedback from these open-ended responses is shown in Table 148. We consistently found that a subset of teachers

found Lesson 4 (the design challenge) to be difficult or confusing. The percentage of teachers making this comment ranged from 7% to 18%. Another common complaint was that lessons took too much time, though not all unit / lessons received this complaint. Finally, Lesson 3 and Lesson 4 was commonly criticized for being difficult to implement. Zero percent to 28% of teachers made this complaint about Lessons 3 and 4.

Table 148. Summary of Coding of Feedback Form Responses—Negative Responses

CO RA SB KB LS # Units Overall # Units

Students were not engaged/activities too easy or boring

L1 6 10 * * 8 3

4 L2 * * * * * 0 L3 7 * * * * 1 L4 * * 9 * * 1

Activities/lessons too difficult or confusing for students and/or teacher

L1 * * * * * 0

5 L2 7 8 * * 11 3 L3 * * * 21 36 2 L4 7 11 18 13 18 5

Time constraints/takes too much time

L1 17 18 19 * * 3

5 L2 15 * 26 12 * 3 L3 15 13 * * 19 3 L4 9 * * * 15 2

Criticism of supporting materials/difficult to implement

L1 * * * * * 0

5 L2 15 * * * 8 2 L3 22 10 * 21 28 4 L4 14 8 24 22 * 4

Other than science, many teachers mentioned content areas that were supported or reinforced by the EiE units they field tested. The most common of these was language arts, mentioned by 50% to 79% of teachers across units. Mathematics was mentioned by 20% to 33% of teachers across units. These results are shown in Table 149.

Table 149. Summary of Coding of Feedback Form Responses—Content Area Connections CO RA SB KB LS Overall #

Units Connected to other content area(s) (beyond science)

Language Arts 57 79 50 73 72 5Social Studies 31 26 16 10 22 5Mathematics 25 24 26 20 33 5

4.1.2 Summative Assessment For all scales for all five units, we found that students overall improved from the pre-test to the post-test (see Table 150 and Figure 57). This improvement was always highly significant (p < .001), and correspondingly, the 95% confidence intervals always excluded zero. This indicates that students learned about engineering and science while they were participating in EiE. Though we cannot say that EiE caused these outcomes, the evidence is promising for the efficacy of EiE. Not all demographic groups improved equally, however. Demographic groups which improved significantly less (or more) than the reference group as defined by each model are indicated in Table 151.

Tab

U

CO

RA

SB

KB

LS

Figu

le 150. Sum

Unit/Scale

ESPP

RoESA

BMMoSci

OEP

BM

ure 57. Summ

mmary of ImpCh(R

Eng Sci Pol PM ocks Eng Sci All ME odels ience OE ED PL ML

mary of Imp

provement: Range in Scor

Reference Gp)1.65 2.00 1.15 1.20 2.84 1.86 1.27 3.46 1.50 1.09 0.95 0.76 0.38 0.59 0.91

provement: R

Reference Gre ) 95%

[1.47, [1.74, 2[1.01, [1.01, [2.60, 3[1.57, 2[1.01, [3.20, 3[1.40, [0.97, [0.81, [0.56, 0[0.29, 0[0.47, 0[0.73,

Reference Gr

Group Chang

CIs

1.82] 182.27] 151.30] 161.39] 123.08] 232.15] 121.53] 93.72] 261.59] 311.22] 171.08] 130.96] 70.47] 80.71] 91.10] 10

roup Change

ge in Score fo

t df

8.711 57 5.052 57 6.062 58 2.565 60 3.598 61 2.876 36

9.880 37 6.864 6041.862 6057.797 6043.983 604

7.527 55 8.556 55 9.540 56 0.060 53

e in Score fo

or Each Unit

p-val

<0.00<0.00<0.00<0.00<0.00<0.00<0.00

4 <0.005 <0.004 <0.004 <0.00

<0.00<0.00<0.00<0.00

or Each Unit

t

ue

01 01 01 01 01 01 01 01 01 01 01 01 01 01 01

t

Table 151. Demographic Groups Included in Reference Groups for Each Scale

Unit Outcome Variable IEP LEP

Race Gender (Male) White Black Asian Hispanic Other

CO PostEng X X X X

PostScience X X X X X PostPol X X X X X X

RA PostPM X X X X PostRocks X X X X X

SB PostEng X X X X X X PostScience X X X X X X

KB

PostAll X X X X X PostBME X X X X X X

PostModels X X X X X PostScience X X X X X

LS

PostOE X X X X PostED X X X X X X PostPL X X X X X

PostBML X X X X X X X X=included in Reference Group

4.2 Recommendations Despite the evaluation being conducted on “draft” versions of the EiE units, the five units reviewed show promise as means for helping students learn about science and engineering—both content and skills such as discussion, teamwork, and problem solving. Teachers consistently reported that their students were engaged and motivated to participate in EiE lessons and design challenges. Though some demographic groups improved less than others, still outcomes were consistently positive, with only a few instances where improvements could not be detected.

Because no control or comparison group was detected, we cannot say that EiE is the cause of the score changes were measured. However, our findings across teacher qualitative data and student quantitative results are strong and consistent enough to be encouraging about the promise of EiE for engaging students and helping them to learn science and engineering.

5 References

Cunningham, C. M., & Hester, K. (2007). Engineering is Elementary: An engineering and technology curriculum for children. Paper presented at the ASEE Annual Conference & Exposition, Honolulu, HI.

Lachapelle, C. P., Cunningham, C. M., Jocz, J., Kay, A. E., Lee-St. John, T. J., Mabikke, H. N., . . . Sullivan, S. (2011). Engineering is Elementary: An evaluation of years 4 through 6 field testing. Boston, MA: Museum of Science.

Lachapelle, C. P., Cunningham, C. M., Lee-St. John, T. J., Cannady, M., & Keenan, K. (2010). An investigation of how two Engineering is Elementary curriculum units support student learning. Paper presented at the P-12 Engineering and Design Education Research Summit, Seaside, OR. http://www.mos.org/eie/pdf/research/Lachapelle_et_al_R1358.pdf

Date post:	26-Apr-2020
Category:	Documents
Upload:	others
View:	1 times
Download:	0 times

eie.orgeie.org/sites/default/files/2011_IMD_Yrs_7-8_report_FINAL.pdf · E M B T G e v An ngineerin...

Documents