Date post: | 05-Jan-2016 |
Category: |
Documents |
Upload: | pauline-bryan |
View: | 213 times |
Download: | 1 times |
Evaluating the Impact of Change in Curriculum and Teaching
Bob delMas, Elizabeth Fry, Laura Le and Anelise Sabbag
Funded by NSF DUE-1043141
Session Overview
• Introduce three assessment instruments• Participants examine subsets of items
• Background on assessment instruments
• Participants explore possible use of the assessment instruments in their curriculums
• Wrap-Up: recommendations for using the instruments
But First, Some Light Entertainment
Three Assessment Instruments• GOALS: 27 forced-choice items
Study design Reasoning about variability Sampling and sampling variability Interpreting confidence intervals and p-values Statistical inference Modeling and simulation
• MOST: Measure of statistical thinking 4 real-world contexts open-ended and forced-choice items
• Affect survey: attitudes and perceptions
Review Assessment Items
• GOALS items 15, 16, 17 and 18
• MOST items 7, 9, 10 and 11
• Affect items 1, 3, 4, 5 and 9
• Answer these questions for each item
– What do you think the item is trying to assess?
– How do you think students would answer?
– Are there options that you think students would choose/not choose?
Small Groups
Form a group with one or two other people and discuss your perspectives about the items
• What do you think the item is trying to assess?
• How do you think students would answer?
• Are there options that you think students would choose/not choose?
GOALS ItemsInterpret Confidence Interval Context
High School Class
Wants to estimate average wt. of
chocolate in Chocolate Chip cookies
Rinsed away
dough & weighed
chocolate
Estimated 95% Confidence Interval
GOALS Items 15 through 18Interpreting Confidence Intervals
For questions 15 to 18, indicate whether the interpretation of the interval provided is valid or invalid.
15. With 95% confidence, we can infer that each cookie for this generic brand has approximately 5.65 to 6.35 grams of chocolate.
Valid Invalid
16. With 95% confidence, we can infer that the average weight of chocolate per cookie in this generic brand is 6 grams plus or minus 0.35 grams.
Valid Invalid
17. We can infer that 95% of all cookies from this generic brand will have between 5.65 to 6.35 grams of chocolate. Valid Invalid
18. With 95% confidence, we can infer that the interval of 5.65 to 6.35 grams includes the true average weight of chocolate per cookie. Valid Invalid
MOST Item: Comparing Groups Context
20 students
Random assignment
10 students: Strategy A
10 students: Strategy B
Mean for A = 5 points higher than mean for B
MOST Item: Comparing Groups Context
7. Explain how the researcher could determine whether the difference in means of 5 points is large enough to claim that one preparation strategy is better than the other. (Be sure to give enough detail that someone else could easily follow your explanation in order to implement your proposed analysis and draw an appropriate inference (conclusion).)
100 students: Strategy A
100 students: Strategy B
MOST Item: 2nd Comparing Groups
9. Imagine that a p-value was produced for the situation described in Questions 7 and 8 where there were 10 students in each group. Would you expect to get the same p-value in this situation where there are 100 students in each group, or would you expect a larger or smaller p-value? Circle the letter of the best response to the question.
• Smaller p-value• Same p-value • Larger p-value
MOST Item: 2nd Comparing Groups
10. Circle the letter of the best explanation for your answer in Question 9.
a) Because a larger sample size produces a more accurate p-value.
b) Because a larger sample size decreases the amount of variation in the distribution of statistics.
c) Because a larger sample size allows us to generalize better to a whole population.
d) Because a larger sample size will give us more variation in the results.
e) Because the observed mean difference and null model didn’t change.
f) Other_______________________________________
MOST Item: 2nd Comparing Groups
MOST Item: Estimation Context
11. Explain how you could help them come up with a good estimate of how many hours a student at that school typically works at outside jobs based on their sample data. (Be sure to give enough detail that someone else could easily follow your explanation in order to implement your proposed analysis and draw an appropriate inference (conclusion).)
MOST Item: Estimation
Affect Items
1. This course helped me understand statistical information I hear or read about in the media.
3. This course helped me learn how to make better decisions when faced with uncertain outcomes.
4. This course helped me realize that how data are produced or collected has an impact on the scope of conclusions that can be made.
8. Learning to (use software/create models in TinkerPlots) helped me learn to think statistically.
9. I believe I am well prepared for future classes that require an understanding of statistics.
RESPONSE SCALE
Strongly Disagree Disagree Agree
Strongly Agree
Instrument Background
• General background on CATALST
• For each instrument– Assessment focus
– Design features
• Use of instruments in evaluation of CATALST curriculum
CATALST Project
• 4-year NSF-funded project (DUE-0814433)
• Radically different intro statistics course
– No t-tests; Use of probability for simulation and modeling
– Coherent curriculum that builds ideas of models, chance, simulated data, inference from first day
– Immersion of students in statistical thinking
– Activities based on real problems, real data
GOALS (Goals and Outcomes Associated with Learning Statistics)
• Focus on types of reasoning to be developed in a first statistics class
• 27 forced-choice items
• 2 versions, same except for items 19-22– One for students in course that teaches
randomization methods for inference– One for students in a course that teaches
traditional methods of inference
MOST (Models of Statistical Thinking)
• Focus on how students think about problems that involve statistical inference
• 4 real-life contexts designed to elicit statistical thinking
• 11 items– 4 open-ended questions, one for each context– 7 forced-choice questions to provide more detail
• 1 version– Can be used in traditional or randomization-based courses
Affect Survey
• Focus on attitudes and beliefs at the end of an introductory statistics class
• 12 items, each using 4-point scale:Strongly Disagree – Disagree – Agree – Strongly Agree
– 4 items about the course– 4 items about using software– 4 items about statistics
• 2 versions– One for classes (e.g, CATALST) that use TinkerPlots™ software– One for classes that use any other software
General Purpose and Use of Instruments• Use of instruments
– To evaluate important student outcomes: reasoning, thinking and attitudes/beliefs
• In CATALST, 3 groups were compared:– CATALST class at U of M (Spring 2012)
• 3 sections all taught the same way, with final version of materials
• Similar students enrolled in all sections
– CATALST classes at 4 other schools (Spring 2012) • Different types of students, requirements
– Non-CATALST: from 6 schools (Fall 2011 and Spring 2012)• Served as a type of control group
GOALS: Bootstrap Confidence Intervals of difference in Mean Proportion Correct for Each Item
(CATALST: n = 289; non-CATALST: n = 440)
Comparison: Confidence Intervals ItemsMean Percent Correct
Comparison: Confidence Intervals ItemsMean Total Number Correct
N MEAN SD
UofM CATALST
138 2.67 0.80
Non-UofM CATALST
151 2.31 0.85
Non-CATALST
440 2.10 0.97
Analyzing MOST Responses
• Statistical Thinking Checklist– Wild and Pfannkuch’s (1999) framework of statistical
thinking– Thinking like an expert from expert-novice literature
Students in Randomization-based Courses
Students in Traditional Course
Modeling Modeling
Integration of statistical and contextual
Integration of statistical and contextual
Domain-specific knowledge Domain-specific knowledge
Statistical Models
MOST Results:Exam Strategy, Description Item
CATALST UofM students
(n = 138)
CATALST non-UofM students (n = 120)
ModelingStudents described a Null Model 39.49% 21.25%
Integration of Statistical and ContextualStudents described the Context within Model
48.19% 32.92%
ModelingStudents provided a Description of the Simulation 43.84% 35.42%Domain-Specific KnowledgeStudents knew how to Make Conclusions based off of the (possible) statistical results
38.77% 24.58%
Integration of Statistical and ContextualStudents described the Context within Conclusion
28.62% 14.17%
MOST Results:Exam Strategy, Description Item
non-CATALST students (n = 187)
Modeling Students described a Null Hypothesis 4.55%Modeling Students described an Alternative Hypothesis 1.87%Integration of Statistical and ContextualStudents described the Context within Hypotheses
3.21%
Domain-Specific KnowledgeStudents described needing to Check Assumptions about the Data
0.80%
Statistical ModelsStudents Stated type of test (z, t, etc.) that would be appropriate for the problem
17.11%
Domain-Specific KnowledgeStudents knew how to Make Conclusions based off of the (possible) statistical results
17.11%
Integration of Statistical and ContextualStudents described the Context within Conclusion
5.35%
MOST Results:Exam Strategy, M-C items
Group
Students who chose “smaller p-value” and correct reason
CATALST UofM students (n= 138)
38.41%
CATALST non-UofM students (n = 120)
15.83%
non-CATALST students(n = 187)
18.18%
MOST Results:Study hours, interval estimate item
CATALST UofM students (n = 138)
CATALST non-UofM students (n = 120)
ModelingStudents described using the Data to find an Interval Estimate
30.07% 15.42%
Integration of Statistical and ContextualStudents described the Context within Model 23.19% 12.92%ModelingStudents provided a Description of the Simulation 36.23% 17.92%VariationStudents mentioned the Need for a variability measure for the CI calculation
48.91% 17.92%
Statistical ModelsStudents knew the CI calculation 40.22% 12.50%Domain-Specific KnowledgeStudents knew how to Make Conclusions based off of the (possible) statistical results and Interpret the interval
22.83% 7.92%
Integration of Statistical and ContextualStudents described the Context within Conclusion 28.99% 13.33%
MOST Results:Study hours, interval estimate item
non-CATALST students (n = 187)
Statistical ModelsStudents Stated type of test (z, t, etc.) that would be appropriate for the problem
3.74%
Domain-Specific KnowledgeStudents described needing to Check Assumptions about the Data
0.27%
VariationStudents mentioned the Need for a variability measure for the CI calculation
4.81%
Statistical ModelsStudents knew the CI calculation 1.60%
Domain-Specific KnowledgeStudents knew how to Make Conclusions based off of the (possible) statistical results and Interpret the interval
3.48%
Integration of Statistical and ContextualStudents described the Context within Conclusion 7.49%
Affect Survey Results
Evaluating the Impact of Change
• Look at the full instruments!
• Questions to think about:– Would these provide useful information to you about your
students if you knew how they performed on these assessments?
– What could you learn from data that you would gather from your students using these assessments?
– What kind of data would you want to compare your student data to? (e.g., national sample, other statistics courses at your institution, other future statistics courses)
Summary and Final Thoughts
• Project Goal: Develop good assessment instruments that provide useful information for curriculum change and development
• Purpose of these instruments: research and evaluation projects
Summary and Final Thoughts
• Instrument development is time consuming, taking several years of testing and revision
• Still under development
Summary and Final Thoughts
• Not designed to replace other kinds of assessments in course (e.g., quizzes, minute papers, projects)
• We want people to use the instruments• Or, you can use some of the items from
an instrument• BUT…
If you use only some items from an instrument
DON'T use them for program evaluations or for research projects
DO use them as part of assessments in your own course
Thank You for your Participation
If you have comments, issues, ideas of what is missing in the instruments, please send one of the PIs an email:
Joan Garfield: [email protected] delMas: [email protected] Zieffler: [email protected]