Measuring confidence to teach statistics to
middle & high school grades:
The development & validation of the SETS instruments
The Research Team
Leigh M. Harrell-Williams M. Alejandra SortoVirginia Tech Texas State Georgia State University University
Rebecca L. Pierce Lawrence M. Lesser Teri J. MurphyBall State University The University of Texas Northern Kentucky at El Paso University
Supported in part by CAUSE (under NSF DUE #0618790)
Why measure middle and high school grades pre-service teachers’
Self-Efficacy to Teach Statistics?• Teachers who are prepared to teach mathematics are
expected to teach statistics as outlined by the PreK-12 Guidelines for Assessment and Instruction in Statistics Education (GAISE) (Franklin et al., 2007) and the Common Core State Standards for Mathematics (CCSSM) (National Governors’ Association, 2010).
• Self-efficacy is task specific; any instrument that assesses self-efficacy needs to be task specific as well.
Self-Efficacy, Attitudes, and Beliefs
Teacher efficacy affects:- teacher motivation- willingness to use more innovative techniques- student achievement- time spent teaching certain concepts(Czerniak, 1990; Riggs & Enochs, 1990; Wenta, 2000).
Existing Instruments:- attitude towards statistics (SATS, ATS)- efficacy for learning/doing statistics (CSSE, SELS)- statistical knowledge (SCI)
No prior instrument measures self-efficacy for future teachers.
Three Developmental
Levels
A Five
Objectives
B Six
Objectives
C Nine
Objectives
GAISE PreK-12 Curriculum Framework
Process Components
1. Formulate Question2. Collect Data3. Analyze Data4. Interpret Results
“Although these three levels may parallel grade levels, they are based on
development in statistical literacy, not age. Thus, a middle-school student who
has had no prior experience with statistics will need to begin with Level A concepts and activities before moving to
Level B.” (p. 13)
CCSSM Framework
VERSUSGrade-Specific Standards for Each
Grade Level
Topic-Specific Standards
Similar to the 4 processes in the GAISE, there are 8 mathematical practices that are threaded throughout the CCSSM.
Identified representative behaviors from
GAISE items
Determined alignment of GAISE
report to state standards
Draft items created for instrument using
language aligned with GAISE and state
standards
Development Process for Middle Grades SETS Instrument
Spring/Summer 2008
Revised item wording based on input from
practicing elementary & middle school
teachersFall 2008/Spring 2009
Pilot Study 2009
Data Collection Study for Validation Purposes
2010 - 2011
Development Process for Middle Grades SETS Instrument
The Middle Grades SETS Instrument
26 Likert scale items in this format:
Please rate your confidence in teaching middle grades students the skills necessary to complete the following tasks successfully:
Scale of 1 to 6
14 Demographic items
Statistics Standards in the Common Core, Grades 2 - 6
Grades 2 - 5:Represent and interpret dataCollecting measurements and creating line plot and bar graphs
Grade 6: Develop understanding of statistical variability: • Recognize statistical question• Data has distribution with specific
center/variation/shapeSummarize and describe distributions• Create boxplots/histograms• Summarize data numerically
Selected SETS Items based on GAISE Level-A
12 items
• Collect data to answer a posed statistical question in contexts of interest to middle school students.
• Recognize that there will be natural variability between observations for individuals.
• Select appropriate graphical displays and numerical summaries to compare individuals to each other and an individual to a group.
Statistics Standards in the Common Core, Grades 7 - 8
Grades 7: Use random sampling to draw inferences about a population.• Inferences can be made from sample about
population if sample is representative.• Generate multiple samples to gauge accuracy.Draw informal comparative inferences about two populations.• Use graphs to estimate differences.• Use numerical values to assess differences.
Grade 8: Investigate patterns of association in bivariate data. • Construct and interpret scatterplots.• Informally fit, assess and interpret a linear
relationship.• Use contingency table to evaluate relationship.
Selected SETS Items based on GAISE Level-B
15 items
• Recognize the role of sampling error when making conclusions based on a random sample taken from a population.
• Recognize that a sample may or may not be representative of a larger population.
• Recognize sampling variability in summary statistics such as the sample mean and the sample proportion.
• Use interquartile range, five-number summaries, and boxplots for comparing distributions.
• Interpret measures of association.
2010 – 2011 Validation Study
• Four US public institutions of higher education with significant proportion of students pursuing degrees in education
• 309 participants enrolled in either an intro statistics course or a math education course
Validation Study - Methods
• Confirmatory Factor Analysis
• Item Analysis
• Rating Scale Analysis
• Reliability
Validation Study - CFA
• Compared unidimensional and two-dimensional factor structure using Multidimensional Random Coefficient Multinomial Logit Model as implemented in Conquest software
• Two dimensions: (Friel, Curcio, & Bright, 2001)– Efficacy to Teach “Reading the Data”– Efficacy to Teach “Reading Between the Data”
• AIC and BIC confirmed two dimension structure• 0.85 between-dimension correlation
Validation Study – Item AnalysisIndex Statistic Composite Reading the Data Reading Between
The Data
rpoint-polyserial Mean 0.66 0.67 0.69
SD 0.05 0.06 0.05 Min 0.55 0.59 0.62 Max 0.73 0.73 0.77 Nitems 26 11 15MSweighted Mean - 1.00 1.00 SD - 0.13 0.23 Min - 0.71 0.71 Max - 1.17 1.52 NiMS > 1.5 - 0 1
MSunweighted Mean - 1 0.98
SD - 0.12 0.21 Min - 0.78 0.72 Max - 1.18 1.47 NiMS > 1.5 - 0 0
Validation Study – Rating Scale Analysis
Criterion Met? Essential?
N > 10 for each response category Yes Yes
Unimodal distribution for each response category Yes Yes
Average measures increase with category Yes Yes
Outfit MNSQ < 2 for each category Yes Yes
Category thresholds increase with category Yes NoMeasure implies Category & Category implies Measure (Coherence) No No
Category thresholds increase by 0.81 logits No No
Category thresholds don’t increase by more than 5 logits Yes No
Validation Study - Reliability
Reliability of Separation - Analogous to Cronbach’s Alpha
Composite Score: 0.94
Subscales:“Reading the Data” (Level A): 0.87 “Reading Between the Data” (Level B): 0.91
The High School Grades SETS Instrument
• Items completed Spring 2012• Based on both GAISE (all levels) and two of the four
strands for CCSSM for High School Statistics & Probability• Interpreting Categorical & Quantitative Data• Making Inferences & Justifying Conclusions
• Data collection• In-service: Summer 2012• Pre-service: Fall 2012 & Spring 2013
• Analysis during Spring/Summer 2013
The High School Grades SETS Instrument
44 Total items:• 26 items from Middle Grades SETS instrument• 18 “new” items based on level C of GAISE and the two
strands of the CCSSM
Item format:• Please rate your confidence in teaching high school students the
skills necessary to complete the following tasks successfully:• Scale of 1 to 6
14 Demographic items
CCSSM “Making Inferences & Justifying Conclusions” Strand
Understand and evaluate random processes underlying statistical experiments• S-IC.1. Understand statistics as a process for making
inferences about population parameters based on a random sample from that population.
• S-IC.2. Decide if a specified model is consistent with results from a given data-generating process, e.g., using simulation.
Make inferences and justify conclusions from sample surveys, experiments, and observational studies• S-IC.3. Recognize the purposes of and differences
among sample surveys, experiments, and observational studies; explain how randomization relates to each.
• S-IC.4. Use data from a sample survey to estimate a population mean or proportion; develop a margin of error through the use of simulation models for random sampling.
• S-IC.5. Use data from a randomized experiment to compare two treatments; use simulations to decide if differences between parameters are significant.
• S-IC.6. Evaluate reports based on data.
Selected SETS High School Items Based on
GAISE Level C and CCSSM
• Evaluate whether a specified model is consistent with data generated from a simulation.
• Explain the role of randomization in surveys, experiments and observational studies.
• Estimate a population mean or proportion using data from a sample survey.
• Evaluate how well the conclusions of a study are supported by the study design and the data collected.
Potential Uses
• Research
• Assessment of Teacher Preparation Programs
• Analysis of Need for In-Service Professional Development Programs
Current Projects Using SETS Instruments
Jean LinnerLassiter High School & GaDOE/GCTM Academy
“Recognizing that low teacher efficacy can inhibit effective teaching as well as student learning, we hope to use the SETS instrument for high school in a longitudinal study to identify and target those professional learning experiences that increase teacher efficacy for teaching statistics.”
Current Projects (continued)
Stephanie Casey & Andrew RossEastern Michigan University
Scores from the SETS for pre-service secondary mathematics teachers enrolled in a statistical methods course were used to assess the pre-service teachers' confidence in teaching statistical concepts in general. Additionally, scores were used to compare the treatment (reform-oriented instruction) section and the control (traditional instruction) section. The results will be used to inform the instruction of the course and assess whether there was a significant difference between the treatment and control groups with respect to their self-efficacy for teaching statistics.
Current Projects (continued)
Salome Martínez & Eugenio Chandía Universidad de Chile & Pontificia Universidad
Católica de Chile
The government of Chile is funding a curriculum project for the preparation of future elementary teachers. The curriculum consists of a series of textbooks for all the content areas, including the Data and Chance strand of the national standards. The SETS instrument will be used to evaluate the impact of the implementation of the textbooks in 10 Chilean institutions of teacher preparation.
Information on Using SETS Instruments
E-mail Rebecca Pierce at [email protected]
to request “Terms of Use” form
Thank you for attending our webinar!
We’d like to open the floor to discuss the following:• How would you envision using one of the SETS
instruments? • What else would you like to know about the
SETS instruments?
Related Papers Hilton, S., Kaplan, J., Hooks, T., Harrell, L. M., Fisher, D., & Sorto, M. A. (2008).
Collaborative projects in statistics education. In Proceedings of the 2008 Joint Statistical Meetings, Section on Statistical Education (pp. 752-756). Alexandria, VA: American Statistical Association.
Harrell, L. M., Pierce, R. L., Sorto, M. A., Murphy, T. J., Lesser, L. M., & Enders, F. B. (2009). On the importance and measurement of pre-service teachers’ efficacy to teach statistics: Results and lessons learned from the development and testing of a GAISE-based instrument. In Proceedings of the 2009 Joint Statistical Meetings, Section on Statistical Education (pp. 3396-3403). Alexandria, VA: American Statistical Association.
Sorto, M. A., Harrell, L. M., Pierce, R. L., Murphy, T. J., Enders, F. B., & Lesser, L. M., (2010). Experts’ perceptions in linking GAISE guidelines to the self-efficacy to teach statistics instrument. In Proceedings of the 2010 Joint Statistical Meetings, Section on Statistical Education (pp. 4289-4294). Alexandria, VA: American Statistical Association.
Harrell-Williams, L. M., Sorto, M. A., Pierce, R. L., Lesser, L. M., & Murphy, T. J. (Under Review). Validation of Scores from a New Measure of Pre-service Teachers’ Self-efficacy to Teach Statistics in the Middle Grades.
REFERENCES
Czerniak, C. M. (1990). A study of self-efficacy, anxiety, and science knowledge in preservice elementary teachers. Paper presented at the National Association for Research in Science Teaching, Atlanta, GA.
Enochs, L.G., Smith, P.L., Huinker, D. (2000). Establishing factorial validity of the mathematics teaching efficacy beliefs instrument. School Science and Mathematics, 100, 194-202.
Finney, S. J., & Schraw, G. (2003). Self-efficacy beliefs in college statistics courses. Contemporary Educational Psychology, 28, 161-186.
Franklin, C., Kader, G., Mewborn, D., Moreno, J., Peck, R., Perry, M., & Scheaffer, R. (2007). Guidelines for Assessment and Instruction in Statistics Education (GAISE) Report: A Pre-K-12 Curriculum Framework. Alexandria, VA: American Statistical Association. (Also available at http://www.amstat.org/education/gaise/)
Friel, S. N., Curcio, F. R., & Bright, G. W. (2001). Making sense of graphs: Critical factors influencing comprehension and instructional implications. Journal for Research in Mathematics Education, 32(2), 124-158.
REFERENCES (continued)
Lutzer, D., Rodi, S., Kirkman, E., & Maxwell, J. Statistical Abstract of Undergraduate Programs in the Mathematical Sciences in the United States, Fall 2005, CBMS Survey, American Mathematical Society, Providence, R.I., 2007.
National Governors Association (2010). Common Core State Standards for Mathematics. http://www.corestandards.org/assets/CCSSI_Math%20Standards.pdf
Riggs, I. M., & Enochs, L. G. (1990). Toward the development of an elementary teacher's science teaching efficacy belief instrument. Science Education, 74(6), 625-637.
Watson, J. (2001). Profiling Teachers’ Competence and Confidence to Teach Particular Mathematics Topics: The Case of Chance and Data. Journal of Mathematics Teacher Education, 4, 305–337.
Wenta, R. G. (2000). Efficacy of preservice elementary mathematics teachers. Unpublished doctoral dissertation, Indiana University.