Date post: | 06-Apr-2018 |
Category: |
Documents |
Upload: | todd-avila |
View: | 220 times |
Download: | 0 times |
of 28
8/2/2019 Massachusetts Math and Science Initiative Evaluation
1/28
Commissioned by Mass Insight Education
Phase 1 Evaluation Report
February 2012
MassMath+ScienceInitiative
AnevaluationoftheimpactoftheMassMath+Science
InitiativeonstudentsparticipationandsuccessonAdvancedPlacementexamsandenrollmentinpost-secondaryeducation
8/2/2019 Massachusetts Math and Science Initiative Evaluation
2/28
The Institute for Strategic Leadership and Learning (INSTLL, LLC) is an education research and consulting
firm that works with educational organizations, state education agencies, districts, and schools to promote
meaningful improvements to our system of public education.
INSTLL works to support the development and spread of innovative ideas to improve public education by
cultivating strategic leadership and learning, supporting the construction of a policy environmentconducive
to innovation and successful implementation of powerful ideas, and engaging in meaningful evaluations of
the various strategies and interventions employed to support teaching and learning.
www.instll.com
2012 Institute for Strategic Leadership and Learning. All rights reserved.
INSTLL,LLC
Acknowledgments
The author formally thanks William Guenther, President, Mass Insight Education and Morton Orlov II,
President, Mass Math + Science Initiative for commissioning this evaluation and providing considerable time
with the author answering questions, discussing preliminary findings, and contributing insights regarding the
Mass Math + Science Initiative. Many thanks also go to Wesley Chin, MMSI Operations Manager, for
providing all data sets used in the evaluation and supporting the analysis of data.
AbouttheAuthor
Brett Lane, President and Founding Member of INSTLL, LLC, an education consulting firm, has over 15
years experience working with state education agencies, districts, and schools on a wide range of research,
consultative and policy-oriented educational initiatives. As an evaluator and policy analyst, Mr. Lanes areas
of expertise include district and school improvement, state and district turnaround strategies, state
accountability systems, teacher effectiveness, high school reform, and charter schools. Mr. Lanes current
work involves researching and informing the policy conditions necessary for states and districts to effectivelysupport local district and school improvement efforts. Mr. Lane is currently working with leaders in multiple
states, including Massachusetts, to develop and implement policies and strategies that will improve public
education and close achievement gaps.
Mr. Lane has authored articles and policy briefs on a variety of education issues, including charter schools,
school turnaround, state systems of support, and district improvement. His most recent publications include
Rapid District Improvement: How districts engage in rapid and sustainable improvement efforts and
Transforming a Statewide System of Support: The Idaho Story, both published through the Center on
Innovation & Improvement.
Mass Insight Education, a 501(c)(3) non-profit organization based in Boston, MA, was founded in 1997. Itslaunch reflected the high priority that business, government, and education leaders placed at that time on the
success of Massachusetts' nascent standards-based reform drive, set in motion by the passage of the Education
Reform Act of 1993. It is the sister organization ofMass Insight Global Partnerships, which has worked since
1989 to keep Massachusetts and its businesses and institutions globally competitive. Mass Insights national
work focuses on district and state strategies to turn around low performing schools.
http://www.instll.com/http://www.instll.com/http://www.instll.com/http://www.massinsight.com/http://www.massinsight.com/http://www.instll.com/http://www.instll.com/8/2/2019 Massachusetts Math and Science Initiative Evaluation
3/28
Abstract 1
Introduction 2
Closing Achievement Gaps and Improving College Readiness and Success 3
The MMSI Theory of Action 4
Distinguishing Features of the Mass Math + Science Initiative 5
Evaluation Overview 6
Evaluation Analysis and Findings 7
Evaluation Finding 1 9
Evaluation Finding 2 11
Evaluation Finding 3 12
Evaluation Finding 4 13
Exploring the hypothetical impact of the MMSI in current, non-MMSI schools 14
College Readiness and Success: Preliminary observations 15
References 16
Appendix A: Summary of @Scale STEM Programs
Appendix B: Evaluation Methodology and Comparison School Selection
Appendix C: Listing of Colleges Receiving MMSI Graduates
TableofContents
A note about the College Boards Advanced Placement (AP) Program
The Mass Math + Science Initiative (MMSI) as a program and as evaluated in this report, is designed to increasestudents participation and success in AP courses, specifically Math, Science, and English courses. All references toAP courses in this report refer to the College Boards Advanced Placement program, which enables students topursue college-level studies while still in high school. The Advanced Placement program includes more than 30college-level courses, each culminating in a rigorous exam; the MMSI program supports students taking andsucceeding in Math, Science, and English AP courses. As documented by the College Board, AP courses providestudents with the opportunity to earn college credit, advanced placement or both. Each AP course is modeled upon a
comparable college course. College faculty ensure that AP courses align with college level standards by definingcurricular expectations for each course and through a review of all AP teachers syllabi.
Participation in an AP course culminates with students taking a college-level assessment that is scored by collegeand university faculty and experienced AP teachers. The College Board verifies that an AP Exam score of 5 isequivalent to a grade of A in the corresponding college course. An AP Exam score of 4 is equivalent to grades of A-,B+ and B in college, and a score of 3 is equivalent to grades of B-, C+ and C in college. Most four-year collegesand universities grant students credit based on successful AP Exam scores or 3 or greater. College Board research hassubstantiated that students who score a 3 or higher on AP Exams typically experience greater academic success incollege and are more likely to graduate on time than otherwise comparable non-AP peers.
Adapted from the The College Boards 7th Annual AP Report to the Nation
8/2/2019 Massachusetts Math and Science Initiative Evaluation
4/28
Abstract
The Mass Math + Science Initiative (MMSI) is a
statewide program, currently comprised of 53 schools in
four cohorts, that is designed to dramatically increase
students participation and performance in Advanced
Placement (AP) courses (Math, Science, and English)
and to increase the number of students matriculating to
and graduating from college. Initiated in 2007 and ledby Mass Insight Education, MMSI is part of a National
Math and Science Initiative to address the national
decline in math and science education. Building upon its
success, MMSI is in the midst of scaling up its program
by expanding to additional schools and leveraging
private and public funding to do so. Documenting and
evaluating the progress that MMSI has made in meeting
program goals is a key step in making this transition.
The evaluation is organized in two phases: Phase 1
focuses on the impact of the MMSI program on
improving students participation and success on AP
exams, as well as providing a preliminary analysis of
college readiness and attendance. Phase 2, due out in
June 2012, will assess the efficacy of the MMSI
program as it is implemented in various high school
contexts.
Since its inception in 2008-09, students in MMSI
schools have taken 13,969 AP exams and have scored 3
or better on 6,426 AP exams.
There is strong longitudinal evidence that the
participation rate (the number of exams taken per1000 students) and the success rate (the number of
exams scoring 3 or better per 1000 students) in
MMSI schools has increased significantly over time.
MMSI schools have madesignificant gains in AP
participation and success in relationship to their
own baseline and they are outperforming similarly
situated groups of non-MMSI schools, grouped
according to need, demographics, and students
income.
The impact of the MMSI program is most pronounced
among African American and Hispanic students in
high-need (e.g., urban, high-poverty schools) and
mid-need (schools between 35 and 60 percent low
income) schools.High- and mid-need MMSI schools
are successfully increasing the number of African
American and Hispanic students enrolling in AP
classes while maintaining success rates that are on
par with similar schools that have not increased
student participation.
The ability of MMSI schools to dramatically increase
African American and Hispanic students participation
in AP courses without a corresponding decline in
students success on AP exams suggests that there is
tremendous untapped capacity among students in
schools across the state that is not being accessed.
There are thousands of students who are not being given
the opportunity to take and succeed in advanced math,
science, and English courses, a crucial contributing
factor to improving college and career success.
1
Illustrative data points and observations:
In 2010, over 7,800 exams were taken in MMSI schools, out
of total eligible student population of 18,955. In contrast, only
3,685 AP exams were taken in the comparison, non-MMSI
schools, out of total eligible student population of 22,911.
Measured as a participation rate, 412 exams were taken for
every 1000 students in MMSI schools, compared to 160exams taken for every 1000 students in non-MMSI schools.
Using the growth rate observed in non-MMSI schools and
controlling for the different starting points of schools, the
following are estimates of the impact of the MMSI program,
in real numbers.
In the 2010-11 school year:
Students in all MMSI schools took approximately 3,473
additional exams that may not have been taken if the
MMSI program had not been implemented.
Students in high- and mid-need MMSI schools took
approximately 1,616 additional exams that may not have
been taken if the MMSI program had not been
implemented.
African American and Hispanic students in high-need
MMSI schools took approximately 335 additional exams
that may not have been taken if the MMSI program had
not been implemented.
If the high-need non-MMSI schools were to have had a
participation rate AND a performance success rate
similar to the MMSI high-need schools, an additional
683 exams may have scored 3 or better in 2010-11.
Additional evaluation findings and data points that merit
further investigation and will be explored in Phase 2 of the
evaluation include:
Significant differences in course-taking patterns between
high- and mid-need MMSI schools.
Wide between-school variance in AP participation and
success rates.
8/2/2019 Massachusetts Math and Science Initiative Evaluation
5/28
Introduction
The Mass Math + Science Initiative (MMSI) is a
replication program designed to dramatically increase
students participation and performance in Advanced
Placement (AP) courses (Math, Science, and English)
and to increase the number of students matriculating
and graduating from college, particularly among
traditionally underserved populations. In 2007,
Massachusetts was one of six states selected to
participate in an innovative program led by the
National Math and Science Initiative to address the
decline in math and science education in the United
States. A multi-year initiative, MMSI was organized by
Mass Insight Education in partnership with the
Commonwealth of Massachusetts and private funders
and is currently Massachusetts largest academic high
school math and science program focused on increasing
student access to, and preparation for, STEM careers.
Current Scope and Demand for MMSI
The MMSI program currently includes 53 schools from
rural, suburban and urban locales across the state and is
comprised of four cohorts of schools and nearly 8000
students participating in AP Math, Science, or English
courses. Each school enters into a formal performance
partnership with the Mass Math + Science Initiative
that describes mutual expectations and performance
goals. Cohort 1 began implementation in 2008-09 andsubsequent cohorts (Cohorts 2, 3, and 4) have entered
the program on an annual basis. An additional 24
schools are seeking access and funding to participate in
MMSI beginning in 2012-13.
Why Advanced Placement?
Research shows that students who enroll in and take AP
exams while in high school tend to perform better in
college (e.g., they have higher college GPAs and are
more likely to graduate in four years) than students that
do not take AP courses (Murphy & Dodd, 2009;
Hargrove, et al., 2008). As a state, Massachusetts has a
history of being relatively successful in having students
enroll in and perform well on AP exams (e.g., score 3 or
higher on an AP exam), compared with other states. In
2010, Massachusetts ranked 5th in the nation in AP
success (as measured by the College Board), with 23
percent of graduating seniors scoring 3 or better at least
once during high school. However, Massachusetts is
not faring nearly as well in supporting African
American, Hispanic, and low income students to
achieve success on AP exams. In 2010, Massachusetts
ranked 17th and 48th (out of 50 states and the District
of Columbia) in the percent of African American and
Hispanic students having success on an AP exam.
Improving Equity and Excellence in Massachusetts
Massachusetts has considerable room to improve in
supporting low income, African American, and
Hispanic students in gaining the requisite skills and
knowledge, particularly in STEM related courses,
necessary to succeed in college. This crucial point has
not been lost by Massachusetts leaders, as Governor
Deval Patrick and Secretary of Education Paul Reville
have set high-visibility goals for STEM success related
to increased student interest, performance, and
graduation from college (Executive Order #513.)
Illustrating the states commitment to STEM programs,
Governor Patricks supplemental FY12 budget includes$500,000 for STEM and the Governor has requested
$1.5 million in the FY13 budget and $2.4 million for
AP initiatives.
Evaluating What Works
Through its first three years of operation, the MMSI
program has provided strong evidence that it is capable
of dramatically supporting the states goals related to
STEM and reducing the achievement gaps. Schools are
eager to participate, AP enrollment in MMSI schools
has increased, and the number of students in MMSI
schools having success on AP exams has increased. As
the only comprehensive, widely applicable high school
academic program with college going impact and
metrics that is available to schools in Massachusetts, it
is important to understand exactly how, and to what
extent, MMSI is currently contributing to closing the
achievement gap and improving students readiness
for college.
Table 1. Massachusetts AP uccess Rate ranking, 2010
Massachusetts AP Success Rate
All Students 5th in nation
Hispanic Students 48th in nation
African American Students 17th in nation
Source: College Board 7th Annual AP Report to the Nation
2
8/2/2019 Massachusetts Math and Science Initiative Evaluation
6/28
The Mass Math + Science Initiative is situated at the
nexus of two critical challenges facing public education
and has taken a systems-based approach to driving
significant change and addressing these challenges. Thetwo challenges that MMSI directly addresses are: (1) the
ever-growing urgency to close achievement gaps,
especially among low income, African American, and
Hispanic students and (2) the crucial economic
imperative to fully prepare students to attend and
succeed in college, especially in STEM related courses
and career pathways. In addressing these dual
challenges, MMSI utilizes multiple levers for change and
sets precise metrics for measuring impact.
Closing the Achievement GapDespite Massachusetts consistently high scores on the
National Assessment of Education Progress (NAEP),
achievement gaps persist across a variety of performance
indicators. Table 2 provides a broad overview of the
achievement gaps in performance on the MCAS (as
measured by the Composite Performance Index) and the
percent of students enrolled in grade 8 Algebra, a widely
accepted indicator of college readiness.
Table 3 displays the stark disparity in students relative
success on AP exams, in this instance success on a Math
AP exam. In 2010-11, only 116 of the Math exams taken
by African American students and 211 of the Math
exams taken by Hispanic studentsin each case less
than 1 percentscored a 3 or better on a Math AP exam.
In contrast, over 5 percent of the exams taken by White
students scored a 3 or better on a Math AP exam. Of
particular note, the percent of exams taken by AfricanAmerican and Hispanic students in MMSI schools is
significantly higher than the rate observed in all other
non-MMSI schools across the state.
College Readiness and Success and STEM Pathways
Massachusetts has been and continues to be a leader
among states in setting goals and taking action to close
achievement gaps. In 2010, theAct Relative to the
Achievement Gap was signed into law, which provided
districts with unprecedented flexibility to change the
conditions in which schools operate. More recently (inNovember of 2011,) Governor Patrick announced four
goals and related strategies that reinforce efforts to close
achievement gaps, one of which is to ensure that students
are prepared for both college and career success.
The urgency to improve college readiness and success is
reinforced by the ever-increasing need to encourage and
prepare all students for career pathways in science,
technology, engineering and math. From the funding of
the STEM Pipeline and PreK-16 Regional Networks to
more recent efforts, such as leveraging federal Race tothe Top funding to enhance STEM teacher training,
Massachusetts has continued to develop policy, provide
funding, and support programs aimed at increasing both
the number of students preparing for and entering STEM
career pathways and the number of qualified STEM
teachers. However, improving college readiness and
success, especially in STEM, remains a pressing
challenge.
Table 2: Two measurand college readines
es of studes, 2006 and
nts acade2010
ic perf rmance
EnrollGrade 8
mentlgebra*
3rd GraE
e CPI**:A
2006 2010 2006 2010
African American 41% 36% 72 71
Hispanic 33% 28% 68 71
White 48% 47% 88 88
Low Income 34% 29% 72 73
* Percent of students en
** The Composite Perfo
extent to which a group
Source: 2011 District An
olled in Gra
mance Inde
is approachi
alysis and R
de 8 Algebr
is a 1-100 i
ng proficie
view Tool,
a
ndex sho
cy, a scor
MA ESE.
ing the
of 100.
Table 3:or be
umber andtter on a M
percent of exams tath AP exam, 2010-1
en that scored a 3, by Ethnicity
Number of exams scoringof group scoring 3
3 or better andor better
In MMSISchools In all non-MMSISchools, statewide All school (MMSIand non-MMSI)
AfricanAmerican
42/2442(1.72%)
74/11889(.62%)
116/14331(0.81%)
Hispanic72/4354(1.65%)
139/17420(.79%)
211/21774(0.96%)
White539/10916
(4.93%)5341/99115
(5.39%)5880/110,031
(5.34%)
Source: College Board, 2 10-11
Dual Challenges: Closing Achievement Gaps and
Improving College Readiness and Success
3
http://www.mass.gov/governor/pressoffice/pressreleases/2011/111109-education-summit-2011.htmlhttp://www.mass.gov/governor/pressoffice/pressreleases/2011/111109-education-summit-2011.htmlhttp://www.mass.gov/governor/pressoffice/pressreleases/2011/111109-education-summit-2011.htmlhttp://www.mass.gov/governor/pressoffice/pressreleases/2011/111109-education-summit-2011.htmlhttp://www.mass.gov/governor/pressoffice/pressreleases/2011/111109-education-summit-2011.html8/2/2019 Massachusetts Math and Science Initiative Evaluation
7/28
The MMSI Theory of Action
The Mass Math + Science Initiative endeavors to
directly address the described challengesclosing the
achievement gap and improving college readiness and
success in STEMthrough a systems-based approach
that uses multiple levers to drive improvement.
On a programmatic level, MMSI is replicating a
research-based initiative with defined program elements
that incorporate multiple levers for changing teacher
and student behaviors and that is linked with specific
measurable goals and targets to gauge progress.
Interested schools submit a competitive proposal to
participate. Once accepted, participating schools enter
into aperformance partnership with MMSI that
articulates expectations for the partnership and sets
goals for program participation. Teachers receive 7 to 9
days of teacher training and students participate in up to
18 additional hours of study sessions during the school
year. Teacher and student incentives, as awards for
participation and success on AP exams, supplement the
teacher and student support and serve as an additional
lever for change.
At a strategic level, MMSI employs strategies designed
to ensure the scalability and sustainability of MMSIwithin and across multiple schools. In doing so, MMSI
has intentionally integrated strategies that characterize
successful reform efforts, including Massachusetts own
effort to raise academic standards.
A significant infusion of national investment, more than
$16 million through 2013, will have been used to
jumpstart the MMSI effort in Massachusetts. To
capitalize on its investment and initial success in 53
schools, MMSI is currently in the process of
transitioning to in-state public/private financing.MMSIs initial infusion of funding, combined with its
intensive and ongoing focus on goals and measuring
progress, has positioned MMSI to successfully expand
to new schools and sustain efforts in schools
demonstrating success.
The Mass Math + Science Initiative:
Provided a significant infusion of money and
investment to accelerate the implementation of the
program;
Developed mutually agreed upon goals and targets
(e.g., benchmarks) and is evaluating progress in
order to adjust and improve, as needed; and
Is transitioning to public/private financing.
Increase participation, as measured by increased
student enrollment in math science, and English AP
courses
Increase performance, as measured by an increase in
the number of qualifying scores (scores of 3, 4, or 5)on AP exams.
Increase college success, as measured by an increase
in the number of students matriculating to and
graduating from college.
Program Elements
Lead to specific goals and the
use of precise metrics
That are reinforced by strategies intended
to expand and sustain successes
4
8/2/2019 Massachusetts Math and Science Initiative Evaluation
8/28
There are a variety of educational programs and
initiatives that have been developed and are being
used to promote college readiness and success inSTEM. Each program comes with its own unique
approach, theory of action, and capacityfeatures
that distinguish one program from another. For
instance, some programs focus primarily on
building student interest in STEM, while others
focus more intensively on developing students and
teachers content knowledge, as a means of building
capacity around STEM courses and content.
To better understand the variety of STEM programs
being used in Massachusetts and to inform this
evaluation, we provide a brief summary of MMSIs
distinguishing features, based on the questions
listed below. A detailed overview of @Scale STEM
programs in Massachusetts is provided in Appendix
A.
Distinguishing Features of the Mass Math + Science Initiative
Academic or Interest Focus: Is the program
Comprehensive or Targeted with respect to
academics and content? Or is the program focused
primarily on building student interest?
Scale: Is the program statewide or located in a
geographic area, district, or school?
Goals and Metrics: Does the program have goals
that are measurable and that provide evidence of
impact or are goals and metrics difficult to
measure, or not directly linked to college success?
Incentives: Does the program include incentives
to ensure that all students have equal access and
opportunity to participate and experience success?
Questions to ask about STEM programs
5
1. MMSI s program is designed to directly improve students
academic competency and performance in STEM content
and courses, rather than focusing solely on generating
student interest in STEM career pathways.
2. MMSI is a comprehensive curriculum-based program that
includes incentives for teachers and students.
3. MMSI is one of two programs (the other being Project Leadthe Way) that are available statewide and that have goals
and metrics for measuring program outputs.
4. MMSI has specific goals related to STEM (e.g., increasing
AP enrollment in math, science and English), academic
achievementand closing achievement gaps, and increasing
college success.
MMSIs Distinguishing Features
In the growing field of STEM-
related programs, and in
particular the @Scale STEM
programsthose programs
endorsed by Governor Patricks
STEM Advisory Councilthe
MMSI program distinguishes
itself in four areas.
What are the Distinguishing Features of the Mass Math + Science Initiative in relationship to
other STEM and College Readiness programs?
Understanding the differences among STEM programsthe levers that they employ to promote college
readiness, the scale of the various programs, and the precision of the measures (e.g., metrics) used to
evaluate impactis crucial when considering the efficacy and cost benefit of various programs.
8/2/2019 Massachusetts Math and Science Initiative Evaluation
9/28
Evaluation Purposes
The purposes of this evaluation are to:
Evaluate the extent to which the Mass Math + Science Initiative is meeting its stated program goals,
focusing on increased enrollment, AP success, and college success among African American and Hispanic
students.
Explore, through interviews and data analysis, how the MMSI program elements are understood and
leveraging changes in teacher and student actions.
Provide information that can be used to inform conversations regarding policy development.
Evaluation Overview
To better understand and to document the extent of
MMSIs success, Mass Insight Education commissioned
INSTLL, LLC to conduct an external evaluation of the
MMSI program. Evaluation questions were developed in
collaboration with Mass Math + Science Initiative
leaders. Building upon MMSIs stated program goals, as
well as issues related to fidelity of implementation, the
purposes of the evaluation were articulated and key
evaluation questions identified, as listed below.
A description of the evaluation methodology, including
the process used to select the comparison schools used
in the analysis and identified in the body of this report,
is provided in Appendix B.
Evaluation Questions
1. How effective is the MMSI program in increasing
students participation and performance on AP exams,
especially among students from diverse backgrounds?
2. To what extent does participation in the MMSI
program increase students readiness for, and success
in, 2- and 4-year colleges?
3. Are there differences in MMSI program impactacross schools? What are the emerging trends and
how can Mass Insight use this information to improve
the MMSI program?
4. How are the components of the MMSI performance-
based, integrated program working together to
improve students participation, performance, and
college readiness and success?
Data Sources
College Board Data: Number of Math,
Science, and English Exams taken and Number
of Exams scoring 3 or higher.
College Board Data for MMSI schools and
Comparison schools.
National Student Clearinghouse: Verification
of MMSI students attending and graduatingfrom college.
Interviews and focus groups with school
leaders, teachers, and students.
6
Two Phase Evaluation
Phase 1, the subject of this report, focuses on
evaluation questions #1 and #2 and examines the
extent to which the MMSI program is meeting
program goals.
In Phase 2, we will conduct focus group sessions
and interviews with key stakeholders (district and
school staff, students, policymakers) to better
understand how the MMSI program is working in
particular settings, and how the program elements
are working together to achieve program goals.
8/2/2019 Massachusetts Math and Science Initiative Evaluation
10/28
Important Evaluation Terms
Performance measures are displayed as raw numbers(e.g., the number of exams taken) and as rates (e.g., 150 out of 1000 students). The two key performance measures are
Participation and Success.
AP Participation: Student participation in AP is based on the number of Math, Science, and English exams
taken in a school year and is expressed as a Participation Rate (number of exams taken for every 1000 students.)
AP Success: Student success in AP is based on the numberof Math, Science, and English exams that receive a
score of 3 or better and is expressed as a Success Rate (number of exams scoring 3 or better for every 1000
students.)
A separate AP Performance Success Rate is a measure of the rate of success among students that take exams. Listed
as a percentage, the AP Performance Success Rate is as follows:
AP Success/AP Participation = AP Performance Success Rate
Evaluation Design In Brief
A quasi-experimental design was employed to assess the impact
of MMSI on students participation and success in AP classes.
Cohort Analysis
Using data from 2006-07 to 2011-12, we compared the
participation and success of students in MMSI schools (Cohorts1, 2, and 3) with the participation and success of students in all
non-MMSI schools across the state.
Comparison Analysis
MMSI program schools represent a range of communities across
the state and serve different, and diverse, student populations. In
order to more accurately measure the impact of MMSI in
schools that serve a particular student population, we set criteria
for grouping schools based on student income and the schools
accountability status, as determined by the state. Using the
criteria for each group (defined below), we separated the MMSIschools into three distinct groups, identified groups of similarly
situated comparison schools, and then analyzed the differences
in performance between MMSI and non-MMSI comparison
schools, within each grouping.
Group A Criteria: Low Income greater than (>) 60 percent orSchool Accountability Status: Level 3 or 4
Group B Criteria: Low Income greater than (>) 35 percent andDoes not meet Group 1 Criteria
Group C Criteria: Low Income less than (
8/2/2019 Massachusetts Math and Science Initiative Evaluation
11/28
Evaluation Questions in Detail
1. How effective is the MMSI program in increasing student participation and performance, especially among
students from diverse backgrounds?
a. How effective is the MMSI program in increasing the participation of students (All students, African
American students, and Hispanic students) in Advanced Placement classes?
b. How effective is the MMSI program in increasing student performance (All students, African Americanstudents, and Hispanic students) on AP exams?
2. What is the effect of the MMSI program on students AP participation and success, in comparison with non-
MMSI schools across the state and with similarly situated (e.g., comparison) schools?
3. How are the components of the MMSI performance-based, integrated program working together to improve
students participation, performance, and college readiness? (To be addressed in Phase 2 of the evaluation)
High Need/Low Income (Group A)MMSI Program Schools
Enroll% LowIncome
High Need/Low Income (Group A)Comparison Schools
Enroll% LowIncome
Boston - Brighton High 1233 79.5 Boston - Burke High 697 76.0
Boston - East Boston High 1382 84.2 Boston - Charlestown High 942 79.7
Chelsea - Chelsea High 1353 75.1 Boston - Madison Park Voc Tech 1286 69.8
Fall River - Durfee HS 2258 70.2 Boston - The English High 777 71.7
Revere - Revere High 1474 68.8 Brockton - Brockton HS 4145 71.4
Salem - Salem High 1231 53.2 Everett - Everett High 1710 61.6
Springfield - HS of Science and Technology 1267 83.7 Fitchburg - Fitchburg High 1146 62.9
Springfield - Springfield Central HS 2046 73.5 Holyoke - Holyoke HS 1268 64.0
Worcester - Burncoat HS 1072 59.3 Lowell - Lowell HS 3403 66.0
Worcester - North HS 1149 77.1 Lynn - Classical High 1401 77.0
Worcester - South HS 1297 72.4 Lynn - Lynn English High 1739 73.5
Lynn - Lynn Voc Tech Institute 809 87.1
New Bedford - New Bedford HS 2711 63.2
Somerville - Somerville High 1344 70.8
Springfield - HS Of Commerce 1286 80.9
Springfield - Putnam Voc Tech HS 1545 79.8
Mid Need (Group B)MMSI Program Schools
Enroll% LowIncome
Mid Need (Group B)Comparison Schools
Enroll% LowIncome
Athol-Royalston - Athol High 451 49.2 Adams-Cheshire - Hoosac Valley High 668 35.9
Boston - O'Bryant Math & Science 1234 49.8 Haverhill - Haverhill High 1748 41.1
Boston Collegiate Charter 554 41.2 Holbrook - Holbrook Jr Sr High 472 37.5
Gill-Montague - Turners Fall High 294 50.7 Leominster - Leominster Center Tech 655 47.8
Greenfield - Greenfield High 483 61.1 Mohawk Trail - Mohawk Trail Reg'l High 566 32.0
Malden - Malden High 1799 61.0 North Adams - Drury High 578 52.2
Marlborough - Marlborough High 1457 36.4 North Central Essential 368 46.2
Quaboag Regional - Quaboag Regional High 582 39.7 Palmer - Palmer High 558 34.6
Randolph - Randolph High 744 55.2 Pioneer Charter Science 294 51.7
Salem Academy Charter 309 42.1 Pioneer Valley - Pioneer Valley Reg'l 536 25.4
Ware - Ware High 498 46.8 Pittsfield - Pittsfield High 976 44.1
Worcester - Worcester Tech HS 1400 61.2 Quincy - Quincy High 1441 53.9
Taunton - Taunton High 1920 39.9
Winchendon - Murdock Middle/High 755 49.5
Tables 4 and 5: Listing of High Need/Low Income (Group A) and Mid Need (Group
B) schools used in Comparison Analysis*
*Note: An Analysis of Low Need (Group C) MMSI schools is notincluded because all low need MMSI schools are in Cohort 2 or 3
8
8/2/2019 Massachusetts Math and Science Initiative Evaluation
12/28
Snapshot Analysis - MMSI Cohorts
Table 6 and Chart 1 depict the change in the number of Science AP exams taken per 1000 Juniors/Seniors for MMSI
Cohorts and for non-MMSI schools, statewide. In MMSI schools, participation rates in Science have increased at a
significantly higher rate
than in non-MMSI schools.The upward trend in
participation rates in Math
and English AP Exams in
MMSI schools mirrors the
growth in the number of
Science AP exams taken, as
displayed in Chart 1.
Statewide, overall AP
participation rates rose
slightly between 2007 and
2011, with approximately
60 to 70 exams being taken
for every 1000 students, in
Science, Math, and English.
*Data Note: The Error Bars used in Chart 1
(above) and in subsequent charts represent 3
Standard Deviations from the statewide
average, and are intended to illustrate the
scope of the increases observed in MMSI
schools.
How effective is the MMSI program in increasing the participation of students in
Advanced Placement classes? (Cohort Analysis)
Key Finding 1: MMSI Cohort Schools have significantly increased the number and percent of
Advanced Placement (AP) Math, Science, and English exams taken by students, in relation to each
Cohorts baseline year and compared with non-MMSI schools.
Data Highlight:In Cohort I schools in 2010-2011, 196 Science AP exams were taken for every 1000
students, which represents a real increase of 570 additional exams that were taken for that year (out of a
total of 918) that would not have been taken if participation rates had not increased.
Baseline Year One Year Two Year Three
Cohort 1 (8 Schools) 75 115 162 196Cohort 2 (11 Schools) 77 127 147 -
Cohort 3 (26 Schools) 51 87 - -
Statewide, Non-MMSI Schools
65 72 71 77
0"
20"
40"
60"
80"
100"
120"
140"
160"
180"
200"
Baseline" Year"One" Year"Two" Year"Three"
Numbero
fExamsTakenper1000Students
Chart'1.'All'MMSI'Cohorts:'Number'of'Science&AP'Exams'Taken''
per'1000'Juniors/Seniors,'by'Cohort'and'Year'in'Program'
Cohort"1"(8"Schools)"
Cohort"2"(11"Schools)"
Cohort"3"(26"Schools)"
Statewide,"non?MMSI"Schools*"
Table 6. Number of Science AP Exams taken per 1000 Juniors/Seniors, byCohort and Year in Program
9
8/2/2019 Massachusetts Math and Science Initiative Evaluation
13/28
Snapshot Analysis - Comparison
Schools
An analysis of participation rates among
Group A (High-need) and Group B(Mid-need) MMSI schools and their
comparison schoolsshows that MMSI
schools are improving AP participation
rates and consistently outperform
similar, non-MMSI schools. Chart 2
displays the number of exams taken per
1000 students for MMSI and non-MMSI
school. High-need schools (both MMSI
and non-MMSI) are represented in Red
and mid-need represented in Blue.
High- and mid-need MMSI schools have experienced consistent growth in AP participation rates since beginning the
program. High-need MMSI schools increased AP participation rates from a baseline of 182 exams per 1000 students
to 346 exams per 1000, in Year 3 of the program. Mid-need MMSI schools increased participation rates from a
baseline of 243 per 1000 to 612 per 1000 in Year 3. Of particular note is the fact that the AP participation rate in high-
need MMSI schoolsthose schools with the most
diverse and low income studentsconsiderably
outpaces the participation rate in mid-need non-MMSI
schools, which tend to serve a less diverse student
population.
Variance in course-taking patterns. A comparison
of course-taking patterns among high- and mid-need
MMSI schools highlights an additional piece of
information that may be important for ongoing
program improvement. As displayed in Chart 3, high-
need MMSI schools have experienced the most
success in increasing the rate of students taking
English AP exams, with smaller increases in Math
and Science. Mid-need schools, in comparison, have
seen dramatic increases in the number of Science
and Math exams being taken, as displayed in Chart4. Overall, mid-need MMSI schools are exhibiting
higher overall participation rates and more
accelerated growth than observed in high-need MMSI
schools, although each group of schools is having
success in meeting MMSI program goals and is
outperforming its comparison group.
How effective is the MMSI program in increasing the participation of students in
Advanced Placement classes? (Comparison Analysis)
10
Baseline( Year(1( Year(2( Year(3(
Math(Exams(Taken( 82.9( 117.5( 170.5( 155.5(
Science(Exams(Taken( 71.0( 119.7( 230.0( 289.0(
English(Exams(Taken( 92.8( 166.5( 182.7( 167.1(
0.0(
50.0(
100.0(
150.0(
200.0(
250.0(
300.0(
Numberper1000
Chart'4.'Number'of'Exams'Taken'per'1000'Students'by'AP'
Exam'Content'Area,'mid@need'MMSI'Schools'
0"
100"
200"
300"
400"
500"
600"
700"
Baseline" Year"1" Year"2" Year"3"
Chart&2.&Number&of&exams&taken&per&1000&students&in&All&Subjects,&MMSI&and&
NonMMSI&high&and&midneed&schools.&&
Highneed"MMSI:"Exams"Taken" Highneed"NonMMSI:"Exams"Taken"
Midneed"MMSI:"Exams"Taken" Midneed"NonMMSI:"Exams"Taken"
Baseline( Year(1( Year(2( Year(3(
Math(Exams(Taken( 39.6( 62.4( 85.1( 79.0(
Science(Exams(Taken( 69.9( 75.4( 81.5( 90.1(
English(Exams(Taken( 73.5( 129.3( 141.9( 176.6(
0.0(
50.0(
100.0(
150.0(
200.0(
Number
per1000
Chart'3.'Number'of'Exams'Taken'per'1000'Students'by'
AP'Exam'Content'Area,'highneed'MMSI'Schools'
8/2/2019 Massachusetts Math and Science Initiative Evaluation
14/28
How effective is the MMSI program in increasing students performance on the
AP exam?
Key Finding 2. MMSI schools are experiencing significant growth in the real number of exams scoring 3 or
higher and in the number of exams scoring 3 or better for every 1000 students (known as a schools success
rate) in relation to each Cohorts baseline year and compared with the success rate in non-MMSI
comparison schools.
Data Highlight:If the high-need non-MMSI schools were to have had a participation rate AND a performance
success rate similar to the MMSI high-need schools, an additional 683 exams may have scored 3 or better in
2010-11.
Evidence of Latent Student Capacity: The significantly higher success rate in MMSI schools demonstrates that
MMSI schools are enrolling and supporting students that have not typically accessed AP courses. Likewise, the data
suggests that there are students in comparison non-MMSI schools who are capable of performing AP course work
but are not afforded the opportunity to do so.
Snapshot Analysis
MMSI schoolswithin cohorts and
organized according to needhave increased
students participation rate and the rate at
which students score 3 or better on AP
exams. Charts 5 and 6 display the
participation and success rates for high- and
mid-need MMSI schools and comparison
schools.
In each group of MMSI schools, we see
evidence of a rise in participation rate
coupled with a significant increase in
success rate, over time. In high-need MMSI
schools, the Year 3 success rate has doubled
since the baseline (from 52 to 115 per 1000
exams scoring 3 or better); in contrast, the
non-MMSI success rate increased from 42 to
52 per 1000. Mid-need MMSI schools have
experienced even greater improvement over
time, moving from a baseline success rate of
112 per 1000 students scoring 3 or better to a
success rate of 283 per 1000 in Year 3. As
displayed in Chart 8, the success rate in mid-
need MMSI schools has dramatically
increased to the extent that thepercent of
students passing AP exams in MMSI
schools has exceeded the percent of students
taking exams in non-MMSI schools.
11
Baseline( Year(1( Year(2( Year(3(
Non0MMSI:(Exams(Taken( 97( 17( 119( 126(
Non0MMSI:(Exams(>(3( 42( 44( 44( 52(
MMSI:(Exams(Taken( 182( 274( 38( 346(
MMSI:(Exams(>(3( 52( 87( 116( 115(
(
5(
1(
15(
2(
25(
3(
35(
4(
Numberper1000
Chart'5.'Number'of'AP'Exams'taken'and'number'of'AP'Exams'
scoring'>'3'per'1000'students:'HighBneed'MMSI'and'nonBMMSI'
schools;'All'students'and'subjects'
Baseline( Year(1( Year(2( Year(3(
Non0MMSI:(Exams(Taken( 166( 198( 221( 228(
Non0MMSI:(Exams(>(3( 87( 113( 127( 137(
MMSI:(Exams(Taken( 243( 44( 583( 612(
MMSI:(Exams(>(3( 112( 177( 261( 283(
(
1(
2(
3(
4(
5(
6(
Numberper1000
Chart'6.'Number'of'AP'Exams'taken'and'number'of'AP'Exams'
scoring'>'3'per'1000'students:'MidBneed'MMSI'and'nonB
MMSI'schools;'All'students'and'subjects'
8/2/2019 Massachusetts Math and Science Initiative Evaluation
15/28
How effective is the MMSI program in increasing the enrollment of African
American and Hispanic students in Advanced Placement classes?
Key Finding 3: MMSI Cohort Schools have significantly increased the number and percent of Advanced
Placement (AP) Math, Science, and English exams taken by African American and Hispanic students, in
relation to each Cohorts baseline year and compared with non-MMSI schools.
Data Highlights:In Year 3, 239 AP exams were taken for every 1000 African American and Hispanic students
attending high-need MMSI schools; in contrast, only 63 exams were taken for every 1000 African American and
Hispanic students in high-need non-MMSI schools.
African American and Hispanic students in high-need MMSI schools are almost 5 times as likely to take an AP
Science exam as are students attending high-need non-MMSI schools.
If the non-MMSI high-need schools were to have had a participation rate similar to the MMSI high-need schools, an
additional 1,016 AP exams would have been taken by African American and Hispanic students in 2010-11.
Snapshot Analysis
Charts 7 and 8 display the AP
participation rate for high- and mid-
need schools for all AP exams and
for Science AP exams. In addition to
the visible and significant
differences in participation rates
between MMSI and non-MMSI
schools, the data shows that MMSI
and non-MMSI schools had
different starting points, orbaseline measures of AP
participation. The difference in
initial participation rates suggest that
schools that applied for and were
selected for MMSI entered the
program with substantially higher
levels of readiness, as measured
by the baseline AP participation
rate.
Despite their already higher AP
participation rates for African
American and Hispanic students,
MMSI high- and mid-need schools
exhibited significant growth in AP
participation rate over time, in
comparison to non-MMSI high- and
mid-need schools.
12
Baseline( Year(1( Year(2( Year(3(
Highneed(MMSI:(Exams(Taken( 109( 167( 170( 239(
Highneed(NonMMSI:(Exams(Taken( 42( 56( 51( 63(
Midneed(MMSI:(Exams(Taken( 159( 286( 378( 370(
Midneed(NonMMSI:(Exams(Taken( 45( 61( 72( 86(
0(
50(
100(
150(
200(
250(
300(
350(
400(
NumberofExamsTakenper1000Students
Chart'7.'Number'of'Exams'taken'per'1000'African'American'and'Hispanic'Students'in'high@'and'
mid@need'MMSI'and'Non@MMSI'schools'
Baseline( Year(1( Year(2( Year(3(
Highneed(MMSI:(Exams(Taken( 50( 58( 54( 99(
Highneed(NonMMSI:(Exams(Taken( 14( 18( 14( 19(
Midneed(MMSI:(Exams(Taken( 67( 99( 146( 202(
Midneed(NonMMSI:(Exams(Taken( 10( 22( 19( 25(
0(
50(
100(
150(
200(
250(
300(
350(
400(
NumberofExamsTakenper1000Students
Chart'8.'Number'of'Science'AP'Exams'taken'per'1000'African'Amercian'students'in'high@'
and'mid@need'MMSI'and'Non@MMSI'schools'
8/2/2019 Massachusetts Math and Science Initiative Evaluation
16/28
How effective is the MMSI program in increasing the performance of African
American and Hispanic Students?
Key Finding 4. MMSI schools are dramatically and significantly increasing the number of African Americans
and Hispanics scoring 3 or better on a Math, Science, or English AP exam, in MMSI schools and relative to
schools across the state.
Data Highlights: In 2010-11, students at MMSI schools comprised 19 percent of the Hispanic and African American
student population in the state,yet accounted for 43 percent of African American and Hispanic AP exam taking
and 36 percent of exam takers scoring a 3 or better.
If the high-need non-MMSI schools were to have had a participation rate AND a performance success rate similar to
the high-need MMSI schools, an additional 206 exams taken by African American and Hispanic students may have
scored 3 or better in 2010-11.
Snapshot Analysis
Charts 9 and 10 display the growing
participation and success ratesamong African American and
Hispanic students in high- and mid-
need MMSI schools. The rate of
exams taken by African American
and Hispanic studentsscoring 3 or
higherin MMSI schools (at 60 per
1000 students in high-need schools
and 142 per 1000 in mid-need
schools) is equal to or, in the case of
mid-need schools, substantially
higher than the rate at which examsare taken, among similar students in
non-MMSI schools.
As noted in Finding #3, high- and
mid-need MMSI schools have
significantly increased the
participation rate among African
Americans and Hispanics. The
increase in the raw numbers of
exams taken by African American
and Hispanic students has
subsequently contributed to a
significant increase in the actual
numbers of exams scoring 3 or
higher and growth in the AP success
rate since the beginning of the
MMSI program.
13
Baseline( Year(1( Year(2( Year(3(
Non0MMSI:(Exams(Taken( 45( 61( 72( 86(
Non0MMSI:(Exams(>(3( 9( 16( 31( 29(
MMSI:(Exams(Taken( 159( 286( 378( 37(
MMSI:(Exams(>(3( 48( 86( 117( 142(
(
5(
1(
15(
2(
25(
3(
35(
4(
Nubmerper1000
Chart'10.'Number'of'AP'Exams'taken'and'number'of'AP'Exams'scoring'>'3'per'1000'
students:'Midneed'MMSI'and'nonMMSI'schools;'African'American'and'Hispanic'Students''
Baseline( Year(1( Year(2( Year(3(
Non0MMSI:(Exams(Taken( 42( 56( 51( 63(
Non0MMSI:(Exams(>(3( 8( 9( 1( 13(
MMSI:(Exams(Taken( 19( 167( 17( 239(
MMSI:(Exams(>(3( 16( 27( 44( 6(
(
5(
1(
15(
2(
25(
3(
35(
4(
Numberper1000
Chart'9.'Number'of'AP'Exams'taken'and'number'of'AP'Exams'scoring'>'3'per'1000'students:'
HighBneed'MMSI'and'nonBMMSI'schools;'African'American'and'Hispanic'Students''
8/2/2019 Massachusetts Math and Science Initiative Evaluation
17/28
What is the hypothetical impact of the MMSI in current, non-MMSI schools?
Exploring the hypothetical impact of the MMSI program.
Table 8 displays the actual numbers of eligible students (e.g., eligible juniors and seniors) in MMSI and non-MMSI
comparison schools for the 2010-2011 school year, the number of exams taken, and the number of exams scoring 3 or
better. The last two columns, Participation Rate and Performance Success Rate, provide a measure of the extent to
which schools in a particular group are successful in having students enroll in and take AP exams (the ParticipationRate) and the extent to which schools are able to support students in scoring 3 or better (the Performance Success
Rate.) The top set of rows display information forall students. The bottom set of rows focuses on the performance of
African American and Hispanic students.
A primary goal of the MMSI program is to increase enrollment and participation, especially among diverse students.
The data shows that MMSI schools are dramatically increasing participation rates. It seems logical (and in fact it was
expected) that increased participation rates would likely result in decreased performance success ratesas schools
enroll more students into AP courses (some of whom may not have been targeted for such classes), a smaller
percentage of these students would score 3 or better. However, dramatically increasing participation rates (as has been
accomplished) would theoretically overcome declining performance success rates and still contribute to increased
overall numbers of students taking AP courses and scoring 3 or higher.
All StudentsEligibleStudents
# of ExamsTaken
# of ExamsScoring >3
ParticipationRate
PerformanceSuccess Rate
Non-MMSI (All Groups) 22911 3658 2012 16.0% 55.0%
MMSI (All Groups) 18955 7812 3519 41.2% 45.0%
Non-MMSI (High- and Mid-) 15923 2444 1186 15.3% 48.5%
MMSI (High- and Mid-) 11054 4331 1621 39.2% 37.4%
Non-MMSI High-need 11687 1477 607 12.6% 41.1%
MMSI High-need 7047 2335 778 33.1% 33.3%
African American andHispanic Students
EligibleStudents
# of ExamsTaken
# of ExamsScoring >3
ParticipationRate
PerformanceSuccess Rate
Non-MMSI (All Groups) 8211 522 137 6.4% 26.2%
MMSI (All Groups) 6609 1582 421 23.9% 26.6%
Non-MMSI (High- and Mid-) 7487 486 109 6.5% 22.4%
MMSI (High- and Mid-) 5722 1380 335 24.1% 24.3%
Non-MMSI High-need 6826 429 90 6.3% 21.0%
MMSI High-need 4177 884 181 21.2% 20.5%
14
Table 8. Summary Table of 2010-11 Participation Rates and Performance Success Rates, MMSI and Non-MMSI Schools
Analysis
The numbers for all students (the top set of rows) follows the above stated logic, as MMSI groups exhibit substantiallyhigher participation rates and significantly lower performance success rates. However, our analysis of the experience
of African American and Hispanic students tells a different story and provides compelling and provocative evidence of
the real impact of the MMSI program related to efforts to close the achievement gap. In the aggregate and by need-
based group, MMSI schools are realizing dramatically higher participation rates for African American and
Hispanic Students in comparison with non-MMSI schools, while maintaining similar performance success rates.
Instead of experiencing a decline in the rate of exams scoring 3 or higher, MMSI schools are increasing the
numbers of students taking AP exams and keeping the performance success rate steady. This finding has significant
implications for the potential expansion of the MMSI program in additional schools across the state.
Note the similarPerformance Success Ratecoupled with MMSI's high
Participation Rates
Significantly higherParticipation Rates
correlated with smallerPerformance Success Rat
8/2/2019 Massachusetts Math and Science Initiative Evaluation
18/28
College Readiness and Success: Preliminary Observations
Tracking Students from High School through College
Measuring the real impact of the MMSI program on individual students requires tracking students as they graduate
from high school, enroll in a 2- or 4-year college, and subsequently graduate (or not) from college. The Mass Math +
Science Initiative team is working carefully with partner schools and with the National Student Clearinghouse to
track the college pathway of those students that take one or more AP exams while enrolled in a MMSI program
school.
Between 2008-09 and 2010-11, 5,269 MMSI students graduated from a MMSI program school, of which 4,024 have
been identified who were accepted to and attended college the first year after graduating from high school, for an
overall college attendance rateof 76 percent. According to National Student Clearinghouse data, MMSI students
(to reiterate, those students who took at least one AP exam while attending a MMSI school) are currently attending
384 different 2- and 4 -year colleges. The primary receiving colleges include the University of Massachusetts
(Amherst, Boston, Dartmouth, and Lowell campuses) and other major colleges and universities in Massachusetts, as
Table 9. Most Frequently Selected Universities/Colleges, MMSI Schools (2008-11)
University/CollegeNumber of MMSI schools*
sending MMSI students
UNIVERSITY OF MASSACHUSETTS ATAMHERST
41/44
FRAMINGHAM STATE UNIVERSITY 31/44
UNIVERSITY OF MASSACHUSETTS-BOSTON 31/44
UNIVERSITY OF MASSACHUSETTS-DARTMOUTH
31/44
BRIDGEWATER STATE UNIVERSITY 28/44
UNIVERSITY OF MASSACHUSETTS ATLOWELL
27/44
SUFFOLK UNIVERSITY 25/44
WESTFIELD STATE UNIVERSITY 25/44
NORTHEASTERN UNIVERSITY 23/44
WORCESTER POLYTECHNIC INSTITUTE 23/44
WORCESTER STATE UNIVERSITY 23/44
MASSACHUSETTS COLLEGE OF PHARMACY 22/44
WENTWORTH INSTITUTE OF TECHNOLOGY 22/44
* Data was a ailable for 44 of the 45 MMSI schools
Measuring the impact of the MMSI program on students readiness for and success in college
Measuring the actual impact of the MMSI program on increasing the number of students matriculating to and
graduating from college requires a metric that takes into account two factors that influence whether or not students
attend college and that are under the control of the MMSI program. For instance, some schools may be very good at
increasing AP enrollment, but not as attentive to supporting students matriculation to college. Other schools may
provide excellent support to students applying for college, but fail to actively enroll sufficient numbers of students in
AP courses. The success of the MMSI program is a function of (1) a schools ability to enroll students in AP courses
(AP participation rate) and (2) the capacity of the school to support students taking AP courses towards applying for
and attending college (college attendance rate.)
A College Success Report, to be produced in Summer 2012, will measure a schools success in having students
participate in AP courses and matriculate to college to quantify the impact of the MMSI program.
15
8/2/2019 Massachusetts Math and Science Initiative Evaluation
19/28
References
College Board.(2011). The 7th Annual Report to the Nation. New York, NY: The College Board
Commonwealth of Massachusetts. (2010). A Foundation for the Future: Massachusetts Plan for Excellence in
STEM Education (Science, Technology, Engineering and Mathematics),
Hargrove, L., Godin, D. & Dodd, B. (2008). College Outcomes Comparisons by AP and Non-AP High School
Experiences. New York, NY: The College Board
Murphy, D. & Dodd, B. (2009).A Comparison of College Performance of Matched AP and Non-AP Student
Groups. New York, NY: The College Board
16
http://www.nga.org/files/live/sites/NGA/files/pdf/1112STEMSUMMITCEDRONE.PDFhttp://www.nga.org/files/live/sites/NGA/files/pdf/1112STEMSUMMITCEDRONE.PDFhttp://www.nga.org/files/live/sites/NGA/files/pdf/1112STEMSUMMITCEDRONE.PDFhttp://www.nga.org/files/live/sites/NGA/files/pdf/1112STEMSUMMITCEDRONE.PDF8/2/2019 Massachusetts Math and Science Initiative Evaluation
20/28
Appendix A
At-a-glance overview of @Scale and selected initiatives
focused on STEM and College Readiness
A - 1
Considering the Scope, Scale, and Measurable Impact of @Scale Initiatives
Dramatically increasing the numbers of low income and diverse students attending and succeeding in
college, especially in Science, Technology, Engineering, and Math (STEM) is crucialboth for
individuals and for the economic health and sustainability of the Commonwealth. There are a variety of
educational programs and initiatives that have been developed and are being used to promote college
readiness and success in STEM. Understanding the differences among these programsthe levers that
they employ to promote college readiness, the scale of the various programs, and the precision of the
measures (e.g., metrics) used to evaluate impactis crucial to consider when making decisions. To the
point, identifying those programs that use multiple levers to effect change (e.g., to contribute to system
change) and that have measurable evidence of impact and considering how to best support these
programs is part of the role of our elected officials.
Which programs are working and having success in preparing students to attend and succeed in
college?
To assist policymakers and constituents in exploring and discussing some of the distinctions amongavailable programs (in this case, the @Scale Initiatives), INSTLL, LLC and Mass Insight have prepared
this at-a-glance overview of the @Scale programs. The @Scale programs are organized according to
the following dimensions, building upon research on program implementation.
Academic or Interest Focus: Is the program Comprehensive or Targeted with respect toacademics and content? Or is the program focused primarily on building student interest?
Scale: Is the program statewide or located in a single district or school (or geographic area)? Goals and Metrics: Does the program have goals that are measurable and the provide evidence
of impact or are goals and metrics difficult to measure, or not directly linked to college success?
@Scale Programs in Brief: Focus, Goals and Metrics, and Scale
Goal and Metrics Scale
Academic Approach:
Comprehensive
Mass Math + Science InitiativeMultiple Goals
Measurable OutcomesStatewide
Project Lead the WaySingle Goal
Measurable OutcomesStatewide
Academic Approach:
Targeted
Science Transfer InitiativeMultiple Goals
Measurable OutcomesLocal
BioTeach Single GoalOutcomes Local
Interest-Based Approach
Advanced Robotics InitiativeSingle Goal
OutcomesLocal
DIGITSSingle Goal
OutcomesStatewide
8/2/2019 Massachusetts Math and Science Initiative Evaluation
21/28
Appendix A
At-a-glance overview of @Scale and selected initiatives
focused on STEM and College Readiness
A - 2
Policy Considerations and Information Display
The following pages provide a selection of questions that can be used to prompt discussion and a
comparison of @Scale programs, followed by a set of graphic organizers that provides additional
information.We offer the following topics and questions for consideration and deliberation regarding STEM related
initiatives.
Addressing Goals related to STEM, College Success, and Closing the Achievement Gap
Does the program address multiple goals (e.g., STEM state goals, College Success, andClosing the Achievement Gap) or is the program focused on a particular goal?
How does the program address the Governors goals of eliminating achievement gaps andensuring college readiness?
Program impact and the capacity of the program to measure impact
How much of an impact does the program have with low income and diverse studentpopulations? Does the program have specific measures to assess the impact of its program with respect to
improving college readiness, decreasing the achievement gap, and increasing participation and
enrollment in STEM related courses?
Using multiple levers to accelerate and sustain improvement
Does the program have the capacity to expand, and to do so with fidelity? Does the program utilize multiple levers (e.g., teacher training, improved curriculum, teacher
and student incentives) to achieve its goals?
Does the program involve a multi-year engagement, or performance agreement, with thedistrict or school?
Does the program contribute to building the capacity of the district or school?Overall cost and cost related to program scalability and effectiveness
What is the cost per student, per teacher, and per school? What would it cost to scale the program statewide? What is the cost effectiveness of the program? Specifically, what is the return on investment for
the program?
8/2/2019 Massachusetts Math and Science Initiative Evaluation
22/28
Appendix B
Evaluation Methodology
B - 1
Data Set
The data set used in the analysis contained school-level counts ofthe number of eligible students (defined as
the number of juniors and seniors in a school, and by ethnicity and gender), the number of exams taken, and
the number of exams scoring 3 or higher for MMSI schools and Comparison Schools. The data set included
separate counts for each school by ethnicity (African American, Hispanic, White, Asian) and gender for each of
the three AP content areasMathematics, Science, and Englishthat are part of the MMSI program. For
instance, the data set used in the analysis included the number of exams taken (and the number of exams scoring
3 or higher) forMath, but not separately for AB Calculus, BC Calculus, and Statistics . Also, data on low-
income students was not available for all schools and subsequently was not used in the analysis.
Calculating Participation and Success Rates
Participation Rates were computed by (1) dividing the number of exams taken (for a particular group) by the
total number of eligible students for a particular group and then (2) multiplying by 1000.
Success Rates were computed by (1) dividing the number of exams scoring 3 or higher (for a particular group)
by the total number of eligible students for a particular group and then (2) multiplying by 1000.
Cohort Analysis
The MMSI program has accepted and initiated work with a new cohort of schools on an annual basis, beginning
with Cohort 1 in 2008-09. In the analysis, the baseline for each cohort was computed by aggregating the
number of eligible students (and related number of exams taken and number of exams scoring 3 or higher) from
the two years prior to the school beginning the MMSI program. The latest year for which data is available is the
2010-11 school year.
Table B1. Computation of Baseline Data for MMSI Cohort Schools
Baseline Years in MMSI
Cohort 1 (n=8) 06-07 and 07-08 08-09, 09-10, 10-11 and 11-12
Cohort 2 (n=11) 07-08 and 08-09 09-10, 10-11 and 11-12
Cohort 3 (n=26) 08-09 and 09-10 10-11 and 11-12
Chart 1 and Table 6 (in the body of the report) present baseline and annual data for cohorts 1, 2, and 3. As
displayed in Table B1, the baselines do not come from the same years; rather, they are the baseline for the
particular cohort. The data for statewide, non-MMSI schools uses the same baseline and annual data as Cohort
1. To the point, a valid statistical comparison can be made between Cohort 1 and the non-MMSI schools, as is
presented on Chart 1. However, we also wanted to look at the relationship among cohorts, which is why we
combined the cohort data in one chart, with each cohorts baseline presented as the same baseline year.
Comparison Analysis
The District Analysis and Review Tool (DART) developed by the Massachusetts Department of Elementary and
Secondary Education was used to identify the comparison schools used in the analysis. Extensive information on
the DART, including the statistical method used by DART to identify comparable districts and schools, is
located at: http://www.doe.mass.edu/apa/dart/ .
While the DART is designed to identify comparable districts and schools for a single district (or school), the
tool is not specifically designed to identify a set of comparison schools for a group of schools. However, the
DART provides an excellent starting point for identifying a potential pool of comparison schools. The following
is the process used by INSTLL to use the DART to develop a meaningful set of comparison schools.
1. We grouped the MMSI schools into three groups, based on percentage of low-income students, each schools
accountability status, and based on discussions with MMSI program staff. As described in the body of the
report, we used the following criteria:
Group A Criteria: Low Income greater than (>) 60 percent or School Accountability Status: Level 3 or 4
8/2/2019 Massachusetts Math and Science Initiative Evaluation
23/28
Appendix B
Evaluation Methodology
B - 2
Group B Criteria: Low Income greater than (>) 35 percent and does not meet Group 1 Criteria Group C Criteria: Low Income less than (
8/2/2019 Massachusetts Math and Science Initiative Evaluation
24/28
C - 1
APPENDIX C
High school students who graduated between 2008-2011 from an high school that is or was participatingwith the Mass Math + Science Initiatives (MMSI) Advanced Placement Training and Awards Program
(APTAP), and who were enrolled in an AP math, science and/or English course(s), are or were enrolled at
the colleges/universities listed below.
The color gradient column indicates the number of MMSI APTAP high schools that the
college/university has received graduates from.
ADRIAN COLLEGE 1
ALBANY COLLEGE OF PHARMACY AND HEALTH
SCIENCES2
ALBERTUS MAGNUS COLLEGE 1ALVERNIA UNIVERSITY 1
AMERICAN INTERNATIONAL COLLEGE 7
AMERICAN UNIVERSITY 5
ANDREWS UNIVERSITY 2
ANNA MARIA COLLEGE 7
ARIZONA STATE UNIVERSITY 2
ASHLAND UNIVERSITY 1
ASNUNTUCK COMMUNITY COLLEGE 1
ASSUMPTION COLLEGE 18
ATLANTA METROPOLITAN COLLEGE 1
AUBURN UNIVERSITY 1
BARNARD COLLEGE 1
BARRY UNIVERSITY 1
BATES COLLEGE 4
BAY PATH COLLEGE 3
BAY STATE COLLEGE 5
BECKER COLLEGE 11
BENJAMIN FRANKLIN INSTITUTE OF TECHNOLOGY 1
BENNINGTON COLLEGE 1
BENTLEY UNIVERSITY 11
BERKLEE COLLEGE OF MUSIC 5
BOSTON COLLEGE 17
BOSTON UNIVERSITY 18
BOWDOIN COLLEGE 3
BRANDEIS UNIVERSITY 13
BRIDGEWATER STATE UNIVERSITY 28
BRIGHAM YOUNG UNIVERSITY 2
BRIGHAM YOUNG UNIVERSITY -IDAHO 1
WINTER/SPRING
BRISTOL COMMUNITY COLLEGE
BROWN UNIVERSITY
BRYANT UNIVERSITY 1
BRYN MAWR COLLEGE
BUCKNELL UNIVERSITY
BUFFALO STATE COLLEGE
BUNKER HILL COMMUNITY COLLEGE 1
BUTTE COMMUNITY COLLEGE
CALIFORNIA INSTITUTE OF TECHNOLOGY
CALIFORNIA STATE UNIVERSITY - FULLERTON
CAMBRIDGE COLLEGE
CAPE COD COMMUNITY COLLEGE
CAPITAL COMMUNITY COLLEGE
CARLETON COLLEGE
CARNEGIE MELLON UNIVERSITY
CASTLETON STATE COLLEGE
CENTRAL CONNECTICUT STATE UNIVERSITY
CENTRAL PIEDMONT COMMUNITY COLLEGE
CENTRE COLLEGE
CHAMPLAIN COLLEGE
CHARLESTON SOUTHERN UNIVERSITY
CHATTAHOOCHEE TECHNICAL COLLEGE
CLARK UNIVERSITYCLARKSON UNIVERSITY
CLEMSON UNIVERSITY
CLEVELAND INSTITUTE OF ART
COASTAL CAROLINA UNIVERSITY
COLBY COLLEGE
COLBY SAWYER COLLEGE
COLGATE UNIVERSITY
8/2/2019 Massachusetts Math and Science Initiative Evaluation
25/28
C - 2
COLLEGE OF CHARLESTON 2
COLLEGE OF THE HOLY CROSS 19
COLLEGE OF WILLIAM & MARY 2
COLLEGE OF WOOSTER 1
COLORADO COLLEGE 1
COLORADO STATE UNIVERSITY 1
COLUMBIA COLLEGE CHICAGO 2
COMMUNITY COLLEGE OF RHODE ISLAND 1
COMMUNITY COLLEGE OF VERMONT 1
CONNECTICUT COLLEGE 2
CORNELL UNIVERSITY 3
CULINARY INSTITUTE OF AMERICA 1
CUNY BERNARD M. BARUCH COLLEGE 1
CUNY HUNTER COLLEGE 1
CUNY QUEENS COLLEGE 1
CURRY COLLEGE 14
DANIEL WEBSTER COLLEGE 2
DARTMOUTH COLLEGE 1
DEAN COLLEGE 8
DENISON UNIVERSITY 4
DEPAUL UNIVERSITY 1
DES MOINES AREA COMMUNITY COLLEGE 1
DICKINSON COLLEGE 1
DRAKE UNIVERSITY 1
DREXEL UNIVERSITY 2
EARLHAM COLLEGE 1EASTERN CONNECTICUT STATE UNIVERSITY 3
EASTERN NAZARENE COLLEGE 8
ECKERD COLLEGE 1
ELIZABETH CITY STATE UNIVERSITY 1
ELMIRA COLLEGE 5
ELMS COLLEGE 11
ELON UNIVERSITY 4
EMBRY-RIDDLE AERONAUTICAL UNIVERSITY -
ARIZONA1
EMBRY-RIDDLE AERONAUTICAL UNIVERSITY -
DAYTONA1
EMERSON COLLEGE 16
EMMANUEL COLLEGE 18
ENDICOTT COLLEGE 10
ENTERPRISE STATE COMMUNITY COLLEGE 1
EVANGEL UNIVERSITY 1
EVEREST COLLEGE - CHELSEA 1
FAIRFIELD UNIVERSITY 3
FAIRLEIGH DICKINSON UNIVERSITY - TEANECK 2
FISHER COLLEGE-TRADITIONAL
FITCHBURG STATE UNIVERSITY 1
FLORIDA A&M UNIVERSITY
FLORIDA INSTITUTE OF TECHNOLOGY
FLORIDA INTERNATIONAL UNIVERSITY
FLORIDA STATE UNIVERSITY
FORDHAM UNIVERSITY
FRAMINGHAM STATE UNIVERSITY 3
FRANKLIN PIERCE UNIVERSITY
GEORGE MASON UNIVERSITY
GEORGE WASHINGTON UNIVERSITY
GEORGETOWN UNIVERSITY
GEORGIA HIGHLANDS COLLEGE
GEORGIA INSTITUTE OF TECHNOLOGY
GETTYSBURG COLLEGE
GORDON COLLEGE
GOUCHER COLLEGE
GREENFIELD COMMUNITY COLLEGE
GRINNELL COLLEGE
GROVE CITY COLLEGE
GUILFORD COLLEGE
HAMILTON COLLEGE
HAMPSHIRE COLLEGE
HAMPTON UNIVERSITY
HARTWICK COLLEGE
HARVARD UNIVERSITYHARVARD UNIVERSITY - CONTINUING ED 1
HAVERFORD COLLEGE
HESSER COLLEGE - SALEM
HOBART & WILLIAM SMITH COLLEGES
HOFSTRA UNIVERSITY
HOLYOKE COMMUNITY COLLEGE
HOPE COLLEGE
HOUGHTON COLLEGE
HOWARD UNIVERSITY
HUSSON COLLEGE
IONA COLLEGE
ITHACA COLLEGE
JOHNSON & WALES UNIVERSITY 1
JOHNSON STATE COLLEGE
JUNIATA COLLEGE
KEENE STATE COLLEGE 1
KENT STATE UNIVERSITY
KENYON COLLEGE
8/2/2019 Massachusetts Math and Science Initiative Evaluation
26/28
C - 3
LA SALLE UNIVERSITY 2
LAKE FOREST COLLEGE 1
LASELL COLLEGE 3
LE MOYNE COLLEGE 2
LEE UNIVERSITY 1
LEHIGH UNIVERSITY 1
LESLEY UNIVERSITY 10
LEWIS & CLARK COLLEGE OF ARTS & SCIENCES 1
LINDENWOOD UNIVERSITY 1
LIPSCOMB UNIVERSITY 1
LONG ISLAND UNIVERSITY 1
LOYOLA UNIVERSITY IN NEW ORLEANS 1
LYCOMING COLLEGE 1
LYNCHBURG COLLEGE 1
LYNDON STATE COLLEGE 2
LYNN UNIVERSITY 1
MACALESTER COLLEGE 2
MACOMB COMMUNITY COLLEGE 1
MAINE MARITIME ACADEMY 2
MARIST COLLEGE 4
MARLBORO COLLEGE 1
MARYLAND INSTITUTE, COLLEGE OF ART 1
MARYMOUNT MANHATTAN COLLEGE 2
MASSACHUSETTS BAY COMMUNITY COLLEGE 9
MASSACHUSETTS COLLEGE OF ART 11
MASSACHUSETTS COLLEGE OF LIBERAL ARTS 14MASSACHUSETTS COLLEGE OF PHARMACY 22
MASSACHUSETTS INSTITUTE OF TECHNOLOGY 4
MASSACHUSETTS MARITIME ACADEMY 3
MASSASOIT COMMUNITY COLLEGE 8
MERCYHURST COLLEGE 1
MERRIMACK COLLEGE 12
MIAMI UNIVERSITY 1
MICHIGAN STATE UNIVERSITY 1
MIDDLEBURY COLLEGE 2
MIDDLESEX COMMUNITY COLLEGE 5
MINNESOTA STATE UNIVERSITY - MANKATO 1
MISSISSIPPI STATE UNIVERSITY 1
MONTANA STATE UNIVERSITY - BOZEMAN 1
MONTCLAIR STATE UNIVERSITY 1
MONTSERRAT COLLEGE OF ART 3
MORRIS COLLEGE 1
MOUNT HOLYOKE COLLEGE 4
MOUNT IDA COLLEGE 8
MOUNT WACHUSETT COMMUNITY COLLEGE
MUHLENBERG COLLEGE
NAZARETH COLLEGE OF ROCHESTER
NEW ENGLAND COLLEGE-SEMESTERS
NEW ENGLAND INSTITUTE OF TECHNOLOGY
NEW RIVER COMMUNITY COLLEGE
NEW YORK UNIVERSITY
NEWBURY COLLEGE
NHTI - CONCORD'S COMMUNITY COLLEGE
NICHOLS COLLEGE
NORMANDALE COMMUNITY COLLEGE
NORTH CAROLINA STATE UNIVERSITY
NORTH PARK UNIVERSITY
NORTH SHORE COMMUNITY COLLEGE
NORTHEASTERN UNIVERSITY 2
NORTHEASTERN UNIVERSITY - LAW/SPCS
NORTHERN ESSEX COMMUNITY COLLEGE
NORTHWESTERN UNIVERSITY
NORWICH UNIVERSITY
NYACK COLLEGE - ROCKLAND UG
OBERLIN COLLEGE
OZARKS TECHNICAL COMMUNITY COLLEGE
PACE UNIVERSITY
PACE UNIVERSITY - PLEASANTVILLE
PENNSYLVANIA STATE UNIVERSITY 1
PHILADELPHIA UNIVERSITYPITZER COLLEGE
PLYMOUTH STATE UNIVERSITY
POLYTECHNIC INSTITUTE OF NEW YORK
UNIVERSITY
POMONA COLLEGE
PRATT INSTITUTE
PRINCETON UNIVERSITY
PROVIDENCE COLLEGE 1
PURDUE UNIVERSITY - WEST LAFAYETTE
QUEENS UNIVERSITY OF CHARLOTTE
QUINCY COLLEGE
QUINNIPIAC UNIVERSITY 1
QUINSIGAMOND COMMUNITY COLLEGE
REGIS COLLEGE 1
RENSSELAER POLYTECHNIC INSTITUTE
RHODE ISLAND COLLEGE
RHODE ISLAND SCHOOL OF DESIGN
RICHLAND COLLEGE-DALLAS CC DISTRICT
RIVIER COLLEGE
8/2/2019 Massachusetts Math and Science Initiative Evaluation
27/28
C - 4
ROANOKE COLLEGE 3
ROCHESTER INSTITUTE OF TECHNOLOGY 4
ROGER WILLIAMS UNIVERSITY 9
ROLLINS COLLEGE 1
ROOSEVELT UNIVERSITY 1
ROXBURY COMMUNITY COLLEGE 7
SACRED HEART UNIVERSITY 4
SAINT ANSELM COLLEGE 7
SAINT JOSEPH'S COLLEGE OF MAINE 3
SAINT MICHAELS COLLEGE 6
SALEM STATE UNIVERSITY 17
SALVE REGINA UNIVERSITY 6
SANTA MONICA COLLEGE 1
SANTA ROSA JUNIOR COLLEGE 1
SARAH LAWRENCE COLLEGE 1
SAVANNAH COLLEGE OF ART & DESIGN 1
SCHOOL OF VISUAL ARTS 2
SEMINOLE STATE COLLEGE OF FLORIDA 1
SHENANDOAH UNIVERSITY 1
SIENA COLLEGE 1
SIMMONS COLLEGE 12
SIT GRADUATE INSTITUTE NON TRADITIONAL 1
SKIDMORE COLLEGE 3
SMITH COLLEGE 10
SOUTHERN MAINE COMMUNITY COLLEGE 2
SOUTHERN NEW HAMPSHIRE UNIVERSITY-
UNGRAD DAY 8
SOUTHERN UTAH UNIVERSITY 1
SOUTHERN VERMONT COLLEGE 3
SOUTHWESTERN ADVENTIST UNIVERSITY 1
SOUTHWESTERN OKLAHOMA STATE UNIVERSITY 1
SPELMAN COLLEGE 1
SPRINGFIELD COLLEGE 8
SPRINGFIELD TECHNICAL COMMUNITY COLLEGE 8
ST EDWARDS UNIVERSITY 1
ST JOHNS RIVER COMMUNITY COLLEGE 1
ST JOHNS UNIVERSITY 5
ST JOSEPH COLLEGE 1
ST LAWRENCE UNIVERSITY 2
ST MARY'S COLLEGE OF CALIFORNIA 2
ST MARYS COLLEGE OF MARYLAND 1
ST OLAF COLLEGE 1
STANFORD UNIVERSITY 1
STETSON UNIVERSITY 2
STEVENS INSTITUTE OF TECHNOLOGY 2
STONEHILL COLLEGE 1
SUFFOLK UNIVERSITY 2
SUNY COLLEGE AT FREDONIA
SUNY COLLEGE AT PURCHASE
SUNY COLLEGE PLATTSBURGH
SUNY FARMINGDALE
SUNY FASHION INSTITUTE OF TECHNOLOGY
SUNY FULTON-MONTGOMERY COMMUNITY
COLLEGE
SUNY MOHAWK VALLEY COMMUNITY COLLEGE
SUNY OSWEGO
SUSQUEHANNA UNIVERSITY
SWARTHMORE COLLEGE
SYRACUSE UNIVERSITY 1
TEMPLE UNIVERSITY
TEXAS A&M UNIVERSITY - COMMERCE
TEXAS CHRISTIAN UNIVERSITY
THE CATHOLIC UNIVERSITY OF AMERICA
THE CITADEL MILITARY COLLEGE OF SOUTH
CAROLINA
THE EVERGREEN STATE COLLEGE
THE NEW SCHOOL
THOMAS NELSON COMMUNITY COLLEGE
TRINITY UNIVERSITY
TUFTS UNIVERSITY 1
TULANE UNIVERSITY
TULSA COMMUNITY COLLEGE
UNION COLLEGE
UNIVERSITY OF ALABAMA
UNIVERSITY OF ARIZONA
UNIVERSITY OF BRIDGEPORT
UNIVERSITY OF CALIFORNIA-BERKELEY
UNIVERSITY OF CALIFORNIA-SAN DIEGO
UNIVERSITY OF CALIFORNIA-SANTA BARBARA
UNIVERSITY OF CALIFORNIA-SANTA CRUZ
UNIVERSITY OF CENTRAL FLORIDA
UNIVERSITY OF CHICAGO
UNIVERSITY OF CONNECTICUT 1
UNIVERSITY OF DAYTON
UNIVERSITY OF DELAWARE
UNIVERSITY OF FLORIDA
UNIVERSITY OF HARTFORD 1
UNIVERSITY OF HAWAII AT MANOA
UNIVERSITY OF ILLINOIS @ URBANA
UNIVERSITY OF MAINE, FARMINGTON
8/2/2019 Massachusetts Math and Science Initiative Evaluation
28/28
UNIVERSITY OF MAINE, ORONO 3
UNIVERSITY OF MARYLAND - COLLEGE PARK 3
UNIVERSITY OF MARYLAND - UNIVERSITY
COLLEGE1
UNIVERSITY OF MASSACHUSETTS AT AMHERST 41
UNIVERSITY OF MASSACHUSETTS AT LOWELL 27
UNIVERSITY OF MASSACHUSETTS BOSTON 31
UNIVERSITY OF MASSACHUSETTS-DARTMOUTH 31
UNIVERSITY OF MIAMI 4
UNIVERSITY OF NEW ENGLAND 8
UNIVERSITY OF NEW HAMPSHIRE 17
UNIVERSITY OF NEW HAVEN-SEMESTERS 9
UNIVERSITY OF NEW MEXICO 1
UNIVERSITY OF NORTH CAROLINA-PEMBROKE 1
UNIVERSITY OF NORTH CAROLINA-WILMINGTON 4
UNIVERSITY OF OREGON, MAIN CAMPUS 1
UNIVERSITY OF PENNSYLVANIA 3
UNIVERSITY OF PHOENIX 2
UNIVERSITY OF PITTSBURGH 3
UNIVERSITY OF RHODE ISLAND 16
UNIVERSITY OF ROCHESTER 5
UNIVERSITY OF SAN DIEGO 1
UNIVERSITY OF SAN FRANCISCO 2
UNIVERSITY OF SOUTH CAROLINA 1
UNIVERSITY OF SOUTH FLORIDA 3
UNIVERSITY OF SOUTHERN CALIFORNIA 2
UNIVERSITY OF SOUTHERN MAINE 3
UNIVERSITY OF TAMPA 2
UNIVERSITY OF THE ARTS 1
UNIVERSITY OF THE PACIFIC 1
UNIVERSITY OF VERMONT & STATE
AGRICULTURAL COLLEGE11
UNIVERSITY OF VIRGINIA 2
UNIVERSITY OF WISCONSIN - MADISON 1
URSINUS COLLEGE 2
UTAH STATE UNIVERSITY 1
UTICA COLLEGE 1
VASSAR COLLEGE 2
VENTURA COLLEGE
VILLANOVA UNIVERSITY
VIRGINIA COMMONWEALTH UNIVERSITY
VIRGINIA POLYTECH AND STATE UNIV
VIRGINIA STATE UNIVERSITY
WAKE FOREST UNIVERSITY
WASHINGTON ADVENTIST UNIVERSITY
WASHINGTON UNIVERSITY
WELLESLEY COLLEGE
WENTWORTH INSTITUTE OF TECHNOLOGY 2
WESLEY COLLEGE
WESLEYAN UNIVERSITY
WESTERN NEW ENGLAND UNIVERSITY 1
WESTFIELD STATE UNIVERSITY 2
WESTMINSTER COLLEGE OF SALT LAKE CITY
WHEATON COLLEGE
WHEELOCK COLLEGE 1
WILLIAMS COLLEGE
WINGATE UNIVERSITY
WOFFORD COLLEGE
WORCESTER POLYTECHNIC INSTITUTE 2
WORCESTER STATE UNIVERSITY 2
WYOTECH - BLAIRSVILLE
YALE UNIVERSITY
YESHIVA UNIVERSITY
Total Colleges/Universities: 384