1

Alliances for Graduate Education and the Professoriate

Comparison Groups and Other issues

September 18, 2008

byCatherine M. Millett, Ph.D.

Policy Evaluation & Research, ETS

2

Educational Realities

The Circumstances:• Some may believe that graduate education is a luxury• Graduate degree holders could be viewed as an elite population –

Among people 25 and over 1.3% have a Doctorate degree and 7% a Master’s degree

• NSF receives 2.5% of the $166.5 billion federal budget for education in 2006

The Result:• All of us have to continually get out the word that graduate

education is important and AGEP is important.

3

About AGEP

• October 1998, NSF awarded 8 universities nearly $2.5 million each to significantly increase the number of African American, Hispanic and Native American students receiving SME degrees.

• As of June 2008, there are 28 AGEP & SBE alliances• AGEP seeks to determine the college and university

policies, practices, and support services that lead to increased numbers of minority SME undergraduates that then enter into graduate school and professorial careers.

4

Roosevelt’s Words of Wisdom

• 2 cycles of the program have been completed. What do you expect?

• Evolution of external perception– “This won’t work”– “AGEP isn’t working”– “While AGEP isn’t responsible for changes

it is making a difference”• Now is the time to take stock of what has

been accomplished and what needs to be accomplished.

5

Micro and Macro View of AGEP

AGEP

Alabama AGEPMichigan AGEP

AlliancePROMISE

Rice-Houston AGEP

Who can best tell the Macro successes?Who can best tell the Micro successes?

Macro View - AGEP Alliance Models:

Program Elements Alabama Michigan Rice Houston

Promise

Recruitment trips Yes Yes Yes Yes

On-campus multi-week orientation program

Yes

Mentoring Yes Yes Yes Yes

Annual student reviews Yes Yes

Financial support to attend professional conferences

Limited Yes Yes

Publications – sole authored, co-authored with faculty and students

Yes Yes Yes

Time to Degree - Complete degree plus/minus 1.5 years of national average within field

Yes Yes Yes

Earn a tenure track job at XXX? Yes Yes Yes Yes

• Common goal: increase degree recipients

5

FABRICATED DATA

7

Individual Alliance Reports

• Showcase the Alliance’s accomplishments – Veronica told us to “Tell Your Story”

• What is your AGEP student experience?• Think about the various audiences for your reports – NSF, public

policy makers, university administrators, other faculty, the public, prospective and current graduate students.– What are we paying for and why? (e.g. trips to conferences, faculty

recruiting trips?)

• Provide information to NSF and other evaluators that may not be in AGEP wide report:– Students’ presentations and awards– Press coverage– NSF employees participation in Alliance events– How you are leveraging AGEP funding

8

Figure 2 – Hierarchy of Study Designs for Evaluating the Effectiveness of a STEM Education Intervention, by Expected Distribution of Study Type

EXPERIMENTAL:

Such as Well-designed Randomized Controlled Trials

QUASI EXPERIMENTAL:

Such as Well-matched Comparison-Group Studies

OTHER DESIGNS:

Such as Pre-Post Studies, and Comparison-Group Studies without careful matching

Source: Coalition for Evidence-Based Policy, 2006U.S. Department of Education. (May 2007) Report of the Academic Competitiveness Council

9

Putting Your Date in Context - Graduate Education Sources

• Graduate Record Examinations – Guide to the Use of Scores 2207-2008.

• US Department of Education (NCES 2007-162). The Path Through Graduate School; A Longitudinal Examination 10 Years After Bachelors’ Degree

• US Department of Education (NCES2006-185). Student Financing of Graduate and First-Professional Education, 2003-2004: Profiles of Students in Selected Degree Programs and Part-Time Students

• Survey of Earned Doctorates: 2006 Doctorate Recipients from United States Universities: Summary Report

• http://nces.ed.gov/das/library/nedrc_tables.asp• Professional societies may have data.

10

Variations in Defining AGEP Population

Comparison Groups – The Goal Fill in the sentence: AGEP could be making a difference in….

Answer the question: Is your program making a difference?

Goal of using comparison groups – provide credible evidence that your program could be making a difference

The Ideal Members:• Graduate students who have not participated in AGEP• Graduate students who are similar to AGEP participants• Have characteristics that the research base demonstrates are important

relative to the intended outcomes of the program.

Random Assignment to AGEP Treatments• Do you have an adequate number of students to do random assignment?• Do faculty and administrators support random assignment (e.g. in

admissions?)• Could you randomly assign students to other experiences in the AGEP

program (e.g. half of students receive full conference funding versus the other half who get 50% of conference support).

12

Other issues to consider

Naturally Found Groups• Applicants accepted to your graduate program but not

supported by AGEP.• Graduate students from pre-AGEP years• Graduate students who participate in other educational

programs from other agencies (e.g. NIH, Energy).• Students in graduate schools who are not in the AGEP AlliancesPossibilities:• Students who attend summer orientation versus those who do

not.• Students who attend semester long series on writing research

papers compared to those who do not.• Students who take a particular academic class (.e.g. statistics)

versus those who do not.

13

Practical Issues

Securing Human Subjects Approval

Accessing data

Planning for staff time and ability

Incentives

Response rates

Cost – if you have an evaluation budget remember to report how it is being spent and if you need more talk to your program officer. All they can say is no.

Confidentiality in reporting data

14

Get the A for Effort!