Date post: | 19-Dec-2015 |
Category: |
Documents |
View: | 216 times |
Download: | 0 times |
National Research CouncilNational Research Council
Assessment of Research Assessment of Research Doctorate ProgramsDoctorate Programs
Summary of Methodology
http://www.nap.edu/catalog.php?record_id=12676
Michigan State University
What is it?• National Academies of Science “rate” research
doctorate programs approx. every 13 years (1982, 1995, 2009).
http://sites.nationalacademies.org/pga/Resdoc/index.htm
1995 was largely a reputational (only) survey.
• In 2003, NRC published a study on the methods used in 1995 with recommendations on changes for 2005-06 data collection (for 2009 ratings).
http://books.nap.edu/catalog.php?record_id=10859
2
Scope and Coverage of the Current Study
• 222 institutions
• 61 fields (MSU ranked in 54 fields)
• 5006 programs across the 61 fields
• Each field of study had produced at least 500 Ph.D.s in the 5 years prior to 2004-05.
• To be included, each University program must have granted 5 Ph.D. degrees in the 5 years prior to 2005-06.
3
Current NRC study
• The taxonomy of fields was hotly debated. Used CIP codes (US Dept of Ed taxonomy), NSF fields, & info from scholarly societies.
• It is what it is! Not very useful for interdisciplinary programs as faculty effort was sub-divided. Universities use different names for similar programs.
• Learn more:http://sites.nationalacademies.org/PGA/Resdoc/PGA_044478 4
54 MSU programs in current NRC studyAgricultural Economics* Electrical Engineering Mechanical Engineering
American Studies* English Microbiol & Molec Genetic
Animal Science Entomology* Music Education
Anthropology Environmental Tox icology Neuroscience
Astrophysics & Astronomy* Fisheries & Wildlife Pathobiol & Diagnostic Inv
Biochem & Molecular Biology Food Science* Pharmacology/Toxicology
Biosystems Engineering Forestry* Philosophy
Cell & Molecular Biology Genetics Physics
Chemical Engineering Geography Physiology
Chemistry Geological Sciences Plant Biology
Civil & Environmental Eng History Plant Breeding & Genetics
Communications Horticulture Plant Pathology
Comm Arts Media & Info Studies Human Nutrition Political Science
Computer Science Kinesiology Psychology
Criminal Justice Large Animal Clinical Sci Sociology
Crop & Soil Sciences Linguistics Spanish Lang, Lit, Culture
Ecology, Evol Bio, & Behavior Materials Science & Eng Statistics
Economics Mathematics Zoology
5
Current NRC studyFaculty effort assignment to program(s) (not always
based on tenure/salary home unit):
– CORE (supervise dissertations and/or on admissions/curriculum committees for PhD)
– NEW (hired in previous 3 years & expected to become Core)
– ASSOCIATED (not Core in program, but regular faculty at institution)
Note: A faculty member’s program assignment may be divided between various programs, but will always total 100% Dept chairs made final decisions.
6
Current NRC study
• Data collection was in 2005-06
• Data points collected over various spans of time (Question G)
• Approx. 25%+ of MSU faculty changed between 05-06 and present
• Approx. 94% questionnaire participation rate by MSU faculty
7
Five Questionnaires1. Institutional Questionnaire: U practices and a list
of doctoral programs
2. Program Questionnaire: Characteristics of students, faculty, program
3. Faculty Questionnaire: Faculty work history, grants, pubs, and important characteristics of a quality program (Question G)
4. Student Questionnaire: Students (post comps in English,
Chem Eng, Economics, Physics, Neuroscience) background, faculty interactions, & post-graduation plans
8
Questionnaires
5. Rating Questionnaire: Asked faculty to rate programs in their field
• Raters classified by rank, geographic region, faculty size in program. Each faculty member rated 15 programs with data provided. Not asked about basis for rating.
9
Current NRC study
• In addition to the questionnaires, publications and citations data were collected thru ISI database http://isiwebofknowledge.com/.
• Humanities CVs submitted with faculty questionnaire were used to count books and publications.
• Honors and awards data came from 224 scholarly societies.
10
The 20 Variables (Question G on Faculty Questionnaire)• Publications per Allocated Faculty*, 2001-2006 (going back to 1986 for faculty in
humanities fields)
• Average Citations per Publication (citations in 2001-2006 to articles dating back to 1981—for all fields except the humanities)
• Number of Grants per Allocated Faculty*
• Percent Interdisciplinary (% Associated Faculty)
• Percent Non-Asian Minority Faculty for Core or New Faculty, 2006**
• Percent Female Faculty for Core or New Faculty, 2006**
• Awards per Allocated Faculty*
• Average GRE, 2004-2006 (Verbal measure for the humanities, Quantitative measure for all other fields)
• Percent students receiving full support in the first year, (fall 2006)
• Percent first year students with external funding, 2006
* Faculty members who served in more than one program were allocated to those programs based on whether they were core in the program and the share of that program of total dissertations supervised. 11
The 20 Variables (continued)• Percent Non-Asian Minority Students, 2006
• Percent Female Students, 2006
• Percent International Students, 2006
• Average annual PhDs graduated 2002 to 2006
• Average completions (8 year completion percentage for humanities fields, 6 years for other fields)
• Time-to-Degree (for Full and Part Time graduates)
• Percent PhDs with definite plans for an academic position, 2001-2005 (including postdoctoral fellowships)
• Student Work Space [1 = 100% of students with workspace, -1 if <100% of students with workspace]
• Health Insurance [1= provides health insurance, -1 = does not provide health insurance]
• Student Activities (number offered from a list of 18)
** “Core” Faculty are those whose primary appointment is in the doctoral program. “New” faculty are faculty with tenure track appointments who were appointed in the past 3 years.
12
2009 NRC Methodology report
MSU task force of statistical/survey experts to study the report and provide input:
Brian Silver -- Director, Center for Statistical Consulting, Political Science
Neal Schmitt -- Chair, Department of Psychology
Les Manderscheid -- Ag, Food and Resource Economics and the Graduate School
Mary Black -- Assistant Director, Office of Planning and Budgets
Kyle Sweitzer -- Data Analyst, Office of Planning and Budgets
CONVENER: Karen Klomparens -- Dean, Graduate School
Methodology was complex and valid, with sources of uncertainty dealt with appropriately.
PROGRAMS WILL PROVIDE FURTHER INTERPRETATION OF THE DATA 13
Ranges of overall ratingstwo methods
1. Directly: Faculty chose the most important quality characteristics from Question G
2. Reputational rating: statistically related faculty ratings to the Question G variables.
Combined the direct and reputational weights, rank ordered. Used first and third quartiles of these ratings to yield a RANGE OF RATINGS for each program
15
Additional information• No student outcomes are reflected in the overall
rating, so 3 supplemental measures added
• NRC preparing a way for us to weight variables differently and run own ratings
• Not reporting data with less than 5 per cell• Combined small progs with others to calculate
weights
• Preparing a way for students to use data when considering programs
16
Supplemental RatingsResearch Impact
– Publications/faculty member– Citations/publication– Percent of faculty holding grants– Honors and awards per faculty member
Student Support and Outcomes
– Percent of students having full support in first year– Percent of students with portable fellowships in first year– Percentage of students with RAships– Percentage of students with TAships– Time-to-degree– Percent who complete in 6 years (sciences) or 8 years (humanities)– Placement in an academic position or postdoc after graduation
Source: C. Kuh, NAS
17
Supplemental Ratings (2)Diversity of the Academic Environment
– Percent of students who are female
– Percent of faculty who are female
– Percent of students from underrepresented minority group
– Percent of faculty from underrepresented minority group
– Percent of students who are international
Source: C. Kuh, NAS
18
Ranges of Ratings for a Field
• Programs will be arranged alphabetically and the range of ratings will be given for each.
• Ranges overlap other ranges for most programs. This means that there may be a number of programs of roughly the same quality.
• You should identify those similar programs in discussing the quality of your programs.
Source: C. Kuh, NAS19
What information will I receive about the rating calculation for my programs?
1) A list of the values of variables that your program supplied to the NRC or that was calculated from those variables
2) The normalized values for those variables
3) The median combined coefficient (statistical + direct) for each variable and its standard deviation
4) The range of the normalized variable values
5) The range of the combined effects of the coefficients in the random halves calculation
Source: C. Kuh, NAS 20
21
TABLE 5-1 Data and Coefficient Table for a Program in Economics
Standardized Program Values and Range of Combined Coefficients Institution Name: xxx Program Name: yyy
Publications per Allocated Faculty V1 1.074 2.180 0.118 0.132Cites per Publication V2 1.171 -0.234 0.276 0.307Percent of Faculty with Grants V3 25.50% -0.583 0.084 0.091Percent of Faculty Interdisciplinary V4 5.90% -0.641 n.s.# n.s.#Percent of Non-Asian Minority Faculty V5 7.70% 0.547 n.s.# n.s.#
Percent Female Faculty V6 12.50% -0/440 n.s.# n.s.#Awards per allocated faculty V7 0 -0.546 0.043 0.060Average GRE-Q V8 746 -0.165 0.092 0.096Percent 1st yr. Students w/full support V9 100.00% 0.980 0.036 0.056
Percent 1rst yr. Students w/portable fellowships V10 0.00% -.544 0.021 0.033
Percent non-Asian Minority Students V11 10.00% 0.069 n.s.# n.s.#
Percent Female Students V12 44.40% 0.678 -0.038 -0.030Percent International Students V13 53.30% -0.509 n.s.# n.s.#Average PhDs 2002 to 2006 V14 5.4 -.0355 0.120 0.144Percent Completing within 6 years V15 27.60% -0.638 n.s.# n.s.#Time to Degree full and Part time V16 5.67 0.232 -0.028 -0.017Percent students in Academic Positions V17 11.10% -1.405 0.049 0.065
Student Work Space V18 1 1 n.s.# n.s.#Health Insurance V19 1 1 n.s.# n.s.#Number of Student Activities Offered V20 17 0.439 0.026 0.037
Column 1 Column 2 Column 3 Column 4 Column 5 Column 6 Combined Coefficients**
Program Program Value Description Variable Value Standardized Minus 1 SD TO Plus 1 SD
*Col 3 is based on data submitted by the program or calculated from these data.+ Col 4 is standardized across all program values in the field, with mean of 0 and variance of 1.** Col 5 is Minus 1 Standard Deviation from the Mean for the combined coefficients for the field as a whole** Col 6 is Plus 1 Standard Deviation from the Mean for the combined coefficients for the field as a whole# n.s. in a cell means the coefficient was not significantly different from 0 at the p=.05 level.
Analyzing the data by program
1. Identify variables with the largest contribution. These had the greatest effect on the range of ratings by program.
2. Compare your variable values with programs in other institutions. These will be available in an online database.
3. Consider any additional relevant comments about your program and the NRC methods.
26
Next steps• The data will be made available 24-72 hours before
they are publically released, and programs may review their data during this time. COMMENTS TO KK!
• Additional analyses will be conducted upon release of the data (by CIC, AAU, and by you!)
• This study improved MSU’s ability to have data readily available in a consistent fashion about students and committee memberships. Data collection has expanded to include ALL graduate programs (Master’s and Doctoral) via the GradInfo database.
www.gradinfo.msu.edu See next slide…27
Weights for Supplemental MeasuresResearch Activity
Pubs per Cites per Pct Fac Awards
Broad Field Faculty Pub w grants per fac
Biological Sciences 0.30 0.21 0.36 0.13
Health Sciences 0.35 0.16 0.37 0.11
Engineering 0.29 0.25 0.29 0.17
Physical Sciences 0.28 0.26 0.29 0.17
Agricultural Sciences 0.34 0.18 0.34 0.13
Social Sciences 0.36 0.26 0.22 0.16
Humanities 0.53 ------ 0.15 0.32
Source: C. Kuh, NAS
30
Weights for Supplemental MeasuresStudent Support & Outcomes
Pct full Finish Median Student Collect
Broad Field Supp 6 Yrs TTD Placmt Outcome
Biological Sciences 0.26 0.26 0.14 0.17 0.17
Health Sciences 0.24 0.29 0.14 0.16 0.16
Engineering 0.34 0.20 0.10 0.18 0.18
Physical Sciences 0.29 0.23 0.12 0.18 0.18
Agricultural Sciences 0.28 0.23 0.12 0.18 0.18
Social Sciences 0.27 0.24 0.12 0.18 0.18
Humanities 0.29 0.25 0.11 0.17 0.17
Source: C. Kuh, NAS
31
Weights for Supplemental MeasuresDiversity of Academic Environment
Pct fac Pct fac Pct stud Pct stud Pct stud
Broad Field Minority Female Minority Female Int’l
Biological Sciences 0.14 0.23 0.30 0.25 0.09
Health Sciences 0.24 0.14 0.38 0.18 0.07
Engineering 0.13 0.16 0.26 0.27 0.18
Physical Sciences 0.10 0.20 0.20 0.29 0.21
Agricultural Sciences 0.15 0.17 0.30 0.22 0.15
Social Sciences 0.22 0.20 0.26 0.17 0.16
Humanities 0.21 0.24 0.20 0.18 0.16
Source: C. Kuh, NAS
32