Date post: | 28-Dec-2015 |
Category: |
Documents |
Upload: | mercy-strickland |
View: | 215 times |
Download: | 0 times |
Access to the World and Its Languages
www.ed.gov/ope/iegps
Evaluation
Access to the World and Its Languages www.ed.gov/ope/iegps
I N T E R N A T I O N A L E D U C A T I O N
P R O G R A M S S E R V I C E
CIBE, FLAS, IIPP, LRC, NRC Technical Assistance Workshop
February 2-3, 2009
Access to the World and Its Languages
www.ed.gov/ope/iegps
Title VI 50th Anniversary Update• Madeline Albright is the Keynote
speaker• Please make separate reservations
– Conference registration– Hotel registration– Rooms are going quickly
• Contacts Rob Glew and Tomas Hult, Michigan State University
• http://titlevi50th.msu.edu
Access to the World and Its Languages
www.ed.gov/ope/iegps
Panelists
• Eleanor Briscoe, Senior Budget Analyst• John Clement, Director, Strategic
Planning Services• Susan Lehmann, Evaluation Analyst,
FIPSE• Karla Ver Bryck Block, Senior Advisor,
IEPS
Access to the World and Its Languages
www.ed.gov/ope/iegps
Why???
• ED believes in and supports high quality program evaluations to help inform budgetary and program decisions
• ED Strategic Plan– Goal 3, Objective 1: Increase success in and
completion of quality postsecondary education.• Strategy 3. Prepare more graduates for
employment in areas of vital interest to the United States, especially in critical-need languages, mathematics, and the sciences.
Access to the World and Its Languages
www.ed.gov/ope/iegps
Administration Update
• “Accountability and Transparency”• GPRA continues• PART may or may not continue but
the need to measure program effectiveness remains
Access to the World and Its Languages
www.ed.gov/ope/iegps
Congress
• Clear interest in program evaluations
• Directed a $1.5 million review of Title VI and Fulbright-Hays programs by the National Research Council of the National Academies– International Education and Foreign
Language – Keys to Securing America’s Future (March 2007)
Access to the World and Its Languages
www.ed.gov/ope/iegps
Keys to Securing America’s Future
• National Academies recommendation– The Department of Education should
commission independent outcome and impact evaluations of all programs every 4 to 5 years
Access to the World and Its Languages
www.ed.gov/ope/iegps
IEPS Program Level Initiatives• Evaluations are underway with AIR
– Language Resource Centers– Fulbright-Hays Group Projects Abroad –
Overseas Intensive Language Projects
• Analysis of the International Research and Studies Program is concluding
• National Resource Centers are next• Future plans include Centers for
International Business Education
Access to the World and Its Languages
www.ed.gov/ope/iegps
IRIS
• IEPS gathers information via IRIS that is used for program evaluations
• Breakouts for 4 programs today (CIBE, LRC) and tomorrow (NRC, FLAS) to discuss:– Current and future program performance
and efficiency measures– Changes to IRIS for the next clearance– Evaluation efforts– Trivia, prizes, chocolate!!!
Access to the World and Its Languages
www.ed.gov/ope/iegps
Keys to Securing America’s Future
• National Academies Recommendation– The Department of Education should work
with universities to create a system of continuous improvement for Title VI and Fulbright-Hays programs. The system would help develop performance indicators and other improvement tools and should include networks of similar centers and university officials with overall responsibilities in language, area, and international studies.
Access to the World and Its Languages
www.ed.gov/ope/iegps
Your Project
• Think…– Goal-oriented– Data driven
ED does not have specific models, timelines, budgets, etc for your evaluation plans. Develop a plan that serves your needs.
Access to the World and Its Languages
www.ed.gov/ope/iegps
Performance Objectives & Measures
• A measurable indication used to determine how well goals are being met– How will progress be measured?– How much progress will constitute success?– How will it be known if an objective or part
of an objective has been achieved?
• Are we measuring process or outcome?
Access to the World and Its Languages
www.ed.gov/ope/iegps
Objective & Measure Types
• A specific achievement/accomplishment
• Change in a level of achievement/accomplishment
• Achievement/accomplishment (or change) compared with other students or institutions
Access to the World and Its Languages
www.ed.gov/ope/iegps
Components of Objectives & Measures
• What will change• How much change is expected• Who will achieve the change• When the change will take place
Access to the World and Its Languages
www.ed.gov/ope/iegps
Performance Objective Example
“20% (how much) of local businesses with more than 200 employees (who) will open an office in a foreign country or enter into a joint venture with a foreign company (what) within two years (when).”
• Help local businesses to internationalize
Access to the World and Its Languages
www.ed.gov/ope/iegps
Performance Objective Example
• Teacher workshop – Developing Materials for LCTLs
“50 (how much) LCTL teaching professionals (who) will complete LCTL workshop (what) annually (when).”
“90% (how much) of LCTL workshop participants (who) respond with 4 or higher on survey (what) one year after completing program (when).”
Access to the World and Its Languages
www.ed.gov/ope/iegps
Performance Measure Example
• Revise public health curriculum to enhance international knowledge“The required MPH courses (who) will be revised (what) to cover the 10 important international health areas (how much) by next fall (when).”
Access to the World and Its Languages
www.ed.gov/ope/iegps
Performance Measure Example
•Improve enrollment in summer language institute“Undergraduate student enrollment in the summer language institute (who) will increase (what) by five percent (how much) next summer (when).”
Access to the World and Its Languages
www.ed.gov/ope/iegps
Improving Performance Measures
• Original Performance Measure: Provide overseas internships.
• What/Who:– Provide dynamic student internships with partners in a
foreign country.
• When:– Provide dynamic student internships with partners in a
foreign country each semester.
• How Much:– 20 students will engage in dynamic student internships
with partners in a foreign country each semester.
Access to the World and Its Languages
www.ed.gov/ope/iegps
Improving Performance Measures• Original Performance Measure: Enhance
Business Resource Center.
• What/Who:– Enhance Business Resource Center to advise
businesses on international trade regulations.
• When/How Much:– Business Resource Center documents and staff will
be updated on new international trade regulations within one week of passage.
Access to the World and Its Languages
www.ed.gov/ope/iegps
Remember!!!
• Activities are not performance measures!
Access to the World and Its Languages
www.ed.gov/ope/iegps
Resolving the “Activity-as-Measures” Dilemma
• “Create opportunities for local faculty to strengthen area skills.” (activity)– Create “visiting” faculty training programs
and train 2 per year. (outcome)
• “Enhance study abroad.” (activity)– Increase number of students studying
abroad by X% each year. (outcome)
Access to the World and Its Languages
www.ed.gov/ope/iegps
Summary: Good Objectives & Measures
• Each objective should have one or a few specific performance measures to demonstrate how progress will be measured toward meeting the objective
• More objectives and measures ≠ better objectives and measures!
• Ensure data is collectible
• Quantify and define success
Access to the World and Its Languages
www.ed.gov/ope/iegps
Summary: Strong Evaluation• Partner with a competent evaluator• Develop quantifiable objectives and
measures• Measure what is important• Measure what can be changed• Get frequent progress reports• Keep key stakeholders informed of
results• Use results to adjust project regularly• Keep analyzing…