Date post: | 03-Jan-2016 |
Category: |
Documents |
Upload: | claire-wright |
View: | 213 times |
Download: | 0 times |
Getting Started With Your ATE Evaluation
ATE PI ConferenceOctober 24, 2012
This material is based upon work supported by the National Science Foundation under Grant Nos. 0802245 0802245 and 1204683. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of NSF.
To promote the goals of the ATE program by- partnering with ATE projects and centers to
strengthen the program's evaluation knowledge base
- expand the use of exemplary evaluation practices
- support the continuous improvement of technician education throughout the nation
’s Mission
- Understand the role evaluation should play in your ATE project
- Have clear expectations for your evaluator
- Know where to turn if you need more guidance about your ATE evaluation
Session Objectives
Formative EvaluationLearn how the project could be improved as it is implemented
Summative EvaluationProvide evidence of the quality and impact of your achievements
AccountabilityDocument what you did with the grant money
Evaluation Purposes
The most important purpose of evaluation is not to prove, but to improve.
―Daniel Stufflebeam
Formative Evaluation
Project staff…- make time for evaluation- are open to hearing
negative findings - use feedback to inform
decision making
Formative Evaluation
Evaluator…- learns about the project,
its context, and stakeholders
- gathers data in sound, efficient ways
- provides timely feedback- guides project staff in
understanding and using results
Formative Evaluation
What is the project’s quality? What are the project’s impacts?
Summative Evaluation
What is the project’s quality?What are the project’s impacts?- changes in knowledge, skills, abilities, attitudes,
performance, practices, or policies- show up at the individual, program, institution,
organization, regional, or national level- occur in the short term, long term, or in between
Summative Evaluation
Document… - What you did- Who you reached- Who you worked with - What you produced
(outputs)
Accountability
Accountability
www.evalu-ate.org/about_us
Create a project vita- Mission- Goals- Funding- Activities- Products- Staff- Students- Contributors
& collaborators- Consultants
Formative Evaluation
How is the project doing and how can
it improve?
AccountabilityWhat was done, who was served, who contributed,
what are the outputs?
Summative EvaluationWhat is the quality and
impact of the project?
Evaluation Purposes & Uses
Formative Evaluation
How is the project doing and how can
it improve?
AccountabilityWhat was done, who was served, who contributed,
what are the outputs?
Summative EvaluationWhat is the quality and
impact of the project?
Evaluation Purposes & Uses
- Feedback to internal stakeholders
- Reflection and redirection
Formative Evaluation
How is the project doing and how can
it improve?
AccountabilityWhat was done, who was served, who contributed,
what are the outputs?
- Summative report to key stakeholders- FastLane/annual report to NSF - ATE Impacts book - Annual ATE survey - New proposals- Promotion and dissemination
Summative EvaluationWhat is the quality and
impact of the project?
Evaluation Purposes & Uses
- Feedback to internal stakeholders
- Reflection and redirection
Report Sections- Participants- Activities & findings- Publications & products- Contributions
Annual Reports to NSF
www.fastlane.nsf.gov
See also: EvaluATE’s summary of FastLane report components
evalu-ate.org/resources Keyword: FastLane
Annual Reports to NSF
For example, # of students and their
demographic characteristics
# of professional development participants
# of materials developed
www.evalu-ate.org/annual_survey
Annual ATE Survey
Student impact data
Workforce impact data
www.atecenters.orgwww.ateprojectimpact.org
ATE Projects/Centers Impact Book & Website
“The Project Description must begin with the subsection on Results of Prior Support…. This subsection must contain specific outcomes and results including metrics to demonstrate the impact of the activities undertaken including evidence of the quality and effectiveness of the project's deliverables.
—ATE Program Solicitation
“
Results of Prior NSF Support
Be an Informed ConsumerJoint Committee on Standards for
Educational EvaluationThe Program Evaluation Standards
www.jcsee.orgAmerican Evaluation Association Guiding Principles for Evaluators
www.eval.org
Canadian Evaluation SocietyCompetencies for Evaluation Practice
www.evaluationcanada.ca
What should we expect to receive from our evaluator?
― Participant in the 2010 ATE PI Conference
Getting Started Workshop
- Detailed evaluation plan- Opportunities to review
and provide feedback on draft plans and materials
- Interim reports- Annual reports- Knowledge and insights
you could not have obtained on your own
Evaluation Deliverables
EVAL REPORT
EVALUATOR PI
PROGRAM OFFICER
OTHER PROJECT STAKEHOLDERS, E.G.:partners, advisors, participants,
host administrators
Reporting
EVAL REPORT
EVALUATOR PI
PROGRAM OFFICER
OTHER PROJECT STAKEHOLDERS, E.G.:partners, advisors, participants,
host administrators
Reporting
EVAL REPORT
EVALUATOR PI
PROGRAM OFFICER
OTHER PROJECT STAKEHOLDERS, E.G.:partners, advisors, participants,
host administrators
Reporting
EVAL REPORT
EVALUATOR PI
PROGRAM OFFICER
OTHER PROJECT STAKEHOLDERS, E.G.:partners, advisors, participants,
host administrators
Reporting
EVAL REPORT
Ways of reporting evaluation results to your program officer
PI PROGRAM OFFICER
Ask your program officer how he/she prefers the evaluation results to be conveyed.
Complete original report
Originalexecutive summary
Evaluators’ summary in
annual report
Reporting
EVAL REPORT
Ways of reporting evaluation results to your program officer
PI PROGRAM OFFICER
Ask your program officer how he/she prefers the evaluation results to be conveyed.
Complete original report
Originalexecutive summary
Evaluators’ summary in
annual report
ASK!
Reporting
Using EvaluATE
www.evalu-ate.org
NewslettersFeature articles by experienced ATE PIs and evaluators
Q & A
Definitions & discussion of evaluation terms
Resource highlights
Resource Library
Search by - Evaluation task- ATE topic- Keyword- Author
Evaluator Directory
Search by - location- discipline- project type- keyword
Annual Survey
Survey form FAQsReports
Webinars
Upcoming eventsMaterials from past events:- Recordings- Slides- Handouts
Next Webinar
Register at www.evalu-ate.org/events
Conference ActivitiesConcurrent with this sessionPI and evaluator workshop on evaluationTonight Showcase #209-210 Tomorrow Afternoon Birds of a Feather Session:Beyond Satisfaction and Short-Term Self-Reports: Evaluating the Impact of Your ATE Grant
Thank You!
Advice from the Trenches Elaine’s Golden Rule:
The one who has the gold rules!i.e., don’t disperse grant funds to contractors or partners until obligations have been met
“ “
Elaine CraftATE PI & Evaluator
Advice from the Trenches Set up regular data
collection milestones or checkpoints so that data is being collected as part of the normal process of carrying out the grant and doesn't have to be ‘bolted’ onto the process of the grant after it is underway.
““
Karl KappATE Evaluator
Advice from the Trenches The evaluator is the expert
in evaluation but you are the expert on your project. If you feel that the evalua-tion is not providing the information you need, then talk to your evaluator. Do it immediately. … You can change evaluators but first, communicate, communicate. communicate.
““Jane Ostrander
ATE PI & National Visiting Committee chairperson