Susan Carter, J.D.Office of Research Development Services
University of California Merced
May 29, 2014
University of California at Merced 1
Introductions
How can RDS help?
Quick Overview of Early Career Opportunities
Discussion/planning activity
Focus on NSF CAREER
Discussion/next steps/close
Overview of Today’s Overview of Today’s WorkshopWorkshop
University of California at Merced 2
Identification of funding opportunities; Support for Program Officer contacts; Connections to educational activities, evaluators,
collaborators; Assistance drafting Dean/Dept. letters; Strategies to integrate education/research activities
(NSF CAREER); Feedback on objectives, approach, activities; Editing suggestions geared to solicitation; Post-doc mentoring and data management plans; Resource section; Budget/Budget justification; Sample proposals, help with internal review
How we can helpHow we can help
University of California at Merced 3
Geared to tenure track, not yet tenured faculty.
Most from private foundations and professional societies (COS/Pivot search ‘young’ and ‘investigator’ for federal opportunities: n= 118; for all opportunities n=2339)
The opportunities are a good way to build a relationship with an agency; some agencies encourage ‘first time’ applicants/recipients.
Overview: Early Career OpportunitiesOverview: Early Career Opportunities
University of California at Merced 4
Should advance you toward your academic goals.
Should be compatible with your institution’s goals.
Should represent a contribution to society at large.
Your proposalYour proposal
University of California at Merced 5
NIH Career Development (K series) Awards:◦ http://grants.nih.gov/training/careerdevelopmentawards.
htm DOE Early Career Research Program:
◦ http://science.energy.gov/early-career/◦ 2014 Recipient: Florin Rusu, UC Merced, ‘Scalable and
Energy-efficient Methods for Interactive Exploration of Scientific Data’ (Office of Advanced Scientific Computing Research)
Air Force Young Investigator Research Program:◦ https://www.fbo.gov/index?
s=opportunity&mode=form&id=c9b1754afe23dd6231f6ba85299a5218&tab=core&_cview=0
DARPA Young Faculty Award:◦ http://www.darpa.mil/Opportunities/Universities/Young_Fa
culty.aspx
Early Career Opportunities: A Very Partial Early Career Opportunities: A Very Partial ListList
University of California at Merced 6
Most agencies have extended webpages; e.g. NIH K Kiosk linked above, and other internet resources and publications.
NIH has a YouTube Channel:◦ https://www.youtube.com/user/nihgrants
Professional societies.
Program Officers.
Colleagues.
Early Career ResourcesEarly Career Resources
University of California at Merced 7
Presentations from NSF hosted workshops:http://aries.imse.ksu.edu/nsf/nsfcareer2013/resource.htmhttp://cs.gmu.edu/events/nsfcisecareer2014/wp-content/uploads/2014/03/slides.pdfhttp://aries.imse.ksu.edu/nsf/NSF2014/Workshopdoc.htm
You Tubehttp://www.youtube.com/watch?v=taYMgx_U3YY&feature=share&list=PLS7JwklCxwEYiXvzKJD3IjKj7a2gF0lfx
AND sample NSF Briefings for review panels:http://crewman.uta.edu/NSF-CAREER-Workshop2013/Files/CAREER-Panel-Briefing.pdfhttp://cs.gmu.edu/events/nsfcisecareer2014/?page_id=15
University of California at Merced 8
Presentation on what NOT to do:◦ http://www.umass.edu/research/system/files/Rutgers_Pazzani_n
ear-misses.pdf
Sample outline:◦ http://www.ohio.edu/engineering/grants/upload/CAREER-Sample
-Outline-2.pdf
Information from NSF on its review process:◦ http://www.nsf.gov/bfa/dias/policy/outreach/grantsconf/meritrev
iew_march13.pdf◦ See also: http://www.nsf.gov/nsb/publications/2012/nsb1228.pdf
(appendices include information on funding rates, often by Directorate)
Some successful CAREER proposals in Geosciences: ◦ http://serc.carleton.edu/NAGTWorkshops/earlycareer/research/N
SFgrants.html
NSF CAREER Resources, contdNSF CAREER Resources, contd..
University of California at Merced9
RDShttp://rds.ucmerced.edu/
UC Educational Evaluation Center: http://education.ucsb.edu/ucec/faq.html
Institutional Research and Decision Support: (institutional data repository; can do survey design and administration): http://ipa.ucmerced.edu/
Assessment at UC Merced (includes links to programs, people, resources at school and institutional level).◦ http://assessment.ucmerced.edu/
Campus and UC ResourcesCampus and UC Resources
University of California at Merced 10
UC Merced Center for Research on Teaching Excellence (CRTE): http://crte.ucmerced.edu/faculty-services◦ Program Learning Outcomes Resources:
Assessment Guide: http://crte.ucmerced.edu/program-learning-outcomes-resources
◦ The MERLOT ELIXR Initiative: offers a digital case story repository that hosts more than 70 discipline-specific multimedia stories.
◦ Students Assessing Teaching and Learning (SATAL): http://crte.ucmerced.edu/node/64 Upon faculty request, SATAL collects and analyzes evidence of
student learning for formative assessment purposes
CRTE also offers individual consultations.
Campus resources, contd.Campus resources, contd.
University of California at Merced 11
Discussion/Questions this Discussion/Questions this far?far?
University of California at Merced 12
Your five, ten and twenty year plans: where will you be, what will you be doing?
CAREER Proposal Planning ActivityCAREER Proposal Planning Activity
University of California at Merced 13
Where are you today?◦ List three areas of research expertise and interest◦ List three areas of education expertise and
interest◦ List three resources that you/ your institution
bring to the table List three career/life goals: where do you
want to be:◦ In five years?◦ In ten years?◦ In twenty years?
How do you get from here to there?
Your CAREER PlanningYour CAREER Planning
University of California at Merced 14
Sketch out your five year career plan.◦ Where do you want to be in five years:
Your research agenda, goals, projects, collaborators, publications?
Your teaching: goals, courses, collaborators, curriculum development?
Outreach: goals, activities, collaborators, integration with your research and teaching?
Where do you want to be in ten years? Twenty years? (Remember: Support for a lifetime of achievement).
Where do your early career submissions fit into this plan?
Discuss your goals and plans with your Dean, Dept. head, mentors, peers.
Early Career Proposal Planning: Initial Steps Early Career Proposal Planning: Initial Steps (an iterative process)(an iterative process)
University of California at Merced 15
Before you do much writing…Before you do much writing…
Know your field:
◦ What is the current state-of-the-art?◦ Who are the top ten researchers?◦ What they are doing right now?◦ Where they get their funding?◦ What they consider to be the key research issues?◦ Who would likely review your proposal?
Source, Hazelrigg, NSF
◦ What is it about (objectives)?
◦ How will you do it (methodology, technical approach, activities, timeline, integration of research and educational activities)?
◦ Can you do it (you and your facilities)?
◦ Is it innovative, exciting and worth doing? Will it contribute to national needs and priorities? (intellectual merit and broader impact)?
◦ How does it fit with your 5/10/20 year plans and goals?
Before you do much writing….Ask yourself Before you do much writing….Ask yourself these basic questions (and then answer them these basic questions (and then answer them in your project description!):in your project description!):
University of California at Merced 17
If you haven’t already done so, identify the most appropriate Division/Program and email and then call the Program Officers◦ Do this SOON (a month before the deadline would be considered
‘too late’.
Be prepared to answer in 25 words or less:◦ What are your research objectives for this funding?
Questions to ask the Program Officer:◦ Do my research objectives fit well with your program?◦ If my project is not a good fit, are there other programs I should
consider?◦ What is the maximum size of your Division/Program’s early career
awards? ◦ How are early career proposals submitted to your program
reviewed?
Initial StepsInitial Steps
University of California at Merced 18
Questions this far?Questions this far?
University of California at Merced 19
Focus on the NSF CAREERFocus on the NSF CAREER
University of California at Merced 20
From the Solicitation:
“CAREER: The Faculty Early Career Development (CAREER) Program is a Foundation-wide activity that offers the National Science Foundation's most prestigious awards in support of junior faculty who exemplify the role of teacher-scholars through outstanding research, excellent education and the integration of education and research within the context of the mission of their organizations. Such activities should build a firm foundation for a lifetime of leadership in integrating education and research.”
Why CAREER?Why CAREER?
University of California at Merced 21
Minimum award: $400K over five years; Bio or PLR $500K (includes IDC.)
Funding rates vary by Directorate, but about 20% overall (per year).
Eligibility: Untenured, in tenure track; can only apply three times.
No Co-PIs or Senior Persons named or in budget. Requires letter from Dean/Dept. head to
demonstrate institutional support and verify eligibility. (At UC Merced this is By-Law 55 unit Chair other than in ENG).
In any area normally supported by NSF.
A quick overview….A quick overview….
University of California at Merced 22
Mark your Calendars for the Due Dates:◦ Monday July 21: BIO, CISE, EHR◦ Tuesday, July 22: ENG◦ Wednesday, July 23: GEO, MPS, SBE
Division and Directorate Contacts:http://www.nsf.gov/crssprgm/career/contacts.jsp
More including abstracts of recent awards:◦ https://www.nsf.gov/funding/pgm_summ.jsp?
pims_id=503214
Quick Overview continued.Quick Overview continued.
University of California at Merced 23
Quick Overview, contd.Quick Overview, contd. Funds the academic career development
of new faculty; it is not just a research award;
All CAREER proposals must have an integrated research and education plan at their core.
“NSF encourages all applicants to think creatively about how their research will impact their education goals and, conversely, how their education activities will feed back into their research. These plans should reflect both the proposer's own disciplinary and educational interests and goals, as well as the needs and context of his or her organization.”
Cover Page (generated in FastLane). Project Summary (1 page equivalent) Description (15 page limit): PI Biosketch: Standard NSF format, but include both
research and educational activities and accomplishments. References. Budget and budget justification. Facilities and resources. Current and pending support. Post-doc mentoring plan (if applicable; 1 page limit). Data Management plan (2 page limit); Supplemental Documents:
Departmental/Dean Letter: (2 pages) Letters of Collaboration (1 page each).
An Overview of the CAREER Proposal An Overview of the CAREER Proposal ComponentsComponents
University of California at Merced 25
From the Solicitation:“Proposers are encouraged to communicate
with the CAREER contact or cognizant Program Officer in the Division closest to their area of research to discuss the expectations and approaches that are most appropriate for that area.”
http://www.nsf.gov/crssprgm/career/contacts.jsp
Don’t forget….Don’t forget….
University of California at Merced 26
Stable support for junior PIs. Career development of outstanding ‘teacher-
scholars’ within the context of their institution. Build a foundation for a lifetime of integrated
contributions to research and education. Give universities incentives to value
integration of research and education. Increase participation of traditionally
underrepresented groups in STEM.◦ Source: Fisher, NSF CISE; wman.uta.edu/NSF-CAREER-
Workshop2013/Files/NSF-Presentation-CAREER-Program.pdf
Don’t forget The Funder’s Goals!Don’t forget The Funder’s Goals!
University of California at Merced 27
Past UC Merced CAREER Recipients include:
◦ Asmeret Asefaw Berhe, LES (EAR) ◦ Elliott Campbell, Environmental Engineering
(CBET)◦ Miguel Carreira-Perpinan, EECS (IIS)◦ Alberto Cerpa, EECS (CNS)◦ Sayantani Ghosh, Physics (DMR)◦ Linda Hirst, Physics (DMR)◦ Christopher Kello, Cognitive Science (BCS)◦ Kevin Mitchell, Physics (PHY)◦ Shawn Newsam, EECS (IIS) (PECASE Recipient)◦ Lin Tian, Physics (DMR)◦ Ming-Hsuan Yang, EECS (IIS)
CAREER Awards at UC MercedCAREER Awards at UC Merced
University of California at Merced 28
FROM THE SOLICITATION:
“The Project Description should be developed in consultation with the department head or equivalent…”
Therefore:◦ Consult with RDS;◦ Develop a written outline;◦ Email and make an apt. to call the Program Contact;◦ Make an apt and discuss with your Department
head or Dean.
Elements of the Project Elements of the Project DescriptionDescription
University of California at Merced 29
Description of the proposed research, including:◦ expected significance;◦ objectives;◦ Preliminary data;◦ methods and approach.
Description of the proposed educational activities; ◦ including evaluation and assessment.
Description of how the research and educational activities are integrated;
Must address intellectual merit and broader impacts;
Results of prior NSF support, if applicable.
Elements of the Project Description, Elements of the Project Description, contd.contd.
University of California at Merced 30
Statement that PI is eligible for the CAREER;
Indication that proposed research and education activities are supported by and integrated into educational and research goals of the department and UC Merced;
Indication that department/school is committed to the support and professional development of the PI.
◦ See handout
Elements of departmental letterElements of departmental letter
University of California at Merced 31
A description of the relationship between the CAREER project, the PI's career goals and job responsibilities, and the goals of his/her department/UC Merced;
A description of ways in which the department head (or equivalent) will ensure mentoring and professional development of the PI;
◦ From the RFA: “A letter that fails to acknowledge institutional commitment to the professional development and mentoring of the PI in both research and education may disadvantage an otherwise outstanding proposal.”
Elements of departmental letter contd.Elements of departmental letter contd.
University of California at Merced 32
Questions this far?Questions this far?
University of California at Merced 33
University of California at Merced 34
Basic two criteria required by NSB since 1997
Requirement that both be explicitly stated in Project Summary and Description since 2002; if not to be returned without review (see http://www.nsf.gov/pubs/2002/iin127/imptnot.pdf)
Emphasis on transformative work in both intellectual merit and broader impacts since 2007;
Criteria revised effective January 2013; even more explicit about the need to address both intellectual merit and broader impacts.
Background and a bit of historyBackground and a bit of history
University of California at Merced 35
NSF wanted to retain researchers’ freedom and creativity to posit their own broader impacts, rather than relying on a predetermined list of ‘National Needs’,’ which was considered but abandoned in the revised, final report.
Report included a call for developing better metrics for broader impacts activities and noted that broader impacts may be better measured at levels beyond the individual project — such as at the institutional level.◦ Source: Holbrook, J.B.; “Re-assessing the science – society relation: The case of
the US National Science Foundation’s broader impacts merit review criterion (1997 – 2011)”; Center for the Study of Interdisciplinarity; http://www.peerev.org/data/7cce53cf90577442771720a370c3c723/PE_doc_28403.pdf
Background, NSB Merit Review TF Report, Background, NSB Merit Review TF Report,
contdcontd..
University of California at Merced 36
Revisions based on recommendations from the NSB appointed Task Force on Merit Review: http://www.nsf.gov/nsb/publications/2011/meritreviewcriteria.pdf
TF Report conclusion: The Intellectual Merit and Broader Impacts review criteria together capture the important elements that should guide evaluation of NSF proposals.
TF Report recommended revisions to the descriptions of the Broader Impacts criterion and how it is implemented.
Use of the two basic review criteria should be informed by a guiding set of three core principles and five review elements.
Background, contd.Background, contd.
University of California at Merced 37
All NSF projects should be of the highest quality and have the potential to advance, if not transform, the frontiers of knowledge.
NSF projects, in the aggregate, should contribute more broadly to achieving societal goals.
Meaningful assessment and evaluation of NSF funded projects should be based on appropriate metrics, keeping in mind the likely correlation between the effect of broader impacts and the resources provided to implement projects.
Three Guiding Core Principles from the Merit Review TF Three Guiding Core Principles from the Merit Review TF ReportReport
University of California at Merced 38
• Applies to all NSF Proposals, including CAREER:
• Intellectual Merit: potential to advance knowledge and understanding;
• Broader Impacts: potential to benefit society and contribute to the achievement of specific, desired societal outcomes.
• NSF will sometimes use additional criteria as required to highlight the specific objectives of certain programs and activities.
Reminder, Two Basic NSF Review CriteriaReminder, Two Basic NSF Review Criteria
University of California at Merced 39
Do the proposed activities suggest and explore creative, original, or potentially transformative concepts?
Is the plan for carrying out the activities well-reasoned, well-organized, and based on a sound rationale?
How well qualified is the individual, team, or organization to conduct the proposed activities?
Five Elements of the Two Criteria: Some Five Elements of the Two Criteria: Some Questions the reviewers will ask:Questions the reviewers will ask:
University of California at Merced 40
Are there adequate resources available to the PI (either at the home organization or through collaborations) to carry out the proposed activities?
Does the plan incorporate a mechanism to assess success?
Five elements, contd.Five elements, contd.
University of California at Merced 41
Because Congress cares! It is a question of policy AND politics.
◦ See e.g.:◦ Holbrook, 2012 (citation above);
◦ Schienke, E. W., Tuana, N., Brown, D. A., Davis, K. J., Keller, K., Shortle, J. S., ... & Baum, S. D. (2009). The Role of the National Science Foundation Broader Impacts Criterion in Enhancing Research Ethics Pedagogy. Social Epistemology, 23(3-4), 317-336.
◦ Rothenberg, M. (2010). Making Judgments about Grant Proposals: A Brief History of the Merit Review Criteria at the National Science Foundation. Technology & Innovation, 12(3), 189-195.
A bit more history: Why does NSF care about A bit more history: Why does NSF care about Broader Impacts?Broader Impacts?
University of California at Merced 42
So, where do I start when So, where do I start when drafting the Broader Impacts drafting the Broader Impacts portion of my proposal?portion of my proposal?
University of California at Merced 43
“Both criteria are to be given full consideration during the review and decision-making processes; each criterion is necessary but neither, by itself, is sufficient.”
“Reviewers will be asked to consider what the proposers want to do, why they want to do it, how they plan to do it, how they will know if they succeed, and what benefits could accrue if the project is successful. These issues apply both to the technical aspects of the proposal and the way in which the project may make broader contributions.”
NSF 14 - 532, emphasis added
Start by reading the SolicitationStart by reading the Solicitation……
University of California at Merced 44
“I shall not today attempt further to define the kinds of material I understand to be embraced within that shorthand description; and perhaps I could never succeed in intelligibly doing so. But I know it when I see it.” (emphasis added)
◦ Justice Potter Stewart,concurring opinion in Jacobellis v. Ohio 378 U.S. 184 (1964)
An analogy?An analogy?
University of California at Merced 45
“Broader impacts may be accomplished through the research itself, through the activities that are directly related to specific research projects, or through activities that are supported by, but are complementary to, the project.”
However, a note of caution: Don’t confuse educational activities and/or outreach with ‘broader impacts.’
You must be specific about how the project activities support and encompass both IM and BI!
More from the Solicitation:More from the Solicitation:
University of California at Merced 46
Then look at the Grants Proposal Then look at the Grants Proposal Guide (GPG):Guide (GPG): GPG Chapter III.C outlines the changes to the
merit review process effective January 14, 2013.◦ ALL applicants should read this, several times:
http://www.nsf.gov/pubs/policydocs/pappguide/nsf13001/gpg_3.jsp
◦ Refers to the three merit review principles, two criteria, five elements.
GPG Chapter II.C.2.d.i. contains information for use by proposers in development of the Project Description section of the proposal.
“Reviewers are strongly encouraged to review the criteria, including GPG Chapter II.C.2.d.i., prior to the review of a proposal.” (NSF 11-690)
NSF values the advancement of scientific knowledge and activities that contribute to achievement of societally relevant outcomes, including, but not limited to:
full participation of women, persons with disabilities, and underrepresented minorities in STEM;
improved STEM education and educator development at any level; increased public scientific literacy and public engagement with
science and technology; improved well-being of individuals in society; development of a diverse, globally competitive STEM workforce; increased partnerships between academia, industry, and others; improved national security; increased economic competitiveness of the United States; enhanced infrastructure for research and education.
From the GPG:From the GPG:
University of California at Merced 48
NSF Presentation on changes to Merit Review: http://www.nsf.gov/bfa/dias/policy/merit_review/overview.pdf
“Revised Merit Review Resources for the External Community” (1/14/2013):◦ http://www.nsf.gov/bfa/dias/policy/merit_review/
resources.jsp◦ Includes FAQs:
http://www.nsf.gov/bfa/dias/policy/merit_review/mrfaqs.jsp
A Few Resources and References on Broader A Few Resources and References on Broader ImpactsImpacts
University of California at Merced 49
Frodeman, R., Holbrook, J. B., Bourexis, P. S., Cook, S. B., Diederick, L., & Tankersley, R. A. (2013). Broader Impacts 2.0: Seeing—and Seizing—the Opportunity. BioScience, 63(3), 153-154.
Center for the Study of Interdisciplinarity, University of North Texas:◦ http://www.csid.unt.edu/topics/NSF-broader-
impacts-criterion.html
Resources, contd.Resources, contd.
University of California at Merced 50
Questions this far?Questions this far?
University of California at Merced 51
The Educational Component: What works, what doesn’t
University of California at Merced 52
Activities “should be consistent with research and best practices in curriculum, pedagogy, and evaluation” (from the Solicitation).◦ Read (and cite, where appropriate!) the resources listed in the
Solicitation!◦ Cite educational research and research on human behavior;
remember that you are expected to include both research and educational citations in your proposal.
Proposed activities can be in a broad range of areas and can by directed at any level;◦ K-12; u/g; grad students and/or the general public.◦ See examples in Solicitation, but don’t limit your activities to
those!
Developing the Educational Component; the basics.Developing the Educational Component; the basics.
University of California at Merced 53
Education and research activities must be integrated.◦ Ask yourself and answer: how are they related and how do they
support each other?◦ Ask yourself: what are creative ways to reach out to underserved
groups and to disseminate research findings?
Activities should be developed and described within the context of your institution.◦ Hint: build on UC Merced’s strengths and existing resources
◦ UC Merced’s HSI status in this context.
Funds should be included in budget to support outreach and education activities; (can include funds for evaluation.)
Developing the Educational Component, contd.Developing the Educational Component, contd.
University of California at Merced 54
Education and research activities must be integrated.◦ Ask yourself and answer: how are they related and how do they
support each other?◦ Ask yourself: what are creative ways to reach out to underserved
groups and to disseminate research findings?
Activities should be developed and described within the context of your institution.◦ Hint: build on UC Merced’s strengths and existing resources
◦ UC Merced’s HSI status in this context.
Funds should be included in budget to support outreach and education activities; (can include funds for evaluation.)
Developing the Educational Component, contd.Developing the Educational Component, contd.
University of California at Merced 55
Research, field work and lab involvement (UG, G, K12, adult learners, the public);
Implement existing instructional innovations in your own courses; work with future teachers (pre-service) – research experiences, course content;
Work with in-service teachers – workshops, curriculum units, research experiences;
Work with high school/middle school students;
Provide content expertise to existing education projects;
Workshops or tutorials
Examples of integrated educational activitiesExamples of integrated educational activities
University of California at Merced 56
Curriculum development (UG, G, K12, 6-12 teachers, inter- or cross-disciplinary course development, online course development);
Assessment of curriculum;
Mentoring (UG, G, 9-12);
Linking activities to industrial, international, or cross-disciplinary work.
More examples:More examples:
University of California at Merced 57
Describe how the activities will improve your teaching (develop your own pedagogical content knowledge as it relates to your field);
Describe how new learning materials or strategies are guided by research on teaching and learning, as well as advances within your discipline;
Describe how the specific activities support your educational goals;
Show you can do it! (preliminary results count here too!);
Highlight what you’ve already done: include the obvious but extend your reach to include innovative change and outreach.◦ Important, however, that the activities be doable.
Educational Activities: what worksEducational Activities: what works
University of California at Merced 58
Partner with communities traditionally underrepresented in STEM
Think systemically◦ i.e. how to involve teachers and students if looking at K-
12 activities.◦ How will your activities have the broadest impact?◦ Who is the intended audience?◦ Nice to refer to alignment of K-12 activities with State
curriculum standards See http://www.cde.ca.gov/be/st/ss/ Consult with the teachers and administrators you will
be involving in your work. But don’t forget to think outside the box in delivery!
Assess: How will you know that your work was successful (how will you know that you achieved the goals)?
Educational Activities: what works, contd.Educational Activities: what works, contd.
University of California at Merced 59
Don’t over-commit!
Don’t only outline activities that are part of your current position (such as current courses taught).
Has it been done a million (or even a hundred) times?◦ If you can catch the reviewers’ attention they are more
likely to believe that you can catch the intended audience’s attention.
Don’t try to do it alone: you probably aren’t an expert, but even if you are, your reach will be limited if you don’t involve others.
Educational Component: what doesn’t workEducational Component: what doesn’t work
University of California at Merced 60
◦ RDS◦ Undergraduate Research Opportunities Center
(UROC): http://uroc.ucmerced.edu/◦ School based instructional assessment◦ Institutional Planning and Analysis:
http://ipa.ucmerced.edu/◦ UC Educational Evaluation Center
http://education.ucsb.edu/ucec/ At UCSB, UCM is a member
◦ Center for Educational Partnerships http://cep.ucmerced.edu/
◦ Center for Research on Teaching Excellence: http://crte.ucmerced.edu/
Educational/outreach resources at UC Educational/outreach resources at UC MercedMerced
University of California at Merced 61
◦ RDS (to make connections)◦ Undergraduate Research Opportunities Center (UROC):
http://uroc.ucmerced.edu/◦ STEM Resource Center:
http://stemresourcecenter.campuscms.ucmerced.edu/◦ School based instructional assessment◦ IRDS:
http://ipa.ucmerced.edu/◦ UC Educational Evaluation Center
http://ucec.gseis.ucla.edu/ At UCSB, UCM is a member
◦ Center for Educational Partnerships(K-12 outreach) http://cep.ucmerced.edu/
◦ Center for Research on Teaching Excellence: http://crte.ucmerced.edu/
Educational/outreach resources at UC MercedEducational/outreach resources at UC Merced
University of California at Merced 62
Questions this far?Questions this far?
University of California at Merced 63
University of California at Merced 64
The 2002 User Friendly Handbook for Project Evaluation (NSF 02-057)
http://www.nsf.gov/pubs/2002/nsf02057/start.htm
Includes a glossary and references to the scientific literature in education and in evaluation.
Start here:Start here:
University of California at Merced 65
Online Evaluation Resource Library: http://oerl.sri.com/
Practical Assessment, Research and Evaluation (an online peer reviewed journal): http://pareonline.net/
National Institute for Science Education: http://archive.wceruw.org/nise/
National Research Council. (2001). Knowing what students know: The science and design of educational assessment. Committee on the Foundations of Assessment. Pellegrino, J., Chudowsky, N., and Glaser, R., Editors
UC Merced Office of Institutional Assessment:◦ http://assessment.ucmerced.edu/assessment-resources
A few more resources:A few more resources:
University of California at Merced 66
Be specific about how you will know whether your research and education plans were successful.
◦ For example, if you state that your goals include improved skill development and retention in STEM, reviewers will want to know how you will show that your project succeeded. Will you track students?
Your assessment plan should restate your project’s major objectives and ideally should describe the metric for each objective.
Limit the objectives to be measured to the most important for the overall success of your project.
Try to use multiple measures for the same objective so you can assess different aspects to “triangulate” your data.
A few tips:A few tips:
University of California at Merced 67
Encouraged by the CAREER Solicitation! While your research goals can be measured in traditional ways (findings, publications, new directions), NSF expects you to develop a plan for assessment of your educational goals. Note: you may want to include funding for this, even if you don’t include an outside evaluator.
Provides information to help improve the project; you can re-assess how the project is going and whether goals are being met as you develop your activities.
Often provides new insights or new insights. “Unanticipated consequences” of a program may be the most useful outcomes of the assessment enterprise.
Source: NSF 02-057; User Friendly Handbook.
Why should you evaluate your educational activities?Why should you evaluate your educational activities?
University of California at Merced 68
What are the goals of your educational activities and component?
Why are they important?
How will you know that your work was successful and you achieved your goals?
Ask Yourself:Ask Yourself:
University of California at Merced 69
“A CAREER proposal must indicate the goals and objectives of the proposed education activities, how it will be integrated with the research component, and the criteria for assessing how these goals will be met. Principal investigators are strongly encouraged to describe how the impact of the educational activities will be assessed or evaluated. A helpful document for information on evaluating educational activities is the NSF publication The 2002 User-Friendly Handbook for Project Evaluation (NSF 02-057).”
From the NSF CAREER From the NSF CAREER Solicitation….Solicitation….
University of California at Merced 70
Q: Do I need to involve an outside evaluator for the education component?
A: You do not need to utilize an outside evaluator. However, you should have some sort of evaluation plan to provide feedback that could lead to future improvements. See if you can find someone within your organization to help you if you have never done an evaluation. Most academic institutions will either have an education department where you might find collaborators, or have some sort of an evaluation system in place for coursework that could be modified to suit your needs.
From the NSF CAREER FAQs…From the NSF CAREER FAQs…
University of California at Merced 71
Discussion…Discussion…
University of California at Merced 72