NSF ECCS-1641369 Project Report NSF Workshop on Defining Broader Impact Activities of ECCS/ NSF Grants May 12-13, 2016 Arlington, Virginia edited by Zhi Tian, PI George Mason University September 2016 This report on the results of the Workshop on Enhancing the Broader Impacts of ECCS Research was sponsored by the National Science Foundation under grant NSF-ECCS-1641369. Staff support was provided by the Department of Electrical and Computer Engineering of George Mason University.
Transcript
NSF ECCS-1641369 Project Report
NSF Workshop on Defining Broader Impact Activities of ECCS/ NSF Grants
May 12-13, 2016
Arlington, Virginia
edited by
Zhi Tian, PI George Mason University
September 2016
This report on the results of the Workshop on Enhancing the Broader Impacts of ECCS Research was sponsored by the National Science Foundation under grant NSF-ECCS-1641369. Staff support was provided by the Department of Electrical and Computer Engineering of George Mason University.
Table of Contents Executive Summary ............................................................................................................................... 3
Workshop Overview .............................................................................................................................. 5 1. Understanding the economic and social value of ECCS projects ............................................ 7
1.1 Clarification of the Broader Impact Criterion, Goals, and Policies .................................................... 7 1.2 Communication of the BI Policies to PIs and Reviewers .................................................................. 9 1.3 Communication of Short-Term and Long-Term Broader Impact Results to the Public .................. 10
2. Meaningful metrics to assess and promote Broader Impacts of ECCS projects ............... 11 2.1 Assessment Metrics for ECCS Projects ............................................................................................ 11 2.2 Improving promotion of Broader Impacts ........................................................................................ 13
3. New approaches and institutional initiatives on Broader Impacts ....................................... 13 3.1 Conveying the importance of Broader Impacts to investigators and reviewers ............................... 14 3.2 Incentivizing investigators and their institutions .............................................................................. 15 3.3 Identifying Broader Impact Grand Challenges ................................................................................. 16
List of Participants ............................................................................................................................... 20 Abstract of Plenary Talks on Broader Impacts ............................................................................. 22
Executive Summary This document reports on the workshop “Defining Broader Impact Activities of ECCS/NSF Grants” held at Ballston, Arlington Virginia on May 12-13, 2016. The goal of this workshop was to help promote consistent practice in applying the Broader Impacts (BI) criterion in NSF proposal evaluation, with focus on understanding the effects of Broader Impacts and the resources provided to implement research projects supported by the Electrical, Communications and Cyber Systems (ECCS). This workshop brings together researchers from the Electrical Engineering community to plan out a roadmap and promote best practice in Broader Impact contributions within the ECCS community. The workshop participants have demonstrated innovations and leadership in research activities of broader impacts. They worked together during the workshop to examine three primary themes:
• delineate the economical and societal value of research and education activities in the fields of ECCS; • create meaningful metrics and incentives to assess and promote the broader impacts of ECCS
projects; and, • stimulate new thinking and institutional initiatives on Broader Impacts that can effectively enhance
the human capital and societal value from the resources provided to implement ECCS projects. The workshop included both plenary talks and breakout discussions within theme groups. Participants were asked to develop recommendations to transform the climate for enhancing broader impacts in the ECCS community. Key recommendations are summarized next to provide a pathway for engaging ECCS researchers in Broader Impact innovations. Key Recommendations First, there should be a follow-up process by the ECCS division that improves the communication of Broader Impacts concepts to various stakeholders, particularly NSF proposal reviewers and principal investigators (PIs). It is recommended that the ECCS prepare an ECCS Broader Impacts Addendum to the NSF Grant Proposal Guide (GPG) that succinctly describes how ECCS panels will evaluate proposals relative to the Broader Impacts criterion. In the ECCS BI addendum, separate the eight goals of the America Competes Act into two categories that address (i) technology & economic impacts, and (ii) human impacts. It should be made clear that proposers shall address both major categories, but not necessarily every goal. All acceptable areas of broader impacts as defined in America Competes should be explicitly stated in this document, in order to avoid the perception that Broader Impacts are limited to the human aspect of “education and outreach.” The NSF should make funding choices to ensure all eight goals are achieved on a global basis within ECCS across all funded programs.
Second, the communication modes among proposers, reviewers and program officers need to be improved. The ECCS BI Addendum, once developed, shall be regularly updated, referenced in solicitations, and located in a prominent place on the ECCS website. It is recommended that that NSF have Fastlane add an inserted page (by default) to the PDF copy of the proposal downloaded by the reviewer, that defines the review criteria (for Intellectual Merit and Broader Impacts). Each panel briefing should review Broader Impacts assessment goals so that panel discussions will be informed. During panel evaluation, the ECCS division shall request a satisfactory summary evaluation of broader impacts for each submitted proposal. The review criteria should include assessment of the proposed timescales over which the broader impacts will be evaluated. This evaluation would be implemented as part of the written summary assessment by the panel. NSF program officers should inform reviewers in advance that this broader impact summary evaluation will be decided during the panel review. This change to the panel review process will encourage greater uniformity in panel deliberation, and provide useful feedback to the PIs as to the perceived merit specific to the broader impact portion.
Third, PIs should be encouraged to leverage existing institutional infrastructure to support their Broader Impacts plan, and to build on their own previously successful methods. During panel briefing, training for proposal reviewers should emphasize placing value on what can be achieved best from the proposed approach (as opposed to a “bean counting” approach), favoring impact by leveraging pre-existing programs and/or impact by creating a novel approach that can eventually lead to impactful solutions when evaluating broader impact plans. Fourth, the BI review and implementation practices should be improved to encourage the development of meaningful and long lasting broader impact programs, rather than favoring novelty. The panel summary of each proposal should include assessment of the proposed or described timescales over which the broader impacts will be evaluated. The ECCS division shall examine program impact beyond the 5-year horizon by looking at enabled commercial products, patent licensing, and human impact as a first step towards communicating long-term impact to the public. The PIs shall be encouraged to update online abstracts to reflect long-term impacts beyond the project period. Fifth, it is recommended that the ECCS division develop training materials for PIs and reviewers related to the ECCS implementation of the BI criterion. Such materials may include a FAQ sheet that is regularly updated as questions and requests for clarification come in related to BI expectations. It would be ideal to develop short (10-15 min) web-based training modules that all reviewers would be required to complete before they are assigned proposals to review. The NSF may also post a webinar on “How to Write the BI Section” of a proposal. The ECCS community will also benefit considerably from having available a compendium of best practices from ECCS grants. Sixth, the societal value of ECCS research needs to be communicated to the public in a proactive manner in order to gain sustainable support and broaden participation from both stakeholders and underrepresented groups. To this end, the ECCS division may: i) create documentation that communicates how the broader impacts criterion can be best met by ECCS projects; ii) develop an “NSF ECCS Speakers Bureau” to raise public awareness about the BI of ECCS work; iii) solicit optional BI nuggets as part of the review process of ongoing grants, and broadly disseminate them to the ECCS community and the general public; iv) create online, streamable, professionally-produced webinars or videos available on the NSF website that describe important NSF-funded innovations; and v) improve the channels of communication between K-12 STEM teachers and PIs, such as involving a social networking vehicle to connect PIs to teachers interested in their work. Seventh, to influence a culture change in investigators and their institutions, it is recommended that NSF utilize two incentive mechanisms: financial leverage and recognition. The ECCS division may provide supplements to NSF awards for meritorious BI programs. This may be coupled to proposals receiving the strong BI evaluation distinction in panel summaries; or, be open to the support of new ideas during the project. NSF may also provide selective recognition of notable Broader Impacts by investigators and institutions. This incentivizes creativity and recognizes innovative or impactful programs, as well as providing the PI additional recognition of their leadership that would be valued at their home institution A more substantial option to financially incentivize investigators and their institutions is to establish a new mid-career award that serves to provide opportunities for professional promotion to mid-career faculty while rewarding a track record of Broader Impact contributions. Finally, it is recommended that the NSF and ECCS define Grand Challenges in Broader Impacts with the idea that this may facilitate request for further financial support for BI and may encourage the formation of BI networks throughout the community. This would facilitate more coherent BI activities between PIs, institutions, and research areas.
Workshop Overview NSF investments are expected to “advance the frontiers of knowledge, contribute to the economic and social wellbeing of the nation, and advance the preeminent and leadership position of the United States in science, technology, engineering and mathematics” [1]. The ECCS Division of the NSF recognizes the importance of promoting Broader Impact innovations in supporting the NSF mission. The Electrical Engineering fields are well positioned to making impactful research to the society and the nation, yet such impacts need to be properly measured, stimulated and promoted in order to maximize the societal value of the manpower and resources spent on implementing ECCS funded projects. The objective of this workshop is to lay out a clear roadmap to engage ECCS researchers in Broader Impact innovations and promote the best practice in Broader Impact contributions within the ECCS community. The document provides a detailed report on the workshop. The workshop included plenary talks and breakout panels. Eleven participants, including ECCS principal investigators (PIs) and administrators in engineering departments and schools across the nation, provided formal presentations to share their experiences and thoughts on Broader Impacts. These presentations voiced on the needs for improvement in current policies and practice, and provided samples of best practice by electrical engineers. Following these presentations, workshop participants formed three ad-hoc workgroups that engaged in in-depth discussions in three parallel Breakout Sessions (c.f. Workshop Agenda) and reported back to the general assembly on three thematic topics:
1) The economical and societal value of research and education activities in the fields of ECCS;
2) Meaningful metrics and incentives to assess and promote the broader impacts of ECCS projects;
3) New thinking and institutional initiatives that engage ECCS PIs in broader impact innovations.
Through the three breakout sessions, participants were asked to develop recommendations to transform the climate through a clear roadmap, as well as consistent metrics and incentives, for advancing and catalyzing Broader Impacts innovations in ECCS research, such that researchers and institutions can align their Broader Impact activities with greater societal return for NSF investments. With this goal in mind, the workgroups examined the current language on Broader Impacts in the NSF Grants and Proposal Guide (GPG) [2]. The GPG states the following:
“The Project Description must contain, as a separate section within the narrative, a section labeled “Broader Impacts”. This section should provide a discussion of the broader impacts of the proposed activities. Broader impacts may be accomplished through the research itself, through the activities that are directly related to specific research projects, or through activities that are supported by, but are complementary to the project. NSF values the advancement of scientific knowledge and activities that contribute to the achievement of societally relevant outcomes. Such outcomes include, but are not limited to: full participation of women, persons with disabilities, and underrepresented minorities in science, technology, engineering, and mathematics (STEM); improved STEM education and educator development at any level; increased public scientific literacy and public engagement with science and technology; improved well-being of individuals in society; development of a diverse, globally competitive STEM workforce; increased partnerships between academia, industry, and others; improved national security; increased economic competitiveness of the US; and enhanced infrastructure for research and education.”
The Broader Impacts review criterion was developed in response to the America Competes Act [7, 8], which states the following:
SEC. 526. BROADER IMPACTS REVIEW CRITERION. (a) GOALS — The Foundation shall apply a Broader Impacts Review Criterion to achieve the
following goals: (1) Increased economic competitiveness of the United States. (2) Development of a globally competitive STEM workforce. (3) Increased participation of women and underrepresented minorities in STEM. (4) Increased partnerships between academia and industry. (5) Improved pre-K–12 STEM education and teacher development. (6) Improved undergraduate STEM education. (7) Increased public scientific literacy. (8) Increased national security.
Provisions of the America Competes Act state the following:
(b) POLICY. — Not later than 6 months after the date of enactment of this Act, the Director shall develop and implement a policy for the Broader Impacts Review Criterion that — (1) provides for educating professional staff at the Foundation, merit review panels, and
applicants for Foundation research grants on the policy developed under this subsection; (2) clarifies that the activities of grant recipients undertaken to satisfy the Broader Impacts
Review Criterion shall— (A) to the extent practicable employ proven strategies and models and draw on existing
programs and activities; and (B) when novel approaches are justified, build on the most current research results;
(3) allows for some portion of funds allocated to broader impacts under a research grant to be used for assessment and evaluation of the broader impacts activity;
(4) encourages institutions of higher education and other nonprofit education or research organizations to develop and provide, either as individual institutions or in partnerships thereof, appropriate training and programs to assist Foundation- funded principal investigators at their institutions in achieving the goals of the Broader Impacts Review Criterion as described in subsection (a); and
(5) requires principal investigators applying for Foundation research grants to provide evidence of institutional support for the portion of the investigator’s proposal designed to satisfy the Broader Impacts Review Criterion, including evidence of relevant training, programs, and other institutional resources available to the investigator from either their home institution or organization or another institution or organization with relevant expertise.
The statements on Broader Impacts Review Criterion in the NSF GPG and those in the America Competes Act formed the basis of panel discussions and recommendations. A summary of the outcomes and key recommendations from the workshop is presented in the Executive Summary Section. The reports from the three workgroups follow, which elaborate on the rationale and details of these recommendations.
1. Understanding the economic and social value of ECCS projects To promote innovations and best practice of research with broader impacts, the first step is to gain in-depth understanding of the broad economical and societal value of research activities and outcomes in the research fields of ECCS. General areas of broader impacts include broadening participation of underrepresented groups [4], enhancing infrastructure for research and education [5], broadening dissemination to enhance scientific and technological understanding, and advancing discovery and understanding while promoting teaching, training and learning [2, 3]. These areas are highly pertinent to ECCS areas, where the situation with underrepresented groups is perplexing and yet there are excellent opportunities for enhancing the infrastructure to facilitate broader impacts. Evidently, raising the public awareness of the societal value of ECCS research is essential for broadening participation from both stakeholders and underrepresented groups. To this end, the workshop participants in the Breakout Session I discussed the following main issues. 1.1 Clarification of the Broader Impact Criterion, Goals, and Policies The presentation of the Broader Impacts (BI) review criterion in the NSF Grant Proposal Guide (GPG) can lead to misinterpretation of objectives by PIs preparing proposals and panel reviewers alike. The language used to communicate the BI goals in the America Competes Act has greater clarity, but an explicit clarification of how specific policies will be used to achieve these goals is required. Compared with the America Competes Act that clearly spelled out eight BI goals (see the Overview section), there are several major issues in the presentation of the Broader Impacts criterion in the NSF GPG. Policy clarification. When inspecting the current language in the GPG (quoted in the Overview section), each member of the workgroup interpreted the statement differently, with all panel members unclear on whether or not the “and” in this statement was a directive to address all eight BI goals in each proposal. Anecdotal evidence suggested similar confusion across and within panels reviewing proposals. The general view of the workgroup was that the eight BI goals in America Competes can be separated into two primary areas:
(i) Goals and objectives related to technology and economic impact; and (ii) Goals and objectives related to human impact on the economy and societal values.
A policy clarification is suggested that requires proposals that are funded to state how they will address both areas, but any given proposal should not need to address all eight goals outlined in America Competes. The NSF should make funding choices to ensure all eight goals are achieved on a portfolio basis within ECCS. Broader Impact priority for ECCS grants. Among the stated goals for broader impacts, there is currently no guidance provided on the relative importance of one goal relative to another. This leads to variability in interpretation by PIs and reviewers. The participants in Session I also had diverse views. One view was that all the goals should have equal weighting. Another was that as an engineering directorate, ECCS should emphasize long-term economic impact. Yet another was that increased participation by women and underrepresented minorities should be emphasized. The lack of consensus and differing views among workshop participants was itself insightful, validating the lack of consensus on BI priorities. Role of established programs. Proposers are in some cases being penalized by reviewers for using organizational infrastructure to realize BI goals, when in fact this should be encouraged when it is clearly shown by the PI that such infrastructure directly contributes to the desired outcomes. Using established programs is expected to provide leverage that can increase the potential impact of a given PI’s efforts. Using established programs may also enable outreach activities that may not otherwise be practical on
small grants (i.e., with only 1 or 2 PIs). Policy clarification is therefore needed to improve consistency in the reviewing process. When Provisions of the America Competes Act (b) Policy is inspected, it becomes evident that important elements of the act supporting the suggested clarification are clauses (2) and (4). These clauses support the position that PIs should utilize existing methods and institutional infrastructure to realize their specific BI goals. Novel approaches are not required and, as indicated in (2)(B), should be supported by research on their effectiveness if employed. Reviewer criticisms based on lack of novelty in the BI plan are therefore inconsistent with guidance provided by America Competes. Clause 1 directs the training of reviewers such that inappropriate metrics for the BI plan and goals are avoided.
Involvement of students at various levels. Direct outcomes such as graduate research assistants and research participation by undergraduates from the PI’s institution are in some cases being discounted as insufficient in realizing BI goals. It is suggested that such direct outcomes not be discounted when compliance with BI goals are being evaluated by the reviewers.
Direct involvement of K-12 students in research programs is not always practical. In addition to liability issues, legislation that has been put in place to protect minors can make the training and background check requirements for those involved in K-12 directed programs impractical to navigate. Research Experiences for Teachers (RET) programs are therefore the best vehicle for supporting this America Competes goal, but institutional support is typically needed. In instances where these issues can be navigated, methods other than RET programs are supported.
Workforce reeducation. The workgroup also debated whether workforce re-education may be missing as a BI goal. As technology and economic forces displace older workers, educating these workers for STEM-related jobs may become as important as K-12 education.
Recommendations Based on the above analysis, the following recommendations are made to improve the clarity of the ECCS division’s guideline on Broader Impacts.
• Create an addendum to the GPG that clarifies ECCS policy with regard to the 8 broader impact goals outlined in the America Competes Act. This addendum should be used by both PIs and reviewers.
• In the ECCS BI addendum, separate the 8 goals into categories that address (i) technology &
economic impact, and (ii) human impact. The suggested division of the 8 goals into the two major categories is as follows:
(i) Technology, Economic, and Social Impact (1) Increased economic competitiveness of the United States. (2) Increased partnerships between academia and industry. (3) Increased public scientific literacy. (4) Increased national security. (5) *Improved undergraduate STEM education.
(ii) Human Impact (1) Development of a globally competitive STEM workforce. (2) Increased participation of women and underrepresented minorities in STEM. (3) Improved pre-K–12 STEM education and teacher development. (4) *Improved undergraduate STEM education.
• In the addendum the specific ordering of these goals within the categories could be chosen based
upon priority of policy emphasis, with an order that may be updated on a regular basis by the NSF. For example, item (2) in category (ii) might be moved to the top of the list if that reflects the immediate policy priority.
• Allowing individual PIs freedom of motion to focus on goals that they find personally motivating
and that they feel they can impact may be the best way to create the greatest impact. A policy is therefore recommended that requires proposals to address both major categories, but not necessarily every goal.
• The NSF should make funding choices to ensure all 8 goals are achieved on a portfolio basis within
ECCS. Proposals from individual PIs and small collaborative proposals should not be expected to address all 8 goals. Large center proposals, by contrast, may be expected to address all 8 BI goals.
• Scale BI expectations for junior versus senior faculty, allowing junior faculty to focus more on
technical merit and setting higher BI expectations for mid-career to senior faculty.
• ECCS policy should encourage PIs to utilize existing institutional infrastructure to support their BI plan, and to build on their own previously successful methods. Encourage use of novelty when institutional infrastructure is lacking and/or when the novelty will address issues not currently achieved by existing institutional solutions.
1.2 Communication of the BI Policies to PIs and Reviewers The workgroup expressed concern that the BI criterion and goals are not being uniformly interpreted and applied. Clarification of expectations for both PIs and reviewers would be expected to result in both higher quality BI sections in proposals and more consistent reviews relative to the BI goals of the America Competes Act. Materials available to guide PIs and reviewers on BI expectations are limited: not much is available that provides guidance on ECCS-specific priorities and practices relative to the BI criterion. Recommendations A number of concrete measures have been recommended to clearly communicate the BI policies and expectations to PIs and reviewers.
• Prepare an ECCS addendum to the GPG that describes for PIs and reviewers alike how ECCS panels will evaluate proposals relative to the BI criterion and the 8 BI goals. This should be regularly updated, referenced in solicitations, and located in a prominent place on the ECCS website.
• Prepare a FAQ sheet that is regularly updated as questions and requests for clarification come in
related to BI expectations. This might include “how to's,” what "not to do," and any other material deemed necessary to keep the community up to date regarding BI standards and expectations.
• Develop training materials for PIs and reviewers related to the ECCS implementation of the BI criterion. These would ideally be short (10-15 min) web-based training modules that all reviewers would be required to complete before they are assigned proposals to review
• The NSF should post a webinar on "How to Write the BI Section" of a proposal. • Improve communication modes between proposers, reviewers and program officers. For reviewers,
one possibility may be to use a portal similar to a classroom management tool or web link that houses all the material reviewers should use while drafting reviews, provided it is presented in a concise, navigable form. In the interim, sending relevant materials and links to reviewers when they are assigned proposals to review may be an option.
• Each panel briefing should review BI assessment goals so that panel discussions will be informed. • Training for proposal reviewers should emphasize placing value on what can be achieved best from
the proposed approach (as opposed to a “bean counting” approach), favoring impact by leveraging pre-existing programs and/or impact by creating a novel approach that can eventually lead to impactful solutions when evaluating broader impact plans.
• Develop a mentoring plan for junior faculty that helps them to develop a sense for what the real
economic and technical impact of a technology might be. 1.3 Communication of Short-Term and Long-Term Broader Impact Results to the Public
The societal value of ECCS research needs to be communicated to the public in a proactive manner in order to gain sustainable support and broaden participation from both stakeholders and underrepresented groups. Key questions are: How do we raise the public awareness of the broader impacts of ECCS research? What are the effective ways to communicate to the public the excitements, benefits and long-term impacts of ECCE research and education? There are several major issues in current practice. First, formal communication of results through reports is not always effective in broad dissemination of knowledge. Generating interest in new technologies may require a more proactive approach. Second, new modes of communication (“social media”) are not being fully exploited as a tool for educating the public about BI results. Third, impact is not always measurable within the reporting horizon required by the NSF. Looking back beyond 5 years can reveal more about the real impact of work, especially with regard to broader economic and societal metrics. Recommendations
• Develop an “NSF ECCS Speakers Bureau” to raise public awareness about the BI of ECCS work. Presentations at small venues (high schools, community colleges, etc.) would be handled through discretionary grant supplements applied for by PIs, whereas speakers for large venues (webinars, “TED Talks,” etc.) would be selected by NSF program directors. This effort might be coordinated with professional societies where appropriate, or in addition to efforts to reach the general public.
• Create online, streamable, professionally-produced webinars or videos available on the NSF
website that describe important NSF-funded innovations. There might be a series for the general public, and a separate “deep-dive” series for a technical audience.
• On a trial basis, have the next round of nugget requests have PIs provide separate slides for technical and BI highlights. Determine if publicizing any of these would be of benefit in increasing public awareness.
• Examine program impact beyond the 5-year horizon by looking at enabled commercial products,
patent licensing, and human impact as a first step towards communicating long-term impact to the public.
• Improve the channels of communication between K-12 STEM teachers and PIs. This might involve
a social networking vehicle to connect PIs to teachers interested in their work, or weekday programs for K-12 students that bring in NSF PIs to discuss their work.
2. Meaningful metrics to assess and promote broader impacts of ECCS projects Meaningful assessment and evaluation of NSF funded projects should be based on appropriate metrics, keeping in mind the expected correlation between intellectual merit and broader impacts of the project activities [6]. On the one hand, a research project cannot have strong broader impacts without strong intellectual merit, because there is no intellectual root for creating added societal value. On the other hand, intellectual merit without broader impact eventually evanesces in isolation. The balance and integration of both intellectual merit and broader impacts need to be reflected in the merit review process. To do so, well-thought-out evaluation metrics need to be defined and implemented and accounted for. Further, meaningful incentives are useful in stimulating and supporting the efforts of PIs and institutions that contribute to the broader impacts of engineering research. 2.1 Assessment Metrics for ECCS Projects The workgroup in Breakout Session II was charged with identifying means to improve the assessment of broader impacts. Questions that aimed to foster discussion on Broader Impact metrics were: How can Broader Impact be classified? What metrics are appropriate for measuring the Broader Impacts of ECCS research in conjunction with the evaluation of Intellectual Merit? In current merit review practice for NSF proposals, are there factors that lead to the schism of intellectual merit and broader impacts? If so, how may they be mended? Through deliberation on these questions, the workgroup identified major issues in current assessment metrics and practices, as follows.
A common perspective within the community of proposers and reviewers is that Broader Impacts fall almost exclusively under the categories of education and outreach. In fact, student training and economic impact are related to the majority portion of typical proposal budgets and already address America Competes requirements.
Inconsistencies are rampant in how panel deliberations handle Broader Impacts, reflecting the related imbalance in how reviewers evaluate them relative to the proposal as a whole. One aspect the working group discussed at length was the relative weighting between novelty and long-term broader impacts. The inconsistencies are exacerbated by the absence of relevant metrics or expectations pertaining to the division of effort for Broader Impacts as part of proposed overall effort, including a lack of clarity regarding the timescales over which they should or could be measured.
Many reviewers note an apparent or perceived disconnect between Broader Impact categories of student training and economic impact, and current Broader Impact criteria such as those addressing the qualifications of the principal investigator, the organization of the research and education plan, and the adequacy of resources. The evaluation criteria for intellectual merit and broader impacts are phrased the same, and it is unclear if reviewers understand how they are to be applied per NSF expectations. Recommendations The workgroup makes the following recommendations for improving the assessment of broader impacts within NSF and ECCS.
• As there seems to be a large variability in what reviewers consider appropriate for broader impacts and to avoid the perception that they are limited to “education and outreach,” it is recommended that that NSF have Fastlane add an inserted page (by default) to the PDF copy of the proposal downloaded by the reviewer, that defines the review criteria (for Intellectual Merit and Broader Impacts). This is the file/format most used by reviewers and if this information is included with each downloaded proposal, it may lead to less variability in the interpretation of broader impacts. It is suggested that all acceptable areas of broader impacts as defined in America Competes be explicitly stated in this document.
• During panel evaluation, the ECCS division shall request a satisfactory/strong summary evaluation
of broader impacts for each submitted proposal. This evaluation would be implemented as part of the written summary assessment by the panel. NSF program officers should inform reviewers in advance that this broader impact summary evaluation will be decided during the panel review. This change to the panel review process will encourage greater uniformity in the panel deliberations, and provide useful feedback to the proposer(s) as to the perceived merit specific to the broader impact portion relative to the other proposals that were reviewed.
• When meaningful assessment metrics are defined, they need to be communicated effectively to the
ECCS community. It is recommended that ECCS and NSF update the review criteria for broader impacts.
o The current wording found in the GPG and/or supplemental documentation does not encourage the development of meaningful and long lasting broader impact programs, but rather favors novelty. As a result, the review criteria should be modified to accommodate one or both broader impact characteristics. For example: If a novel plan for broader impact is proposed, is the rationale sound and well-supported? If the plan builds on prior activity, is there ample evidence of past success and a plan for meaningful expansion or continuation?
o The current wording should also be modified to provide improved guidance as to the expected balance between intellectual merit and broader impact discussions within the proposal, as per recommendation (i) of Section 3.
o The review criteria should include assessment of the proposed or described timescales over which the broader impacts will be evaluated.
o The review criteria should be modified to embody flexibility in how some criteria, such as PI qualifications or the organization of a research plan, are applied. For example: the current criterion regarding organization of a plan may not be relevant for a broader impact emphasizing economic or national security impact. Rather, the clarity and justification of such an impact may be a more relevant measure.
2.2 Improving promotion of Broader Impacts The workgroup charged with identifying means to improve the promotion of broader impacts noted issues with the current promotion and communication. Discussions revolved around the following questions: What incentives can be developed in the merit review criteria to stimulate Broader Impact innovations? What are appropriate resource commitments to facilitate Broader Impact contributions? How do we assess the Broader Impact contributions during and after the funding period of a grant and ensure accountability? In current practice, there are several major issues that hinder the promotion of broader impacts innovation. First, while there are funds provided to perform some aspects of broader impacts (STEM workforce training, economic impact, etc.), there are not sufficient funds provided in typical single PI proposals to undertake substantial outreach and education activities. Second, the GPG fails to convey to reviewers and proposers the essential elements of broader impacts included in the America Competes Act. Third, it is debatable whether broader impacts should be PI-centered or created as part of a network. Efforts should be made to minimize redundancy in the creation of broader impact programs. It should be recognized by the broader community that leveraging existing infrastructure could lead to innovative broader impacts. Recommendations
• It is recommended that ECCS provide selective recognition of outstanding proposed broader impact efforts to newly awarded programs. This incentivizes creativity and recognizes innovative or impactful programs, as well as providing the PI additional recognition/distinction of their leadership that would be valued at their home institution
• To incentivize the creation of innovative or impactful programs, it is recommended that ECCS provide supplements for meritorious BI programs. That this may be coupled to proposals receiving the strong BI evaluation distinction in panel summaries; or, be open to the support of new ideas during the project, which could be awarded at the program director’s discretion.
• ECCS may request optional BI nuggets from PIs, as part of the review process of ongoing grants.
• It is recommended that the NSF and ECCS define Grand Challenges in BI with the idea that this may facilitate request for further financial support for BI and may encourage the formation of BI networks throughout the community. This would facilitate more coherent BI activities between PIs, institutions, and research areas.
3. New approaches and institutional initiatives on Broader Impacts Inconsistency over the expectations and prioritization of Broader Impacts exists on all sides of the NSF funding process, leading to difficulty in assessing the value of Broader Impact plans as well as an often under-commitment of such broader efforts from both investigators and institutions. Additionally, the disciplines encompassed by the ECCS division face a unique set of challenges to maintain the trajectory of their societal status and value. The creation of new approaches and initiatives can be utilized to clarify and incentivize Broader Impacts at both the agency and ECCS levels, and such approaches are discussed in this section. As depicted in Figure 1, prioritizing Broader Impact activities through the use of incentives requires a three-pronged approach, involving the participation of PIs, reviewers, and departments/universities, with
the NSF agency at the core. The key recommendations for new approaches and initiatives emerging from this discussion lie at the interface of each set of two prongs. At the Reviewer/Investigator interface, a proposal paradigm shift is suggested to reflect the importance of the Broader Impacts in an NSF grant proposal. At the Reviewer/Institution interface, the grand challenges central to the ECCS disciplines should be conveyed and understood. At the Investigator/Institution interface, a strategy for incentivizing both parties to place the appropriate value to Broader Impacts is proposed.
Figure 1. Prioritizing Broader Impacts by (top) defining Broader Impact priorities to inform institutions and review panels, (bottom right) incentivizing institutions and investigators, and (bottom left) conveying the importance of broader impacts to investigators and review panels.
3.1 Conveying the importance of Broader Impacts to investigators and reviewers Although the terms “Intellectual Merit” and “Broader Impacts” describe the two review criteria applicable to all NSF proposals, it is clear that Intellectual Merit dominates the proposal preparation and review process, usually at the expense of broader impact considerations. In terms of proposal preparation, the Broader Impacts are often described in short concluding passages. In contrast to intellectual merit that is articulated in great detail with specific goals and supporting evidence, the Broader Impacts are often described in generic terms without such goals or evidence. The review process mirrors (and likely perpetuates) this emphasis. Individual and panel reviews tend to focus on intellectual merit, where reviewers are comfortable critically examining proposal details. As a result, the review process inherently places much lower emphasis on Broader Impacts. There is no question that Intellectual Merit should be the primary criteria for all proposal reviews, but it should not be the sole criteria. The challenge is how to communicate the importance of broader impacts in a manner that motivates both investigators and reviewers to address this criterion. If empowered to dedicate more attention to broader impacts, the proposal preparation and review process could become an opportunity to help define, examine, and promote more impactful projects. Recommendations While there have been adequate discussions and recommendation in improving the clarity, the clarification also needs to be emphasized to convey the importance of BI to investigators and reviewers. In addition to the recommendations elsewhere, ECCS should create documentation that communicates
how the broader impacts criterion can be best met by ECCS projects. This documentation should include highlights about exceptional broader impact programs (similar to the research Highlights promoted by the NSF Discoveries website, but specifically for broader impacts) as well as a broader impacts Frequently Asked Questions (FAQ) document that could serve as a quick reference to both investigators and reviewers. The inclusion of specific examples or best practices from other projects may be useful to the investigators. 3.2 Incentivizing investigators and their institutions While clarifying the importance and relevance of Broader Impacts will create a potential for improved Broader Impacts within ECCS, the majority of investigators may still fall short on the expectations of NSF until the culture of Broader Impacts within universities changes. The perception in many institutions today is that Broader Impacts are not valued by departments, colleges, or universities, as they often do not lead to the factors that improve ranking, visibility, and recruiting. If no benefit is conveyed to institutions in the terms of financial advancement or prestige, then it is unlikely that these institutions will value Broader Impacts at the level that NSF expects. If institutions do not place a value on Broader Impacts, then investigators may hesitate to invest in Broader Impacts in fear of damaging their career trajectories. Moreover, if institutions do not invest in resources to promote Broader Impacts among its investigators, then the burden is left to the investigators, who are already stretched too thin to establish unique foundations for their Broader Impacts. Recommendations To influence a culture change within institutions, it is recommended that NSF utilize two mechanisms: financial leverage and recognition. To begin, NSF can make it clear to all institutions that Broader Impacts are now an important element in determining whether an award is funded, and that investigators who can demonstrate that their institution has well established resources and networks to support Broader Impacts will be more likely to receive NSF awards. A list of examples can be provided, but creativity should be encouraged. Furthermore, it should be clarified that simply appointing an administrator is not sufficient; rather, tangible resources and networks that promote authentic engagement of investigators should be visible. Because institutions benefit in terms of ranking and visibility from investigator awards, in particular CAREER awards, it is anticipated that they will respond to directions that improve investigators’ opportunities for funding. One opportunity to supply tangible financial support to this message is to create Broader Impact supplements for NSF awards. Currently investigators can apply for REU and RET supplements. It is recommended to generalize the supplement to include supplements for REU, RET, and any other activity that falls within the scope of Broader Impacts, including K-12 STEM outreach, technology transfer, communications to the public, etc. To be effective, supplements should be up to 20% of the initial award size and should allow for modest salary support such that the investigator and a graduate student or post-doc can dedicate significant effort to the Broader Impact activity. A more substantial option to financially incentivize investigators and their institutions is to establish a new mid-career award that serves to provide opportunities for professional promotion to mid-career faculty while rewarding a track record of Broader Impact contributions. The mid-career award could not only require a significant Broader Impact component, but could state in the solicitation that investigators who can leverage existing resources and networks at their institution will receive the strongest consideration, thus sending the message to institutions that such established resources will lead to new opportunities for awards.
However, given the current climate of budget constraints, financial incentives alone may not be sufficient to change the culture, indicating that NSF should pursue other means as well. Investigators value national recognition, as this strengthens their opportunities for promotion, and institutions likewise value national recognition, as increased visibility improves department ranking and prestige. NSF recognition of notable Broader Impacts by investigators and institutions would constitute a strong component in such a portfolio. These awards, which may have a small financial prize or none at all, would be announced through ECCS and potentially through relevant societies. Importantly, the awards would enable institutions to promote the work and achievements of their faculty, which incentivizes both the investigator and institution. 3.3 Identifying Broader Impact Grand Challenges While motivating institutions to establish resources may engage institutions and their investigators in Broader Impact activities, misconceptions and lack of direction cloud the foundation’s desired targets and outcomes for Broader Impacts. This presents an opportunity to clarify Broader Impact targets to institutions and their investigators while establishing ECCS-specific Broader Impact outcomes. Furthermore, by presenting review panelists with these ECCS-specific Broader Impact opportunities, proposed Broader Impact directions can be judged uniformly across proposals and programs. Recommendation To direct institutions and their investigators towards a clear set of Broader Impact targets it is recommended that NSF work with the ECE Department Heads Association (http://www.ecedha.org/) and the relevant professional societies to publish and distribute a collection of Broader Impacts Grand Challenges for Electrical and Computer Engineering. In particular this list would couple Broader Impacts with the future of Electrical and Computer Engineering (ECE). Some examples of Broader Impact Grand Challenges include, but are not limited to:
• Educating and exciting the general public on what ECE is and what ECEs do. • In particular how ECE will contribute to new high-impact societal developments (biomedical,
robotics, Internet of Things, etc.). • Increasing enrollments in ECE programs. • Diversifying the population in ECE careers. • Training the workforce to leverage new advances developed by ECE. • Focus on unique populations, such as veterans, displaced populations, and underrepresented groups. • Envisioning a post-Moore’s Law era.
Summary The workshop participants have consistently identified issues with current Broader Impacts criterion and practice, and made a number of useful recommendations to the ECCS division. Discussions during the workshop provided in-depth understanding of the goals and value of Broader Impacts activities in ECCS research. Recommendations include concrete measures for the ECCS to clarify NSF policies and expectations on Broader Impacts to reviewers and investigators, through clear documentation, effective communication and proper training both during panel deliberation and on a regular basis. The workshop outcomes lead to meaningful metrics and tangible procedural changes that may improve the consistency of panel evaluation and ensure the long-term broader impacts of ECCS funded projects. Incentives are suggested to advance and catalyze Broader Impacts innovations in ECCS research, such that researchers and institutions can align their Broader Impact activities with greater societal return for NSF investments.
References
[1] “Investing in Science, Engineering, and Education for the Nation’s Future,” NSF Strategic Plan for 2014-2018, March 2014.
[2] Grant Proposal Guide, NSF 16-1, January 2016. OMB Control Number 3145-0058.
[3] Perspectives on Broader Impacts, NSF 15-008, 2015. http://www.nsf.gov/od/oia/publications/Broader_Impacts.pdf
[4] Institute for Broadening Participation http://www.ibparticipation.org/
[6] Framework for Evaluating Impacts of Broadening Participation Projects: Report from a National Science Foundation Workshop http://www.nsf.gov/od/broadeningparticipation/framework-evaluating-impacts-broadening-participation-projects_1101.pdf
[7] The America Creating Opportunities to Meaningfully Promote Excellence in Technology, Education, and Science Act (America COMPETES Act), P.L. 110-69, August 2007.
[8] America COMPETES Reauthorization Act, P.L. 111-358, 2010.
Workshop Agenda
Thursday, May 12
7:30 am Registration & Continental Breakfast
8:30 am Broader Impacts: Achieving Greater Goals Pramod Khargonekar, Assistant Director For Engineering
Introduction and Overview of Workshop Goals Samir El-Ghazaly, Division Director, ECCS Division, NSF
Zhi Tian, Workshop Chair, ECE Department, George Mason University
9:20 am
9:40 am
10:05 am
10:25 am
10:50 am
Nanoelectronics from the Lab to the Marketplace, Classroom and Society: the Texas Story
Sanjay Banerjee, Director, Microelectronics Research Center, Univ. of Texas at Austin
Adding Rigor to the Broader Impact Review in ECCS Thomas Weller, Chair, Department of Electrical Engineering, Univ. of South Florida
Creating Broader Impacts using Social and Mass Media Pradeep Fulay, Associate Dean for Research, Benjamin M. Statler College of Engineering and Mineral Resources, West Virginia University
Creating a Culture that Celebrates Broader Impact Initiatives with Longevity Christine Grant, Associate Dean of Faculty Advancement, College of Engineering, North Carolina State University
Enhancing the Values of Impacts Beyond Research in NSF Program Nelson Tansu, Director, Center for Photonics and Nanoelectronics, Department of ECE, Lehigh University
11:15 am Break
11:25 am
11:50 am
12:10 pm 12:30 pm
GOALI: Undergraduate Broader Impacts André Knoesen, ECE Department, University of California, Davis
A Personal Perspective on Broader Impacts in Science and Engineering Jingyu Lin, Linda F. Whitacre Chair in ECE, Texas Tech University
Beyond the classroom: from ubiquitous learning to residential communities of entrepreneurs for a new generation of innovators
Gianluca Lazzi, Chair, Department of ECE, University of Utah
Creating Broader Impacts to Promote the Future of Electrical and Computer Engineering John Papapolymerou, Chair, Department of ECE, Michigan State University
12:45pm Lunch Provided at Workshop
Thursday, May 12 -- Afternoon
2:00 pm 2:20 pm
Broader Impact: An EPIC Event Sheldon H. Jacobson, Director, Simulation and Optimization Laboratory, University of Illinois - UIUC
Broader Impacts: Integrating Research and Teaching in a Wide Variety of Communities & Settings
Bozenna Pasik-Duncan, Department of Mathematics, University of Kansas
2:40 pm Guidance for Break-Out Sessions - Z. Tian, Workshop Chair 2:45 pm Break-Out Session I - Understanding the economic and social value of ECCS projects
Facilitator: Rhonda Franklin, University of Minnesota Scribe: John Dallesasse, University of Illinois at Urbana-Champaign
Break-Out Session II – Meaningful metrics to assess and promote Broader Impacts activities
Facilitator: Tom Wellner, University of South Florida Scribe: Douglas Irving, North Carolina State University
Break-Out Session III – New approaches and institutional initiatives on Broader Impacts Facilitator: Rashid Zia, Brown University Scribe: Ian White, University of Maryland
4:00 pm Break
4:15 pm Break-Out Sessions - Continue
5:30 pm Open Discussion – What are we learning? Facilitator: Zhi Tian, Workshop Chair
6:00 pm Adjourn
7:00 pm Group Dinner
Friday, May 13
8:00 am Continental Breakfast
8:45 am Report from Scribes – Session I - John Dallesasse Session II - Douglas Irving
Session III - Ian White
10:00 am Preparing Draft of Workshop Report
10:15 am Break
10:30 am Preparing Draft of Workshop Report - Continue
11:45 am Open Discussion – What do we recommend?
12:00 pm Lunch Provided at Workshop
1:00 pm Workshop Ends
List of Participants
Abed, Eyad National Science Foundation
Baheti, Radhakishan National Science Foundation
Banerjee, Sanjay University of Texas at Austin
Bank, Seth University of Texas at Austin
Brown, April Duke University
Cheng, Susan George Washington University
Culbertson, Joanne National Science Foundation
Dagenais, Dominique National Science Foundation
Dallesasse, John University of Illinois at Urbana-Champaign
El-Ghazaly, Samir National Science Foundation
El-Masry, Nadia National Science Foundation
Fallahi, Mahmoud National Science Foundation
Fay, Patrick University of Notre Dame
Foster, Amy Johns Hopkins University
Franklin, Rhonda University of Minnesota - Twin Cities
Fulay, Pradeep West Virginia University
Goodings, Deborah National Science Foundation
Grant, Christine North Carolina State University
Goldberg, Larry National Science Foundation
Gonzalez, Cecile National Science Foundation
Haddad, George University of Michigan - Ann Arbor
Harvey, James Army Research Laboratory
Henderson, Rashaunda University of Texas at Dallas
Irving, Douglas North Carolina State University
Jacobson, Sheldon University of Illinois at Urbana-Champaign
Kar, Swastik Northeastern University
Khargonekar, Pramod National Science Foundation
Kim, Seongsin University of Alabama
Knoesen, Andre University of California - Davis
Lazzi, Gianluca University of Utah
Lee, Charles Air Force Office of Scientific Research
Lin, Jingyu Texas Tech University
Ling, Hao National Science Foundation
Manitius, Andre George Mason University
Palmer, William DARPA
Papapolymerou, John Michigan State University
Pasik-Duncan, Bozenna University of Kansas
Pavlidis, Dimitris National Science Foundation
Sorger, Volker George Washington University
Tansu, Nelson Lehigh University
Tian, Zhi (Gerry) George Mason University
Varshney, Usha National Science Foundation
Wasserman, Dan University of Illinois at Urbana-Champaign
Weller, Tom University of South Florida
White, Ian University of Maryland
Zavada, John National Science Foundation
Zhirnov, Victor Semiconductor Research Corporation
Zia, Rashid Brown University
Abstract of Plenary Talks on Broader Impacts
Nanoelectronics from the Lab to the Marketplace, Classroom and Society: the Texas Story Sanjay Banerjee
Microelectronics Research Center University of Texas at Austin
It is obviously incumbent on academic engineers and scientists to try to translate their research for the betterment of society in terms of commercial products and K-gray education. There are challenges and opportunities that are unique to a large, sparsely populated state like Texas, with a strong high tech hub in Central Texas, abundant sunshine and a growing minority population.
We will discuss how to enable and foster breakthroughs in nanoelectronics in the high-tech
Texas triangle around Austin-Dallas-Houston for applications in tele-medicine to serve poor, isolated communities. Because of her geography, there are tremendous opportunities in energy generation through photovoltaics, as well as opportunities in smart lighting and cooling.
Texas will soon become a majority-minority state, with a growing Hispanic population, many of whom are economically and educationally dis-advantaged. There are opportunities to perform academic research in nanoelectronics, while engaging with under-represented minorities through education, and contribution to societal and ethical issues as a necessary part of nanoscience.
Broader Impact: An EPIC Event Sheldon H. Jacobson
Simulation and Optimization Laboratory University of Illinois at Urbana-Champaign
Intellectual Merit and Broader Impact both require significant attention if NSF is to continue to contribute to the economic and social well-being of the United States. Classifying Broader Impact based on the type of activity and the time line for its implementation will provide a mechanism by which PIs can better frame their Broader Impact activities, allow such activities to be more easily evaluated, and enable institutions to better support such efforts.
GOALI: Undergraduate Broader Impacts André Knoesen
Department of Electrical and Computer Engineering University of California, Davis
A GOALI project between UC Davis and IBM Almaden extensively involved undergraduate students from San Jose State University, as well as other universities, preparing them for
graduate school and careers in industry. This talk focuses on how a small K-12 outreach effort started in this GOALI grew into a much larger ongoing effort, known as EE-Emerge, to engage junior students, specifically under-represented minorities (URM) and first generation students, in hands-on engineering projects that provide context for the fundamental knowledge gained in the classroom. EE-Emerge prepares the students for their senior design project experience, and all are engaged in public outreach activities to promote engineering to K-12 students.
Adding Rigor to the Broader Impact Review in ECCS Thomas Weller
University of South Florida
In ECCS it is common practice for the Intellectual Merit and Broader Impact review criteria to be primarily associated with the proposed research plan and the proposed education and/or outreach plans, respectively. And while not explicitly stated in the NSF Grant Proposal Guide, the panel review process weighs heavily on Intellectual Merit and generally treats Broader Impact as a secondary discriminator. This weighting is appropriate given the Division’s primary mission to address fundamental research issues in the electrical, communications and cyber systems domain. Yet the Division also has a goal to integrate education into research and to ensure the development of a diverse workforce, and so the evaluation of Broader Impact should be addressed with similar rigorous standards as those applied to Intellectual Merit. Appropriate changes to the review process may include assigning a separate rating (Excellent/Very Good/Good/Fair/Poor) to each criteria, and for Broader Impact outlining expectations for measurable results and/or assessment, sustained impact, novelty and clear ties to the research plan. Consideration should also be given to clearly stating whether traditional academic activities such as the development of new courses and training graduate students are meritorious, as reviews often describe such efforts as normal expectations of faculty members.
Creating a Culture that Celebrates Broader Impact Initiatives with Longevity Christine S. Grant
Associate Dean of Faculty Advancement, College of Engineering North Carolina State University
During the process of writing grants, faculty often are determined to create new and innovative initiatives in the realm of Broader Impacts (BI). They write a research proposal and develop a creative BI initiative that may be a cross-disciplinary collaborative (e.g., humanities, education) with grand aspects that appear to be compelling to reviewers. Sometimes during the course of the actual grant execution, the BI Initiative may (or may not) be as “fruitful” or as “successful” as the reviewers anticipated. The faculty may then write subsequent proposals that create research directions that build on past research success (Research+). These proposals, however, may have brand new BI components (BI:B) in an effort to again stand out during the review process. The NSF ECCS Division could create a new paradigm rewarding and celebrating
researchers who establish a strong initial BI Initiative that becomes an essential aspect of subsequent research proposals; and highly regarded research grants. This new culture would incentivize researchers to adopt BI initiatives that become a core component in their scholarly activities. This would also create an environment connecting multidisciplinary researchers in BI collaborative with longevity.
Creating Broader Impacts using Social and Mass Media Pradeep Fulay, Associate Dean for Research
Benjamin M. Statler College of Engineering and Mineral Resources West Virginia University
In this presentation, I would share ideas on how investigators can use social media and mass media to communicate the importance and significance of their research. I would also describe examples of how broader impacts and outreach can be localized in certain geographical areas and how in some cases the broader impact created is not always linked to the level of funding available.
Enhancing the Values of Impacts Beyond Research in NSF Program Nelson Tansu
Director, Center for Photonics and Nanoelectronics Department of Electrical and Computer Engineering (ECE)
Lehigh University
I will focus on how the importance of research impacts which will be part of the Intellectual Merits, and how the beyond research impacts will be important for societal and educational purposes. The importance of aligning these impacts with the research portion for NSF program is essential given the boundary conditions defined from the budget in each program. The approaches to execute these goals will be relevant given the time period and financial constraints in each program. The methods to enhance its impacts and examples of such impacts will be provided.
