Evaluation Report of On the Cutting Edge
February 5, 2010
Submitted by
John McLaughlin, External Evaluator
McLaughlin Associates, Lanexa, Virginia
Data collection and analyses conducted by the Cutting Edge Evaluation Team of
Ellen Iverson, Evaluation Director
Randahl Kirkendall, Evaluation Specialist
Monica Bruckner, Evaluation Assistant
Science Education Resource Center, Carleton College, Northfield, Minnesota
Principal Investigator Liaison to the Evaluation Team
Cathy Manduca, Director
Science Education Resource Center, Carleton College, Northfield, Minnesota
Page 2 of 63
Contents
Executive Summary 3
2009 External Evaluation of On the Cutting Edge Executive Summary 7
Introduction 7
Internal Evaluation 7
External Evaluator Data Collection 8
Summary 12
The Cutting Edge Structure for Learning 14
An Integrated Professional Development Program 14
Cutting Edge Workshop Participation 15
Cutting Edge Website Contributions and Use 17
Website and Workshop Use Together 19
Faculty Response to Cutting Edge 20
The Cutting Edge Impact on Teaching 32
Changes in Teaching 32
Shifting how Faculty Think About Teaching 40
Impact of Cutting Edge Experience on Student Learning 44
The Cutting Edge Impact on the Geoscience Education Community 45
Networking 45
Contributing to the Geoscience Education Community 49
Result of the Cutting Edge Impact on the Geoscience Education Community 54
Appendix: Evaluation Methodology 56
Large-scale survey of Geoscience Community 56
Surveys of all Cutting Edge Workshop Participants and Website Users 57
In-depth Small Samples of Interviews 59
Report on Statistical Findings 60
Limits of this Evaluation 61
References Cited 63
Page 3 of 63
Executive Summary
This report summarizes the evaluation of the On the Cutting Edge Faculty Professional
Development program (http://serc.carleton.edu/NAGTWorkshops) from 2002 to 2009. On the
Cutting Edge (Cutting Edge) is a comprehensive program of workshops and related web-based
resources to support faculty professional development at all stages of their careers. The project is
funded by the National Science Foundation Directorate for Education and Human Resources
Division of Undergraduate Education as part of the Course, Curriculum, and Laboratory
Improvement program (CCLI) and managed collaboratively by PIs Heather Macdonald (College
of William and Mary), Cathy Manduca (Carleton College), Dave Mogk (Montana State
University) and Barb Tewksbury (Hamilton College. The Cutting Edge proposal responded to
Program Solicitation NSF 05-559, specifically to increase faculty participation in the cycle of
educational innovation, to provide support in developing teaching skills and knowledge, to
develop leaders in professional development and innovation, and to develop networks and
communities in which a culture of educational innovation can thrive and grow. The methods,
tools, and limitations of this evaluation are described in the appendix.
The Cutting Edge program works via a learning structure built from an integrated workshop
series and a website of teaching resources. Cutting Edge developed models for both face-to-face
and virtual workshops ranging from one to six days in length and offered 56 workshops between
2002 and 2009. 1751 faculty, post-docs, and graduate students participated in one or more
workshops. More than 20% of the participants came to two or more workshops for a total of
2246 workshop participant-seats. Approximately 25% of the geoscience faculty in the U.S. and
more than half the departments in the United States have participated. The reach of the
workshops was in proportion to the different types of departments, with the exception of two-
year colleges which were underrepresented. Cutting Edge workshop participants included a
higher percentage of women and members of underrepresented groups than in the overall
population of U.S. geoscience faculty. End-of-workshop evaluations indicate a high level of
participant satisfaction with a median of 9.1 on a 10-point scale.
The Cutting Edge online collection of teaching activities and other resources supported those
who attended the workshops, provided a venue for sharing teaching resources, and extended the
reach of Cutting Edge to those who had not attended a workshop. The web collection includes
over 4000 pages, 33 topical collections, and more than 1200 community-contributed teaching
activities. In 2008, more than half a million users visited the website and approximately 12,000
visited 10 or more pages in a single session. Approximately 25% of users are geoscience faculty
with the remaining 75% comprised of faculty from other disciplines, K-12 teachers from
geoscience and other disciplines, undergraduate and graduate students, postdocs, and others.
Approximately 30% of U.S. geoscience faculty use the website to find teaching ideas and
Page 4 of 63
materials, to compare their own teaching to that of others, to learn about teaching methods or
geoscience topics, and to obtain information about career planning. Users report that the website
increases their confidence in trying new teaching methods.
Impact on Participants Teaching
Comparison of faculty who have participated in the On the Cutting Edge program, either using
the website, or by attending a workshop and making use of the website, to faculty who had not
participated in the On the Cutting Edge program, shows that participants were more likely to
converse with other faculty about teaching, make more use of educational research, and more
likely to seek information about others‟ instruction (often through the website, Fig. I).
Participants were more likely than non-participants to have changed their teaching methods in
the past two years, and specifically, they were more likely to have decreased the amount of time
lecturing and to have added group work or small group activities. Participants reported more use
of in-class questioning, small group discussion and in-class exercises. Participation also appears
to have impacted the evaluation strategies and assessment methods that faculty used. In many
cases, impacts were more pronounced for those who both attended a workshop and made use of
the website (Figs. II and III). Qualitative data indicate that workshop participants underwent a
shift in their teaching philosophy to an approach that was more focused on student-centered
learning. This change in philosophy was foundational to subsequent learning and changes in
teaching practice.
Figure I. Cutting Edge website use and changes in teaching.
70%
37%47%
32%25%
35%25%
54%
21%
32%
15% 15%22%
17%
0%
10%
20%
30%
40%
50%
60%
70%
80%
Made changes in teaching
methods used in intro courses over past two
yrs
Spent less time lecturing
Increased questioning of
students during lecture
Added group work/small
group activities
Spent more time on class/small
group discussions
Changed assessment
tools/strategies
Added assignments
Impacts of CE Participation on TeachingWeb
No Web
Page 5 of 63
Figure II. Cutting Edge website use, workshop participation and changes in teaching.
Figure III. Cutting Edge website use, workshop participation and use of teaching strategies.
15%
22%25%
32%
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
50%
Added group work or small group activties
Spent less time lecturing
Changes made in last two years by Cutting Edge exposure
No Cutting Edge exposure
Website only
Website and Workshop
40%43%
42%
17% 19%
53%
28%34%
65%
44%50%
0%
10%
20%
30%
40%
50%
60%
70%
Posing questions answered by entire
class2
Small group discussion or think-
pair-share
In-class exercises
Teaching Strategies Used Weekly or Nearly Every Class by Cutting Edge Exposure
No Cutting Edge exposure
Website only
Website and Workshop
Page 6 of 63
Impact on the Geoscience Education Community
On the Cutting Edge has increased networking and interaction on topics of teaching throughout
the geoscience community. Workshops provided valuable opportunities to network with other
faculty and with experts in education and cognitive science. Workshop participants talked more
to colleagues about teaching after their workshop experience; they interacted with colleagues
across the nation in addition to those on their campus, and they made use of the website to find
out what others were doing. An important subset of participants built new networks for career
planning and made use of the website as a career development networking tool.
Increased interaction and networking supported ongoing scholarly work in geoscience education.
Participants specifically attribute 40 successful grant proposals to participation in the On the
Cutting Edge program. Cutting Edge participants were more likely than others to contribute to
the community discussion of teaching both through presentations and publications, as well as by
sharing teaching materials on-line.
There also appears to be a clear dose response of reported changes at the department level in
response to the number of faculty participating in workshops. The 2008 Participant Survey found
that participants from departments in which at least 3 department members had attended a
workshop were 3 times more likely to report that the workshop had changed their department‟s
collective approach to teaching compared to participants coming from departments in which 2 or
fewer members had attended a workshop.
The impact of the program clearly extended beyond the geoscience community. The project PIs
contributed to national conversations on faculty professional development. An area not yet
evaluated is the impact that Cutting Edge has on faculty participants who are not from the
geosciences as they also develop new insights into teaching and expand their professional
networks. This population represents 75% of website users.
In sum, as a result of the On the Cutting Edge program:
Education is now an accepted topic for discussion and work among geoscience researchers from
all types of institutions;
Research networks have become more supportive and conducive to education research;
A national perspective on geoscience teaching has been created and is being used to benchmark
teaching norms and behaviors at the institutional and individual levels;
Geoscience faculty members are connecting with new colleagues from different institutions and
different disciplines;
There is a culture of sharing experiences, successes, and new ideas in education and teaching
practice; and
The sphere of influence and leadership roles for education specialists in geoscience have
expanded.
Page 7 of 63
2009 External Evaluation of On the Cutting Edge
Executive Summary
John A. McLaughlin
Introduction
On the Cutting Edge (CE) is a comprehensive program of workshops and related web-based
resources to support geoscience faculty in their professional development at all stages of their
careers. Cutting Edge specifically aims to increase faculty learning about the cycle of educational
innovation, to provide support in developing teaching skills and knowledge, to develop leaders in
professional development and innovation, and to develop networks and communities in which a
culture of educational innovation can thrive and grow.
The evaluation of the CE project is the responsibility of an evaluation team whose members are
Ellen Iverson, Randahl Kirkendall, and Cathryn Manduca. John McLaughlin was the external
evaluator and supervised the program evaluation design, implementation, analyses, and
reporting. In addition, the external evaluator collected evidence through in-depth interviews with
key stakeholders, participants, and observers of the CE project. The purpose of this report is to
summarize the external evaluator‟s view of the project performance.
Internal Evaluation
As noted above the impacts of emerging from the CE project were projected to occur in two
domains: a) individual workshop participants and website users, and b) geoscience community
impacts following from individual changes and applications. This section presents the external
evaluator‟s observations based on his review of the annual CE evaluation report.
Individual Impacts
One of the most powerful evaluation tools employed by the CE evaluators was a survey of the
total geoscience faculty community. The survey was conducted by an independent survey group
and allowed a comparison of faculty members who participated and did not participate in CE
activities – workshop or web site. In addition to this survey, the team used interviews, surveys,
observations, and document reviews to inform them of CE performance.
It is clear from a review of the evidence produced by these multiple methods that participants in
CE activities changed the way they think about teaching, how they approach planning and
assessment of learning, and how they deliver learning in their classrooms. One of the key aspects
of this shift in teaching practice was toward student-centered learning -- changing from „what do
they teach‟ to what are their students learning‟. In addition, evidence demonstrates that
Page 8 of 63
participants increased their level of teaching confidence. They increased their knowledge and
skills/tools related to their planning, teaching, and assessment and they increased their
pedagogical expertise.
Community Impacts
The evidence found in the various data collection strategies indicates that Cutting Edge has
played an important role in creating an environment that is characterized by the enhanced use of
established networks and new information pathways in a way that has overcome the traditional
barriers between research and education, across disciplines, and among different institutional
types. This, in turn, has nurtured and encouraged faculty members to learn from each other and
share their experiences with new teaching methods and technologies.
Participants became more likely to take part in the cycle of innovation in which they developed
and tested new teaching approaches and tools in their classrooms, shared successes with
colleagues in their institution, and contributed new learning to the CE web and the geoscience
literature through publications and presentations.
Further, participants were more likely to learn form others rather than self-invent. They were
more motivated to adopt new ideas from colleagues and to seek resources from the Internet,
including CE. They were more likely to attend professional workshops and seminars that focused
on pedagogy. These changes in approach to course design and assessment occurred for both
introductory and advanced courses.
External Evaluator Data Collection
The external evaluator provided technical consultation to the CE staff members on the internal
evaluation design, collection, analyses, and reporting strategies. In addition, he collected data
from participants in the emergent CE Leadership Development program and a number of well-
known geoscience leaders across the nation. This section summarizes the findings from those
interviews.
Leader Interviews
The Leadership Development component of the CE project is best characterized as emergent.
Aas the CE project advanced, the need to expand the offerings through the workshops and
website grew, and thus CE needed to develop other leaders to meet the need. In addition, the
Leadership program was initiated to enable a broader impact on the community at large. Ten
participants in the Leadership program were interviewed by phone. It should be noted that CE
was just beginning a formal Leadership development program that was designed to include a
formal induction workshop and continuous coaching by the PI‟s. The results of these interviews
were meant to contribute to the design work. Those interviewed had participated in a less formal
induction into their leadership roles.
Page 9 of 63
When asked why they appreciated the CE workshops in comparison to others they had attended,
they interviewees noted that the CE workshop presenters did not use the „expert‟ model but “had
a genuine desire to share their experience.” The workshops were applied in nature and focused
on shared learning. “It was the right mix of getting the ideas and being given the time to produce
something.” “It is like the feeling I get after a talk when I realize that the students are really
listening and I have learned something from them too.” “I got turned on to pedagogy and how it
works. There are all these tools and techniques out there – different ways of teaching and
learning – in all of the workshops it is about engaging people and being engaged by the
material.”
Next, faculty participating in the Leadership Development Program were asked how their
involvement in CE influenced their career path. Influences were deep, lasting and ranged from
changes in their pedagogy and their impact outside of their classes. In addition, their
participation increased their teaching confidence. “The Cutting Edge serves as the model of the
way instruction should be going. I have gotten a lot more active in issues of pedagogy.” “Early
on it cemented my feelings that I wanted to teach and that there were many ways to do that – and
that I could be creative.” “I did have some anxiety about transitioning back to the teaching world.
The reason I went to the Cutting Edge is because I knew I needed a jump start and it did that.” “I
was able to make major curricular changes in our program that was designed based on the
principles that I learned through Cutting Edge colleagues and putting the audience/students‟
needs first. It is truly a student centered program.” “I am a department chair now and I have
encouraged my faculty to invest time in attending these workshops and to invest time in
adjusting their curricula.”
Participants also gained an appreciation for what makes a successful CE workshop. It is
important to get the right mix of people. “It is useful to get experiences from a heterogeneous
group of people – when you are teaching about something topical it is useful to get segments of a
population that is going to be responsible for teaching that.” Starting with the end in mind was
also viewed as important. “Getting the goal right, keep a level of respect for people‟s time, and
make sure the group dynamic is positive with high energy.” Organization from planning to
implementation is critical. Skills, logistics, encouraging people to interact & perform during the
workshop, the amount of work PI‟s have gotten people to do in preparing and following up on
the workshops.” “You need to have a program that mixed the old and new stuff with reflection.
If they do not have the time or space for reflection – the ideas will not be implemented.”
When asked how they were inducted into their leadership roles most suggested that the
experience was incremental – like boiling a frog! “They ask you to present during a session, they
recognize you are doing something special and they want you to share it with others. It is baby
steps. You can volunteer to present – you don‟t have to and the feedback is helpful. This builds
Page 10 of 63
your confidence. Then I was asked to co-facilitate with experienced presenters who I was
comfortable with. There is a clear place as to your piece.”
When asked how they were selected, participants suggested that they had demonstrated
leadership potential through their participation in CE workshops. “They recognized that I “got it”
and they recognized that I was able to see its potential. I was going to run with the ball. That
was really all they “did”. They did not use me as an example or include me in the conversations.
I am a little extroverted and was able to convey my enthusiasm. They were paying attention.”
“PI called me and said, “I think you would be good at this.” “It seems well thought out – given a
small topic at first and pair with people who knew what they were doing. Doing joint things
where there is someone else in charge and you are there to assist – which teaches you how to do
it. Later pairing experience and less experienced leaders together – co-leading. Finally, I was
leading and coordinating workshops. There is always support.”
All of the ten interviewees indicated that they intend to continue their participation as CE
Leaders. Many have actually taken the initiative to design and offer workshops outside of the CE
organization but still focused on pedagogy in geoscience undergraduate and graduate learning. It
was clear from the interviews that the Leadership development program was beginning to
emerge as an effective way to extend the CE impact.
Gateway Interviews
The external evaluator interviewed a number of nationally recognized leaders in the geoscience
community to determine their impressions of the impact that CE has had on the broader
community and the extent to which the need addressed by CE was still relevant. In the spring of
2009, CE PI‟s identified six recognized leaders in the geoscience field who were knowledgeable
of CE but were not closely affiliated with implementation. The CE external evaluator conducted
the interviews; one person was not available for an interview.
Interviewees had a very broad perception of the CE project that extended beyond the workshops
and website. They viewed CE as having an impact on the geoscience community at large.
“Reaching out to the broaden geoscience community. Providing an opportunity for the
geoscience community to have some sort of common binder from a number of different
perspectives. Making sure geoscience communities are viewed strongly at their institutions.”
“My perception is trying to prevent geoscientists from having to re-invent the wheel from all
aspects by allowing us to communicate with each other in a variety of ways (online, in person) to
help improve the state of university work in Earth Science.”
Interview participants were asked to identify the impact of program activities (workshops,
website). They identified CE as a valuable information resource for geoscientists and the
broader scientific community. The website provides a wealth of information for geoscience
Page 11 of 63
educators and is rapidly becoming a primary resource for geoscience faculty. One participant
reported: “They speak of CE like it is one of their fingers – one of their regular tools. The people
I work with seem to take it for granted like it is a needed tool.” In addition, CE has improved the
image of teaching geoscience among geological scientists. As a result, interest in the Education
Division of the Geological Society of American (GSA) has grown substantially during the
annual conference. Although one participant had not seen a specific impact of CE on the broader
geoscience community, the individual indicated that CE has had an impact on the teaching and
career development education in a variety of scientific societies.
Leaders recognized the importance of building a community of practice within the geoscience
field. Workshops for students and faculty, and community building were noted as the two major
objectives of CE. The significance of community building was described as: “The community is
a pathway by which we achieve something – what are we trying to achieve? I would still say it
is a savvy, smart, skilled group of people in the geoscience community that is extremely
important for the researchers. Mainly it is the people.”
A second national leader noted: “What is interesting about our relationship with CE, it has
enabled two communities to work parallel and in unity (research and teaching). We have used
Barbara Tewksbury as a national treasure. This isn‟t just somebody coming from an educational
school, but we have a practicing geologist. It conveys an amount of legitimacy. It also opened
up an avenue for people who are professionally dedicated to open up things like these. They can
join our groups and be taken seriously without that „snootiness‟ of educational faculty. The
bridge has been created to enable our people to walk over it.”
The opportunity to disseminate information via the CE website provided a pathway for
developing connections among practitioners. Once the process was in place and information
became available online the website became common ground for both educators and researchers.
Increasing participation on policy-making boards by members of CE is evidence of a growing
community of practice. One participant agreed stating: “Yes – more so in the areas of
geoscience education. The growing emphasis by NSF has already helped. They realized the
need for scientists to get the word out, not just doing the work. They have created educational
programs within what used to be purely scientific disciplines. When they look around to see who
to do to, they naturally go to CE.”
All national leaders agreed that there is a long-term need for CE and that the National Science
Foundation should continue funding the project. CE serves the NSF well in supporting research
and education. CE serves as a clearinghouse of information for geoscience where practitioners
can access timely information quickly. At least one individual judged the website as a critical
resource even more important than the workshops. The CE program is unique in addressing the
Page 12 of 63
needs of the scientific community, and should be expanded to other disciplines. One participant
noted, “There isn’t a group to develop younger scientists other than CE.” Another said,
“Absolutely – I think the rotating workshops (most of my exposure is from this) to different
disciplines every year. When they did it in geophysics, we had never had anything like that – we
could come together with groups of like-minded professionals. By the time it rotates back to
geophysics there will be all new participants, research, and points of view.” Another referred to
CE as a vital resource to the community when he said, “There are all these ideas out there. How
do you choose? Where do you go? You go to the CE website. You know your colleagues share
the activities and ideas. In this world of ever expanding information, there is a need for accurate
and reliable information.”
Finally, one national leader noted that there are no other alternative projects that address the
same need. “Yes – if they do not, I don‟t know who will. The agency desires we all are able to
carry out research with a real impact and that education is a real part of what we do. They
should be interested in funding CE. I know they don‟t typically invest long term, but they
should. I don‟t know of other funding sources out there - for example foundations. The other
question – is there a way to make it self-sustaining, which is not realistic since they are trying to
make it accessible to everyone.”
Summary
The external evaluator examined all evidence collected through the CE internal evaluation
strategies and collected additional evidence through in-depth interviews with key stakeholders
and participants in the emergent Leadership initiative. The internal evidence is convincing with
respect to the impact CE is having on its workshop and web site participants. There is sufficient
evidence that participants are changing their philosophy of teaching, their approaches to
planning, delivering, and assessing their instruction. Further, their comments indicate that
participation increases their pedagogical confidence. Impacts extend beyond the individual
participants. When compared to non-CE participants they are active participants in the cycle of
innovation.
Participants in the emergent Leadership development program are very supportive of their
experiences. They reported a deep understanding on what makes the CE experience work and
noted the importance of PI continued support. It is clear that CE has been successful in
developing a set of leaders who can design and deliver workshops, contribute to the web site and
impact the broader geoscience community within and beyond their institution.
Finally, national leaders suggested that CE is unique in providing a valuable service to
geoscience professionals particularly those working in the education arena. Leaders in the field
of geoscience recognize the program as the sole source for professional development training
and resources. CE is credited for reducing the gap that exists between research and teaching
Page 13 of 63
through information dissemination and community building, and increasing interest in the
teaching of geoscience. Interview participants were unanimous in their views that CE merits
long-term support from the NSF.
Page 14 of 63
The Cutting Edge Structure for Learning
An Integrated Professional Development Program
The Cutting Edge program is based on the concept that, if faculty are engaged in a community
where they can share and discuss their teaching and learn from others within and beyond the
geosciences, then the quality of teaching will rise for individual faculty, ultimately improving
student learning across the country. The Cutting Edge program was designed to develop and
support such a learning community via three elements: an integrated workshop series, a website
of teaching resources, and a leadership development program.
An integrated workshop series
Cutting Edge developed models for both face-to-face and virtual workshops ranging from one to
six days in length and has offered 56 workshops since 2002, including 27 in the last three years.
All workshops were designed to model and promote best practice in pedagogy, to help faculty
stay current in geoscience and educational research advances, and to build community resources
and networks. Workshops of different types engaged faculty with different interests and at
different points in their careers:
Emerging theme workshops accelerated the introduction of new geoscience research or
a new aspect of pedagogy into the undergraduate curriculum. Workshop topics through
2010 include biocomplexity, the deep Earth, discoveries from Mars, the early Earth,
energy, geology and human health, hurricanes and climate change, ocean systems, public
policy, rates and time, the affective domain, assessment of learning, metacognition,
teaching complex systems, teaching with data, teaching with models, teaching online,
teaching with games, urban students and urban issues, visualizations, and web design.
Workshops about teaching a core geoscience topic brought together faculty who teach
a topic that is part of the geoscience major to explore issues in teaching this topic and to
share teaching resources. The phrase 'Teaching X' is used to refer to these workshops.
Past workshops included topics such as geomorphology, geophysics, hydrogeology,
paleontology, petrology, sedimentary geology, and structural geology.
Annual workshops that supported faculty at various career stages included a
workshop for early career geoscience faculty, a workshop for graduate students and
postdoctoral fellows on preparing for an academic career, and a workshop on effective
course design (including course "redesign").
Page 15 of 63
Large workshops disseminated best practices in teaching (e.g., Teaching Introductory
Geoscience; Teaching with Visualizations, Models, and Online Data; Teaching in the
Field).
Workshops held in conjunction with meetings of professional societies and research
consortia brought a pedagogical dimension to research meetings for faculty who would
not otherwise attend a "teaching" workshop (e.g., workshops at AGU, GSA, AMQUA
and Goldschmidt meetings).
Follow-on workshops repurposed resources developed for a particular workshop and
brought them to a new audience (e.g., Student Motivation and the Affective Domain;
Pursing an Academic Career).
A website of teaching resources
One of the hallmarks of On the Cutting Edge has been the extensive and widely used online
collection of teaching activities and other resources. The collection supported those who attended
the workshops, provided a venue for sharing teaching resources, and extended the reach of On
the Cutting Edge beyond those attending a workshop. The web collection includes over 4000
pages, 33 topical collections, and more than 1200 community-contributed teaching activities.
A leadership development program
In order to increase the number of leaders prepared to offer professional development for
geoscience faculty, On the Cutting Edge provided leadership experiences and targeted leader
training for more than 100 co-conveners and facilitators. In addition, it provided additional
opportunities for several individuals to develop substantial modules of topical web resources and
for several others to design and lead follow-on workshops that built upon and repurposed
resources developed initially for other Cutting Edge workshops.
Cutting Edge Workshop Participation
The 56 workshops in the Cutting Edge series have attracted a wide range of participants: 1751
faculty, post-docs, and graduate students from the full spectrum of geoscience disciplines
participated in one or more workshops. More than 20% of the participants came to two or more
workshops for a total of 2246 workshop participant-seats. The 1751 workshop participants
included 1376 faculty (out of an estimated 5,600 geoscience faculty in the U.S.) and 375
graduate students and post-docs. Participants came from all 50 states and from 467 different
institutions. As figure 1 shows, 8% of the faculty participants were from two-year colleges; 34%
from undergraduate-only departments; 15% from departments offering MA/MS degrees as their
highest degree, 39% from departments offering PhDs, and 4% classified as other, which included
scientists in research institutions and government agencies. Workshops that addressed the
teaching of a specific core geoscience specialty (e.g. Teaching Structural Geology) drew
Page 16 of 63
participants representing 10-15% of departments teaching these specialties. For comparison, the
AGI Directory of Geoscience Departments lists 757 geoscience departments, with approximately
18% as two-year colleges (134), 33% as four-year institutions (250), 14% offering a masters
degree (109), 33% offering a doctorate (247) and 2% classified as other (17).
Figure 1: Cutting Edge participation by highest degree granted by department.
Thus, Cutting Edge workshops made significant inroads into the geosciences community,
reaching approximately 25% of the geoscience faculty in the U.S., and more than half the
departments in the United States. The faculty reach of the workshops was in proportion to the
different types of departments, with the exception of two-year colleges, where the reach was
disproportionately small compared to the percentage of faculty in two-year colleges.
By two measures, the Cutting Edge workshops participants were more diverse than that of
geoscience faculty in the U.S. overall. While less than 15% of geoscience faculty were women
(AGI,2009; Fig. 2), and only 2% of recent PhD graduates in the geosciences were racial/ethnic
minorities (NSF 04-590), Cutting Edge workshops attracted 47% women and 6% minorities
underrepresented in geoscience (Black/African American, Hispanic/Latino, Native American,
Pacific Islander), more than double the percentage in the faculty population. Several people with
disabilities have also participated in Cutting Edge workshops.
8%
34%
15%
39%
4%
Cutting Edge Participation by Highest Degree Granted by Department
2 Year College (110)
4 Year Degree granted (468)
MA/MS Degree granted (206)
PhD Degree granted (537)
Other (55)
Page 17 of 63
Figure 2: Workshop participation gender compared to breakdown among all geoscience faculty.
Cutting Edge Website Contributions and Use
The Cutting Edge website aims to support workshop participants and extend the reach of the
program by providing a venue for learning about and sharing teaching resources. An important
goal has been to involve the geoscience community in developing the website in order to make it
an accurate, living, and up-to-date representation of current understanding in each topical area.
The website content is developed through the workshop process including major contributions by
workshop conveners, speakers, and participants. The project has successfully completed a
section of the website for each workshop topic. Together these comprise more than 4000 pages
of content in 33 topical collections with more than 1200 community-contributed teaching
activities reflecting extensive involvement of the community in generating an on-line resource
for use by geoscience faculty.
The website is heavily used both within and beyond the geosciences and usage has grown
consistently since it was launched. In 2008, more than half a million users visited the website
and approximately 12,000 visited 10 or more pages in a single session. For comparison, there are
47%
15%
53%
85%
0%
20%
40%
60%
80%
100%
Workshop participants (including students)
Total geoscience faculty, by gender
Workshop Participation by Gender
Women
Men
Page 18 of 63
5,600 U.S. based geoscience faculty members. Traffic to the website in 2009 reflects a 27%
growth in visitors over that of 2008. (Table X).
Table 1: Annual Number of Cutting Edge Visitors
Year Cutting Edge
Visitors
2002 1,500
2003 14,400
2004 46,600
2005 241,000
2006 301,000
2007 400,000
2008 550,000
2009 700,000
A picture of website use can be drawn from qualitative interview and survey data combined with
sampled web log analysis and overall web statistics. Users reported visiting the website to find
ideas about teaching and to compare their teaching to what other colleagues were doing in
specific teaching situations. Analysis of the web logs of website user visits reveals that
approximately 40% of all “deep session” visitors (those seeing 10 or more pages) browsed
teaching activities, 30% visited visualizations, and 20% explored a pedagogic approach. The site
also appeared to be useful to those interested in career development with more than 10% of
session visits including pages from one of the Career collections.
The sheer numbers of users, as well as data from multiple pop-up surveys confirms that the site
extends the reach of the Cutting Edge program beyond workshop participants. Approximately
25% of users are geoscience faculty with the remaining 75% comprised of faculty from other
disciplines, K-12 teachers from geoscience and other disciplines, undergraduate and graduate
students, postdocs, and others. Using these data we estimate that 3,000 of the 12,000 deep
sessions users are geoscience faculty. Overall web usage statistics suggests that 60% of the
visitors to the website were from within the United States, which applied to the 3,000, provides
Page 19 of 63
an estimate that 1,800 of the 5,500 (33%) U.S. geoscience faculty were regular users of the
Cutting Edge website. This estimate is comparable to results from the 2009 Faculty Survey
where 30% of responding faculty indicated that they had used the Cutting Edge website. 2009
faculty survey responses also indicate that 45% of those who report using the website have also
attended a Cutting Edge workshop.
Figure 3. Geoscience Users of the Cutting Edge website.
Website use and workshop participation together
Data from the 2009 Faculty Survey revealed that faculty members who participated in a Cutting
Edge workshop were more than four times as likely as non-participants to have reported ever
using the Cutting Edge website (87% vs. 20%). And faculty members who reported ever using
the website were fifteen times more likely than non-users to have participated in a workshop
(45% to 3%). Thus, there is a very high correlation between website use and workshop
participation. The following table shows the frequency of website use and workshop
participation among the respondents to the survey.
Table 2: Frequency of website use and workshop participation among respondents to 2009 Faculty Survey
Ever used the CE website Never used the CE website
Any CE workshop participation 288 44
No CE workshop participation 358 1443
12%
27%
61%
Of Those Who Teach Geoscience2007-2008
Student
K-12 Teacher
College/University Faculty
Page 20 of 63
It should be noted that because website use among workshop participants is very high (87%), any
of the findings in this report that include workshop participation should be interpreted with the
consideration that website use may also be a factor. As already demonstrated, a sizable number
of U.S. geoscience faculty members are using the website and the majority of them (55%) had
not participated in a workshop. Thus, the primary mechanisms by which the structure for
learning operated were via the website alone and the website with workshop participation.
Faculty Response to Cutting Edge
The evaluation found that faculty members who use the Cutting Edge website or participate in a
Cutting Edge workshop (which includes website use) show an increase in their learning from
others and in their use of online resources for teaching. These behaviors are the means by which
faculty with Cutting Edge exposure seem to build upon and use their Cutting Edge experiences
to make the teaching changes that will be discussed in the next section.
Increased learning about teaching from others
In general, faculty members with Cutting Edge exposure were associated with an increase in
learning about teaching from others, via different sources. For example, the findings of the 2009
Faculty Survey indicate that when designing a new activity for an introductory course, Cutting
Edge website users and workshop participants were more likely than non-users and non-
participants to look on the web for others‟ activities for introductory courses and majors courses,
and to read education research papers about the methods they were considering for introductory
courses and majors courses (Figs. 4 and 5). Additionally, Cutting Edge website users were more
likely than non-users to look for activities in texts, lab manuals, or instructor guides (Fig. 4).
Regarding the sources of their learning, the 2009 Faculty Survey found that Cutting Edge
website users and workshop participants were more likely than non-users and non-participants to
learn about new teaching methods from professional meetings or workshops, from publications,
from discussions with colleagues in other institutions, and from on-line resources (Figs. 6 and 7).
Cutting Edge website users were also more likely to learn from discussions with colleagues on
campus.
The specific behaviors indicating increased learning from others reported in the 2009 Faculty
Survey preferentially by faculty members with Cutting Edge experience included:
Seeking advice on teaching Both Cutting Edge website users and workshop participants
were more likely than non-users and non-participants to seek advice on teaching from
colleagues at the campus teaching/learning center, colleagues outside of their institution
who they knew from their geoscience research, colleagues outside of their institution who
they met through their interest in teaching, and nationally known leaders on education
(Figs. 8 and 9).
Page 21 of 63
Utilizing research. When designing a new activity for an introductory course or majors
course, Cutting Edge website users and workshop participants were more likely than non-
users and non-participants to indicate in the 2009 Faculty Survey that they read education
research papers about the methods they are considering (Fig. 10 and 11).
Motivation from others Reflecting on the last time they made a substantive revision to a
course in the 2009 Faculty Survey, Cutting Edge website users were more likely than
non-users to be motivated by a great new idea from a colleague, a publication, or the
web; and finding the ideal activity on a website or in a publication and adopting it
wholesale (Fig. 12). Workshop participants were more likely than non-participants to be
motivated by a great new idea from a colleague, a publication, or the web; and being
inspired by attending a workshop or other professional development activity (Fig. 13).
In looking at the additive effects on motivation of workshop participation to website use,
the data show that the percentage of faculty members who reported that they were
motivated by attending a workshop or other professional development opportunity that
inspired them to make changes doubled from 13% for those with no Cutting Edge
experience to 26% for website users. The percentage then jumped to 60% for website
users who also participated in at least one workshop (Fig. 14). Conversely, being
motivated to make substantive course revisions because the content needed to be updated
was less likely for Cutting Edge website users than for faculty members with no Cutting
Edge experience (52% vs. 68%). And this percentage falls again with the addition of
workshop participation, down to 34% (Fig. 15).
Looking at what others have done and making comparisons Cutting Edge website
users and workshop participants were more likely than non-users and non-participants to
consider what their colleagues were doing when revising an introductory course or a
course for majors (Fig. 16 and 17).
Page 22 of 63
Figure 4: Sources of teaching activities for Cutting Edge web users and non-users.
Figure 5: Sources of teaching activities for workshop participants vs. non-participants.
81% 79%
57%63%
32% 29%
54% 52%46% 46%
15% 13%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90% Teaching Activity SourcesWeb
No Web
INTRO INTRO INTROMAJORS MAJORS MAJORSLook on web to see what
activities others have developed
Read education research papers to learn about methods considered
Look for activities in texts,lab manuals, or instructor's
guides
78%71%
34%27%
60% 59%
19% 16%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%Teaching Activity Sources
WorkshopNo Workshop
Look on web to see what activities others have developed
Read education research papers to learn about methods considered
INTRO INTROMAJORS MAJORS
Page 23 of 63
Figure 6: Comparison of sources for learning about new teaching methods between website users and non-
users.
Figure 7: Comparison of sources for learning about new teaching methods between workshop participants and
non-participants.
73%
50%56%
62% 65%
35%27%
48% 47%37%
0%
10%
20%
30%
40%
50%
60%
70%
80%
Professional meetings/workshops
Publications Discussions with other colleagues on
campus
Discussions with other colleagues in other institutions
On-line resources
Sources of Learning about New Teaching MethodsWeb
No Web
83%
46%
61% 63%
40%32%
50%42%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
Professional meetings/workshops
Publications Discussions with other colleagues in other
institutions
On-line resources
Sources of Learning about New Teaching Methods
Workshop
No Workshop
Page 24 of 63
Figure 8: Places web users vs. non-users turn to for teaching advice.
Figure 9: Places where workshop participants vs. non-participants turn to for teaching advice.
24%
54%
35%
11%
17%
43%
14%
4%
0%
10%
20%
30%
40%
50%
60%
Colleagues at the campus teaching/learning center
Colleagues outside my institution whom I know
from my geoscience research
Colleagues outside my institution whom I met through my interest in
teaching
Nationally known leaders on education
Sources of Teaching Advice for Website UsersWebNo Web
24%
55%
40%
14%18%
45%
17%
5%0%
10%
20%
30%
40%
50%
60%
Colleagues at the campus teaching/learning center
Colleagues outside my institution whom I know
from my geoscience research
Colleagues outside my institution whom I met through my interest in
teaching
Nationally known leaders on education
Sources of Teaching Advice for Workshop ParticipantsWorkshop
No Workshop
Page 25 of 63
Figure 10: Comparison between web users/non-users for likelihood for reading education research papers to
learn about teaching methods.
Figure 11: Comparison between workshop participants/non-participants for likelihood for reading education
research papers to learn about teaching methods.
32%29%
15% 13%
0%
5%
10%
15%
20%
25%
30%
35%
Intro Courses: Read education research papers about considered
methods
Majors Courses: Read education research papers about considered
methods
Likelihood of Reading Education Research Papers to Learn about Teaching Methods
WebNo Web
34%27%
19% 16%
0%
5%
10%
15%
20%
25%
30%
35%
40%
Intro Courses: Read education research papers about considered
methods
Majors Courses: Read education research papers about considered
methods
Likelihood of Reading Education Research Papers to Learn about Teaching Methods
WorkshopNo Workshop
Page 26 of 63
Figure 12: Comparison of activity sources for website users and non-users.
Figure 13: Comparison of activity sources for workshop participants and non-participants.
47%38%
15% 15%
29% 25%
9% 7%0%5%
10%15%20%25%30%35%40%45%50%
INTRO MAJORS INTRO MAJORS
Learn From, Share With, and Use Others' Experiences, Results, and Ideas
Web
No Web
Motivated by new idea from colleague, publication,
or web
Found & adopted activitywholesale from web/
publication
47%
16%
56%62%
33%
8%17% 17%
0%
10%
20%
30%
40%
50%
60%
70%
INTRO MAJORS INTRO MAJORS
Learn From, Share With, and Use Others' Experiences, Results, and Ideas
Workshop
No Workshop
Motivated by new idea from colleague, publication,or web
Inspired by attending workshop/professional development activities
Page 27 of 63
Figure 14: Percentage of faculty members reporting that attending a workshop or other professional
development opportunity motivated them to revise their introductory course by level of Cutting Edge exposure.
Figure 15: Percentage of faculty members reporting that the need to update content motivated them to revise
their introductory course, grouped by amount of Cutting Edge exposure.
13%26%
60%
No Cutting Edge exposure Website only Website and Workshop
Motivated to make a change due to attending a workshop or other
professional development opportunty
68%
52%
34%
No Cutting Edge exposure Website only Website and Workshop
Motivated to make a change due to content needing to be updated
Page 28 of 63
Figure 17. Percent of respondents who consider
what colleagues are doing when revising a course;
compares Cutting Edge workshop participants vs.
non-participants.
31% 31%
24%21%
0%
5%
10%
15%
20%
25%
30%
35%
Intro Majors
Consider What Colleagues are Doing when Revising a Course
Web
35% 34%
25%22%
0%
5%
10%
15%
20%
25%
30%
35%
40%
Intro Majors
Consider What Colleagues are Doing when Revising a Course
Workshop
No Workshop
Increased Use of Online Resources for Teaching
In interviews, Cutting Edge website users described a variety of behaviors that were related to
seeking ideas for teaching including:
browsing through topical modules to find ideas when they were conceiving a class for the
first time,
browsing through activities related to a specific topic,
looking for examples of using a specific teaching method,
looking for new ways to teach, and
looking for data that could be used to teach a particular topic.
Of the 30 website users interviewed, 25 described using the website to find ideas for teaching.
These users described using the website as a resource to learn in depth about a particular
teaching method (e.g. think-pair-share, teaching metacognition), teaching tool (e.g. Google
Earth, rubrics), education or geoscience topics (e.g. early earth, affective domain, or assessment).
Figure 16: Percent of respondents who consider
what colleagues are doing when revising a course;
compares Cutting Edge website users vs. non-users.
Page 29 of 63
Nine of 30 interviewees reported using the site to learn about a new field, to strengthen a content
area of weakness, or to learn more about pedagogy. Comparable findings came from the 2009
Faculty Survey. When asked about their use of on-line teaching resources within the past two
years, 19% of respondents indicated that they learn about the methods they will be using and
45% learn about the content they will be teaching.
The 2009 Faculty Survey shows that Cutting Edge website users and workshop participants were
more likely than non-users and non-participants to have used on-line teaching resources in the
past two years to surf for ideas for teaching, to download materials to use in class, to find
materials for students to use in assignments, and to learn about the methods they will be using
(Fig. 18 and 19). In web user interviews conducted from 2005 to 2009, users reported using the
website to help them to incorporate teaching activities from a given topic, to get information
about course design, to prepare to teach a course for the first time, to prepare to teach a course
ahead of time, and to change or revamp a course.
Figure 18: Uses of online teaching resources by Cutting Edge website users in last two years.
70%85%
64%
31%
52%
77%
51%
17%0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
Ideas for teaching Download materials to use in
class
Find materials for students to use in
assignments
Learn about teaching methods
Uses of Online Teaching Resources in Last Two Years
Web
No Web
Page 30 of 63
Figure 19: Uses of online teaching resources by Cutting Edge participants in last two years.
Responses to the 2008 participant survey confirmed these findings with 61% of respondents
indicating that seeking ideas for teaching was the most valuable aspect of their use of the Cutting
Edge website. When asked about their use of on-line teaching resources within the past two
years, 55% of respondents indicated that they surfed for ideas for their teaching, 78%
downloaded materials to use in class, and 53% found materials for students to use in
assignments. Faculty primarily adapted these ideas for use in their teaching; however, users also
reported using materials wholesale in their courses, particularly if they were teaching out of field.
These descriptions(with the names changed), drawn from the qualitative data (interviews and
open-ended survey questions), exemplify the types of use typically described.
David, an Early Career 07 participant, puts time into preparing his lectures to
emphasize important points, make them more interactive, and keep the
students involved, using techniques such as incorporating science news and
popular media. He went to the Cutting Edge website several times to mine it.
Dot, an Early Career 08 participant, incorporated the use of web-based
practice materials to prepare students for assessments. She administered a
mid-term survey to assess how her contributions and the website influenced
their learning and had them reflect on their effort and performance in class.
She used the Cutting Edge website frequently for lecture content, practice
problems, and ideas for revamping lab activities.
74%85%
64%
30%47%
75%
48%
15%0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
Ideas for teaching Download materials to use in
class
Find materials for students to use in
assignments
Learn about teaching methods
Uses of Online Teaching Resources in Last Two Years
Workshop
No Workshop
Page 31 of 63
The website seemed to be a valuable resource for faculty to explore and learn pedagogy and
educational innovations in depth, which impacted their teaching and supported their ability to
participate in educational innovation in community discussions.
Using Cutting Edge to make comparisons
Evidence that website users were searching for and looking at the contributions made by others
comes from phone interviews in which 12 of 30 interviewees reported using the website to find
out what others were doing or how they were teaching. Users reported a wide variety of
behaviors that can be described as norming: trying to determine how their own behavior
compares to others in the community. Others reported behaviors that included looking at syllabi
and activities to determine if their course covered similar content to those taught by others;
looking at activities to determine how their approach to a specific topic compared to that of
others; looking at activities to determine the level of sophistication or effort that others were
expecting from their students; and looking at examples of teaching methods to determine if
methods were in widespread use or being used to teach the topics that they teach.
These behaviors of checking on what others were doing were also reported in follow-up
interviews specific to workshop Action Plans. The following example is how one workshop
participant (with the name changed) had been using the website in this way.
Sally, an Introductory Geoscience participant, went to the Cutting Edge
website every time she wanted a change to see if someone else had
implemented the change and may have suggestions for a better way to do it.
She used the website to add Think-Pair-Share, Jigsaws, and group experiences
to facilitate student participation.
An interesting finding from the 2009 Faculty Survey was that Cutting Edge website users and
workshop participants were less likely than non-users and non-participants to report that the use
of on-line resources had increased their own knowledge of a particular topic (61% vs. 73% for
website; 60% vs. 71% for workshop). The Cutting Edge programming may be more beneficial
for teaching methods than for geosciences content.
Page 32 of 63
The Cutting Edge Impact on Teaching
On the Cutting Edge aimed to change faculty members‟ teaching practices in order to positively
impacting student learning. Making such changes requires first a change in attitude toward
student-centered approaches to teaching, followed by new knowledge and skills about
appropriate teaching methods. Knowledge and skills in the absence of a student-centered
approach to teaching is unlikely to lead to change (Barr and Tagg, 1995, Pellegrino et al., 2001).
The findings from this evaluation suggest that Cutting Edge programming develops both a
student-centered approach and the knowledge and skills needed to implement appropriate
teaching methods. Participation is associated with changes in classroom teaching that include
course design and goal setting, use of lecture, use of activities, use of tools, and use of student
assessments. Further examination indicates that these changes are grounded in an immediate and
sustained shift in teaching philosophy and confidence that shapes a faculty member‟s approach
to teaching.
Changes in Teaching
Website users, including both those who have participated in Cutting Edge workshops (~45%)
and those who have not (~55%) report impacts to their teaching. In the 2009 Faculty Survey,
Cutting Edge website users were more likely than non-users to report having changed the
teaching methods used in their introductory courses in the past two years. Specifically, they were
more likely to have spent less time lecturing, increased the questioning of students during
lectures, added group work or small group activities, spent more time on class discussions or
small group discussions, changed assessment tools or strategies, and added assignments (Fig.
20). As it did with Cutting Edge website users, the 2009 Faculty Survey found that Cutting Edge
workshop participants were more likely than non-participants to report having made any changes
in the teaching methods used in their introductory courses in the past two years.
Page 33 of 63
Figure 20: Impact of website use on use of various teaching practices in the classroom.
Figure 21: Impact of workshop participation on use of various teaching practices in the classroom.
These data are born out in telephone interviews. 45 of 54 past workshop participants in 2005 and
21 of 30 website users in 2008 and 2009 noted that participation had a major impact on their
teaching. Interviewees described specific changes in teaching techniques and activities in their
70%
37%
47%
32%
25%
35%
25%
54%
21%
32%
15% 15%
22%17%
0%
10%
20%
30%
40%
50%
60%
70%
80%
Made changes in teaching
methods used in intro courses over past two
yrs
Spent less time lecturing
Increased questioning of
students during lecture
Added group work/small
group activities
Spent more time on
class/small group
discussions
Changed assessment
tools/strategies
Added assignments
Impacts of CE Participation on TeachingWeb
No Web
71%
41% 48%30%
38% 37%
57%
25%35%
16% 18% 24%
0%
10%
20%
30%
40%
50%
60%
70%
80%
Made changes to teaching method
in intro course over the past two
yrs
Spent less time lecturing
Increased questioning of
students during lecture
Added group work/small group
activiites
Spent more time on class/small
group discussions
Changed assessment
tools/strategies
Impacts of CE Participation on TeachingWorkshop
No Workshop
Page 34 of 63
courses resulting from their participation in the Cutting Edge program. Similarly, in the 2009
Action Planning Follow-up Survey, 35 out of 43 respondents who participated in an Introductory
Geology or Early Career workshop described specific pedagogical changes that they attributed to
their workshop participation, and 20 made statements about the impact of these changes on
teaching effectiveness. Eleven respondents reported course changes as a result of their workshop
experience. Seventy-one percent of respondents to the 2008 Participant Survey described a
change to their teaching, 85% indicated that as a result of the program they changed their
teaching to incorporate more active learning techniques in their classes, and 87% reported at
least one course was changed as a result of their participation.
As discussed earlier in this report, the use of online resources is an important part of the Cutting
Edge structure for learning. The charts in figures 22 and 23, show that when asked in the 2009
Faculty Survey how the use of on-line resources has positively impacted their teaching within the
past two years, Cutting Edge website users and workshop participants were more likely than
non-users to report that it increased the variety of methods that they use, increased their skill
with a particular teaching method, increased their confidence as a teacher, and increased their
ability to assess student learning.
Figure 22: Impacts of using online teaching resources for web users vs. non-users.
71%
27% 22% 25%
51%
18% 16% 12%0%
10%
20%
30%
40%
50%
60%
70%
80%
Increased variety of methods used
Increased skill with teaching method
Increased confidence as a teacher
Increased ability to assess student learning
Impact on Teaching BehaviorWebNo Web
Page 35 of 63
Figure 23: Impacts of using online teaching resources for workshop participants vs. non-participants.
Areas of particular note are the changes that occurred in course design and goal setting, the use
of lectures, the use of activities, the use of teaching tools, and the use of student assessments.
Changes in course design and goal setting
In the 2008 Participant Survey, 83% of respondents stated that the workshop affected the way
they use goals and objectives to design their course(s). Similarly, in the 2005 Impact Survey, 234
workshop participants (half of whom used the website for course design) were asked to indicate
the degree to which the Cutting Edge program (workshop or website) had impacted the goals and
objectives that they set for their course(s). Sixty-three percent indicated that it caused them to
make changes and an additional 26% indicated that it raised their awareness in the area. For the
56 respondents that checked participation in one of the course design workshops, 51 (92%)
reported that the workshop experience caused them to make changes with the remaining 5
respondents reporting that it raised their awareness.
To get a better picture of the ways in which Cutting Edge impacts goal setting, we can look at the
2009 Faculty Survey results regarding introductory course design. In designing their most recent
introductory course or activities for it, Cutting Edge website users were more likely than non-
users to address student anxiety about specific activities (54% vs. 39%), while Cutting Edge
workshop participants were more likely than non-participants to address student comfort in the
intellectual environment of the course (76% vs. 64%). As reported earlier, Cutting Edge website
users were more likely than non-users to consider the following factors to be of major
importance in setting goals for their most recent introductory course: developing problem solving
skills, developing interpersonal skills, and increasing student awareness of the utility of
geoscience in addressing important problems. And Cutting Edge workshop participants were
72%
30% 28% 26%
55%
19% 16% 14%0%
10%
20%
30%
40%
50%
60%
70%
80%
Increased variety of methods used
Increased skill with teaching method
Increased confidence as a teacher
Increased ability to assess student learning
Impact on Teaching Behavior
Workshop
No Workshop
Page 36 of 63
more likely than non-participants to consider developing problem solving skills and interpersonal
skills to be of major importance.
Changes in classroom activities
Less use of traditional lectures
Findings from the 2009 Faculty Survey reveal that compared to non-users, Cutting Edge website
users were less likely to report using traditional lecture often. When asked about teaching
changes made in their introductory courses in the last two years, website users with no workshop
experience were more likely than faculty with no Cutting Edge experience to report that they
spent less time lecturing (32% vs. 22%) and added group work or small group activities (25% vs.
15%). As shown in Figure 24, these percentages increase further with workshop participation.
Overall, Cutting Edge website users were more likely to report “often” use of:
posing questions that are answered by individual students,
posing questions that are answered simultaneously by the entire class,
small group discussion or think-pair-share, and
in-class exercises (Fig. 25).
While Cutting Edge workshop participants were neither more or less likely than non-participants
to use traditional lectures in their introductory courses, they were more likely to report “often”
use of:
posing questions that are answered simultaneously by the entire class,
small group discussion or think-pair-share, and
in-class exercises (Fig. 26).
This question was also analyzed for possible combination effects of adding workshop
participation to website use. As shown in Figure 27, adding workshop participation to website
use increased the likelihoods of using these techniques often by 12 to 16 percentage points.
Page 37 of 63
Figure 24: Percentage of faculty members reporting changes made in their introductory courses demonstrating
the association with website use and website use with workshop participation.
Figure 25: Impact of website use on use of lecture in the classroom.
15%
22%25%
32%
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
50%
Added group work or small group activties
Spent less time lecturing
Changes made in last two years by Cutting Edge exposure
No Cutting Edge exposure
Website only
Website and Workshop
40%43%
78%71%
59%
35%41%
87%
59%
42%
17% 20%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Traditional lecture Posing questions answered by
individual students
Posing questions answered by entire class
Small group discussion or
think-pair-share
In-class exercises
Cutting Edge Impact on Use of LecturesWebNo Web
Page 38 of 63
Figure 26: Impact of workshop participation on use of lecture in the classroom.
Figure 27: Percentage of faculty members reporting use of teaching strategies in their introductory courses
demonstrating the association with website use and website use with workshop participation.
Increased use of interactive activities
Both web usage statistics and workshop surveys indicate that Cutting Edge influences users and
participants to make changes in their in-classroom activities that increase student involvement.
61%
39%48%44%
19% 23%
0%
10%
20%
30%
40%
50%
60%
70%
Posing questions answered by entire class
Small group discussion or think-pair-share
In-class exercises
Cutting Edge Impact on Use of LecturesWorkshop
No Workshop
42%
17% 19%
53%
28%34%
65%
44%50%
0%
10%
20%
30%
40%
50%
60%
70%
Posing questions answered by entire
class2
Small group discussion or think-
pair-share
In-class exercises
Teaching Strategies Used Weekly or Nearly Every Class by Cutting Edge Exposure
No Cutting Edge exposure
Website only
Website and Workshop
Page 39 of 63
Web statistics reveal that 40% of deep sessions (sessions seeing more than 10 pages) included
viewing activity pages. Over half of the participants attending an Early Career workshop (16 of
27) and almost a third of the Introductory Geology participants (5 of 16) reported in the 2009
Action Planning Follow-up Survey that they had made changes intended to increase student
involvement. Specifically mentioned were the use of demonstrations, group work, ice-breaker
activities, in class assignments, in-class discussions, interactive lectures, lecture tutorials, peer
review exercises, reflective analysis, student presentations, student projects, Think-Pair-Share,
and web-based practice materials.
In their most recent course, Cutting Edge website users and workshop participants (in the 2009
Faculty Survey) were more likely than non-users and non-participants to report using problem
solving activities that addressed a problem of national or global interest (72% vs. 62% for
website users; 75% vs. 63% for workshop participants). Additionally, Cutting Edge website
users were more likely than non-users to report using problem solving activities that students
posed and solved on their own (16% vs. 10%), and in which students developed a geologic
history of a field (45% vs. 33%).
Use of tools to support teaching methods
In addition to providing information and methods for improving teaching, Cutting Edge also
provided tools, such as visualizations, to support website users and workshop participants in
making those teaching changes. In the 2009 Action Planning Follow-up Survey, 11 of 54
respondents reported without prompting that they used tools or resources from the Cutting Edge
website in their teaching such as visualizations, animated demonstration, and clickers. Feedback
from workshop participants suggests that they valued learning about new tools and resources
(particularly being introduced to tools by people who developed the tool or actively use it). In
reviewing end of workshop summaries and road checks for 11 workshops conducted between
2006 and 2009, evidence that participants valued learning about new tools and resources
surfaced in all 11 of these reports. Participants valued the exposure to new tools and resources
and seeing how they could use them in their own teaching.
While finding visualizations was not a major theme in interview and open-ended survey
responses (for example, only six of the 30 website phone interviews), quantitative survey data
and the web use statistics repeatedly show that using the website to find images for use in
teaching was a common type of use. In consecutive pop-up surveys in 2005 and 2006, a little
over a quarter (26% and 28%) of the website users indicated that they came to the site for
visualizations. Visualization collections On the Cutting Edge website, which through March
2009 were identifiable within the Visualization module, were visited in 30% of all the deep
sessions in 2008 and the Visualization module was the most popular single area of the site. Deep
session visits with substantial use of visualization collections show that users were looking
through large numbers of visualizations on a single topic and branching to pages that discussed
Page 40 of 63
the use of those visualizations. On average, users in these sessions saw 7 pages in the
visualization collection and also saw 23 pages from elsewhere on the site.
Changes in assessment
The Cutting Edge exposure may change the way faculty conduct assessments and increase the
breadth of assessment methods that they use. Twenty-five percent of respondents to the 2008
Participant Survey wrote about specific Cutting Edge assessments that they used and discovered
either through workshops or on the website. The 2009 Action Planning Follow-up Survey found
that 16 of 43 respondents who attended an Introductory Geology or Early Career workshop
described making changes to their assessments or rubrics following the workshop.
In 2007 interviews conducted with 25 Cutting Edge workshop participants and 14 non-
participants at the AGU annual meeting, the participants reported a broader range of assessments
that included observing levels of student engagement, conversations with students, course
evaluations, as well as quizzes, reports and exams. The non-participants reported assessments
that only included quizzes, reports, and exams.
Shifting How Faculty Approach Teaching
I got turned on to pedagogy and how it works. There are all these tools and
techniques out there. In all the workshops, it is about engaging people and
being engaged by the material.
– Leadership interview participant
Surveys and interviews found that the Cutting Edge experience, whether via the website or
workshops, was associated with changes that reflected increases in student-centered learning
approaches to teaching. The basis for this shift was most evident in changes in the teaching
philosophy of faculty members, but evidence also indicates that Cutting Edge helped faculty
members to increase their confidence in using new teaching methods.
Changes in teaching philosophy
An indication that Cutting Edge helps to shift the way faculty think about orienting their teaching
comes from the 2009 Faculty Survey questions that asked respondents to reflect on what
motivated them to make a change the last time they made a substantive revision to a course.
Cutting Edge website users were more likely than non-users to report that they were motivated to
make the change for an introductory course because they adopted a new philosophy for their
teaching (31% vs. 18%). In the case of Cutting Edge workshop participants, they were also more
likely than non-participants to report that they were motivated to make the change because they
Page 41 of 63
adopted a new philosophy for their teaching, irrespective of whether it was for an introductory
course (31% vs. 20%) or a majors course (32% vs. 21%).
At the same time, Cutting Edge website users and workshop participants were less likely to
indicate that the need to update the content was the motivation for making a revision whether for
an introductory course (44% vs. 68% for website users; 37% vs. 64% for workshop participants)
or a majors course (46% vs. 71%; 35% vs. 68% for workshop participants).This shift away from
a content focus and attributing a new philosophy for making change is evidence that adopting a
student-centered approach to teaching leads to actual (self-reported) changes in teaching practice.
Suggesting a possible shift in their view of the effectiveness of student-centered teaching
methods, the 2009 Faculty Survey found that Cutting Edge website users and workshop
participants were more likely than non-users and non-participants to indicate that student
learning had been impacted in their most recent introductory course by providing opportunities
for low-stakes practice before high-stakes assignments, activities, or exams; by including
activities that allow students to get to know one another; and by using group projects, field work,
or other activities that promote teamwork and collaborative learning (Figs. 28 and 29).
Figure 28: Affective domain strategies having largest impact on student learning for Cutting Edge website users
vs non-users.
62%54% 56%
66%39% 42%
0%
10%
20%
30%
40%
50%
60%
70%
Practice before exams Activities to get to know each other
Teamwork/collaborative learning
Affective Domain Strategies Having Largest Impact on Student Learning
Web
No Web
Page 42 of 63
Figure 29: Affective domain strategies having largest impact on student learning for Cutting Edge workshop
participants vs non-participants.
Survey results from different sources indicate that the shift toward student-centered learning
experienced by Cutting Edge workshop participants was both immediate and sustained. In a 2004
participant survey in response to an open-ended question asking about the impact of the
workshop on their teaching, 42 out of 95 participants indicated that they thought more critically
about teaching. Of 274 past workshop participants responding to the 2008 Participant Survey,
91% indicated that they shifted their thinking from “What do I teach?” to “What are my students
learning?” Less than 5% reported that the workshop did not shift their thinking in this way. In
2007 face-to-face interviews, Cutting Edge workshop participants offered unsolicited feedback
that the workshops influenced their philosophy to be more student-centered. In 2005, 70% of
past workshop participants (54 in the sample) articulated this change in attitude in telephone
interviews using words such as “eye opener,” “seismic shift,” and “new twist.” At that time, four
percent of the participants were early in their career and did not feel they had a shift in their
attitude because of the research nature of their tenure-track positions.
An example of this shift occurring as a result of a workshop comes from the 2008 Metacognition
workshop embedded assessment, in which the 29 participants were asked before and after the
workshop about the features that they looked for in a strong teaching activity. The number of
participants indicating that they looked for aspects of the importance of students‟ prior
knowledge, beliefs, or self-regulation increased from 3 before the workshop to 25 after the
workshop.
70%59% 56%
48%41% 45%
0%
10%
20%
30%
40%
50%
60%
70%
80%
Practice before exams Activities to get to know each other
Teamwork/collaborative learning
Affective Domain Strategies Having Largest Impact on Student Learning
Workshop
No workshop
Page 43 of 63
Evidence of a sustained shift was found in the action plans developed at workshops, and end-of-
workshop survey write in comments from 2007 and 2008 workshop participants. The contents of
those plans and comments indicated a prevailing change in participants‟ focus from content and
material development to student learning. When a sample of these participants were surveyed
one to two years out from the workshop as to how the changes they anticipated at the workshop
were working, their responses indicated that the Cutting Edge workshop had played an important
role in facilitating the adoption of a new philosophy of student-centered learning.
I used to approach course and lesson development based on delivering
material. Now I feel like I am actively assessing their grasp of concepts and
terms continually, rather than only on exams and graded assignments.
– Action Plan Follow-up Survey Respondent
The workshop had a powerful impact on my thoughts of teaching philosophy,
highlighting the need for different approaches to get students to learn the
material, including group work, and hands-on/inquiry-based approaches to
subject matter that is normally covered in a (dry) lecture-based format.
– Action Plan Follow-up Survey Respondent
Increases in confidence
In the 2009 Faculty Survey, both Cutting Edge website users and Cutting Edge workshop
participants were more likely than non-users and non-participants to report that using online
resources had increased their confidence as a teacher (22% vs. 16% for website users, 28% vs.
16% for workshop participants). In interviews, website users reported that having access to the
resources on the website gave them more confidence in trying new things, more comfort in
teaching topics outside of their sub-discipline, and greater confidence in their own teaching
ability and content. In particular, users noted that knowing the resources on the website had been
used by other colleagues made it a trusted source, such as in the following quote.
I would say that it has built confidence for me, and I guess, mainly, the
confidence to try new things and things that I may not have, you know,
necessarily gone and done. It sort of provides an avenue of, you know, okay,
here is an idea. This is something you can try in your classroom. This has been
done. It has been successful. If it isn’t successful, this is what you can do.
- Website user interviewee
Page 44 of 63
Workshop participants were not asked directly about changes in confidence resulting from their
workshop experience; however, their responses to other questions and their unsolicited
comments would indicate that confidence was enhanced for some of them. In reviewing end of
workshop summaries and road checks for 28 workshops between 2006 and 2009, 21 contained
strong evidence that the participants gained confidence in their ability to teach, their attitude
towards their career, and the material that they use in teaching. The following comments are
characteristic of how participants expressed their increase in confidence.
I've been resisting the move to incorporate more energy in my courses because
of not really knowing the best way to get started. I now have plenty of ideas
and it no longer seems such a daunting task.
- Energy ’09 workshop participant
This workshop has given me the tools I need to really move forward; complete
the transition to a ‘senior’ faculty member; and succeed in becoming the
educator and scientist that I want to be.
- Early Career ‘09 workshop participant
Impact of Cutting Edge Experience on Student Learning
The evaluation for Cutting Edge did not collect data that would directly measure any direct
impacts on student learning. However, anecdotal evidence was revealed in the course of data
collection that would indicate student learning had been impacted by the teaching changes that
faculty members learned from Cutting Edge.
In the 2008 Participant Survey, 61% of respondents indicated that they had seen changes in
content understanding in students as a result of changes they made to their teaching.
Additionally, 45% indicated changes in students‟ skill development, 45% indicated changes in
students‟ attitude toward science, and 47% indicated changes in students‟ motivation to learn.
In 2005 telephone interviews, 33 of 54 workshop participants reported increases in student
participation, student engagement, and student comments for course evaluations that reflected
improved learning. As reported earlier in this report, the 2009 Faculty Survey found that Cutting
Edge website users and workshop participants were more likely than non-users and non-
participants to indicate that strategies that are consistent with a student-centered learning
approach had the largest impact on student learning.
Page 45 of 63
The Cutting Edge Impact on the Geoscience Education Community
To complete the picture of participation in the cycle of educational innovation, the evaluation
looked at the possible impact of Cutting Edge on the broader geoscience education community,
specifically on faculty networking and their contributions which result in new knowledge that
will be accessible to others via the structure for learning.
The findings suggest that just as Cutting Edge exposure is associated with more learning from
others and greater use of the website, it is also associated with an increase in networking and
contributions to the geoscience education knowledge base.
Networking
One of the best parts about this workshop was having so many colleagues to
talk with about teaching. This is what made the experience so much better
than just a compilation of new teaching materials.
- Geomorphology 2008 Participant
Ninety-eight percent of website users and 99 percent of workshop participants reported in the
2009 Faculty Survey that they corresponded with colleagues about teaching over the past two
years. Cutting Edge experience was associated with much higher rates of faculty communication
on a variety of teaching issues. Cutting Edge website users and workshop participants were more
likely than non-users and non-participants to frequently communicate with colleagues about new
ideas in pedagogy, specific assignments or activities, problems encountered in teaching,
classroom management issues, and how to assess student learning (Figs. 30 and 31).
Additionally, website users were more likely to frequently communicate with colleagues about
course design (51% vs. 43%).
Page 46 of 63
Figure 30: Topics of frequent conversation with colleagues for web users and non-users.
Figure 31: Topics of frequent conversation with colleagues for workshop participants and non-participants.
Cutting Edge Workshop participation and networking
Workshop participation appears to have influenced behaviors related to networking with others.
In the 2008 Participant Survey, 87% of Cutting Edge workshop participants reported that they
talked more to their colleagues about teaching after their workshop participation than before.
53%
68% 72%
41%
62%
30%
50%57%
29%
46%
0%
10%
20%
30%
40%
50%
60%
70%
80%
New ideas in pedagogy
Specific assignments or
activities
Problems I encounter in my
teaching
Classroom management issues
How to assess student learning
Topics of Frequent Communication with ColleaguesWebNo Web
53%67%
77%
40%
60%
35%
54% 59%
31%
49%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
New ideas in pedagogy
Specific assignments or
activities
Problems I encounter in my
teaching
Classroom management issues
How to assess student learning
Topics of Frequent Communication with ColleaguesWorkshopNo Workshop
Page 47 of 63
Frequently mentioned in every end-of-workshop summary for 28 workshops between 2006 and
2009, was that one of the most valued aspects of the workshops was the opportunity to
collaborate, network, or to form support groups. In 2009 end of workshop surveys, 91% of
workshop participants indicated that they planned to use the Cutting Edge website to share what
they knew with colleagues and 85% listed specific people whom they planned to contact. For
example, in the 2009 workshop, Teaching About Energy in Geoscience Courses: Current
Research and Pedagogy, 33 of 34 participants said they had formed networks of people that they
would contact in the future and most of these individuals identified members of their network by
name. In 2005, 38 of 54 telephone interviewees indicated that the ability to network with others
was valued and shaped their learning. Twenty percent of those responding to the 2008
Participant Survey indicated in write-in comments that networking (as a means of community
connection) was the most important impact of the program.
Providing an introduction to networking
Some workshops included time for peer review and feedback activities. This provided an
introduction for some, and the opportunity for all, participants to engage in a structured
networking activity around teaching. Evidence that participants valued the activity review and
associated discussion surfaced in 6 of 11 summaries for workshops that offered participant peer-
reviewed activities conducted from 2006 to 2009. The following is an example of a typical
comment on the value of peer review and feedback in the workshops.
The review was a learning experience and confidence builder – I noticed that
others need to make the same adjustments and encounter similar problems
that I do. I will continue to modify and tweak my labs to stimulate critical
thinking and student interest.
- Geomorphology 2008 Participant
Participants valued having their peers critically review their activities and learning from the
group discussions. The activity review and associated discussion at the workshops not only
improved the quality of the activity collection but also helped them to think more critically about
their teaching and what constituted a “good activity.”
Cutting Edge workshops also provide opportunities for geoscientists to interact with experts in
education research and psychology. A review of end of workshop summaries for four emerging
theme workshops specifically related to education research conducted between 2005 and 2009,
found that participants in these workshops on education topics, metacognition, affective domain,
and assessment valued interactions with education researchers and psychologists, as well as
workshop content related to education. As the following comment illustrates, participants valued
Page 48 of 63
the perspectives and insights that educational researchers and psychologists brought to the
discussions on geosciences education.
I am thrilled to find myself in a community of geoscientists interested in
understanding how students learn. The other very positive aspect of this
workshop is the inclusion of educational psychologists and their willingness to
work with our community to increase student success.
- Metacognition 2008 Participant
Expanding the scale of professional networks
In addition to providing networking opportunities and increasing communications about
teaching, Cutting Edge workshops may also influence the extent of professional networks. In
2007 interviews conducted at an AGU meeting, 16 of 24 (67%) Cutting Edge participants
characterized their teaching network on a national scale compared to only 4 of 14 (29%) non
Cutting Edge respondents, and 10 of 24 (42%) Cutting Edge participants characterized their
teaching network as broader than their discipline compared to only 4 of 14 (29%) non Cutting
Edge respondents (Fig. 32).
Figure 32: Extent of professional network for Cutting Edge workshop participants vs. non-participants.
Cutting Edge website use and networking
While the interviews with 30 website users found that they did not seem to think of the Cutting
Edge website as a networking tool, some of them did report that they would look for content or
material from specific individuals and some had made initial contact with colleagues due to
finding them on the website. Additionally, data cited earlier from the 2009 Faculty Survey
67%
42%
29% 29%
0
10
20
30
40
50
60
70
National Broader than discipline
Pe
rce
nt
of
me
mb
ers
su
rve
yed
fo
r p
arti
cip
ants
/no
n-p
arti
cip
ants
Scale of Professional NetworkCE Participants
Non-Participants
Page 49 of 63
demonstrates that Cutting Edge website users were more likely to engage in networking
activities and to use the web for sharing information than were non-users. This sharing and
networking was much more likely to be reported by website users who were also workshop
attendees. Career development was a common use of the website reported by website users.
The Cutting Edge website as a career development networking tool
A sub-group of users reported that they used the website to support their career planning and
networking with other faculty. In phone interviews, 16 of 30 interviewees reported using the
website for professional development or networking purposes. In the 2007-2008 pop-up survey,
22% of survey respondents indicated that career planning was the motivation behind their visit.
These users were geoscience faculty, students, and others. In interviews, faculty described using
the site to identify experts to contact with questions, to share information, and to seek
information for career planning.
Contributing to the geoscience education community
There are several indicators from the 2009 Faculty Survey that point to a relationship between
Cutting Edge and making contributions to the geoscience community. Cutting Edge website
users and workshop participants were more likely than non-users and non-participants to have
presented research on teaching methods or student learning at meetings within the past two
years, published about educational topics within the past two years, shared materials from their
courses in the last two years via talks, and publish materials from their courses in journals. And
when they had done something that was particularly successful in class, Cutting Edge website
users and workshop participants were more likely than non-users and non-participants to tell
colleagues, and publish a paper about their activity (Figs. 33 and 34).
Page 50 of 63
Figure 33: Comparison between web users and non-users for community contributions beyond Cutting Edge.
Figure 34: Comparison between workshop participants and non-participants for community contributions
beyond Cutting Edge.
In interviews conducted in 2005 and 2007, 29 of 79 workshop participants expressed either an
affirmation of their attitude toward teaching or change in attitude toward teaching that motivated
27%
16%
31%23%
71%
8%10%6%
14%6%
54%
4%
0%
10%
20%
30%
40%
50%
60%
70%
80%
Has presented research on
teaching methods or
student learning at meetings
within the past two years
Published about educational
topics 1 or more times within the past two years
Share course materials via
talks
Published in journals
Tell colleagues whom I know
will be interested
Publish a paper about my activity
Community Contributions By Website UsersWeb
No Web
29%
16%
30%22%
70%
9%13%8%
17%9%
57%
4%0%
10%
20%
30%
40%
50%
60%
70%
80%
Has presented research on
teaching methods or
student learning at meetings
within the past two years
Published about educational
topics 1 or more times within the past two years
Share course materials via
talks
Published in journals
Tell colleagues whom I know
will be interested
Publish a paper about my activity
Community Contributions By Workshop ParticipantsWorkshopNo Workshop
Page 51 of 63
them to contribute to the geosciences education community. Their new attitude or newly
invigorated attitude inspired participants and enhanced their ability to join conversations about
teaching and geosciences education on their campus and nationally. Participants reported that
they were motivated or encouraged to present at meetings and at other campuses, to consider
publishing on education, and to pursue grants related to education. While participants reported
that their inherent interest in teaching may have eventually led them down this path, their
participation in the workshops accelerated and motivated this interest.
Evidence of other scholarly output resulting from Cutting Edge workshops comes from a few
sources. Over a quarter of the respondents to the 2008 Participant Survey indicated that they
could attribute presentations (27%) and posters (26%) at regional or national meetings to their
Cutting Edge participation. Slightly less than a quarter of these survey participants attributed
contributions to websites (24%), department presentations (21%), and proposals submitted (22%)
to their participation. In the 2005 Impact Survey, 116 of 230 participants (50%) listed specific
presentations, publications, grants, or grant proposals resulting, either directly or indirectly, from
their participation in the program. An example of how quickly this sharing can occur comes from
the end of workshop survey for the 2007 Affective Domain workshop when three weeks after the
workshop, 9 of 18 participants reported that they had already disseminated what they learned.
Six of them submitted and latter reported that they received grant funding for continued work in
this area.
Conversations started at workshops around topics as diverse as metacognition and teaching
structural geology have continued with new audiences at professional society meetings in
sessions led by workshop participants. From 2003-2007, alumni of Cutting Edge workshops
made over 1100 scholarly contributions (presentations at meetings, journal articles, book
chapters) documenting what these faculty have done in their own settings to support geoscience
education reform (bibliometric survey based on educational topics as defined in Salisbury, 2008).
Forty individuals attribute their successful grant awards to participation in a Cutting Edge
workshop and participants frequently comment on the value of personal connections made
through the workshops.
Sharing on the Website
The 2009 Faculty Survey found that making contributions to the website is still not a highly
prevalent activity among faculty. However, when Cutting Edge website users and workshop
participants had done something that was particularly successful in class, they were much more
likely than non-users and non-participants to report that they added their activity to an online
collection (8% vs. 1% for website users; 11% vs. 2% for workshop participants). While the
percentages are small, the magnitude of the difference compared to non-users and non-
participants is a good sign for the potential to continue to impact this behavior and raise the
norm. An indication of the potential Cutting Edge role for online contributions comes from end
Page 52 of 63
of workshop surveys in 2009, in which 91 percent of participants indicated that they planned to
use the Cutting Edge website to share what they knew with colleagues.
Research and Collaboration
In the 2005 Impact Survey, 116 of 235 participants (50%) listed specific presentations,
publications, grants, or grant proposals resulting, either directly or indirectly, from their
participation in the program. Over a quarter of the respondents to the 2008 Participant Survey
indicated that they could attribute presentations (27%) and posters (26%) at regional or national
meetings to their Cutting Edge participation. Survey respondents also credited their workshop
participation for contributions they made to websites (24%), department presentations (21%),
and proposals they submitted (22%). Forty individuals have attributed successful grant proposals
to participation in a Cutting Edge workshop.
Cutting Edge use and participation is associated with more scholarly activity specifically focused
on pedagogical research. Fifty-nine of the 274 respondents (22%) to the 2008 Participant Survey
attributed proposals that they submitted to their participation in a Cutting Edge workshop and 40
(15%) indicated that they were awarded grants that they also attributed to their workshop
participation. As an outgrowth of the 2007 Affective Domain workshop, 11 of the participants
proposed and received funding for the collaborative research grant GARNET (Geoscience
Affective Research Network) which examined the connection between instruction, student affect,
and geoscience learning outcomes From the 2004 Teaching with Visualizations workshop, 4 of
the participants formed a collaboration with the Spatial Intelligence Learning Center bringing
geoscience education into one of their funded working groups.
Interestingly, the 2009 Faculty Survey found that Cutting Edge website users and workshop
participants were less likely than non-users and non-participants to have published any articles
about their research within the last two years (80% vs. 88% for website; 77% vs. 85% for
workshops). However, they were more likely than non-users and non-participants to have
presented research on teaching methods or student learning at meetings within the past two years
(27% vs. 10% for website; 29% vs. 13% for workshops); and they were at least twice as likely to
have published about educational topics within the past two years (16% vs. 6% for website; 16%
vs. 8% for workshops) (Figs. 35 and 36). Additionally, when they had done something
particularly successful in class, they were more likely to tell colleagues (71% vs. 54% for
website; 70% vs. 57% for workshops), many times more likely to add their activity to an online
collection (8% vs. 1% for website; 11% vs. 2% for workshops), and twice as likely to publish a
paper about the activity (8% vs. 4% for website; 9% vs. 4% for workshops). So while Cutting
Edge may not impact the publication of research in general, it does appear to be associated with
the publication of research specific to education and teaching. While the research article
publication percentages are still fairly high for Cutting Edge website users and workshop
participants, the finding that they are lower than non-Cutting Edge faculty seems counterintuitive
Page 53 of 63
and it does not seem plausible (nor desirable) that Cutting Edge would dampen the publication of
research in general. Thus, other factors or characteristics of website users and workshop
participants that might affect publication behaviors should be explored in subsequent
evaluations. It is worth noting that the relative percentage increase over non-Cutting Edge faculty
in their likelihood to specifically disseminate their classroom successes is much greater than the
decrease in their publications overall.
Figure 35: Comparison between web users and non-users for community contributions beyond Cutting Edge.
27%
16%
31%23%
71%
8%10%6%
14%6%
54%
4%
0%
10%
20%
30%
40%
50%
60%
70%
80%
Has presented research on
teaching methods or
student learning at meetings
within the past two years
Published about educational
topics 1 or more times within the past two years
Share course materials via
talks
Published in journals
Tell colleagues whom I know
will be interested
Publish a paper about my activity
Community Contributions By Website UsersWeb
No Web
Page 54 of 63
Figure 36: Comparison between workshop participants and non-participants for community contributions
beyond Cutting Edge.
Result of the Cutting Edge Impact on the Geoscience Education
Community
Impact at the local and departmental level
In telephone interviews, 42 of 54 2005 workshop participants highlighted outcomes from the
workshop characterized by developing outreach programs for local community education, pre-
service in-service science teacher programs, leading regional workshops, or publishing in the
area of geoscience education. Fifteen of those participants described specific department-wide
results from their participation in the program such as curriculum overhaul for majors, teacher
preparation, and better cross-disciplinary and cross-institutional alignment. Thirty-seven others
found their departments supportive of the changes but that changes were not necessarily adopted
across the department. Two noted their department was hostile and not supportive. There appears
to be a clear dose response of reported changes in the department to the number of faculty
participating in workshops. The 2008 Participant Survey found that participants from
departments in which at least 3 department members had attended a workshop were 3 times more
likely to report that the workshop had changed their department‟s collective approach to teaching
compared to participants coming from departments in which 2 or fewer members had attended a
workshop (48% to 16%).
29%
16%
30%22%
70%
9%
13% 8% 17% 9%
57%
4%0%
10%
20%
30%
40%
50%
60%
70%
80%
Has presented research on
teaching methods or
student learning at meetings
within the past two years
Published about educational
topics 1 or more times within the past two years
Share course materials via
talks
Published in journals
Tell colleagues whom I know
will be interested
Publish a paper about my activity
Community Contributions By Workshop ParticipantsWorkshopNo Workshop
Page 55 of 63
It was unclear from the interviews with website users whether use of the Cutting Edge website
had impacted community and department norms. However, reports from website users that
Cutting Edge content was integrated into on-campus department in-services and faculty meetings
were positive signs that the Cutting Edge reach was being extended.
The broader impact
Cutting Edge provided a venue for collegial exchange and communication that included the
sharing of teaching experiences and learning from others teaching experiences. The contributions
made by Cutting Edge website users and workshop participants extended beyond the platforms
provided by Cutting Edge to bring new knowledge into the community, nurture research
activities, and expand the networks used for professional exchange regarding teaching practices
specifically and pedagogical issues in general. The result has been a shift the geosciences
education community in which:
Education is now an accepted topic for discussion and work among geoscience
faculty from all types of institutions;
Research networks have been developed to support education research;
A national perspective on geoscience teaching has been created and is being used
to benchmark teaching norms and behaviors at the institutional and individual
levels;
Geoscience faculty members are connecting with new colleagues from different
institutions and different disciplines;
There is a culture of sharing experiences, successes, and new ideas in education
and teaching practice; and
The sphere of influence and leadership roles for education specialists in
geoscience have expanded.
Page 56 of 63
Appendix: Evaluation Methodology
We used a tiered evaluation framework that combined mixed-method approaches (Fig. A). We
used small samples of in-depth interviews related to both the workshop and website to inform
larger surveys of all participants as they exit workshops and periodically through the web. The
results of these interviews and surveys also informed the design of a larger survey which we
administered in 2004 and administered again in spring 2009 to the larger geoscience community.
Figure A. Tiered Evaluation Framework
Separate summary reports of the interviews and surveys conducted and results can be viewed at:
http://serc.carleton.edu/NAGTWorkshops/evaluation.html
Large-scale survey of geoscience community
A Faculty survey was conducted by Statistical Research Center of the American Institute of
Physics as a baseline in 2004 and administered again in 2009. The survey focused on what
teaching methods were being used by geoscience faculty, how they learned about new content
and teaching methodology, and how the shared what they knew with their colleagues. The
2009 survey was sent electronically to 5107 geoscience faculty and received 2537 completed
responses (50% response rate). Analysis of survey results as the data pertained to the On the
Cutting Edge program focused on three populations: workshop participants who use the website
(as 87% of workshop participants who responded to the survey also used the Cutting Edge
website), Cutting Edge website users who have not attended a workshop, and survey respondents
who had neither attended a Cutting Edge workshop nor used the Cutting Edge website. The
number of respondents who had attended a Cutting Edge workshop and had not used the website
was too small (44 of 2537 completed responses) to analyze.
Page 57 of 63
Surveys of all Cutting Edge workshop participants and website users
Workshops participants
Three strategies were used to collect a consistent set of evaluation data for all workshops: daily
road checks, end-of-workshop surveys, and evaluator observations. In addition for some
workshops, action plans and embedded assessments were completed by all participants.
Road Checks – administered at the close of each workshop day. The purpose was to obtain a
quick scan of the participant reactions to the workshop and provide real-time formative
evaluation for use in modifying the program. In addition to the basic demographic information
the road check asked participants what aspects of the workshop seem to be working and not
working for them; what needed to be improved; and, their general satisfaction with the workshop
at that time.
Road check surveys were analyzed immediately and results were provided to the workshop
leaders so that when necessary, mid-course corrections were made in the workshop. Brief
feedback was provided to participants the following day so that they could see the impact of their
comments. Road check surveys were also used to gather information about what was learned in
that day to better identify the causal linkages between the workshop and its impact on
participants.
End-of-Workshop surveys – administered at the end of each workshop. The purpose was to
provide leaders with insights into not only what worked and did not work with the workshop that
would guide improvements for future workshops, but also indications of what might be added to
the workshop agenda. The surveys also helped identify immediate impacts – changes in
knowledge, skills, and attitude as well as anticipated behavioral application that could be
attributed to the workshop.
In addition to demographic information, participants rated the degree to which expected
outcomes (workshop objectives) were achieved for them and the usefulness of key strategies or
activities that were used in the workshop. Participants also responded to open-ended questions
about what aspects worked well/did not work well, what topics might be added/dropped, how
they changed, what they planned to apply, and what they planned to share after the workshop.
Information was also collected about the participant‟s use of the website to prepare for the
workshop as well as any other use. Last, participants gave over-all ratings for the given
workshop.
End-of-workshop surveys were analyzed and along with the road check data, observations, and
interviews summarized for workshop leaders. Quantitative data was averaged by demographic
and write-in comments were recorded and reported in aggregate. The qualitative write-in
Page 58 of 63
comments were coded by workshop type and used to better understand what participants learned
and how they anticipated using this new knowledge, skills, or attitude.
Observations and interviews –conducted throughout the workshops by the external evaluator or
others serving in the evaluator role, but also included the observations and input from workshop
leaders and/or facilitators. The purpose of these observations or interviews was to better gauge
how well the workshop was meeting the expectations of participants as well as to understand
which aspects of the workshop catalyzed learning outcomes and take note of any emerging
outcomes.
Observations and interviews were discussed with leaders at regular intervals throughout the
given workshop to provide real-time assessment in addition to the written road checks. These
observations were focused on particular aspects identified by leaders as well as captured
unanticipated events/outcomes. These observations and interviews were summarized with the
end-of-workshop data in the end-of-workshop summary report.
Action plans - completed by participants at selected workshops in 2007-2009. The purpose of
these action plans was to help participants articulate how they would put what they learned into
practice and identify any obstacles for change. An email survey was conducted with a sample of
participants one or two years after the workshop from four of these workshops: Teaching
Introductory Geoscience (2008), Teaching with New Geoscience Tools (2008), Early Career
(2007) and (2008). The purpose of this survey was to solicit feedback on teaching changes that
workshop participants had made since their workshop, their use of the Cutting Edge website in
making those changes, and the role that action planning played in helping to facilitate those
changes.
Embedded assessments – a range of assessments was conducted as pilot evaluation strategies at
selected workshops. The purpose of these embedded assessments was to attempt to move
beyond the end of workshop surveys as self reported measures. As part of the Early Career
workshop, participants generated posters that describe plans for both teaching and research. In
addition to the posters participants identified what aspects of the plans they attributed to their
workshop participation and the value of having them reviewed by colleagues. As part of the
Teaching X workshops, participants submitted teaching activities and then participated in an
activity review session using a specific rubric. As part of the evaluation they reported on the
value of this review process in terms of their own activity as well as how they think about
teaching. In selected workshops participants were asked prior to the workshop and again after
the workshop what features they look for in a strong teaching activity. These responses were
qualitative coded for emerging learning themes.
Page 59 of 63
On-line participant surveys administered in 2004, 2005, and 2008. The purpose of the on-line
survey was to characterize how the different types of Cutting Edge workshops impacted
participants‟ professional development, teaching practice, and courses as well as their
observations of impacts on student learning. The survey also aimed to illuminate other ways that
the program impacted faculty such as leadership opportunities on campus, regionally, and
nationally and means of disseminating their knowledge (attributed to program). Of the roughly
1000 potential participants, 274 participants chose to respond (approximately 27% return rate).
The array of workshops covered and the range of answers to questions provides a more
representative picture than the response rate would suggest.
The 2008 On-line Survey collected responses from participants by workshop. The questions
covered the participants‟ thoughts on the impact of the workshop on their professional
development, philosophy and practice of teaching, how they had shared the knowledge, skills, or
attitudes acquired from the workshop with colleagues, and how they had used the website. It
also asked them to share the most valuable impact related to their workshop experience.
Website users
Web statistics reports produced monthly to better understand the number of users of the website
and the resources viewed.
Pop-up surveys conducted on the serc.carleton.edu website in 2006 and again in 2007 to better
identify the demographics of the website users. The pop-up survey was administered to all users
website who visited 4 or more pages. The survey identified the types of users (student, K-12
teacher, and faculty) as well as for what the website was being used (images, data, teaching
activities). Of the 2105 responses, 657 responded from a Cutting Edge website page. This
sample was 1% of the estimated users of the Cutting Edge website for the period the survey was
active.
Awareness poll conducted from November 2007 to January 2008. The email survey was
administered to a randomly generated sample of 1054 geoscientists. The survey had a 44%
return from email. To reduce the opportunity for bias with non-respondents, the survey was also
administered by telephone to 6% of the non-respondents. The telephone survey found an even
higher percentage of awareness to the On the Cutting Edge program. The confidence level is
95% with a 4.4% margin of error.
In-depth small samples of interviews
Community Interviews conducted by Ellen Iverson and John McLaughlin at the December 2007
meeting of the American Geophysical Union. The 39 participants were a mix of Cutting Edge
participants (25) and geosciences faculty who had never attended a Cutting Edge workshop (14).
The purpose of the interviews was to better understand the nature of the cycle of educational
Page 60 of 63
innovation in geosciences and the specific roles that Cutting Edge activities and resources play in
supporting, strengthening and broadening the cycle of education innovation for individuals and
the geoscience community. In addition, we wanted to start to understand the structure of
subpopulations within the geosciences community with respect to these questions. We were
particularly interested in understanding the faculty who defined/aligned their professional life
and network to the following subpopulations: 1) Science research 2) Education research 3)
Community of geoscience educators (practice of teaching) 4) Isolated faculty.
Workshop Participant Telephone interviews conducted from May to September 2005 by Ellen
Iverson and John McLaughlin. The purpose of the phone interviews was to determine the suite
of important impacts for the various types of workshops. In particular the object was to explore
impacts of new knowledge, networks, and leadership skills on teaching practice, professional
planning, and leadership in community. The 54 telephone interviews were from past (2002-
2004) workshop participants.
Leadership Interviews conducted by John McLaughlin via telephone in March-April of 2009.
The nine participants included past conveners of Cutting Edge workshops, contributors to the
Cutting Edge website, and participants in a co-hort of leaders for regional workshops. The
purpose of the interviews was to understand the role the program is playing in developing
leadership for the geoscience education community.
Geoscience Leader Interviews conducted by John McLaughlin via telephone in July-August
2009. In the spring of 2009, the Cutting Edge Principal Investigators identified six recognized
leaders in the geoscience field who were knowledgeable of the CE but were not closely affiliated
with implementation. John McLaughlin, the external evaluator conducted the interviews; one
person was not available for an interview. The purpose of the interviews was to collect
information related to: the perceptions of the program by leaders in the field, program impact on
geoscience teachers and the broader geoscience community, the long-term need for CE services,
and suggestions for addressing changing needs in the geoscience field.
Web User Telephone Interviews conducted by Linda Goozen and Randy Kirkendall from
September 2008 to April 2009 and by Ellen Iverson in November 2005. The goal of these
phone interviews was to define the types of uses of the Cutting Edge website and its impact on
teaching.
Reporting of Statistical Findings
The responses to the 2009 Faculty Survey were analyzed for differences between Cutting Edge
website users and non-users and between Cutting Edge workshop participants and non-
participants using simple 2x2 chi square tests of association. The results from those analyses
were included in this report if the p-value using Pearson‟s method was less than .01.
Page 61 of 63
For the purposes of exploring possible additive effects due to adding workshop participation to
website use, questions that asked about specific teaching behaviors that faculty members
reported using in their introductory and majors courses in which the 2x2 chi square comparisons
for both website use vs. teaching behavior and workshop participation vs. teaching behavior
were statistically significant (at p<.01) were selected for further analysis. The results of these 2x2
chi squares are reported throughout this report. The tests for website associations were
irrespective of workshop participation and the tests for workshop participation were irrespective
of website use.
In order to determine if there was a combined effect for adding workshop participation to
website use, three more chi square tests were run for each selected question.
Website use only (no workshop) vs. neither website use or workshop participation.
Website use only vs. combined website use and workshop participation.
Combined website use and workshop participation vs. neither website use or workshop
participation.
For reported additive effects in this report, all three of the above comparisons were significant at
the level of p<.05.
Limitations of this Evaluation
Workshop participation and website use are highly correlated with each other. Among the
respondents to the 2009 Faculty, Survey Cutting Edge workshop participants were much more
likely than non-participants to have used the Cutting Edge website at all (87% vs. 20%) and
seven times more likely to have used it on at least a monthly basis (43% vs. 6%). And Cutting
Edge website users were much more likely to have participated in a Cutting Edge workshop
(44% vs. 3%). From a programmatic perspective, this is good news because the workshops
emphasize the use of the website as a tool for continuing the learning and making the teaching
changes presented. However, it presents challenges in trying to determine the separate effects of
workshop participation and website use. In fact, the effect of workshop participation only could
not be determined because of the very few number of workshop participants who did not use the
website (44 out of 2537 responses). Thus, any reported effects of the workshops may inherently
include some undetermined level of website effect.
The small number of workshop participants who did not use the website also meant that
interaction effects could not be sufficiently tested and the only combination effect that could be
tested was of workshop participation in addition to website use.
In addition to the 2009 Faculty Survey results that demonstrated a significant difference between
faculty with Cutting Edge exposure and those without it, there were a number of findings from
Page 62 of 63
the survey in which a statistical difference did not exist. Thus, there are a number of attitudinal
and behavior measures in which Cutting Edge may not have had any impact.
For the evaluation results that rely on small sample sizes or low response rates, it is possible that
the reported effects simply reflect that workshop participants and website users who experienced
the measured effects (i.e., shifts in attitude, changes in teaching practice) were more likely to
respond to the questions and that those who did not experience the effects. This may be true,
however, it is reassuring that the results were consistent across the different evaluation findings
using different tools. Additionally, even if all non-responders were not affected by Cutting Edge,
the magnitude of the measured effect would still be large. For example, the 2008 Participant
Survey which found a greater than 90% shift in attitude had a response rate of 27%. So it is
possible that participants who experienced the shift were more likely to respond than participants
who did not experience the shift. And yet, even if all non-responders did not experience this shift
(an unlikely scenario), there were still 25% of the total 1000 potential responders who reported
that they did experience a shift in attitude.
While faculty may report using educational innovations, their behavior in the classroom may be
different or not in line with the intent of the teaching method. This evaluation did not directly
measure any changes that might have occurred in the classroom. All reported effects on teaching
behavior were based on self-reported data.
Similar to the limitation with respect to demonstrating any effects on teaching behavior, the data
cannot demonstrate that student learning was impacted. While student learning is the ultimate
goal, it was not a direct goal for this project and was not measured for the evaluation. All the
same, the teaching innovations developed, disseminated, promoted, and taught by Cutting Edge
were based on best practices and research that indicated that when the innovations are adopted
and implemented, student learning will be positively impacted.
Page 63 of 63
References Cited
American Geological Institute, 2009, Status of the Geoscience Workforce 2009:
http://www.agiweb.org/workforce/reports.html
Barr, R. B. and Tagg, J., 1995, From Teaching to Learning - a New Paradigm for Undergraduate
Education: Change, v. 27(6), p. 12.
Pellegrino, J. W., Chudowsky, N. and Glaser, R., 2001, Knowing What Students Know: The
Science and Design of Educational Assessment: National Research Council (U.S.). Committee
on the Foundations of Assessment.