TRAILER
Kim Arcand
LOGIC MODEL
Sketching out
how the program
will work and
devising a plan to
measure the
outcomes
(integrative
framework for
analysis)
EVALUATION METHODOLOGIES
FOCUS GROUP,
WORKSHOPS
Organized
discussion with a
small group to
understand the
how’s & why’s.
Requires
mediation, gives
social response
and perspective.
INTERVIEWS,
QUESTIONNAIRES &
OBSERVATIONS
Asking what visitors
think or watching how
they interact: building
your baseline of
qualitative and
quantitative data.
CREATIVE OR
PARTICIPATORY
Capturing open,
qualitative info by
participants drawing,
photographing or
writing responses.
Involving
participants in
creating the eval.
framework
Institution or Program
Perspective
Participant Perspective
TRAILER
Kim Arcand
BASIC LOGIC MODEL
Adapted from University of Idaho: http://www.uiweb.uidaho.edu/extension/LogicModel.pdf
INPUTS OUTPUTS OUTCOMES
What is invested•Time•Money•Partners•Equipment•Facilities
What is done•Exhibits•Workshops•Publications•Etc.
Short-termChange in:•Knowledge•Skills•Attitude•Awareness•Motivation
Long-termChange in:•Environment•Social conditions•Economic cond.•Political cond.•Etc.
Evaluation Study: Measurement of process indicators – Measurement of outcome indicators
AUDIENCES
Who•Internal•External
(Formative, summative and anything in between)
TRAILER
Kim Arcand
SPECIFIC LOGIC MODEL
• Plan programs
• Examine feasibility of implementation & potential success
• Create tools to communicate within & outside
• Facilitate evaluation questions
TRAILER
Kim Arcand
From Earth to the Universe (FETTU) evaluation strategy
FINDING THE RIGHT QUESTIONS…AFTERWARDS?
- Based on NSF ISE evaluation rec., "Framework for Evaluating Impacts of Informal
Science Education Projects" (2008)
- Used observation, survey and interview protocols for viewing audiences, as well as staff
of host-partner institutions and local community partners
- Found out good stuff…but had more questions than answers
http://www.fromearthtotheuniverse.org/
TRAILER
Kim Arcand
From Earth to the Solar System (FETTSS), new exhibit in 2011-2013
SECOND TIME AROUND
- Testing findings from FETTU
- ~80 locations worldwide (From cafes in New Zealand to malls in Canada)
- Audience targeted is the non-science expert. Offer multiple access points. Family-
friendly language/set up; free, public, everyday settings; Spanish translations;
Braille/tactile materials.
http://fettss.arc.nasa.gov/
TRAILER
Kim Arcand
2ND CHANCES: FETTSS EVALUATION STRATEGY
Observer: Date: FETTSS Venue: Time of Day:
Composition of Visitor Group:________________ Key: M = adult male F = adult female
G = teen girl B = teen boy g = girl b = boy
Total Visit Time:
Notes: List any site specific notes (eg., the most visited images were
by the restroom or the entry way, etc.) or general observations here:
Behaviors Observed Read Label
Read Aloud Point at Image feature Talk (about exhibit) Show/Explain to others
Ask Questions Other (describe) Visitor Comments Overheard:
Observation sheet:
TRAILER
Kim Arcand
2ND CHANCES: FETTSS EVALUATION STRATEGY
Survey sheet: (exploring casual vs. intentional; science “identity” & expertise level, demographics, etc.)
TRAILER
Kim Arcand
EVALUATING ON A BUDGET: POSTCARDS
What to watch out forErrors: sample size (n=10+), random selection, etcOver-generalizing, SubjectivityEvaluation does not = Research
TRAILER
Kim Arcand
FOCUS GROUPS
Aesthetics & Astronomy is a research project that
studies the perception of multi-wavelength astronomical
imagery and the effects of the scientific and artistic choices
in processing astronomical data.
Runs series of 2-3 focus groups for the qualitative data to
accompany an online survey and in-person interviews for
quantitative data.
Research questions include:
• How much do variations in presentation of color, explanation, and
scale affect comprehension of astronomical images?
• What are the differences between various populations (experts,
novices, students) in terms of what they learn from the images?
• What misconceptions do the non-experts have about astronomy
and the images they are exposed to?
• Does presentation have an effect on the participant?
http://astroart.cfa.harvard.edu
G292.0+1.820,000 light-years
Where do the calcium in our bones and the oxygen in our lungs come from? From supernovae, the exploding stars that create beautiful “supernova remnants” such as this one. This X-ray image of the supernova remnant G292.0+1.8 shows a rapidly expanding shell of gas containing elements such as oxygen, neon, magnesium silicon and calcium that were created both during the lifetime of the star and in the explosion itself. Explosions like this dispersed elements that were necessary to form our Sun and Solar System.
G292.0+1.820,000 light-years
The oxygen we breathe, the iron in our blood, and the calcium in our bones comes in part from exploded stars such as the Cassiopeia A supernova remnant seen here. Most of the elements throughout the Universe, other than hydrogen and helium, were forged in the cores of stars. These elements were then dispersed into space when the stars exploded, later to be used as the building blocks when new stars and planets formed. Our Sun and Solar System—including Earth—contain the materials for life thanks to such previous generations of stars.
G292.0+1.820,000 light-years
G292.0+1.8 is a young supernova remnant located in our galaxy. This deep Chandra image shows a spectacularly detailed, rapidly expanding shell of gas that is 36 light years across and contains large amounts of oxygen, neon, magnesium, silicon and sulfur. Astronomers believe that this supernova remnant, one of only three in the Milky Way known to be rich in oxygen, was formed by the collapse and explosion of a massive star. Supernovas are of great interest because they are a primary source of the heavy elements believed to be necessary to form planets and life.
TRAILER
Kim Arcand
What questions do you want to ask?
What are the outputs?
Who are your audiences?
What outcomes do you want to measure (short term, long term)?
What outcomes are useful to your funding organization(s)?
EVALUATING: CLASS ARCHIVES PROJECT
INPUTS OUTPUTS OUTCOMESAUDIENCES