How to Assess Student Learning in Arts Partnerships

Part I: Focus Groups

Revised April 2, 2012

Mary Campbell-Zopf, Ohio Arts CouncilMary.Campbell-Zopf@oac.state.oh.us

Craig Dreeszen Ph.D., Dreeszen & Associatescraig@dreeszen.com

Mary Campbell-Zopf is deputy director at the Ohio Arts Council and has been with the agency since 1989. Between 1989 and 2011, Ms. Campbell-Zopf helped secure $12 million for state-level arts, arts education and international programming. She also played a central role in an agency-wide effort to expand the OAC’s International Program through a $1.2 million U.S. Department of Education grant, for which she served as the evaluation manager and participated in professional

development activities with educators and arts administrators in Chile. In 2006, a major milestone in international work was the publishing of The Appreciative Journey: A Guide to Developing International Cultural Exchanges. Campbell-Zopf oversaw the guide’s development and writing and was a major contributor to its content. Ms. Campbell-Zopf has an enduring interest in strategic planning and program evaluation, which led to the publishing of the seven-volume series, Focusing the Light: the Art and Practice of Planning in 2009. Ms. Campbell-Zopf has also been active at the state and national levels in arts education, including serving on numerous state advisory committees for content standards, curriculum, learner assessment, as well as on national advisory committees for the U.S. Department of Education, the National Endowment for the Arts, the National Assembly of State Arts Agencies and the Arts Education Partnership.

Craig Dreeszen directs Dreeszen & Associates, a consulting firm in Northampton Massachusetts. He is a planner, educator, program evaluator, and organizational development consultant. For twelve years he directed the Arts Extension Service at the University of Massachusetts. Since 1986 he has evaluated 30 programs for foundations and state arts agencies and provided strategic

planning support to over 50 public and nonprofit organizations, mostly community based cultural organizations and the agencies that fund and support them.

Dreeszen is a contributing editor of Fundamentals of Arts Management, 5th Edition, author of the chapters, “Program Evaluation,” “Strategic Planning,” and “Board Development.” He wrote the online courses “Strategic Planning” and “Outcome Based Program Evaluation,” the “Learning Partnerships Planning and Evaluation Workbook” and other arts management books and articles. Dreeszen earned his Ph.D. in regional planning and M.A. in organizational development from the University of Massachusetts Amherst.

Your Instructors

Our intended learning outcomes

Understand key concepts of student learning assessment

Discern when interviews and focus groups make sense

Design effective focus groups

Learn to conduct focus groups

Learning Assessment• Assessment of student learning -- a

special kind of outcome-based program evaluation

• Same principles and similar methods as other program evaluations

• Intended outcomes in arts education are student learning

• May also be increased teacher and school capacity to teach arts education

Assessment is a Creative Process

The art-making process employs mental processes that are critical to thoughtful assessment.

Stages of creative process:•Exploring•Decision-making•Art-making—the artistic work takes shape•Refining/rehearsing•Presenting•Cycle begins again

Ohio Arts CouncilOhio Arts Council 55

Three Stages of Assessment

• Define what “good” is – Planning and formative

assessment

• Make the work good– Primarily formative assessment

• Describe how well it turned out– Summative assessment

Ohio Arts CouncilOhio Arts Council 66

Learning Outcomes

Learning Outcomes are the knowledge and skills that you expect participants to learn as a result of participating in an educational program or activity.

Other key terms:• Standards• Performance indicators• Assessment-embedded instruction

Ohio Arts CouncilOhio Arts Council 77

Assessment Process

1. Define intended outcomes2. Articulate assessment questions3. Determine data sources and collection

methods4. Select sample or population to query5. Collect data6. Analyze data7. Report findings

Data Sources

• Existing records• Observation (esp. with rubrics)

• Portfolios• Pre- and post-tests• Interviews

• Focus groups• Surveys

Focus Groups are Qualitative Research

• Focused, moderated, & interactive discussion

• Usually 8-10 people, representative of a specific constituency

• Discussing opinions & attitudes in response to specific questions

• 90 minutes to 2 hours

Interviews -- like simple focus groupsSet up interviews

•Develop interview guide of questions•Determine sample and schedule•Invite to interview

Conduct interview

•Welcome and explain objective of interview•Promise anonymity (and keep that trust!)•Ask permission to take notes or record •Ask questions

– Start simple– Move toward more subtle questions

•Listen well•Probe to clarify answers or elicit details•Take notes•Thank interviewee

Summarize notes after interview

Effective Uses of Focus Groups

• Access constituents & communities• Assess needs• Explore “whys” behind divergent opinions

• Clarify an issue for subsequent research• Clarify questions and identify opinions to explore with surveys

• Generate ideas and potential solutions• Learn unexpected information • Demonstrate respect by asking for advice

Focus Group Advantages• Relatively simple and inexpensive

• Quick results

• Enables access to specific constituents & communities

• Believable and usually understandable results

• Demonstrates willingness to hear constituents

Criticisms and Risks• Soft data• Moderator influences discussion

• Participants may not be candid• Dominant personalities may influence others

• Group think is possible• Biased interpretation of results possible

• Risky to generalize from non-representative sample to larger population

Focus Group Steps• Define learning outcomes• Confirm focus group is

appropriate method

• Develop questions• Set location, date & time• Recruit moderator

• Identify and recruit participants• Offer incentive (food is good)

• Conduct focus groups• Take notes or record

• Analyze and report findings

Writing Questions• Know your intended learning

outcomes

• What are participants likely to know?

• Draft short list of short, open-ended questions

• Shorten list (4 -5 not 10-12)

• Sequence questions

• Write probing questions

• Test and revise

…said Alice, "would you tell me, please, which way I ought to go from here?"

Practice Writing Questions

• Identify an arts education program you’d like to assess.

• What information do you need? From whom?

• Write 2-3 group questions for a focus group

• Try out your questions with colleagues

Leave Some Questions to Policy Makers

•Focus group participants can only speak of what they know.

•Collect information about their experiences.

•But do not ask policy questions:

– How shall we allocate resources among competing interests?

– How shall we solve a complex problem?

– What should be our plan?

A messy planning problemA messy planning problem

Identify Potential Participants

•Determine what sector should participate

– Invitees share common characteristics (all 8th grade students or all teachers)

– Define target population

•Select individuals representative of sector

Recruit Participants

• Letter or email– Explain purpose– Use of information– Importance– Present questions– Offer incentive?

• Phone follow up• Email reminder

Sampling Populations• Select representative sample of

larger population– Random (random number generator)– Regular (every nth person)– Deliberate (pick representative sample)

• Sample should look like population

• Consider statistical significance and margin of error

• Or question entire small population (the whole class)

Select Location

Recruit Moderator• Skilled professionals make a difference

– Corporate market research staff (ask your board)– Marketing or social science faculty– Planning, evaluation, or marketing consultants

• Moderating yourself may work– Testing a program design or policy change– Getting feedback on a draft plan– Asking about need– However, grantees will temper comments to their funder

• Amateur moderators may skew results– It is easy to hear only what you expect or hope for– Discussions need skillful, light-handed facilitation

Moderating Focus Groups• Share refreshments & mingle• Open with purpose & process• Participants introduce themselves• Summarize questions

• Ask each question– Probe for details and explanations– Encourage discussion– Summarize

• Be flexible & responsive• Draw out quiet people & reign in

digressions• Thank people at end

Recording Options

• Direct observation through one-way mirror or participation

• Audio or video taping– Transcribe into text

• Take notes– One or two note takers

• Write up notes

Content Analysis• Read notes or transcript• Look for and define patterns

• Assign codes to patterns• Mark text with codes

• Count instances of coded patterns

• Select typical quotes

• Write summary– Most frequent responses– Typical quotes

Parents’ perception of most important educational benefits for child

Most frequently cited benefits (listed in order of most frequent mention, with numbers indicating relative frequency of each idea)

Access to excellent teachers (cited by 90)“Our kids have loved it. Beth has been instrumental in fostering and interest in music for them.”

Learning good foundation of music skills (50)“…is able to play the instrument and does it with pleasure.”

Community of people who love music (39)“…interacting with a group of peers quite different than the peers she engages with at her middles school.”

Report Results• Type or dictate notes or

transcribe tapes

• Do content analysis

• Summarize

• Draw conclusions

• Report findings

Focus Group Report

Seventeen parents and adult students participated in two 90-minute, focus groups.

Summary Results:

• There is much respect and appreciation for high quality teachers.

• Most feel and value an authentic sense of community.

• Ensembles are an important part of the learning and community.

• Most, but not all, appreciate recital performance opportunities.

Use Results• Compare focus group

results to other data

• Triangulate

• Draw conclusions

• Report to senior staff and/or commissioners

• Use data to inform decisions

Resources

• Focus Groups: A Practical Guide for Applied Research, Third Edition, 2000, Richard A. Krueger

• Focus Groups as Qualitative Research (Qualitative Research Methods), 1997, David L. Morgan

Recommended Practice

To really understand focus groups, design one

Send one-page focus group design by [date]

We provide individual feedback

Program name

Learning outcomes

Potential focus group participants

Focus group questions

Part II Survey Research

• Steps to develop survey

• Writing survey questions

• Online surveys

• Qualitative and quantitative data analysis

• Reporting