Date post: | 25-Dec-2015 |
Category: |
Documents |
Upload: | brice-watson |
View: | 217 times |
Download: | 0 times |
OrientationProject
Implementation
Reporting&
Sustainability
Testing Efficacy To Ensure Applicability and Impact
Module 3, Unit 3, Sub-Unit 1, Session 5
Failed in Focus Group Example:
http://www.youtube.com/watch?v=BmykFKjNpdY
2
Module 3, Unit 3, Sub-Unit 1, Session 5
3
Pretesting
• The process of systematically gathering target audience reactions to messages and materials before they are produced in final form
Module 3, Unit 3, Sub-Unit 1, Session 5
4
Use Pretesting To:
• Determine which of multiple alternative creative directions or messages may be most effective
• Identify strengths and weaknesses in a single execution (e.g., brochure, poster, etc.)
Module 3, Unit 3, Sub-Unit 1, Session 5
5
Reasons For Conducting Pretest
• Identify the benefit that resonates most with the target audience
• Ensure that audience understands the message• Identify any “red flags”• Strengthen your message• Conduct “real world” check• Make the materials more appealing• Identify weak and strong points
Module 3, Unit 3, Sub-Unit 1, Session 5
6
Which Materials Do You Pretest?
• Pamphlets, posters, brochures, booklets, flyers, print ads, and TV and radio spots and dramas
• No need to pre-test:– Bumper stickers or badges– Derivatives of already pretested materials
Module 3, Unit 3, Sub-Unit 1, Session 5
7
Which Materials Do You Pretest?
• Multiple versions (3 to 4) of creative executions with core messages communicating various benefit exchanges
• More elaborated version of the materials – containing supporting facts and information
Module 3, Unit 3, Sub-Unit 1, Session 5
8
Criteria Evaluated During Pretesting• Comprehension – does the Target Audience (TA)
understand the materials?
• Relevance – Is the material and the message important to TA?
• Credibility – have trust in the message
• Acceptability – fitting with the target audience value and culture?
• Attraction – will it hold the audience’s attention?
• Persuasion – is it convincing?
Module 3, Unit 3, Sub-Unit 1, Session 5
13
Pretesting Process
1. Determine test objectives and choose research methods
2. Draft research instruments (e.g., discussion guides, questionnaires)
3. Secure interviewers and space for conducting research (if required) and identify, screen and recruit research participants
4. Conduct testing and analyze results5. Based on consumer feedback revise concepts
and materials
Module 3, Unit 3, Sub-Unit 1, Session 5
14
Determine Test Objectives and Choose Research Methods
• Sample Objectives:• Ensure that the main message of a [poster,
brochure, etc.] is readily understood• Ensure that the language and images used in
a [poster, brochure, etc.] are clear • Select the concept that best communicates the
intended message• Determine what was unclear or hard to
understand• Identify the concerns of respondents that need
to be addressed by the materials
Module 3, Unit 3, Sub-Unit 1, Session 5
15
Determine Test Objectives and Choose Research Methods
• Methods:– Expert and gatekeeper review– Focus groups– In-depth interviews– Central location intercept interviews– Self-administered questionnaires
Module 3, Unit 3, Sub-Unit 1, Session 5
16
Expert and Gatekeeper Review
• Purpose:– Ensure technical accuracy
and acceptability
• To Be Tested:– Content and artistic elements– Print and audio-visual
• Number:– 5 to 10
• Pros and Cons
Module 3, Unit 3, Sub-Unit 1, Session 5
17
Focus Groups• Purpose:
– Group interaction; idea generation; materials evaluation; problem identification
• To Be Tested:
– Print, radio, or TV message concepts, slogans/-tag-lines
– Logos and rough artwork
• Number:– 8 to 10 per group– 2 groups per segment
• Pros and Cons
Module 3, Unit 3, Sub-Unit 1, Session 5
18
In-depth Interviews• Purpose:
– No peer pressure; sensitive issues or complex draft material; hard-to-reach audiences
• To Be Tested:– Draft brochures/pamphlets, including on sensitive or
emotional subjects– Message concepts
• Number:– Minimum 12 to 24
• Pros and Cons• Variations
Module 3, Unit 3, Sub-Unit 1, Session 5
19
Central Location Intercept Interviews
• Purpose:– Short time period and large number of
respondents• To Be Tested:
– Message concepts; print materials (e.g., leaflets, fact sheets, posters); print or radio ads
• Number:– Minimum 50 - 100
• Pros and Cons
Focus Group on Tape:
20
http://www.youtube.com/watch?v=m4qJbW9CDOY
Module 3, Unit 3, Sub-Unit 1, Session 5
21
Self-Administered Questionnaires
• Purpose:– Reactions to materials; large number of respondents
• To Be Tested:– Print materials such as book-
lets, brochures, pamphlets
• Number:– 50 - 100
• Pros and Cons
Module 3, Unit 3, Sub-Unit 1, Session 5
22
Draft Research Strategy
• Outline main features of pretest:– Materials to be tested – Variables to be measured – Method – Sample – Estimated length of each interview – Description
• Control for presentation• Pretesting brochures and radio spots
Module 3, Unit 3, Sub-Unit 1, Session 5
23
• Secure interviewers and space for conducting research (if required) and identify, screen, and recruit research participants. – “Facilities”– Moderators
Pretest Logistics
Module 3, Unit 3, Sub-Unit 1, Session 5
24
Conduct Testing and Analyze Results
• Observers
• Note taking
• Data analysis
• Pitfalls of data analysis
• Report writing
Module 3, Unit 3, Sub-Unit 1, Session 5
25
Revise Concepts and Materials
• Change
• Possibly change
• Keep
Module 3, Unit 3, Sub-Unit 1, Session 5
26
Reporting the data
• Background: research objectives and description of materials tested.
• Method: include sample size, sample description, how the respondent were recruited)
• Findings: included reactions and qoutes
• Conclusions and Recommendation:
Module 3, Unit 3, Sub-Unit 1, Session 5
27
A Word of Caution AboutPretesting
• No guarantees
• Controlled environment
• Depends on execution
• Biases introduced by participants
• Statistically imprecise
• Bad advice?
Module 3, Unit 3, Sub-Unit 1, Session 5
28
Myths and Misconceptions About Pretesting
• “I don’t have the time or money”
• “I can tell the difference between good and bad materials, so I don’t need to pretest”
• “Our artist/producer says that pretesting can’t be used to judge creativity”