REACH MENA Regional Workshop 5-7 March 2014 Jordan

REACH MENA Regional Workshop

5-7 March 2014Jordan

1. Research Ethos2. Terms of Reference3. Secondary data analysis4. Objectives & indicators5. Quantitative & Qualitative

data collection – methods6. Quantitative & Qualitative

data collection – tool design7. Sampling8. Field work preparations9.a. Quantitative data analysis9.b. Qualitative data analysis10. Reporting and representing

data11. REACH products

Assessment Workshop

1. Research Ethos

Slide 2

Who does research quality matter to?

Research community: developing sound research literature; ‘knowing the causes of things’

Research funders: value for money, continued investment,

Research users: confidence in the results, belief that they are relevant

Research respondents: ethical considerations, cooperation

It matters to these groups because quality ensures we are able to inform more effective humanitarian action

1. Research Ethos

Slide 2

How to do quality research?

Develop a strong research question – clear objective(s)

Develop/adopt a strong conceptual and theoretical framework – to identify context, needs, response, gaps and priorities

A fit between the method and the research question/orientating framework

High-quality data and analysis

Firm basis for our conclusions

Being expansive: highlighting the significance of our work - making recommendations

• Primary Data Collection & Entry•Preliminary Data Analysis

•Presenting Initial Analysis •Final Analysis & reporting• Development of REACH Products• Peer Review• Dissemination

The REACH Assessment Life Cycle

Slide 2

• Activation & Terms of Reference

• Secondary Data Review• Assessment

Methodology Design and Planning

• Impact Assessment• Research Appraisal

Evaluation Planning and Design

Field Assessment

Analysis and Documentation of Findings

2. Terms of Reference

Slide 2

2. Terms of Reference

Slide 2

Why do Secondary data analysis?

To provide context

To identify gaps in information needed to measure indicators

To inform sampling

To inform primary data collection tools

Slide 7

3. Secondary data analysis

Slide 8

3. Secondary data analysisSOURCES Syria Humanitarian Assistance Response Plan (SHARP) Regional Response Plan (RRP) National Response Plan (NRP) Comprehensive Regional Strategy (CRS) National institutions (ministries, research institutions, universities, etc.) Large survey (DHS, MICS, censuses, etc.) International development institutions (i.e. World Bank) Sector fact sheets Common Operational Datasets (CODs) United Nations as well as local and international NGOs assessment and

survey reports United Nations global data sets or country portals Online databases (i.e. EM-DAT, PreventionWeb) Previous Flash appeals and Consolidated Appeal Processes (CAPs) WHO country epidemiological profiles ALNAP evaluation reports, After Action reviews DevInfo, World Bank’s world development indicators, Millennium

Development Goals

Slide 9

4. Objectives & indicators

Slide 10

4. Objectives & indicatorsCROSS-CUTTING THEMES

TYPE INDICATORS - Syrian Non-Camp refugees in KRIProtecti

onGende

r AgeGeneral % of individuals by current residence in governorate/district/sub-district General % of HH by time of arrival of first HH member in KRI General % of HH by time of arrival of last HH member in KRI General % of HH by time of arrival of first HH member in district General % of HH by original district of origin in Syria General % of HH residing in other districts since arriving in KRI - by most recent other district General % of HH that are registered/unregistered General % of HH where at least one member holds a KRI identity card General % of HH where at least one member holds a residency card General % of individuals by age group and gender x x xGeneral % of individuals with a permanent disability by type (physical, mental, visual) and gender x x General % of HH by head of household specifics (female-headed HH, child-headed HH, elder-headed

HH) x x xGeneral % of HH caring for unaccompanied minors (aged 0-17) x xGeneral Average number of unaccompanied minors (aged 0-17) per HH x xGeneral % of HH that have immediate family members remaining in Syria Intentions % of HH intending to move within district/to other district/other governorate Intentions % of HH intending to move within KRI - by reason(s) why Intentions % of HH intending to move within KRI- by time of planned move Intentions % of HH intending to leave KRI for Syrian district of origin/not of origin/other country Intentions % of HH intending to leave KRI - by reason(s) why Intentions % of HH intending to leave KRI - by time of planned move

Slide 11

4. Objectives & indicators

DISCUSSION

Why do we need consistent indicators?

Why do we need regional and country specific indicators?

Slide 12

5. Quantitative & Qualitative data collection – methods

Slide 13


QUANTITATIVE researchers typically seek – ►Causal determination, prediction, robust dependence/associations, generalization of findings►Structured and systemizing method►Controlled conditions►Usually large samples►Tests theories and hypotheses

QUALITATIVE researchers typically seek – ►Depth rather than breadth: integrity of perspectives through rich

own-word accounts; description, insight and understanding

►Discovery (iterative) rather than verification (hypothesis testing)• A mainly inductive rather than deductive analytical process: can

develop theory from information collected

►Less structured / exploratory method (NOT UNSYSTEMATIC)

►Uncontrolled conditions, usually small samples, no statistical analysis

Slide 14


QUANTITATIVE: Relational/causal – typically addressed through representative sample surveys and experiments where “robust dependence” is established if a relationship refuses to go away once other factors AND explanations are taken into account

ALWAYS REMEMBER: Correlation is NOT causation > > > >

Slide 15


Decline in pirates causing climate change?

QUALITATIVE: Aim is to draw conclusions about mechanisms within group being studied – NOT to infer to general population

MIXING QUALITATIVE & QUANTITATIVE METHODS: • Qualitative method used to INFORM or EXPLAIN data gathered

through quantitative method• Quantitative method used to MEASURE PREVALENCE of factors

identified in data gathered through qualitative method

Slide 16


Individual or household interviews – structured (survey)• WEAKNESS: Time consuming, expensive, requires specialized

knowledge of survey design to provide valid information. • STRENGTH: When properly done provides hard evidence of

basic statistics (e.g. malnutrition rates; demography; disease rates, etc.) which are representative of the entire population. • Only use a survey

If you are confident of your design and sampling methods If the objective is to produce findings that are representative of the

broader population Key Informant interviews – structured (survey)

• We use a KI survey to gather quantitative data – where we essentially ask KIs to estimate household level information

Slide 17

5. Quantitative data collection – methods

Individual or Household interviews – semi-structured• A cross-section of people interviewed on the same topic to reveal a range of

attitudes, opinions and behaviours. • Interviewees must be selected to give a good-cross section and avoid sample bias.• Enables more private reflections and broader perspective of each individual than in-

group interviews – more likely to reveal conflicts. Key Informant interviews – semi-structured

• Key informants can be specialists in topics you are interested in, outsiders within the community (like teachers) who may give a more objective view, or others who are in some way especially knowledgeable. Beware, key informants can reinforce inherent power structures and existing inequalities within

communities. Bear in mind the local power structure before interacting with key informants. Focus Group Discussions/Interviews – semi-structured

• Small groups (6-12) of people with something in common: special knowledge or interest in certain topics.

• We often want homogeneous groups – e.g. Age/gender to allow free discussion – beware power dynamics

• Facilitator keeps the discussion balanced and on track. • Enables insight into group interactions/social norms • Can help identify themes to measure prevalence of in individual interviews• Can help explain trends in prevalence already measured in individual interviews

Slide 18

5. Qualitative data collection – methods

DISCUSSION

Which methods have you worked with?

What challenges did you find using quantitative methods (structured interviews)

What challenges did you find using qualitative methods (semi-structured interviews)

Slide 19

5. Qualitative & Qualitative data collection – methods

Slide 20

6. Quantitative & Qualitative data collection – tools

Quantitative QualitativeSample Probability Purposive

Representative of wider population

Yes No

Objective Measure prevalence

Explore mechanisms

Questionnaire Structured: Closed questions

Semi-structured: Open questions

Slide 21

6. Quantitative data collection – tools

QUESTIONS MUST… RESPONSES MUST…- be effective at measuring your indicators

- guide enumerators

- be consistent with one another - be of the appropriate type for the analysis, (i.e.numbers, ranges of values, or words)

- be specific

- not be leading and not be judgmental

- Have a response choice of “other_________________________”.

- use simple words, or explain simply any technical terms

- include important responses for clarity and to avoid skipped questions: for example “none” and “don’t know”

- give structured guidance if observations are to be made to avoid subjectivity

- state if multiple responses are allowed- have discreet categories without overlaps: for example: 0-4; 5-9 – NOT 0-5; 5-10- be limited to questions you will use

in your analysis- - be adapted to the context- take 30 minutes maximum to complete

- be realistic and simple

(Source: ACF – Lessons Learned from KAP Survey Failures – January 2013)

STRUCTURED QUESTIONNAIRE– EXAMPLE QUESTION

BAD: How much do you spend on essential needs?

Measurement (IQDs? USD?) Essential needs (what are they?) Spent by who (all household members? The respondent?) Time period (during the past day, the past week?)

BETTER:How much did your household spend on education (school materials, fees) during the most recent 7 days?

Slide 22

6. Quantitative data collection – tools

SEMI-STRUCTURED INTERVIEW/DISCUSSION TOPIC GUIDE

Identify the major objectives – what exactly do I want to know?

Do not ask the research question directly but through indirect questions and conversations around the issue (translate into everyday language)

The topic guide should help to ensure a comfortable conversation

Funnel approach: from general to specific The topic guide should be short (rule of thumb: 5 to 8 questions) but well prepared (piloting)

Moderators should take detailed notes, using the same language as participants to not loose context

1 hour maximum to complete.

Slide 23

6. Qualitative data collection – tools

RANKING & SCORING Placing something in order, reveals differences within a population.

Helps to identify main problems or preferences of people, and the criteria they use when deciding in what order to place things.

Enables the priorities of different people to be compared.

Can be used in interviews or on their own Can lead to more direct and revealing questions (for example, Why is X a more serious problem than Y?).

Slide 24


TYPES OF RANKING & SCORINGPreference ranking (where people vote to select priorities),

Direct matrix ranking or scoring (breaking down criteria for preference and scoring on each, such as scoring different kinds of trees on a scale from 1-4 on their usefulness for fuel wood, building, fruit, medicine etc.)

Pair-wise ranking (where people choose between two options in different combinations)

Slide 25


FOOD EDUCATION WATER ROADS

FOOD

EDUCATION FOOD

WATER WATER WATER

ROADS FOOD ROADS WATER

TYPES OF RANKING & SCORING Wealth (or well-being) ranking:

Can investigate perceptions of wealth differences and inequalities in a community,

to discover local indicators and criteria, and to establish the relative wealth of households in the community.

Done by making a list of all households and asking different people to sort them into categories according to their own criteria of ‘wealth.

The term ‘well-being’ is often used, since perceptions of wealth usually include non-economics criteria.

Often only three categories are needed: the poorest, middle and richest.

MODIFICATION: • When a list of all households is not feasible, as in a situation of

recent displacement: Get people to identify attributes (material and otherwise) of

households in three categories (poorest, middle, richest), e.g. only the richest households have tin roofs, only the poorest use hand-hoes, etc.

Direct observation can assess how many households fall into each category.

Slide 26


MAPS & DIAGRAMS Social maps:

Maps of a village or area showing where groups of people live. Can be combined with wealth ranking exercises to identify which

are the poorest households, landless, female headed households, different ethnic groups, number of children in a household, etc.

Similar maps can show key installations like water points, schools, and children’s play areas.

Seasonal calendars: Ways of representing seasonal variation in climate, crop

sequences, agricultural and income-generating activities, nutrition, health and diseases, debt, etc.

Can help identify times of shortage—of food, money or time Daily routine diagrams:

Can help compare daily routines of different groups of people, and seasonal changes in the routines.

Can help identify suitable times for meetings, training courses, visits, etc.

Slide 27


MAPS & DIAGRAMS

Flow diagrams: • Shows causes, effects and relationships between key variables.

For example: Refugee and IDP movement; Relationships between economic, political, cultural and climatic factors causing environmental degradation; Flow of commodities and cash in a marketing system; Effects of major changes or innovations (impact diagrams); Organisation chart.

Venn diagrams: • Show key institutions and individuals in a community and their

relationships and importance for decision-making. • Different circles indicate the institutions and individuals.

When circles are separate there is no contact between them. When circles touch, information passes between them.

• If circles overlap a little there is some co-operation in decision-making.

• If they overlap a lot there is considerable cooperation in decision-making.

Slide 28


Quantitative or qualitative method?

Slide 29

6. Quantitative & Qualitative data collection – methods & tools

Specific objectives (indicators) Data collection method(s)

% of households that are buying bottled water Quantitative

Reasons why women are not using camp showers Qualitative

Number of children that have received polio vaccine Quantitative

Skills in demand by local business owners Qualitative

% of adults aged over 18 that have completed secondary education

Quantitative

Social norms on child labour Qualitative

DISCUSSION

What tools have you used in the past?

What challenges do you face when designing tools?

Slide 30

6. Quantitative & Qualitative data collection – tools

Sampling parametersPopulation of interest•Size (known/infinite)•Key characteristics

Significance level

Sampling frame – bias

Resources

Slide 31

7. SamplingTypes of samplingPurposiveRandomStratifiedClustered

Random sampling – key concepts•Central Limit Theorem•Confidence level•Confidence interval•Margin of error•Standard Deviation•Kurtosis

Slide 32

7. Sampling

DISCUSSION

How have you been sampling refugees and non-refugees?

How have you been stratifying your sample?

How have you been randomising your sample?

Slide 33

7. Sampling

Slide 34

8. Field work preparations

FIELDWORKPLAN

Outlines logistical issues that need to be followed and considered

during fieldwork

Should include the following decisions:Number, size and make-up of the assessment teams;

Allocation of assessment teams to specific locations;

Proposed itinerary of visits to specific locations;

Frequency of interim reporting from field teams;

Time to allow for fieldwork at each location;

How teams will travel;

Time to allow for travel; and

Where teams will eat and sleep.

Slide 35


FIELD TEAM TRAINING

Should Cover the following topics:Terms of reference for the assessmentPlan of action, including methodology to be used and time frameFlow diagram linked to TOR for each positionWorking relationships: responsibility of each team member, reporting lines, etcLogistical arrangements for the assessment (transport, accommodation, etc.)Security: existing situation and procedures during the assessment

Slide 36


DISCUSSION

Slide 37


PRIMARY DATA ANALYSIS – Often made unnecessarily complicated! No matter what kind of analysis you do (statistical or non-

statistical), the main issues that you would investigate in any REACH assessment are:

1. CHANGE: how the situation is different now compared to before the crisis, crisis impacts and pre-existing vulnerabilities;

2. GROUP DIFFERENCES: compare the situation of different groups (age, gender, ethnicity);

3. GAPS: any holes in the information that you still need.

PRIMARY DATA ANALYSIS PLAN – structure around:4. Your indicator list5. Potential correlations identified during data collection6. Potential correlations identified during initial analysis & when

presenting initial results to specialists7. Potential correlations identified during Secondary Data Analysis

Slide 38

9. Data analysis

DATA CLEANING

DESCRIBING RESULTS

GENERALISING RESULTS TO YOUR POPULATION OF INTEREST

Slide 39

9.a. Quantitative data analysis

DATA CLEANING Missing values (blanks) – how to avoid them in the first place

• Always make sure your categories for each questions includes all possible answers – no one should be forced to leave a question blank!

Missing values (blanks) – how to treat them in your analysis• Identify if non-random (e.g. caused by confusing question) or random

(e.g data entry mistake, interviewee got tired)• IF very few missing values for one question = likely to be random• IF many missing values for one question = check question!

Could it be confusing/difficult to answer for respondents? = non-random

Or is it a question that is part of many options (e.g. individual expenses items that many may not spend on)? = likely to be random

• IF many missing values for one interview = exclude it from the final analysis

Exclude missing values from your final analysis – if you don’t your results may be misleading

Slide 40


DATA CLEANING Frequency errors – out of range entries

• Check variables for out of range entries, e.g. HHs with an abnormally high number of members– can to a large extent be prevented by adding restrictions to ODK

• Note that consistently high numbers in one interview may simply mean that it is a large household (as opposed to having e.g. 2 or 3 in all age groups and then suddenly 50 in males aged 50+ which is clearly and error) – use common sense!

Frequency errors – others nobody ever went to school (schooling Y/N) but schooling

expenditures recorded more people at school than in HH No medical expenditures but someone in HH received care,

treatment, went to the hospital Exclude cases with frequency errors from your final analysis – if

you don’t your results will be misleading!

Slide 41


DATA CLEANING Text entries

• Make sure spelling is consistent• Review all text entries (e.g. ‘OTHER’) and categorize where

enumerators have failed to assign to an already existing category – e.g. where daily labour has been entered as ‘Other’ instead of existing category

• Where no categories exist, categorise by creating new binary variables (e.g. Daily labour) entering ‘0/1’

Slide 42


Describing CONTINUOUS variables – where values are numerical

• E.g. household expenditure, household income, exact age of household head

What to report:• Averages, maximum, minimum, distribution (standard

deviation)

How to show:• Bar charts – to show difference in means, maximums, minimums,

across e.g. governorates• Line graphs – when showing evolution over time• Tables• Maps

Slide 43


Slide 44


2009 2010 2011 2012 2013 20140

5

10

15

20

25

30

35

School attendance rate – by year and governorate

MafraqIrbidBalqa

YEAR

atte

ndan

ce ra

te

Slide 45


2009 2010 2011 2012 2013 20140

5

10

15

20

25

30

35

School attendance rate – by year and governorate

MafraqIrbidBalqa

YEAR

atte

ndan

ce ra

te

Slide 46

9.a. Quantitative data analysisAverages do not tell the whole story - why we need to explore distributions…..One way of doing this is to look at Standard DeviationExplains how far from the overall mean, each individual observation is on averageSo here, the average distance from the mean is 10 IQD in the red population and 50 IQD in the blue population

Slide 47


Slide 48

9.a. Quantitative data analysisOne way of illustrating distributions – box plotsEXAMPLE: Distribution of food consumption scores within governorates

Slide 49

9.a. Quantitative data analysisAnother useful illustrations of distributions – histograms

EXAMPLE: Average HH size is 5

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 24 26 27 28 30 NA0

200

400

600

800

1000

1200

1400

Number of household members

Num

ber o

f hou

seho

lds

Describing CATEGORICAL variables – values are categorical

E.g. pit latrine; flush latrine; etcE.g. 0-3 years old; 4-6 years old; etc

What to report:Proportions

How to show:Bar charts – stacked Pie charts (when few categories or when showing proportions within proportions)

Slide 50


Slide 51


Qushtapa

Kawergosk

Gawilan

Darashakran

Basirma

Arbat Transit

Akre

0% 10% 20% 30% 40% 50% 60% 70% 80% 90%100%

89%

52%

45%

26%

40%

86%

52%

8%

27%

14%

26%

25%

12%

31%

3%

22%

41%

48%

35%

1%

17%

Food consumption score – by Camp

AcceptableBorderlinePoor

Slide 52


Male Female Male Female Male FemalePre-primary Primary Secondary

010203040

School attendance rate – by education level and gov-ernorate

MafraqIrbidBalqa

% a

ttend

ance

rate

Male Pre-

primary

Female

Pre-pri

mary

Male Prim

ary

Female

Primary

Male Sec

onda

ry

Female

Secon

dary

0

10

20

30

MafraqIrbidBalqa

% a

ttend

ance

rate

So now you know the results in your sample, how will you know if these hold true in your population of interest?

Inference for continuous variables – values are numerical E.g. household expenditure, household income, exact age of

household head T-test (for difference between TWO groups) – SPSS ANOVA (for difference amongst THREE OR MORE groups) –

SPSS

Inference for categorical variables – values are categorical E.g. pit latrine; flush latrine; etc E.g. 0-3 years old; 4-6 years old; etc Proportions Chi-square test – SPSS

If we have time -- Correlation and linear regression…

Slide 53


THEMATIC ANALYSIS Latent (interpretive) and manifest (descriptive) observations:

• Manifest observations can be coded first• Latent observations best coded after – often only by looking at

what is manifest in the text that latent themes become visible. • In sum, they are coded in the same way, but not always at the

same time.

Codes and themes:• Codes stand in for and represent themes – a kind of shorthand,

and the relationship works both ways. • Codes will represent themes in the data and then as you look at

the relationship between codes, you will be better able to elaborate how themes interrelate.

• Codes would be amended as you read and re-read the text, and this is generally where latent themes emerge.

Slide 54

9.b. Qualitative data analysis

CONENT ANALYSISBased on assumption that words and phrases used more often

reflects most important concerns held by usersMeasures word frequencies, space measurements (column

centimeters/inches in the case of newspapers), time counts (for radio and television time) and keyword frequencies.

Slide 55


CONTENT ANALYSIS VS THEMATIC ANALYSIS• Content analysis and thematic analysis are related and

sometimes thematic analysis is referred to as 'interpretive content analysis‘

Thematic analysis: interpreting themes Content analysis: counting themes

• To do a content analysis you need to do basic thematic analysis to least identify the key themes which you are going to count.

• The theory about relationships between themes developed through content analysis may be different compared to one developed through a thematic analysis because:

In content analysis you count instances In thematic analysis you identify relationships between themes on the basis

of your topic, emergent themes, secondary data analysis. In thematic analysis you do not usually quantify themes – if you do count

them this is in a limited sense (meaning is not derived specifically from quantifying them as it would be in content analysis).

Slide 56


DISCUSSION

What analysis have you used in the past?

What worked/didn’t work?

Slide 57

9. Data analysis

Reporting census data – actual numbers OR proportions – depending on interest:

1. Overall population finding – e.g. 5,466 (52%) households in Za’atari had an acceptable food consumption score.

2. Disaggregated population finding – e.g. The number of households with an acceptable food consumption score varied from 544 households in District 8 to 2,444 households in District 2.

Reporting sample data – ALWAYS PROPORTIONS:1.Overall population finding – e.g. 52% of refugee households in KRI

have an acceptable food consumption score2.Disaggregated population finding –

1.IF difference is statistically significant – e.g. ‘43% of households in Dohuk governorate had an acceptable food consumption score, compared to 49% in Sulaymanyiah and 54% in Erbil governorates’.

2.IF difference is NOT statistically significant – ‘No statistically significant difference was found when comparing governorates.’

Slide 58

10. Reporting and representing data

ALWAYS REMEMBER – Correlation is not causation…

Slide 59


Consistent phrasing:</> 20% = ‘less than / almost / more than a fifth of households (XX

%) ’</> 25% = ‘less than / almost / more than a quarter of households

(XX%) ’</> 33% = ‘less than / almost / more than a third of households (XX

%)’</> 50% = ‘less than / almost / more than half of households (XX%)’</> 66% = ‘less than / almost / more than two thirds of households

(XX%)’</> 75% = ‘less than / almost / more than three quarters of

households (XX%)’< 100% = ‘almost all / the majority of households (XX%)’

Reporting qualitative data:

Visualising data – beyond graphs and tables:

Slide 60


Presentations Fact sheets Sit-reps Thematic assessment reports Research reports Maps

Slide 61

11. REACH Products

Slide 62

11. REACH Products - Presentations

Tool for Assessment

• Conducting regular presentations will ensure that the process is considered inclusive and transparent to all actors and stakeholders.

• If conducted in suitable forums (clusters, technical working groups, etc.) it can be used as a tool to validate methodologies and scope / terms of reference of the assessment in the early stages of the process.

• In later stages, it can be used as a tool to validate data and address concerns in the validity of any preliminary results through a consultation process.

• In the longer term it can be included as a process towards the establishment of a peer review system (see next section below).

Output / Product of Assessment

• Where buy-in by actors and stakeholders in the early stages of an assessment is limited, the use of presentations to present preliminary findings and illustrate the relevance of the assessment as a tool for more effective decision-making, may help ensure that the assessment generates support and interest.

• Presentations facilitate the dissemination of the assessment findings and products. In particular, it allows for the tailoring of findings and recommendations (if relevant) to specific target audiences thus ensuring the buy-in of actors and stakeholders that may otherwise not actively make use of the outputs.

Slide 63

Statistics

http://www.ted.com/talks/hans_rosling_shows_the_best_stats_you_ve_ever_seen.html

Slide 64

Statistics

Date post:	15-Feb-2016
Category:	Documents
Upload:	marnie
View:	34 times
Download:	0 times

REACH MENA Regional Workshop 5-7 March 2014 Jordan

Documents