AIA Web Conference on Culturally Competent Practices in Program

Evaluation

Considering Culture in Evaluation and Applied Research

Rodney K. HopsonDuquesne University

hopson@duq.edu

2

Acknowledgements Collaborative and funded work from lectures, co-

presentations, research grants, published papers, commissioned evaluation work:

National Science Foundation California Endowment American Evaluation Association/Center for Disease

Control Relevance of Culture in Evaluation Institute Heinz Endowments Lincoln U, Manchester Metropolitan U, Claremont Graduate

U Development of pipeline programs to encourage

additional practice AEA Graduate Diversity Internship:

http://www.eval.org/gedip.htm RWJF Evaluation Fellowship: http://www.rwjf-

evaluationfellows.org/

Learning objectives: Recognize relevance and value-addedness

of culture in evaluation theory and practice Identify strategies for assisting evaluators

and agencies in becoming culturally responsive Encourage the development of tips and tools

for working with agencies who work with traditionally underserved and/or minoritized groups

4

Warm up/Introductions Who are you? What do you do? What disciplines, connections,

experiences do you bring into evaluation? How are your disciplines,…important in the

way you think about evaluation? How do you currently think about the role

of culture in evaluation and applied research? How does your background help you think

about the role of culture in evaluation and applied research?

5

Reflections of an ethnoevaluator/culturally responsive evaluator/sociolinguistevaluator

Curry School bridge at UVa between education & arts/sciences

Linking (trans)disciplines: (educational) evaluation, sociolinguistics, anthropology (Hopson, 2000)

Exploring connections between ethnography and evaluation (Hopson, 2005)

Beyond “technocist imperatives of most program evaluations”

Extending Fetterman’s pioneering work and Stakes’ seminal notions on countenance of educational evaluation

Challenging and resisting mainstream thinking on ways of thinking about cultural diversity and difference (Hopson, 2009)

Legitimizing knowledge of communities of color or traditionally underserved communities

Building spaces of hope, praxis and social action for same communities

New answers to increasing complex questions @ culture

7

Warm up/Introductions/Review

Who are you? What do you do? What disciplines, connections, experiences

do you bring into evaluation? How do they help you think about evaluation? What previous backgrounds or experiences do

you bring that might assist you in thinking culture in evaluation?

How do you currently think about the role of culture in your practice? How do you intend to use this discussion of

culture in evaluation to influence your practice?

8

Culturethe way of life of a group of people, the complex of shared concepts and patterns of learned behavior that are handed down from one generation to the next through the means of language and imitation.

(Barnouw, 1985)

the ever-changing values, traditions, social and political relationships, and worldview created, shared and transformed by a group of people bound together by a combination of factors that include a common history, geographic location, language, social class, and religion…

(Nieto, 1999)

9

Cultural Contexts, Characteristics, Locations, and Perspectives Demographic, sociopolitical, contextual

dimensions Characteristics as dynamic, multifaceted,

learned, created, influenced From personal to global position Majority vs. minority More than about race but inclusive of

other identifications(SenGupta, Hopson & Thompson-Robinson, 2004;

Nieto, 1999)

Emerging approach/model used to guide evaluation

System and culmination of evaluation strategies

Theoretical and political positioned (as are all evaluations) Demographic, sociopolitical, and contextual

dimensions, locations, perspectives, and characteristics of culture matter

Privileging lived experiences, esp. communities and populations of color

Avoiding the phenomenon of “evaluating down”

Culturally Responsive Evaluation

(Hopson, 2009)

Theoretical and Practical Intersection of CRE: Advocacy, Race, Power

Decolonizing/ indigenous positions, epistemologies, and

frameworks

Critical theories and epistemologies of race

Social agenda and advocacy theories,

models and approaches in evaluation

12

Cultural CompetenceA set of academic and interpersonal skills that allow individuals to increase their understanding and appreciation of cultural differences and similarities within, among, and between groups. This requires a willingness and ability to draw on community-based values, traditions, and customs, and to work with knowledgeable persons of and from the community in developing focused interventions, communications and other supports.

(Orlandi, 1992)

Talking as culturally responsive

14

Scenario: Dialogue for Diversity and Social Change (DDSC)first impressions

What elements of culture, at what levels, seem salient to this scenario at first glance? What elements of culture are you assuming will not be as salient, based upon your initial impressions?

What is your own cultural position in relation to these cultural elements?How do we think about the relevance of culture in all stages of evaluation for those in public health, education, and other helping professions?

15

Culturally Responsive Evaluation Framework

Step 1: Prepare for the evaluation. Step 2: Engage stakeholders. Step 3: Identify the purpose of the evaluation. Step 4: Frame the right questions. Step 5: Design the evaluation. Step 6: Select and adapt instrumentation. Step 7: Collect the data. Step 8: Analyze the data. Step 9: Disseminate and use the results.

(Frierson, Hood & Hughes, 2002)

Cultural ResponsiveEvaluation

7Collect the data

8 Analyze the data

3 Identify purpose of the evaluation

4 Frame the

right questions

1 Prepare for

the evaluation

5 Design the evaluation

6 Select and adapt instrumentation

2 Engage

stakeholders

9 Disseminate and

use the results

16

17

1 Prepare for the Evaluation Examine the sociocultural context of

the evaluand, including History Community Intersecting cultural identifications

Assemble an evaluation team whose collective lived experience is appropriate to the context of the evaluand.

18

Engage Stakeholders

Develop a stakeholder group representative of the population served by program.

Seek to include direct and indirect consumers.

Pay attention to distributions of power.

Include multiple voices.

19

Identify Purpose of Evaluation

Alignment of purpose with its intended consumers

Program operates in ways that are respectful of cultural context

Program resources are equitably distributed

Benefits of program evaluation are equitable

Applying Stages 1-3 (LaFrance, 2004)

Reflections on embedding culturally competent evaluation in Indian Country Build understanding of values that underlie

programs and projects and create value-added evaluative contribution

Engage stakeholders in participatory manner Build ethic of participation and capacity building that

values community, relationships, respect Frame purpose by building conceptual

picture/model Careful of “too sequential and narrative driven”

logic model 20

21

Scenario: Stages 1-3 What elements of background and

context are important here? What more would you want to know?

Who was included on the evaluation team and what presumed skills, traits do they bring to the evaluation process?

Based on the stated purpose of this evaluation, who do you understand to be the major stakeholders?

22

Frame the Right Questions Include questions of relevance to

significant stakeholders. Determine what will be accepted as

evidence in seeking answers to the questions.

Examine whose voices are heard in the choice of questions and evidence.

Is the lived experience of stakeholders reflected in these choices?

23

Design the Evaluation Build design appropriate to both evaluation

questions and cultural context. Seek culturally appropriate methods that

combine qualitative and quantitative approaches.

Try to collect data at multiple points in time, extending the time frame of the evaluation as needed.

Construct control or comparison groups in ways that respect cultural context and values.

24

Select & Adapt Instrumentation

Establish reliability and validity of instruments for the local population.

Norms must be appropriate to the group(s) involved in the program.

Language and content of instruments should be culturally sensitive.

Adapt instruments as needed and conduct additional validation studies.

Applying Stages 4-6 (Jay, Eatmon, & Frierson, 2005)

Evaluation of undergraduate STEM research program designed for students of color Deliberate design of evaluation team intimately

connected with program of study and background of program, including similar lived experiences of participants

Questions were sensitive to lived experiences of participants and focused on substance of participant experiences

Beyond attention to traditional issues of success but exploring issues of persistence as students of color

Attempt to address nuances and subtleties relative to experiences and impact of program

25

26

Scenario: Stages 4-6 What/whose perspectives are

represented in the evaluation questions, and what other questions might have been posed?

Whose perspectives are accepted as credible evidence? Credible to whom?

How well does the time frame in this study match the needs and rhythms of this context?

27

Collect the Data Procedures used to collect both

qualitative and quantitative data must be responsive to cultural context.

Nonverbal as well as verbal communications provide qualitative data.

Careful training of data collectors in both technical procedures and culture is key.

Shared lived experience provides optimal grounding for culturally-responsive data collection.

28

Analyze the Data Cultural context is a necessary

component of accurate interpretation. Disaggregate data to examine diversity

within groups. Examine outliers, especially successful

ones. A cultural interpreter may be needed to

capture nuances of meaning. Stakeholder review panels can assist in

accurate interpretation.

29

Disseminate & Use the Results

Cultural responsiveness increases both the truthfulness and utility of the results.

Communication mechanisms must be culturally responsive.

Inform a wide range of stakeholders. Make use consistent with the purpose of

the evaluation. Consider community benefit

Applying Stages 7-9 (Manswell Butty, Reid, & LaPoint, 2004)

Discussion of urban school-to-career intervention program using culturally responsive evaluation approach Input derived from school stakeholders on how

best to analyze and interpret data in ways that provided meaning in particular contexts

Findings disaggregated by gender and age to get breakdown of career attitudes and beliefs for participants

Findings provided to numerous stakeholders in particular ways (e.g. student findings presented in student-friendly manner)

30

31

Scenario: Stages 7-9 What additional data collection

procedures might have been useful to consider in designing a culturally responsive evaluation?

Given the findings briefly summarized, what aspects of cultural context might add meaning to guide recommendations?

Were results shared in a culturally congruent way?

32

Take Home/Next Steps/Questions

Culture is relevant, if not central, to all aspects of the evaluation process. Thinking through project plans Incorporating CRE approaches and thinking

in project planning and evaluation steps What key ideas do you take away from

CRE? How can you inform your own evaluation project with cultural lenses?

What questions do you continue to have about CRE?

33

References Barnouw, V. (1985). Culture and personality (4th

ed.). Homewood, IL: The Dorsey Press.

Frierson, H. T., Hood, S., & Hughes, G. B. (2002). Strategies that address culturally-responsive evaluation. In Frechtling, J., The 2002 user-friendly handbook for project evaluation. National Science Foundation.

Hopson, R. (2009, forthcoming). Reclaiming knowledge at the margins: Culturally responsive evaluation in the current evaluation moment. In Ryan, K. & Cousins, B. (Eds.). International Handbook on Evaluation. Thousand Oaks, CA: Sage.

Hopson, R.K. (2005). Commentary: Reinventing Evaluation. Anthropology and Education Quarterly, 36(3).

References Hopson, R. K. (2003). Overview of multicultural and

culturally competent program evaluation: Issues, challenges and opportunities. Woodland Hills, CA: The California Endowment.Hopson, R.K., Ed. (2000). How and Why Language Matters in Evaluation. New Directions for Evaluation, No. 86. San Francisco: Jossey-Bass.

Jay, M., Eatmon, D., & Frierson, H. (2005). Cultural reflections stemming from the evaluation of an undergraduate research program. In S. Hood, R. K. Hopson, & H. T. Frierson (Eds.) The role of culture and cultural context: A mandate for inclusion, the discovery of truth, and understanding in evaluative theory and practice (pp. 201-216). Greenwich, CT: Information Age Publishing, Inc.

34

References King, J. A., Nielsen, J. E., & Colby, J. (2004). Lessons

for culturally competent evaluation from the study of a multicultural initiative. In M. Thompson-Robinson, R. Hopson & S. SenGupta (Eds.), In search of cultural competence in evaluation: Toward Principles and Practices, New Directions for Evaluation, Number 102 (pp.5-19). San Francisco: Jossey-Bass.

La France, J. (2004). Culturally competent evaluation in Indian Country. In M. Thompson-Robinson, R. Hopson, & S. SenGupta (Eds.), (pp. 39-50). San Francisco: Jossey-Bass.

35

36

References Nieto, S. (1999). Affirming diversity: The sociopolitical

context of multicultural education (3rd ed.). Boston: Allyn & Bacon.

Orlandi, M. A. (Ed.) (1992). Cultural competence for evaluators: A guide for alcohol and other drug abuse prevention practitioners working with ethnic/racial communities. U.S. Department of Health and Human Services, Office for Substance Abuse Prevention. DHHS Publication No. (ADM)92-1884.

SenGupta, S., Hopson, R., & Thompson-Robinson, M. (2004). Cultural competence in evaluation: An overview. In M. Thompson-Robinson, R. Hopson & S. SenGupta (Eds.), (pp.5-19). San Francisco: Jossey-Bass.