The Children & Residential Experiences: The CARE

program model theory and research

Charles Izzo (cvi2@cornell.edu)Martha Holden (mjh19@cornell.edu_

Elliott Smith (egs1@cornell.edu)

Funded through The Duke Endowment & Cornell University

Goals of Workshop

Describe an evaluation that aims for both

high scientific standards AND local utility

Discuss how we tried to: a) produce generalizable knowledge, and b) meet practical needs of agencies

Critically examine our approach and advance our thinking about these issues

2

Overview of Discussion

Provide context for evaluation Objectives: Scientific and Practical Program description

Describe Evaluation Design decisions Measurement Implications for internal validity &

utility Data-based feedback to agencies

3

Some Scientific Objectives

Provide reliable evidence about the outcomes produced by CARE Research design with credible

counterfactual Multi-modal assessment Reflects the Theory of Change

Qualify for review by California Evidence-base Clearinghouse (CEBC)

4

Some Practical Objectives

Provide agencies with insight about Youth, and the services they receive Agency dynamics that affect service

quality Short and long-term effects of CARE

training

Facilitate better CARE implementation by Providing clues for implementation

targets Shaping the internal dialog Motivating agencies with positive

feedback

5

CARE: Children And Residential Experiences

PROGRAM SUMMARY

6

• The ‘training’ is focused on changing a mindset, not about simply adding new information or developing new technical skills.

• The emphasis is on transforming the organization as a whole.

Implementing CARE is about changing the entire operating system

Characteristics of the CARE Approach

The agency is the locus of learning. The agency itself becomes the primary learning site.

The agency is the unit of learning, rather than the individual (or even the team).

The facilitation process involves much more than skills training or knowledge transmission

The CARE consultants are engaged in a co-learning and co-creation process alongside the agency staff members; all participants are learners.

CARE consultants work to realize the potential of adult learners and to align their mindsets with the needs and experiences of the children.

The Implementation Process

Best Interests of Children

Six Core Principles

Beliefs, Attitudes and Assumptions

Change Facilitation

Integrating CARE Model

Staff Development

Organizational Climate &Culture

Interlocking “Nested” Elements in Translating the CARE Program Model

into Practice

The Three Levels of the CARE Philosophy

The first three circles, or levels, represent the foundation of the CARE program model:

The best interests of the child is essentially a commitment or value

The six principles of CARE provide compass points and a focus for thinking and decision-making.

Articulated beliefs, attitudes and assumptions are an additional level of understanding of the principles that direct actions in practice

Putting CARE into practice requires the ability to move beyond technical thinking (“if x, then y”); it requires adaptive thinking (“what is going on here, and how can I be helpful?”)

Ultimately, it is about mindset, and the need to think and act in an integrated manner with an appreciation of complexity.

Adaptive Thinking is Essential

Changing a mindset involves all aspects of one’s being – cognitive (how we think), affective (what motivates us), social (where we find reinforcement) and spiritual (what gives our life meaning).

Changing Mindsets

Staff/Adult Development Domains

Cognitive/intellectual

Affective/emotional

Moral/spiritual

Social/affiliative

MINDSET

thinking

motivation

meaning

reinforcement

Commitment to CARE by agency

Embracing 6 principles

Understanding key concepts(beliefs, attitudes, assumptions)Working

through applications

Re-

Re-

Re-

Re-

Re-

Re-

Integration of the CARE Philosophy

Experiencing CARE effectiveness

Gaining confidence

Elements of the Change Facilitation Process

Validating

Communicating empathy

Modeling

ProbingAffirming

Challenging

Analyzing case examples

Self-disclosure

QuestioningSharing illustrative stories

Paraphrasing

Providing information

Observing

Creating the context for change

Joining in the task of agency evolution and supporting changing mindsets

Description Of

Evaluation

17

18

Selecting Focal Outcomes To Assess

CARE process involves lots of depth and subtlety

Our evaluation targets only selected outcomes that Map onto specific curriculum content Reflect key milestones in the CARE process

(i.e., that reflect our Theory of Change)

Children And Residential Experiences: Measurement

All Staff Training

Technical Assistance Quarterly

Database to track key program events Ongoing

Intervention

CARE-Consistent Practices

Staff survey pre-training, then

semi-annually

Staff Outcomes

Intentions / Motivation to apply CARE

Principles

Staff survey: post-training only

Socio-emotional Well-being

Strengths & Difficulties

Questionnairepre-training,

then annually

Youth Perceptions About Staff

Youth survey pre-training,

then annually

Youth Outcomes

Knowledge of & Belief in

CARE principles

Staff survey: pre & post-training,

then annually

Organizational Context

Organizational Social Context Survey Glisson & Hemmelgarn, 1998 pre-training, then annually

CARE Now Agencies

(N=7)

Wait-List Agencies

(N=8)

Socio-Emot

Adjust (Pre)

CARE Intervention Interval

12 months

CARE Intervention Interval

12 months

Quasi-Experimental Cohort Design

Socio-Emot

Adjust (Pre1)

Interval with No CARE

12 months

Socio-Emot

Adjust (Post)

Socio-Emot

Adjust (Pre2)

Socio-Emot

Adjust (Post)

Mostly In North Carolina

21

22

CA

RE T

rain

ing

Pre Post 6 mos 12 mos

>Knowledge>Beliefs>Curr Practice>Youth Percep

>Knowledge>Beliefs>Intended Practice

>Curr Practice

>Knowledge>Beliefs>Curr Practice>Youth Perc

Training Outcomes:Short Term & Intermediate

Measurement Instruments

23

Key Features of CARE Assessments

Survey items map directly on to curricular material

Use of “priming” to improve response quality

Corresponding ratings by youth and staff

Matching scales and items across instruments

Validation of new instruments with established instruments

Matching across assessments while maintaining “anonymity”

24

Assessment of Beliefs & Attitudes

Several scenarios with challenging situations, followed by possible responses, rated from “Poor” - - - - - - - - - - “Excellent”

25

Ex: Lacey is an 11 year old who consistently seeks attentionfrom you, asking silly questions, clinging to you, andfollowing you around. Below are several ways torespond to a situation like this.

> Spend time with her to help her feel more securely attached.> Give her less attention to avoid reinforcing her clingy behavior.> Try to learn about the reasons for her attention-seeking behavior.> Try to re-direct her to focus more on other youth instead of staff.

Beliefs & Attitudes- SubscalesListening & Understanding

Attention-Seeker: “Try to learn about the reasons for her attention-seeking behavior.”

Inclusion Chore resister: “He should not be allowed to

participate in group activities.” (R)

Investment / Engagement Attention-seeker: “Spend time with her to

help her feel more securely attached.”26

Beliefs & Attitudes - Subscales

Flexibility Homework refuser: “Give him a choice

about doing it now or before school.”

Competence-Building  Won't eat in group: “See if he can

tolerate eating in a different area with just you and another child he knows.”

27

Assessment of Current Practice

34 practices: “How often have you done this in the past month?” Respondents rate fromNever - - - - - - - - - - - - - - - - - Several

times/day

If “Never,” then why not?No opportunity Not my job No Time Disagree with the practice 28

Assessment of Intended Practice

Asks about the same 34 practices:

“Now that you have finished CARE training,please indicate if you will use

these practices . . . ”

Less Often - - - - About the Same - - - - More Often

29

Youth Perceptions about Interactions with Staff

Several scenarios followed by possible responses from staff. Youth indicate how often they acted that way from

“Never” - - - - - - - - - - - - - - - - “Always”

30

Ex: Think about times when you needed help with something that was difficult for you. Maybe you were nervous about a test / had problems with someone who lives here / problem with a family member / just needed staff to do something for you.

> They tried to understand what I wanted.> They tried to be helpful.> If felt that they were happy to help.> They helped me learn how I could handle the situation better myself.

Trade-offs: Validity vs. Pragmatic Concerns

Instruments: Established instruments with general constructs

vs. Tailored instruments with program-specific constructs

Response choices General (agree/disagree) vs. Specific (how often

in past month) Youth report on specific shifts vs. all staff

Minimizing perceived threat Identified vs. Anonymous data: Implications for

matching, statistical power, and candidness31

Data-Based Feedback To Agencies

32

Useful Features of Data Reports

Discussion at both the construct-level (abstract) and the item-level (tangible)

Providing a reference point (other agencies, subgroups, early time points)

Showing variability, not just means. 33

Useful Features of Data Reports

Showing correspondence across measures

Illustrate with respondents’ own words

Situating each finding within the theory of change.

34

OSC Culture Profile

Proficiency Rigidity Resistance10

20

30

40

50

60

70

80

T S

core

45.5932nd percentile

56.6473rd percentile

58.3972nd percentile

Number of Knowledge Questions Answered Correctly (out of 13)

Information0123456789

10111213

6.4

9.1

Pre-Training Post-Training

N

um

ber

of

Corr

ect

An

sw

ers

Sample: Self-Efficacy is best described as: “Believing you can accomplish something.”

37

Listen

ing

& Und

erstan

ding

Show

ing

Inve

stm

ent

Inclus

ion

Flex

ibility

Compe

tenc

e Bu

ilding

1.0

2.0

3.0

4.0

5.0

6.0

7.0

4.8

5.9

3.9 3.7

4.84.65.2

4.7 4.7 4.4

Current Practices of Direct Care Staff

Your Agency Other CARE Agencies

Ne

ve

r

-

- -

-

-

- -

1

/ W

ee

k

-

-

- -

-

1

+/

Da

y

Change in Staff Belief Scales

Under

stan

ding

Inve

stm

ent

Flex

ibility

Rules

Compe

tenc

e1.0

2.0

3.0

4.0

5.0

3.93.7

3.2

2.8

3.2

4.2 4.2

3.8

2.3

3.6

Baseline Post-Training

Poor

-

-

- -

- -

Fair

-

- -

-

Good

39

Average Score0

0.51

1.52

2.53

3.54

4.55

Times when I’m upset: “They listen to try and un-

derstand what I need”Average Score = 3.2

Perc

en

t A

nsw

ere

d

40

1 "Never"

2 "Rarely"

3"Sometimes"

4 "Usually"

5 "Always"

0102030405060708090

100

29.414.7

30.425

16.1

Times when I’m upset: “They listen to try and un-

derstand what I need”Average Score = 3.2

Perc

en

t A

nsw

ere

d

41

31%

69%

BeliefsDidn't finish responsibilities:

"They should be excluded from group activities."

(Direct-Care Only)

Poor or Very Poor

Fair, Good or Excellent

42

27%

73%

Current Practices"Keep youth out of activ-

ity for not completing responsibilities."(Direct-Care Only)

Weekly or more

Less than weekly

24%

76%

Youth ReportWhen you misbe-

haved or didn't finish responsibilities: "I was not allowed to participate in group

activities."

Some-times, Usually or Always

Never or Rarely

Staff Practices & IntentionsSensitivity / Responsiveness

Current Practices (Past Month)Direct Care Staff Only

Sample: “Avoid situations that trigger a child’s stress response.”

Practice Intentions (Future)Direct Care Staff Only

0

1

2

3

4

5

6

7

4.6

5.3

4.54.7

4.4

Never

-

- -

Weekly

-

-

- S

evera

l Tim

es/d

ay

0

1

2

3

4

5

6

7

5.6 5.6 5.7 5.7 5.7

Le

ss O

fte

n

- -

Sa

me

-

- -

Mo

re O

f-te

n

Appendix Slides

44

Current Practices - Subscales

Listening / Understanding “Talk with a child who is not completing

tasks or responsibilities to try and understand why.”

Investment/ Engagement “Make a point of showing a child that you

care about them and/or enjoy being with them.”

Inclusion “Keep youth out of an activity because

they did not complete their responsibilities.”

45

Current Practices - Subscales

Flexibility “Help create expectations or routines

that are tailored to a child’s individual needs.”

Competence-Building “Use a real life situation to teach a child

how to solve a problem.”

46

Youth Perceptions - SubscalesListening & Understanding

Times when you misbehaved: “Tried to understand why I acted that way.”

Inclusion During recreational activities: “Would not let

me participate in group activities.” (R)

Investment / Engagement During recreational activities: “I felt that

they enjoyed spending time with me.”47

Youth Perceptions - SubscalesFlexibility Times when I misbehaved: “Let up

on the rules to give me a break.”

Competence-Building  Times when I was upset: “Showed

me how I could calm down or make myself feel better.”

48

Supplementary Slides

Youth Perceptions of StaffStaff Sensitivity / Responsiveness

Sensitivity Responsiveness Respect for Autonomy

Investment0

1

2

3

4

5

3.23.5 3.2 3.2

Sensitivity/Responsiveness - Subscales

Ne

ve

r -

-

- -

S

om

eti

me

s

-

-

- -

A

lwa

ys

Sample: “They were available to help me whenever I needed it.”

Staff Report of Current Practices

10

1

2

3

4

5

6

7

5.0

3.7

4.84.5

4.34.6

4.85.0

4.44.6

4.3 4.34.1

Sensitivity/Responsiveness Variation in Means Across Sites (Direct Care)

<- - - - - - - - Site - - - - - - - ->

Ne

ve

r -

-

- -

-

-

We

ek

ly

- -

-

Seve

ral/

da

y

Youth Perceptions of Staff

10

1

2

3

4

5

3.7

3.2

3.63.2 3.2

3.3 3.4 3.3

2.4

3.7

2.8

3.4

2.7

3.2

4.4

Sensitivity/Responsiveness Variation Across Sites

Ne

ve

r -

-

-

- S

om

eti

me

s

-

-

- -

A

lwa

ys

<- - - - - - - - Site - - - - - - - ->

Favorite Staff Member: What makes them your favorite?

Understand me and try to work with me instead of just telling me my problems. And treating everyone equal.

Always joking with us. Let us have fun and go places. Doesn’t give me a hard time.

They are amazing people, always there for you, trust you and give you freedom

Actually cares and is like the grandpa I never had. He encourages me.

When I want to talk, they listen most of the time instead of blowing me off.

Listen and always seem to care about our well-being. They always want to help.

Joke and have fun with us. Provide us with structure and routine.

They actually act like they love us and show us affection

Easy going, easy to get along with, if .. problem will help you find a good solution

They help me if I need help with something at school or with my family.