8/2/2019 Designing, Managing, And Analyzing Multisite Evaluation - Munawir_pep

1/18

DEBRA J.ROG

SUMMARIZED BY

MUNAWIR RAZAK

NIM : 1106038513

DESIGNING, MANAGING, AND

ANALYZING MULTISITE EVALUATION

8/2/2019 Designing, Managing, And Analyzing Multisite Evaluation - Munawir_pep

2/18

8/2/2019 Designing, Managing, And Analyzing Multisite Evaluation - Munawir_pep

3/18

EXAMPLES OF MULTISITE EVALUATIONS (MSEs)

8/2/2019 Designing, Managing, And Analyzing Multisite Evaluation - Munawir_pep

4/18

MULTISITE EVALUATIONS (MSEs)

Example 1Evaluation of 4 community coalitions aimes at preventing family violence

a. Each of the Sites was located in a different county of oneMidwestern State

b. The sites are funded by local foundation, received two-year planing grantsm followed by 5 years ofimplementation funding

c. The initiative had a cross-site technical assistance provider

as well as a cross-site evaluation team.d. Evaluation was modest in funding and scopee. Relied primarily on the collection data through interviews,f. Led by a cross-site evaluators, with local evaluators

involved only in the planning stage.g. The design involved longitudinal case studies, with an

emphasis on using the same lens in each site

h. The focus was initially on the implementation of thecoalitions, followed by an assessment to their ability toachieve outcomes.

8/2/2019 Designing, Managing, And Analyzing Multisite Evaluation - Munawir_pep

5/18

MULTISITE EVALUATIONS (MSEs)

Example 2Evaluation of 4 providers of supportive housing for homeles family

a. This SME is relatively small and in progress in Washingtonstate, it may ultimately involve up to ten providers

b. Most of them serving small numbers of families (thirty orfewer)

c. The Study involves both qualitative and quantitive data onthe implementation and fidelity of the housing programs,

and on the experiences and outcomes of familiesd. The evaluation funding is limited, data are being collectedlargely through telephone interviews with the providers,and family data are being collected by providers staff.

e. This MSE provides a good illustration on MSE datacollection, training, and quality control on a small budget

8/2/2019 Designing, Managing, And Analyzing Multisite Evaluation - Munawir_pep

6/18

MULTISITE EVALUATIONS (MSEs)

Advantages

a. Offer the power of results that can riseabove any single investigation

b. We are likely to learn more about the

generalizability of an intervention and itseffect

c. Opportunity to understand the trade-offsand relative advantages and disadvantagesof the different types of program strategies

d. Can create an invisible college

MSEs offer opportunities to accelerate the development of knowledge in

a program area

8/2/2019 Designing, Managing, And Analyzing Multisite Evaluation - Munawir_pep

7/18

MULTISITE EVALUATIONS (MSEs)

Disadvantages

a. Take more fundingb. Take more time to design, implement and

analyze

c. Error that occur in MSE are likely to bemore serious and long lasting

8/2/2019 Designing, Managing, And Analyzing Multisite Evaluation - Munawir_pep

8/18

MULTISITE EVALUATIONS (MSEs)

Multisite Approaches and Designs

a. Laying the foundation for an MSE

b. Determining the MSE Design

c. Sampling Sites

8/2/2019 Designing, Managing, And Analyzing Multisite Evaluation - Munawir_pep

9/18

MULTISITE EVALUATIONS (MSEs)

Multisite Approaches and Designs

a. Laying the foundation for an MSE- Determining who id the lead for the evaluation and

identifying the mechanism by which cross-site decisions aremade

- Determining the nature of the relationship between theevaluation team and the evaluation sponsor

- Communication is the most distinct for MSEs compared tosingle-site evaluations

- The More decentralized the decision making in in the MSE,the more important it is that decisions and otherinformation be communicated regularly and frequently.

- Communication should include a mix of in-person,telephone and electronic communications.

8/2/2019 Designing, Managing, And Analyzing Multisite Evaluation - Munawir_pep

10/18

MULTISITE EVALUATIONS (MSEs)

Multisite Approaches and Designs

b. Determining the MSE Design- The Only distinction that MSEs share is that evaluation

examines an intervention in two or more sites.- Factors that shape the MSE design nature of evaluation

questions; readiness of the field; the nature of problem; thereason of evaluation; nature, diversity, and number of sites;and resources.

- A range of design often required to meet the variety ofevaluation questions and purposes for the information

- The larger the number of sites, the more important arestandards for data collection

( )

8/2/2019 Designing, Managing, And Analyzing Multisite Evaluation - Munawir_pep

11/18

MULTISITE EVALUATIONS (MSEs)

Multisite Approaches and Designs

c. Sampling Sites- Some MSE design may examine representative sites from

clusters of sites sharing similiar charactheristic.- Sampling may be limited to sites that meet a treshold level

of fidelity of implementation

MULTISITE EVALUATIONS (MSE )

8/2/2019 Designing, Managing, And Analyzing Multisite Evaluation - Munawir_pep

12/18

MULTISITE EVALUATIONS (MSEs)

Strategies for Multisite Data Collection

a. Collecting Common Versus Specific Site Data

b. Developing a Common Protocol

c. Maximizing Existing Data

d. Developing a Common Data Collection Tool

MULTISITE EVALUATIONS (MSE )

8/2/2019 Designing, Managing, And Analyzing Multisite Evaluation - Munawir_pep

13/18

MULTISITE EVALUATIONS (MSEs)

Quality Control in MSEs

a. Selecting and Hiring Data Collectors

b. Common Training and Booster Sessions

c. Readiness of Interviewers

d. Communications, Supervision and OngoingReview

MULTISITE EVALUATIONS (MSEs)

8/2/2019 Designing, Managing, And Analyzing Multisite Evaluation - Munawir_pep

14/18

MULTISITE EVALUATIONS (MSEs)

Tips for the MSE evaluator

a. MSEs require interpersonal acumen

b. Should control and accomodate ego,

attending to equity, and being sensitive

c. Maintaing communication, keep everyoneinformed, deliver on promise made

MULTISITE EVALUATIONS (MSEs)

8/2/2019 Designing, Managing, And Analyzing Multisite Evaluation - Munawir_pep

15/18

MULTISITE EVALUATIONS (MSEs)

Data Management

a. Computerizing and Managing QualitativeData

b. Computerizing and Managing Quantative

Datac. IDs and Confidentiality

MULTISITE EVALUATIONS (MSEs)

8/2/2019 Designing, Managing, And Analyzing Multisite Evaluation - Munawir_pep

16/18

MULTISITE EVALUATIONS (MSEs)

Assessing Multisite Interventions

a. Monitoring Fidelity

b. Assessing Common Ingredients

c. Studying Implementation

d. Measuring Program Participation

e. Assessing Comparison as well as TreatmentSites

MULTISITE EVALUATIONS (MSEs)

8/2/2019 Designing, Managing, And Analyzing Multisite Evaluation - Munawir_pep

17/18

MULTISITE EVALUATIONS (MSEs)

Monitoring Multisite Implementation

a. Design Features to Monitor

b. Monitoring Methods

8/2/2019 Designing, Managing, And Analyzing Multisite Evaluation - Munawir_pep

18/18

Thank YOU