Basics of Performance Measurement

Presented by Mark Hardin

National Child Welfare Legal Resource Center on Legal and Judicial Issues

ABA Center on Children and the Law

© MMVI by the American Bar Association

2

WHY DO PERFORMANCE MEASUREMENT?

From Reinventing Government:

What gets measured gets done. If you don’t measure results, you can’t tell

success from failure. If you can’t see success, you can’t reward it. If you can’t reward success, you’re probably

rewarding failure.

3

WHY DO PERFORMANCE MEASUREMENT?

If you can’t see success, you can’t learn from it.

If you can’t recognize failure, you can’t correct it.

If you can demonstrate results, you can win public support.

4

Some benefits of performance measurement for courts:

Guidance for court process improvements and efficient case flow management;

Identification of weaker elements of court performance, encouraging improvement;

Evidence whether reform efforts are client effective and also cost effective;

Identification of reforms that aren’t working well or have limited benefits which leads to their redesign or elimination;

5

Some benefits of performance measurement for courts:

Increased compliance with legal deadlines for court hearings and court decisions;

Decreased delays in scheduling hearings; Fewer hearing delays once hearings have

started; Hearings in which all parties have a better

opportunity to be heard; Better informed court decisions, contributing

to improved outcomes for children;

6

Some benefits of performance measurement for courts:

Increased communication and collaboration between the court, child welfare agency and other agencies that are present in the courtroom;

Increased collaboration between courts, child welfare agencies and service providers;

7

Some benefits of performance measurement for courts:

Support of requests for needed funding to support pilot programs, increased staff;

Assessment of judicial resources necessary to support a timely and fair process;

8

SOME KEY TERMINOLOGY:

Quantitative analysis: Evaluation of phenomena through numerical analysis, based on collecting and calculating measurable data. Examples: Observations involving counting and measuring, surveys asking for numerical or yes/no answers that can be tabulated numerically.

9

SOME KEY TERMINOLOGY:

Qualitative (descriptive) analysis: Evaluation of phenomena through observations and descriptions, such as through observations or interviews. Information is recorded and recounted in descriptive rather than numerical terms.

10

SOME KEY TERMINOLOGY:

Performance standards: list of principles or expectations for performing a specified professional role, generally formally approved by an official body representing the group to whom the standards apply. Examples: ABA standards for agency attorneys in child welfare cases; Child Welfare League of America Standards for Adoption.

11

SOME KEY TERMINOLOGY:

Performance measurement: Broad terminology for scientific process (using methodologies developed or approved by social scientists) of measuring and evaluating performance. Performance measurement may be either ongoing or ad hoc. It may involve qualitative or quantitative analysis or both.

12

SOME KEY TERMINOLOGY:

Quality assurance and quality improvement (QA & QI): ongoing evaluation to determine level of compliance with pre-established performance standards or expectations.

13

SOME KEY TERMINOLOGY:

Automated performance analysis: Measurement of performance through computer operations. Example: Automated performance measurement of dependency cases.

14

Automated Performance Measurement:

Computer generated Every case is counted and calculated Is ongoing and never stops Should be, but isn’t necessarily, an

automated byproduct of other work

15

Non Automated Performance Measurement:

This is done either ad hoc or periodically (QA/QI)

May be quantitative (numerical) or qualitative (descriptive)

16

Non Automated Performance Measurement: Quantitative

Based on adding numbers, such as: Measuring times between events Counting numbers of events Scoring multiple choice tests

Non automated quantitative measurement often uses carefully identified samples rather than counting every case

17

Non Automated Performance Measurement: Quantitative

Methods for non automated judicial performance analysis:

Surveys (questionnaires) mailed or handed out

File reviews Extracting data from computers ad hoc Court observations (counting frequency

of events)

18

Non Automated Performance Measurement: Qualitative

Examples:

Interviews – open ended, as opposed to numerical, descriptions of observations

File reviews (reading about cases and describing them, seeking insights, not counting) – open ended, as opposed to numerical, descriptions of observations

Court observations (describing what happens in court, seeking insights) -- open ended, as opposed to numerical, descriptions of observations

19

Non Automated Performance Measurement: Qualitative

Qualitative work still involves social science

Ask different people the same questions Word questions carefully to avoid ambiguity Use systematic methods to compare what

different people said about the same thing Example: Computer sorts to see how different

people address the same issue Ideally: Checking back to clear up inaccuracies Use “triangulation” or combining qualitative with

quantitative for a full picture of what is being studied

20

ABCs of Automated Performance Measurement: Measures

A numerical statement of some significant dimension of performance.

Examples: Of those children who are adopted from foster care,

the percentage who were adopted within 24 months after their entry into foster care

Of those children who a court has adjudicated to have been abused or neglected, percentage of children whose cases were adjudicated within 60 days after the filing of the petition

Of those children whose court cases were closed in 2005, the percentage who were the subject of a new abuse or neglect petition in 2006

21

ABCs of Automated Performance Measurement: Data Elements

Bits of information needed to fill in a performance measure.

Example of measure: Of those children who are adopted from foster care,

the percentage who were adopted within 24 months after their entry into foster care

Example of data elements: Foster child adopted (A) Date the child entered foster care (B) Date the child was adopted (C)

22

ABCs of Automated Performance Measurement: Business rules (specifications)

Instructions to the programmers how to collect and calculate the information to produce a performance measure

23

ABCs of Automated Performance Measurement: Business rules (specifications)

Example: Percentage of adopted children adopted within 24 months after entering foster care: Identify each foster child adopted, e.g., in 2005 in specific

state, county, etc. (A) For each child subtract the date of entry into foster care (B)

from the date of exit (C) – to calculate the number of days. (D) Count the number of foster children adopted in 2005. (E) Of the children in E, count the number of children where (D)

was less than 24 months. (F) Divide (F) by (E) for the percentage of foster children adopted

in 2005 who were adopted within 24 months of entry into foster care.

24

Automated Performance Measurement by Child Welfare Agencies: NCCANDS

Voluntary Information related to CAPTA Reporting, safety, recurrence of abuse and

neglect Includes specific data elements

25

Automated Performance Measurement by Child Welfare Agencies: AFCARS

Mandatory data elements and measures Information regarding Titles IV-E and IV-B Entry into foster care Length of stay Permanency

26

Automated Performance Measurement by Child Welfare Agencies: CFSR Performance Standards

Includes measures But must reach a certain level based on 75th

percentile nationally If fail to reach level, must do two-year

“Program Improvement Plan” (PIP) If fail to succeed in PIP, face fines

27

Automated Performance Measurement by Child Welfare Agencies: SACWIS

SACWIS (state automated child welfare information system) is a set of functions that computer systems are supposed to fulfill

States receive special Title IV-E matching funds to establish management information systems fulfilling SACWIS requirements

SACWIS includes many mandatory and some optional functions, including optional court functions

28

SACWIS optional requirement 58: preparation of agency documents for courts

Does the system facilitate preparation of all State agency documents for courts, such as petitions, letters, attorney approvals, and supervisory approvals?

Does the system interface with one or more court systems?

Is duplicate data entry required to generate court documents?

29

SACWIS optional requirement 59: notification of court actions

Does the system generate notifications, either manually or automatically of upcoming court actions for all relevant parties?

30

Automated Performance Measurement by Child Welfare Agencies: CFSR Performance Standards for Safety

Discrete safety measures and standards

CFSR S1A - Recurrence rate of abuse and neglect: Of all children who were victims of a substantiated or indicated maltreatment allegation during the first 6 months of FFY 2004, what percent were not victims of another substantiated or indicated maltreatment allegation during a 6-month period? National Standard = 95.2% or more.

CFSR S1B - Rate of child abuse and neglect while the child is in foster care: Of all children in foster care in FFY 2004, what percent were not victims of a substantiated or indicated maltreatment by a foster parent or facility staff member? National Standard is 99.67% or more.

31

Automated Performance Measurement by Child Welfare Agencies: CFSR Composite Performance Standards

Combined results of several related measures

Combined to avoid unintended incentives, like high percentage of reunifications with low percentage of reentries after reunification

Must collectively meet a specific standard

32

Court Improvement Performance Measures:

No national data No standards (too early) Focus on:

Safety Permanency “Due process” and fairness Timeliness

33

Court Improvement Performance Measures: Safety

1A – Maltreatment while under court jurisdiction

“Percentage of children who are abused or neglected while under court jurisdiction.”

1B -- Maltreatment after release from court jurisdiction

“Percentage of children who are abused or neglected within 12 months after case closure.”

34

Court Improvement Performance Measures: Permanency

2A – Achievement of Child Permanency

Percentage of children in foster care who reach legal permanency by reunification, adoption, or legal guardianship.

35

Court Improvement Performance Measures: Fairness

3A – Number of Judicial Officers Per Case

Percentage of cases in which all hearings were heard by one judicial officer. 

3G – Presence of Advocates During Hearings

Percentage of cases in which legal counsel for parents, children and the agency is present at every hearing.

36

Court Improvement Performance Measures: Timeliness

4A – Time to Permanent Placement

Average (median) time from filing of the original petition to permanency.

4B – Time to Adjudication

Average (median) time from filing of the original petition to adjudication.

37

To Contact Us

Jennifer Renne, (202) 662-1731, rennej@staff.abanet.org

Mark Hardin, (202) 662-1750, markhardin@staff.abanet.org

National Child Welfare Resource Center on Legal and Judicial Issues

ABA Center on Children and the Law

740 15th Street, NW

Washington, DC 20005-1022

http://www.abanet.org/child/nrclji