GPMS Scoring Algorithm Comparison (Alpha Version 2.2)

GPMS Scoring Algorithm Comparison (Alpha Version 2.2)NOTE: Key changes in documentation from Alpha Version 2.0 are in RED as of 20210303

Overview:

This is a working document. Its purpose is to provide and initial data model capturing the relationships being developed as part of the new EAS system.

This latest version, per team discussions, incorporated three previously developed components and introduces two new components.

The previously developed components are: The Performance Assessment Matrix (PAM), Performance Measure Weighting (PM Weighting) System, and the working Grantee Scorecard.

The two new components are: A variable QI to PM aggregation relationship and the incoporation of Content Areas (CAs)

The revised EAS system has three four overall key components:

1) Quality Markers Quality Markers (QMs) are the lowest (most elemental) item being reviewed within the proposed EAS system.

Quality Markers have the following properties within the proposed EAS system.

a) Each is intended to capture a distinct item.

b) Each is measured in terms of either its existance or upon a quality scale.

c) Each is either linked to a specific standard/requirement for grantee compliance OR a specific performance/quality concept

Note 1: For purposes of illustration, two types of QMs are included in this working document. These are distinguished by how they are measured (see Notes 2 and 3)

Note 2: The first type of QM included in this example measures existance. QMs measuring existance result in a boolean (1=Yes; 0=No) outcome indicating that the grantee has the system, qualification, measurement, etc. in place or not.

Note 2: The second type of QM included reflects items measures on a scale. For simplicity, we currently assume that rating scale reflects three "states"I: 1=Fails to Meet Requirments or Expectations, 2=Meets Requirements or Expectations, 3=Exceeds Requirements or Expectations

2) Quality Indicators: Quality Indicators are intended to key specific concepts related to grantee quality and performance. Quality Indicators (QIs) are an aggregation of one or more Quality Markers (QMs). Unlike QMs, QIs do not need to be distinct from each other. Instead, each QI is intended to capture a different concept that is thought important. They have the following properties:

a) Each QI is composed of multiple QMs

b) A QIs final value is a function of its component QMs

c) There is a many to many relationship between QMs and QIs. In other words, a given QMs may contribute to multiple Qis. It is this potential "sharing of QMs" that mean that Qis may not be distinct from one another.

d) Quality Indicators are rated on a 5-point scale with 1 indicating lowest performance and/or quality and 5 being highest.

e) The 5-point scale used by QIs is an ordinal and not cardinal scale. The values represent relative ranks tieds to specific definitions, but do not represent points on a continuum

Note 1: The 5-point PM Scale is proposed and assumed here to illustrate how Quality Markers can roll into Quality Indicators which roll into Performance Measures.

Note 2: The sheet, QI_Coding_Range, provides a proposed interpretation of this 5-point QI scale along with how a score can be derived by first combining the two different types of Quality Markers and then combining the score of each type.

Note 3: Additional "types" of Quality Markers can be included in future versions. To do so, two important things are needed: 1) A clear definition of how that new type is measures, and 2) a clear definition of how the QMs of the new type can be combined and contribute to a Quality Indicator.

Note 4: In this version, all QM contribute to their QIs in a similar way. This can be relaxed in later versions (see how QIs aggregate to Performance Measures as a conceptual example)

3) Performance Measures: Performance Measures (PMs) are intended to broader areas of grantee quality and performance. They can exist with specific Content Areas or be applied system-wide. They have the following properties:

a) Each PM is composed of multiple Quality Indicators (Qis)

b) Each PM is intended to measure a different dimension of performance and/or quality.

c) Each PMs value is a function of its components Qis

d) QIs do not need to be exclusive to a PM. There can be a many-to-many relationship between QIs and PMs. In other words, a given QI can contribute to multiple PMs.

e) PMs are rated on a 5-point scale with 1 being lowest and 5 being highest. However, this is an ordinal and not cardinal scale. The values represent relative ranks tieds to specific definitions, but do not represent points on a continuum

f) A given QI can contribute differently to different PMs. For example, an 'electronic invoicing system' QI can be a relatively important component of a performance measure regarding fiscal health, but of relatively low importance relative to other parts of program managment.

Note 1: This 5 point scale is based upon the prior 5-point scale used in previous FA reviews

Note 2: The sheet, PM_Coding_Range, provides a proposed interpretation of this scale incorporated the proposed changes to the EAS and addition of QMs.

4) Content Areas: Content Areas (CAs) reflect important, and targeted, dimensions of grantee performance. They are comprised of one or more Performance Measures (PMs), and have the following properties:

a) Each CA is intended to reflect performance and/or quality across a specific broad dimension. NOTE: In this sense, "System-Wide" performance can be as a unique content area.

b) Each CA is composed of one or more PMs.

c) Each component PM's contribution to a given CA is defined by the assessment approach selected. NOTE: There are currently two proposed assessment approaches: The PAM and PM Weighting

d) While CAs are intended to be unique, they do not need exclusive of one another. In other words, a given PM could contributed to multiple content areas.

Organization of Spreadsheet:

This spreadsheet is organized in the following way. The first several sheets describe the data model, the proposed coding and aggregation approaches for Quality Indicators, Quality Markers, Performance Measures, and Content Areas, respectively.

Following these documentation sheets, an initial working model is presented. This model captures the relationship between Qis, QMs, PMs, and CAs. This current version contains

The Workbook contains the following sheets:

Data_Model: This sheet graphically describes the data model including lists, references, and crosswalks (proposed queries).

QI_Coding_Range: This sheet describing how values for QIs can be derived from its components QMs

PM_Coding_Range: This sheet describes how values for the PMs can be mapped from its component QIs

Overview_PAM: This sheet describes the Performance Assessment Matrix (PAM) approach to scoring.

Overview_PM_Weighting: This sheet describesthe weighting approach to scoring.

Grantee Comparison ScoreCard: This sheet, which is from prior distributed versions, presents and initial GPMS scorecard and compares the PAM and PM Wieghting GPMS approaches

TBL_QM_MasterList: This sheet contains an exhaustive list of all Quality Markers their scale, and their values. There are currently 100 example Quality Markers

TBL_QI_MasterList: This sheet contains an exhaustive list of all Quality Indicators and computes their value (5-point scale) from component QMs. There are currently 50 Quality Indicators.

TBL_PM_MasterList: This sheet continas and exhaustive list of all Performance Measures and computes their values (5-point scale) from component QMs. There are currently 25 Performance Measures.

TBL_CA_MasterList: This sheet contains an exhaustive list of all Content Areas

CR1_QM_QI: This crosswalk table captures the many to many relationship between QMs and component Qis. There are currently 500 QI:QM links.

CR_PM_QI: This crosswork captures the many to many relationship between PMs and component QIs. It also captures information concerning how Qis contributed to PMs. There are currently 250 QI:PM links.

CR_CA_PM: This crosswork captures the many to many relationship between CAs and component PMs.

ReferenceLists: This sheet contains several lists to ensure referential integrity across the working model.

PAM_DataInputs: This sheet, reproduced from previous distributions, contains input data to calculate PAM scores on the Grantee Comparison ScoreCard.

Scoring_DataInputs: This sheet, reproduced from previous distributions, contains input data to calculate PM Weighting scores on the Grantee Comparison ScoreCard.

SimulatedData: This sheet contains several sets of simulated data for the QM values. The different scenarios can be selected on the Grantee Comparison ScoreCard sheet .

Appdx_PermutationsOfQMScores: This appendix sheet illustrates all possible permutations from the provided QM scoring approach discussed in the PM_Coding_Range sheet.

a set of hypothetical Qis and QMs. These aggregate to a group of 25 PMs. This set of PMs contains the 23 PMs reviewed and ranked in earlier work as well as two additional "System PMs".

The addition of the System PMs is intended to capture design consideration noted during the last team discussion.

The Content Areas are the six already established and assessed during FA1/2 reviews and an addition "Systems" content area, which-like System PMs-is intended to capture design considerations noteded during previous team meetings.

As the evolving Alpha Version illustrations, our framework can be quickly adopoted to incorporate actual measures to support simulation and stress testing of functioning.

GPMS Data Model

Overview:

This sheet presents a graphic overview of the proposed GPMS data model that has been implemented in this workbook. Being Excel-based, this alpha version makes extensive use of internal Excel functions such as

Vlookup, SumIfs, and other reference functions. These will likely require specific queries or front-end coding in the IT&AMS system.

A design consideration of a future IT&AMS system is an administration interface that focuses upon the QM:QI, QI:PM, and PM:CA relationships currently captures in the CRX_YY_ZZ tabs.

A design consideration of a future IT&AMS system is a "simulator"/"scenario" interface. As all core data driving the system is captured at the QM level, this interface could support 'scenarios' in the form of 'proposed Qis', 'proposed PMs', and alternative aggregation rules.

Data Model:

Tbl: QM_Master Listan exhaustive list of all Quality Markers, how they are scored, and thier value.

Tbl: QI_Master ListAn exhaustive list of all Quality Indicators, their value and summary data of their component QMs.

CR1: QM to QICaptures Many to Many relationship between QMs and QIs

Tbl: PM_Master Listan exhaustive list of all Performance Measures, their values, and a summary fof their component Quality Indicators.

CR2: PM to QICaptures Many to Many relationship between QIs and PMs

LST: QM_Scoring Ways Quality Markers

could be scored. There are currently two ways.

LST: QI_Scoring Ways QIs could be

scored

LST: PM_Scoring Ways QMs could be scored

LST: QI_Levels and Weights Captures the relative importance

of each QI to different PMs

Tbl: CA_Master Listan exhaustive list of all Content Areas, their values, and a summary fof their component Performance Measures.

CR2: PM to QICaptures Many to Many relationship between QIs and PMs

CURRENT NOT USED

Grantee Comparison ScoreCardA working scorecard that the summarizes grantee performance by content area and compares/contracts the two proposed approaches to GPMS calcuation.

PAM_DataInputsStructure and Inputs needed to

calculate PAM Scores

Scoring_DataInputsStructure and Inputs needed to

calculate PM Weighting Scores

Priority on Severity Priority on

Prevalance

Ref_Code Severity or Importance Scope or Prevalence 1 2* 3 4* 5 1 2* 3 4* 5

High_Often High Often -9 -4.5 0 4.5 9 -9 -4.5 0 4.5 9

High_Sometimes High Sometimes -8 -4 0 4 8 -6 -3 0 3 6

High_Rarely High Rarely -7 -3.5 0 3.5 7 -3 -1.5 0 1.5 3

Mid_Often Mid Often -6 -3 0 3 6 -8 -4 0 4 8

Mid_Sometimes Mid Sometimes -5 -2.5 0 2.5 5 -5 -2.5 0 2.5 5

Mid_Rarely Mid Rarely -4 -2 0 2 4 -2 -1 0 1 2

Low_Often Low Often -3 -1.5 0 1.5 3 -7 -3.5 0 3.5 7

Low_Sometimes Low Sometimes -2 -1 0 1 2 -4 -2 0 2 4

Low_Rarely Low Rarely -1 -0.5 0 0.5 1 -1 -0.5 0 0.5 1

*Not Included in Original Matrix and set to mid-point between adjacent categories

GUIDE CODE Key Performance Measures Step 6a: Normalized

Median Rank

Rank Severity

Category

Frequency of

Findings*

Scope

Category

Ref_Code Baseline Rand

PDM 1 1.000 The grantee’s program structure and design is informed by the community’s strengths and needs. 0.612 3.000 High 68.000 Often High_Often 68.000 38

PMQ 1 2.000The grantee establishes a management structure that consists of staff, consultants, or contractors who ensure high-quality service

delivery; have sufficient knowledge, training, experience, and competencies to fulfill the roles and responsibilities of their

positions; and provide regular supervision and support to staff.

1.000 1.000 High 66.000 Often High_Often 66.00025

PMQ 2 3.000The grantee uses data to identify program strengths, needs, and areas needing improvement; to evaluate progress toward

achieving program goals and compliance with program performance standards; and to assess the effectiveness of professional

development.

0.991 2.000 High 1.000 Rarely High_Rarely 1.00085

PMQ 3 4.000 The grantee maintains a formal structure of program governance to oversee the quality of services for children and families, and

to make decisions related to program design and implementation.0.337 8.000 High 50.000 Sometimes High_Sometimes 50.000 89

FIS 1 5.000

The grantee develops and implements its budget to sustain management, staffing structures, and the delivery of services that

support the needs of enrolled children and families. This entails relating financial data to accomplishments of the grant award and

an awareness of program progress, lessons learned, and needed improvements.0.564 4.000 High 57.000 Sometimes High_Sometimes 57.000

100

FIS 4 6.000 The grantee complies with application, prior approval, and reporting requirements for facilities purchased, constructed, or

renovated with Head Start funds.0.094 25.000 Low 90.000 Often Low_Often 90.000

44

FIS 2 7.000 The grantee plans and implements a fiscal management system that supports the organization’s ongoing capacity to execute its

budget over time and meet the needs of its organization.0.355 7.000 High 38.000 Sometimes High_Sometimes 38.000

74

FIS 3 8.000 The grantee’s financial management system provides for effective control over and accountability for all funds, property, and

other assets.0.285 10.000 Mid 20.000 Rarely Mid_Rarely 20.000

18

ERSEA 1 9.000The grantee enrolls children or pregnant women who are categorically eligible or who meet defined income-eligibility

requirements.0.193 19.000 Low 10.000 Rarely Low_Rarely 10.000 84

ERSEA 2 10.000At least 10% of the grantee’s total funded enrollment is filled by children eligible for services under IDEA or the grantee has

received a waiver.0.126 23.000 Low 26.000 Rarely Low_Rarely 26.000 66

ERSEA 3 11.000 The grantee maintains and tracks enrollment for all participants, including pregnant women. 0.163 20.000 Low 62.000 Sometimes Low_Sometimes 62.000 20

FCE 1 12.000 The grantee collaborates with families to support family well-being, parents’ aspirations, and parents’ life goals. 0.213 15.000 Mid 45.000 Sometimes Mid_Sometimes 45.000 73

HEA 2 13.000 The grantee supports a program-wide culture that promotes mental health and social and emotional well-being, and uses mental

health consultation to support staff.0.224 13.000 Mid 8.000 Rarely Mid_Rarely 8.000 79

FCE 2 14.000 The grantee provides services that strengthen parent-child relationships and support parents in strengthening parenting skills. 0.204 16.000 Mid 23.000 Rarely Mid_Rarely 23.00027

FCE 3 15.000 The grantee’s education and child development services recognize parents’ roles as children’s lifelong educators and encourage

parents to engage in their child’s education.0.216 14.000 Mid 5.000 Rarely Mid_Rarely 5.000 31

ECD 1 16.000School readiness efforts align with the expectations of receiving schools*, the Head Start Early Learning Outcomes Framework

(HSELOF), and state early learning standards.0.202 17.000 Low 3.000 Rarely Low_Rarely 3.000

10

ECD 3 17.000 The grantee ensures teachers are prepared to implement the curriculum and support children’s progress toward school readiness. 0.252 12.000 Mid 2.000 Rarely Mid_Rarely 2.000 99

CLS 2 18.000Teaching practices intentionally promote progress toward school readiness and provide high-quality learning experiences for

children.0.289 9.000 Mid 93.000 Often Mid_Often 93.000

86

HEA 1 19.000 The grantee has an approach to ensuring delivery of high-quality health services. 0.200 18.000 Low 27.000 Rarely Low_Rarely 27.000 63

HEA 3 20.000The grantee maintains and monitors for effective oral health practices and nutrition services that meet the nutritional needs and

accommodate feeding requirements and allergies.0.155 21.000 Low 44.000 Sometimes Low_Sometimes 44.000

30

HEA 4 21.000 The grantee has implemented a process for monitoring and maintaining healthy and safe environments. This includes ensuring all

staff have complete background checks.0.260 11.000 Mid 79.000 Often Mid_Often 79.000 49

ECD 4 22.000 The grantee ensures home-based program services provide home visits and group socialization activities that provide high-quality

learning experiences.0.120 24.000 Low 22.000 Rarely Low_Rarely 22.000

5

HEA 5 23.000For programs serving pregnant women, the grantee provides and monitors for quality services that facilitate enrolled pregnant

women’s access to health care and provide information to support prenatal, postpartum, maternal, and infant health and

emotional well-being.

0.130 22.000 Low 89.000 Often Low_Often 89.00020

Sys_1 24.000 A working System wide PM 0.500 5.000 High 89.000 Often High_Often 89.000 80

Sys_2 25.000 A second working System-wide PM 0.500 5.000 High 89.000 Often High_Often 89.000 61

*Randomly Generated between 0 and 100 to reflect total findings across all grantees

Input Table 1: PAM Matrix

Input Table 2: Performance Measure Serverity and Scope

GPMS Performance Assessment Matrix (PAM) Scoring

Overview:

This document presents a simulated example of how the PAM, originally presented by Dr. Fiene, could be presented in a Grantee Scorecard Approach similar to the GPMS Scoring Calculator

The PAM is organized around the following principles

1. The existing performance measures in the FA 1 and 2 protocols can be grouped in tiers of severity and/or importance.

Note 1: This initial version proposes three tiers of performance

Note 2: Each PM's tier is determined by their median rating from our survey of experts.

Note 3: These surveyed ranking are the same data used to set the weights in the GPMS performance calculator.

2. Performance measure can be grouped in tiers according to the scope/prevalence of findings (i.e. citations pertaining to standards under a performance measure)

Note 1: The logic behind this is that PMs with a greater number of findings are better able to distinguish levels of performance.

Note 2: In this illustration, prevalence refers to the total number of findings across all grantees. Thus, this introduces a level of relative performance to the PAM.

3. Grantee PM Score that rated on a scale of 1-5.

Combined, these three principles create of 3 dimensional "cube" of potential performance values for each PM for each grantee. These dimensions are

In the original presentation, these were illustrated as a 3x3x3 cube in the following way:

1. The "Importance Tier" of the PM as determined by the survey data was presented as three rows: Top, Mid, and Low PMS

2. The "Prevalence Tier" as determined by the number of total findings across all grantees was presented as three columns: Often, Sometimes, and Rarely.

NOTE1: These two dimensions are 'universal' across all grantees in that the PM rankings and prevalence across all grantees of citations is near constant for any particular grantee.

3. There were three possible values for a grantee's score for that PM in this example: Low, Medium, and High.

The values within the PAM reflect the priority given to the three dimensions. This is illustrated in the two tables below.

The table on the left gives priority to the Severity Dimension and has the following ordering: Severity, Prevalence and PM Score. Thus, the top row (highest severity) has the highest values followed by the second and so on.

The table on the right gives priority to the Prevalence Dimension and has the following ordings: Prevalence, Severity, and PM Score. In this tables, the first column has the highest values followed by the second and so on.

On the Grantee Scorecare comparison sheet, the impact of these two orderings can be seen by changing the priority pulldown.

Often (+/-) Sometimes (+/-) Rarely (+/-)Grantee PM Score

(Selection of specific value within

PM's cell)

Often (+/-) Sometimes (+/-) Rarely (+/-)Grantee PM Score

(Selection of specific value within

PM's cell)

Top PMs -9,0,+9 -8,0,+8 -7,0,+7 Low Score: Negative Value in Cell Top PMs -9,0,+9 -6,0,+6 -3,0,+3 Low Score: Negative Value in Cell

Mid PMs -6,0,+6 -5,0,+5 -4,0,+4 Medium Score: Zero Value in Cell Mid PMs -8,0,+8 -5,0,+5 -2,0,+2 Medium Score: Zero Value in Cell

Low PMs -3,0,+3 -2,0,+2 -1,0,+1 High Score: Highest Value in Cell Low PMs -7,0,+7 -4,0,+4 -1,0,+1 High Score: Highest Value in Cell

Methodology applied in this document for Calculating a PAM score

This numerical example applies the approach discussed above and limits calculation to only the use of PM values. These values are contained in Table 1 of the PAM Data Inputs page.

To facilitate comparison between the PAM and the PM_Weighting approach the same QM:QI relationships and QI:PM relationships are used.

A three tier structure for the Severity/Importance dimension is used. Each PM's tier on this dimension is determined by their tercile of the median PM ranks. The highest ranked third is placed in the top PMs row.

The middle third in the Mid PMs row, and the lowest third in the Low PMs row. These are listed in the Severity Category column of Table 2.

A three tier structure is also used for the Prevalence dimension. To populate this dimension, a randomly generated number of findings across all grantees (from 0-100) were created.

Using this value, PMs were placed-by tercile-into the Often, Sometimes, or Rarely column.

The Grantee PM score dimension is expanded to five levels consistent with how PM scores will be generated in the real world.

Values for the two "middling" categories of 2 and 4 were set to the mid-point between the Low and Medium AND Medium and High values, respectively.

These final values are contained within Table 1 on the PAM Data Inputs sheet.

Grantee PAM Scorecard

This sheet illustrates, in a manner similar to the GPMS Scoring Calculator, how the PAM would capture and report grantee performance across different content areas and overall.

One is able to see a variety of potential grantee scores by changing the sample grantee data used. This is done by the drop down menu in cell d5. The baseline is setting all PMs to 3 and a zero PAM score.

Score are updated and presented along with the maximum and minimum possible for each PM grouped by content area.

At the top, the overall PAM score and content area specific scores are presented. These are also illustrated in a Red-Yellow-Green Stoplight graph.

In that graph, Red is low performance, Yellow medium performance, and Green top performance.

Finally, to illustrate how sensitive the range of potential PAM scores (i.e. column determination through the Prevalence Matrix) are to the simulated number of findings, one can toggle from the baseline number of findings to a new random draw by

using the pulldown and selecting 'random' in cell D7. Then, each time F9 is pressed a new random draw is pulled, the PAM recalibrated, and scores updated.

NOTE2: In tandem, the Importance and Prevalence dimensions place each PM on a 3x3 grid and determine how many points it can contribute.

For instance, a "Top PM" that is "Often" cited can contribute up to 9 points if the grantee scores high and deduct 9 is that grantee scores low on that PM.

NOTE3: The grantee's PM score determines the specific value (from three possible)

For instance, a high PM score contributes 9, a low deducts 9 and a medium contributes 0.

Prevalence (Often, Sometimes, and Rarely as columns) Prevalence (Often, Sometimes, and Rarely as columns)

Severity or

Importance (

Top, Mid and

Low as rows)

Severity or

Importance (

Top, Mid and

Low as rows)

Content Area Content Area Text PM PM_Text PM_Value Within Content Area Wieght Adjusted Percentage

ECD Early Childhood Development ECD 1 School readiness efforts align with the expectations of receiving schools*, the Head Start Early

Learning Outcomes Framework (HSELOF), and state early learning standards.

ECD Early Childhood Development CLS 2 Teaching practices intentionally promote progress toward school readiness and provide high-

quality learning experiences for children.

ECD Early Childhood Development ECD 3 The grantee ensures teachers are prepared to implement the curriculum and support children’s

progress toward school readiness.

ECD Early Childhood Development ECD 4 The grantee ensures home-based program services provide home visits and group socialization

activities that provide high-quality learning experiences.

PD Program Design PDM 1 The grantee’s program structure and design is informed by the community’s strengths and needs.

PD Program Design PMQ 1 The grantee establishes a management structure that consists of staff, consultants, or

contractors who ensure high-quality service delivery; have sufficient knowledge, training,

experience, and competencies to fulfill the roles and responsibilities of their positions; and

provide regular supervision and support to staff.PD Program Design PMQ 2 The grantee uses data to identify program strengths, needs, and areas needing improvement; to

evaluate progress toward achieving program goals and compliance with program performance

standards; and to assess the effectiveness of professional development.

PD Program Design PMQ 3 The grantee maintains a formal structure of program governance to oversee the quality of

services for children and families, and to make decisions related to program design and

implementation.

FCE Family and Community Engagement FCE 1 The grantee collaborates with families to support family well-being, parents’ aspirations, and

parents’ life goals.

FCE Family and Community Engagement FCE 2 The grantee provides services that strengthen parent-child relationships and support parents in

strengthening parenting skills.

FCE Family and Community Engagement FCE 3 The grantee’s education and child development services recognize parents’ roles as children’s

lifelong educators and encourage parents to engage in their child’s education.

HS Health and Safety HEA 1 The grantee has an approach to ensuring delivery of high-quality health services.

HS Health and Safety HEA 2 The grantee supports a program-wide culture that promotes mental health and social and

emotional well-being, and uses mental health consultation to support staff.

HS Health and Safety HEA 3 The grantee maintains and monitors for effective oral health practices and nutrition services that

meet the nutritional needs and accommodate feeding requirements and allergies.

HS Health and Safety HEA 4 The grantee has implemented a process for monitoring and maintaining healthy and safe

environments. This includes ensuring all staff have complete background checks.

HS Health and Safety HEA 5 For programs serving pregnant women, the grantee provides and monitors for quality services

that facilitate enrolled pregnant women’s access to health care and provide information to

support prenatal, postpartum, maternal, and infant health and emotional well-being.

FS Fiscal FIS 1 The grantee develops and implements its budget to sustain management, staffing structures, and

the delivery of services that support the needs of enrolled children and families. This entails

relating financial data to accomplishments of the grant award and an awareness of program

progress, lessons learned, and needed improvements.

FS Fiscal FIS 2 The grantee plans and implements a fiscal management system that supports the organization’s

ongoing capacity to execute its budget over time and meet the needs of its organization.

FS Fiscal FIS 3 The grantee’s financial management system provides for effective control over and accountability

for all funds, property, and other assets.

FS Fiscal FIS 4 The grantee complies with application, prior approval, and reporting requirements for facilities

purchased, constructed, or renovated with Head Start funds.

ERSEA ERSA ERSEA 1 The grantee enrolls children or pregnant women who are categorically eligible or who meet

defined income-eligibility requirements.

ERSEA ERSA ERSEA 2 At least 10% of the grantee’s total funded enrollment is filled by children eligible for services

under IDEA or the grantee has received a waiver.

ERSEA ERSA ERSEA 3 The grantee maintains and tracks enrollment for all participants, including pregnant women.

SYS System Level Assessment Sys_1 A working System wide PM

SYS System Level Assessment Sys_2 A second working System-wide PM

PM Scoring

Overview:

This sheet presents a proposed final scoring for Performance Measures based upon the values of its component Quality Indicators.

This current approach assumes each PM will eventually be rated on a five point scale.

The approach presented here expands upon previous versions in one imporant way: it allows for differential QI contributions to given PMs.

Background

Each PM is comprised of multiple Quality Indicators, and combines these result into a final PM Score.

Prior FA reviews rated Performance Measures on a similar scale, but these scores were based upon standards.

PMs are scored on a scale of 1 to 5. However, this is an ordinal and not cardinal scale. The values represent relative ranks tieds to specific definitions, but do not represent points on a continuum

A given QI may contribute to different PMs differently. For some it may be very important (i.e. essential). For others, it may be only moderately or minimally important (i.e. non-essential).

Definition and Interpretation of PM Scores

PM Table 1 lists final PM values, prior definitions used in FA1 and 2 Reviews, and proposed revisions.

These proposed definitions have been changed considerably to address the difference between a PM's essential and non-essential QI

Several comments regarding the proposed Revisions should be made regarding how different QIs can contribute differently to different PMs

1) Several qualifying terms appear in the Proposed Revision. Examples are: Severe, Expected, Exceeded

2) The proposed revision is driven by a separation between Essential and Non-Essential Quality Indicators. This is discussed in detail in a later section.

3) The Proposed Revison to the definitions was developed based upon examining the universe of possible permutations considering different Quality Indicators levels. These levels are discussed in a later section.

4) Depending upon the level of flexibility regarding the QI:PM relationship we want to build into the system, the proposed Revisions may be too specific/precise.

Final Value

1

2

3

4

5

Derivation of PM ScoresOverview:

This approach is intended to meet three design requirements:

The first is that Qis are all scored on a 5-point scale as presented in the QI-Coding Range Tab.

The second is that PMs are scored on a 5 point ordinal scale with an interpretation similar to that provided in PM Table 1 (above).

The third is that a QI can, and likely will, have different importance to different PMs. NOTE: The essential/non-essential wording of the proposed revised definitions in PM Table 1.

Based upon these, it is possible to construct an approach to QI aggregation to the PM level that can be consistently applied regardless the PM's content area, the number of QIs that comprise that PM, and the different importance different QI's have to that PM

Basic Approach to QI aggregation to the PM levelThe first means each PM will have component QIs that have a range of values. However, the second that range will be limited.

More specifically as illustrated in PM Table 2, there are fifteen possible ranges of QI Scores for any given PM.

Assuming all QIs have equal importance to a PM, the proposed definitions of the middle collum of PM Table a can be used to translate QI ranges (left-most collumn of PM Table 2) to Final PM Scores (right-most collumn)This process begins with identification of five "anchor points". An anchor point is where all Quality Indicators are equal, and sets the PM's values exactly. Theseare shaded. Other QI Ranges listed first by lowest QI value and then greatest QI value.

Following a strict interpretation of the definitions proposed in the center collumn of PM Table 1, each PM's value is determined by it's lowest QI value. Note: Alternative approaches to determining PM values are also possible. Examples are the average QM score, the Median Score, or a weighted approach.

The program meets compliance; however there are one or more areas that need

improvement

The program is compliant

All Quality Indicators indicate requirements are exceeded.The program is meeting compliance at a higher level than most grantees. Innovations

or promising practices make the grantee exceptional in this area

All essential Quality Indicators exceed their expected level of performance, AND all non-essential Quality

Indicators are compliant with many exceeding their expected level of performance.

All Quality Indicators indicate the grantee is compliant and/or meets

expected levels of performance, but one or more have identified areas of

improvement

All Quality Indicators indicate the grantee is compliant and/or meets

expected levels of performance with possibly one or more indicating

requirements are exceeded.

One of three scenarios have occurred:

3.1) All essential Quality Indicators indicate compliance and/or the expected level of performance, at least

one identifies an area of improvement.

3.2) All essential Quality Indicators indicate compliance and/or exceed the expected level of performance,

BUT multiple non-essential Quality Indicators indicate severe or systematic issues and/or deficiencies.

3.3) At least one essential Quality Indicators indicates non-compliance and/or severe or systematic issues

and/or deficiencies, AND non-essential Quality Indicators generally indicate compliance and/or exceed their

expected level of performance.

All essential Quality Indicators are compliant and/or meet their expected level of performance with either:

4.1) at least one essential Quality Indicators exceeding their expected level of performance,

4.2) OR, nearly all non-essential Quality Indicators exceeding their expected level of performance.

PM Table 1: Performance Measure ScoresPrior RevisionCurrent Definition

The issue(s) in this area are either severe or systemic, recommended deficiency

The grantee has one or more area(s) of noncompliance

One or more Quality Indicators indicate severe or systemic issues and/or

deficiencies.

One or more Quality Indicators indicate an area or areas of

noncompliance and/or underperformance, but no Quality Indicators

indicates severe issues or deficiencies noted.

Proposed Revision

One or more essential Quality Indicators indicate severe or systemic issues and/or deficiencies while non-

essential Quality Indicators may be compliant or have achieved an expected level of performance.

One of two scenarios have occurred:

2.1) At least one essential Quality Indicators indicates an area or areas of noncompliance and/or

underperformance,

2.2) Multiple non-essential Quality Indicators indicate severe issues or deficiencies.

Range of QI Scores 1 2 3 4 5

Final PM

Scores

All 1's All 1*

1-2 Some 1

1-3 1

1-4 1

1-5 Some 1

All 2's All 2*

2-3 Some 2

2-4 2

2-5 Some 2

All 3's All 3*

3-4 Some 3

3-5 3

All 4's All 4*

4-5 Some 4All 5's All 5*

*Indicate clear 'anchor points' on PM Scoring Scale

Expanding to allow different levels of QI importance to a PM

Defining QI Priority for PMs

Table PM 2 illustrates how an number of QI values can be translated (i.e. aggregated) to a single 5-point PM score. However, this assumes all QIs have equal importance (i.e. weight) to the PM.

The third requirement implies QI aggregation allow for different QI weights. This is reflected in the essential versus non-essential phrasing in the proposed revised PM definitions (right-most collumn of PM Table 1) implies a similar differentiation of QIs.

To meet this requirement, a priority level needs to be assigned to each QI at the PM level. Then, a 'weight' reflecting that QI's relative importance can be applied during aggregation to the PM level.

PM Table 3 illustrates how QI priority levels can be declared and various corresponding sets of wieghts that can be used. Currently, there are three QI levels: Low, Moderate, and High. Each collumn to the left contains a potential set of QI weights that can be applied to each QI priority level.

Within the base case in this spreadsheet the left-most (doubling) option is used. NOTE: Other combinations can also be used if desired.

PM Table 3: QI Priority Levels and Potential WeightsPriority Level Equal Essential/Non-Essential Linear Doubling

InterpretationAll QIs have equal

importance

The categories of Low and Moderate have equal important, and

each of these QI values are 1/2 as important as those in the high

category

A Moderate QI is twice as important as

a Low QI, and a High is 3x's as

important as a Low.

Each level is twice as

important as the prior

level.

Low 1 1 1 1

Moderate 1 1 2 2

High 1 2 3 4

Total 3 4 6 7

Assigning PM specific weights to QI valuesTab CR2_PM_QI defines the linkages between PMs and QIs. PM Table 4 contains and excerpt corresponding to the PM, CLS2.

CLS 2 is comprised of 9 Qis. Three are given high priority two are moderate priority, and the last four low priority. Using the Doubling weight set, each QI is assigned a final wieght in the left most collumn.

In PM Table 4, the QI, QI_38, is given a high priority. However, QI_38 contributes to seven different PMs (PM Table 5) where is has high priority for 2, moderate priority for 3 and low priority for 2.

PM Table 4: PM to QI crosswalk and QI priority levelPM QI QI_Value QI_Level for PM Final Weight for PM

CLS 2 QI_38 1 High 4



CLS 2 QI_2 4 Moderate 2

CLS 2 QI_27 1 Moderate 2

CLS 2 QI_27 1 Low 1

CLS 2 QI_2 4 Low 1

CLS 2 QI_39 1 Low 1

CLS 2 QI_42 1 Low 1

PM Table 5: QI drilldown with different PM priority levelsPM QI QI_Value QI_Level for PM Final Weight for PM


ECD 1 QI_38 1 High 4

ECD 4 QI_38 1 Moderate 2

FIS 2 QI_38 1 Moderate 2

HEA 5 QI_38 1 Moderate 2

PMQ 3 QI_38 1 Low 1

Sys_1 QI_38 1 Low 1

PM Table 2: Quality Indicator Ranges and Proposed Final PM ValuesRange/Combination of Quality Indicator Values

Determining PM Scores across multiple QI levelsPM scores are determined through a two-step process. In the first step, the PM definitions contained in the middle collumn of PM Table 1 are applied to each QI Priority level to arrive at a within priority level PM score.

Then, in the second step, the weighted sum of Within Priority level PM scores is normalized (divided) by the sum of possible weights (set's Total from PM Table 3) and rounded to arrive at an Ordinal PM score.

This process is illustrated in PM Table 6 for three PMs.

The first PM in PM Table 6, CLS 2, has 9 contributing Qis. As indicated in PM Table 4, 3 are High priority, 2 are Moderate priority, and 4 are Low Priority. Starting with the Low Priority Qis, three has a value of 1 and one had a value or 4. Applying the

aggregation approach of PM Table 2, the results in a within priority level PM score of 1 for the Low Priority Qis. Doing the same for the two Moderate Priority Qis (1 and 4) results in another within priority level PM score of 1 as it does for the three High Priority QIs.

The weighted sum of these three scores is: 1*4 (High) + 1*2 (Moderate) + 1*1 (low) = 7, which is divided by 7 (sum of total weights) to arrive at a final PM score of 1.

The second line of PM Table 6, illustrates a PM (ERSEA 2) with 10 Qis (5 High, 2 Moderate, and 3 Low) with Within Priority Level PM Scores of 1, 2, and 2 respectively. This results in a normalized weighted sum of (1*4+2*2+2*1/7) of 1.667, which rounds to a PM score of 2.

Finally, the last line illustrates what happens when a PM has no Qis within a given priority level. In this case, that priority level is ignored and only those levels with Qis are considered. For ECD 4, this amounts to a normalized weighted sum of (1*4 + 1*2 / 6).

Where the denomonator of 6 results from excluding the Low Priority QI category.

PM Table 6: Calculation of PM ScoresHigh Moderate Low

PM ID PM Value PM_WithinLevel_Weighted_Average QI_Tot_Num Score Num 1 2 3 4 5 Score Num M1 M2 M3 M4 M5 Score Num L1 L2 L3 L4 L5

CLS 2 1 1 9 1 3 3 0 0 0 0 1 2 1 0 0 1 0 1 4 3 0 0 1 0

ERSEA 2 2 1.667 10 1 5 4 1 0 0 0 2 2 0 2 0 0 0 2 3 0 3 0 0 0

ECD 4 1 1 8 1 4 4 0 0 0 0 1 4 4 0 0 0 0 . 0 0 0 0 0 0

Considerations and ConclusionsTable PM 2 illustrates that with the proposed 5-point QI scale, there are 15 possible ranges for QI values within each quality level given there are at least two Qis within that level of a PM.

Expanding this to the three priority levels currently considered, results in 15^3 = 3,375 possible permutations of combined QI ranges, and adding additional level increases the combinations accordingly. For instance with four priority levels there would be 50,625 combinations.

Returning to the three-level approach contained here, we see that despite the large number of possibilities, this approach remains tractable and results in PM scores with clear interpretation.

This is illustrated in the Apdx_Permutations tab. There, all 3,375 permutations using the doubling weights are presented along with their resulting PM score. Using the available filters, one can see how many permutations correspond to an Anchor, and how many require interpretation.

Futher, by filtering on final PM Values, one can see how the Proposed Revisions to the PM Definitions (PM Table 1) capture the QI values across levels.

High Moderate Low Alternate Method (Discussed but not currently used)

PM ID PM Text PM Value PM_Weighted_Average QI_Tot_Num Score_High Number High 1 2 3 4 5 Score_Moderate Number_Moderate M1 M2 M3 M4 M5 Score_Low Number_Low L1 L2 L3 L4 L5

Total Wieghted QI Scores

(Numerator)

Total Number of Qis (Weighted)

(Demononator) Weighted Average for PMFinal PM Score (Alt Method)

CLS 2

Teaching practices intentionally promote

progress toward school readiness and

provide high-quality learning experiences for

children. 5 5 9 5 3 0 0 0 0 3 5 2 0 0 0 0 2 5 4 0 0 0 0 4 45 9 5 5

ECD 1

School readiness efforts align with the

expectations of receiving schools*, the Head

Start Early Learning Outcomes Framework

(HSELOF), and state early learning

standards. 5 5 10 5 4 0 0 0 0 4 5 5 0 0 0 0 5 5 1 0 0 0 0 1 50 10 5 5

ECD 3

The grantee ensures teachers are prepared

to implement the curriculum and support

children’s progress toward school readiness. 5 5 13 5 5 0 0 0 0 5 5 5 0 0 0 0 5 5 3 0 0 0 0 3 65 13 5 5

ECD 4

The grantee ensures home-based program

services provide home visits and group

socialization activities that provide high-

quality learning experiences. 5 5 8 5 4 0 0 0 0 4 5 4 0 0 0 0 4 . 0 0 0 0 0 0 40 8 5 5

ERSEA 1

The grantee enrolls children or pregnant

women who are categorically eligible or who

meet defined income-eligibility requirements. 5 5 12 5 6 0 0 0 0 6 5 4 0 0 0 0 4 5 2 0 0 0 0 2 60 12 5 5

ERSEA 2

At least 10% of the grantee’s total funded

enrollment is filled by children eligible for

services under IDEA or the grantee has

received a waiver. 5 5 10 5 5 0 0 0 0 5 5 2 0 0 0 0 2 5 3 0 0 0 0 3 50 10 5 5

ERSEA 3

The grantee maintains and tracks enrollment

for all participants, including pregnant

women. 5 5 9 5 4 0 0 0 0 4 5 3 0 0 0 0 3 5 2 0 0 0 0 2 45 9 5 5

FCE 1

The grantee collaborates with families to

support family well-being, parents’

aspirations, and parents’ life goals. 5 5 10 5 4 0 0 0 0 4 5 4 0 0 0 0 4 5 2 0 0 0 0 2 50 10 5 5

FCE 2

The grantee provides services that

strengthen parent-child relationships and

support parents in strengthening parenting

skills. 5 5 10 5 5 0 0 0 0 5 5 2 0 0 0 0 2 5 3 0 0 0 0 3 50 10 5 5

FCE 3

The grantee’s education and child

development services recognize parents’

roles as children’s lifelong educators and

encourage parents to engage in their child’s

education. 5 5 9 5 5 0 0 0 0 5 5 2 0 0 0 0 2 5 2 0 0 0 0 2 45 9 5 5

FIS 1

The grantee develops and implements its

budget to sustain management, staffing

structures, and the delivery of services that

support the needs of enrolled children and

families. This entails relating financial data to

accomplishments of the grant award and an

awareness of program progress, lessons

learned, and needed improvements. 5 5 9 5 4 0 0 0 0 4 5 4 0 0 0 0 4 5 1 0 0 0 0 1 45 9 5 5

FIS 2

The grantee plans and implements a fiscal

management system that supports the

organization’s ongoing capacity to execute its

budget over time and meet the needs of its

organization. 5 5 10 5 4 0 0 0 0 4 5 3 0 0 0 0 3 5 3 0 0 0 0 3 50 10 5 5

FIS 3

The grantee’s financial management system

provides for effective control over and

accountability for all funds, property, and

other assets. 5 5 11 5 4 0 0 0 0 4 5 4 0 0 0 0 4 5 3 0 0 0 0 3 55 11 5 5

FIS 4

The grantee complies with application, prior

approval, and reporting requirements for

facilities purchased, constructed, or

renovated with Head Start funds. 5 5 9 5 4 0 0 0 0 4 5 2 0 0 0 0 2 5 3 0 0 0 0 3 45 9 5 5

HEA 1The grantee has an approach to ensuring

delivery of high-quality health services. 5 5 9 5 3 0 0 0 0 3 5 4 0 0 0 0 4 5 2 0 0 0 0 2 45 9 5 5

HEA 2

The grantee supports a program-wide

culture that promotes mental health and

social and emotional well-being, and uses

mental health consultation to support staff. 5 5 13 5 5 0 0 0 0 5 5 2 0 0 0 0 2 5 6 0 0 0 0 6 65 13 5 5

HEA 3

The grantee maintains and monitors for

effective oral health practices and nutrition

services that meet the nutritional needs and

accommodate feeding requirements and

allergies. 5 5 9 5 4 0 0 0 0 4 5 3 0 0 0 0 3 5 2 0 0 0 0 2 45 9 5 5

HEA 4

The grantee has implemented a process for

monitoring and maintaining healthy and safe

environments. This includes ensuring all

staff have complete background checks. 5 5 10 5 4 0 0 0 0 4 5 3 0 0 0 0 3 5 3 0 0 0 0 3 50 10 5 5

HEA 5

For programs serving pregnant women, the

grantee provides and monitors for quality

services that facilitate enrolled pregnant

women’s access to health care and provide

information to support prenatal, postpartum,

maternal, and infant health and emotional

well-being. 5 5 8 5 2 0 0 0 0 2 5 3 0 0 0 0 3 5 3 0 0 0 0 3 40 8 5 5

PDM 1

The grantee’s program structure and design

is informed by the community’s strengths and

needs. 5 5 9 5 3 0 0 0 0 3 5 3 0 0 0 0 3 5 3 0 0 0 0 3 45 9 5 5

PMQ 1

The grantee establishes a management

structure that consists of staff, consultants,

or contractors who ensure high-quality

service delivery; have sufficient knowledge,

training, experience, and competencies to

fulfill the roles and responsibilities of their

positions; and provide regular supervision

and support to staff. 5 5 10 5 5 0 0 0 0 5 5 2 0 0 0 0 2 5 3 0 0 0 0 3 50 10 5 5

PMQ 2

The grantee uses data to identify program

strengths, needs, and areas needing

improvement; to evaluate progress toward

achieving program goals and compliance

with program performance standards; and to

assess the effectiveness of professional

development. 5 5 9 5 3 0 0 0 0 3 5 5 0 0 0 0 5 5 1 0 0 0 0 1 45 9 5 5

PMQ 3

The grantee maintains a formal structure of

program governance to oversee the quality of

services for children and families, and to

make decisions related to program design

and implementation. 5 5 8 5 3 0 0 0 0 3 5 1 0 0 0 0 1 5 4 0 0 0 0 4 40 8 5 5

Sys_1 A working System wide PM 5 5 8 5 3 0 0 0 0 3 5 2 0 0 0 0 2 5 3 0 0 0 0 3 40 8 5 5

Sys_2 A second working System-wide PM 5 5 7 5 2 0 0 0 0 2 5 3 0 0 0 0 3 5 2 0 0 0 0 2 35 7 5 5

QI ID QI Text QI_Value QM_Tot_Num Boolean_Rating Range_Rating QM_Boolean_Required_Num QM_Boolean_Required_Sum Boolean Required Rating QM_Boolean_Preferred_Num QM_Boolean_Preferred_Sum Boolean Preferred Rating Column4 Column5 QM_Range_Required_Num QM_Range_Required_1 QM_Range_Required_2 QM_Range_Required_3 Range_Required_Rating QM_Range_Preferred_Num QM_Range_Preferred_1 QM_Range_Preferred_2 QM_Range_Preferred_3 Range_Preferred_Rating

QI_1 QI_Text 1 5 13 5 5 5 5 3 2 2 3 5 0 0 5 5 1 0 0 0 5

QI_2 QI_Text 2 5 7 5 5 1 1 3 3 3 3 1 0 0 1 5 2 0 0 0 5

QI_3 QI_Text 3 5 7 5 5 5 5 3 1 1 3 1 0 0 1 5 0 0 0 0 5

QI_4 QI_Text 4 5 11 5 5 3 3 3 4 4 3 1 0 0 1 5 3 0 0 0 5

QI_5 QI_Text 5 5 15 5 5 4 4 3 6 6 3 3 0 0 3 5 2 0 0 0 5

QI_6 QI_Text 6 5 11 5 5 1 1 3 6 6 3 3 0 0 3 5 1 0 0 0 5

QI_7 QI_Text 7 5 10 5 5 3 3 3 4 4 3 3 0 0 3 5 0 0 0 0 5

QI_8 QI_Text 8 5 11 5 5 6 6 3 1 1 3 1 0 0 1 5 3 0 0 0 5

QI_9 QI_Text 9 5 14 5 5 6 6 3 4 4 3 1 0 0 1 5 3 0 0 0 5

QI_10 QI_Text 10 5 13 5 5 4 4 3 5 5 3 1 0 0 1 5 3 0 0 0 5

QI_11 QI_Text 11 5 6 5 5 0 0 3 1 1 3 3 0 0 3 5 2 0 0 0 5

QI_12 QI_Text 12 5 9 5 5 1 1 3 1 1 3 1 0 0 1 5 6 0 0 0 5

QI_13 QI_Text 13 5 9 5 5 2 2 3 1 1 3 2 0 0 2 5 4 0 0 0 5

QI_14 QI_Text 14 5 14 5 5 4 4 3 2 2 3 4 0 0 4 5 4 0 0 0 5

QI_15 QI_Text 15 5 7 5 5 0 0 3 1 1 3 5 0 0 5 5 1 0 0 0 5

QI_16 QI_Text 16 5 14 5 5 2 2 3 0 0 3 5 0 0 5 5 7 0 0 0 5

QI_17 QI_Text 17 5 8 5 5 1 1 3 0 0 3 4 0 0 4 5 3 0 0 0 5

QI_18 QI_Text 18 5 9 5 5 1 1 3 1 1 3 1 0 0 1 5 6 0 0 0 5

QI_19 QI_Text 19 5 6 5 5 0 0 3 0 0 3 3 0 0 3 5 3 0 0 0 5

QI_20 QI_Text 20 5 15 5 5 2 2 3 2 2 3 4 0 0 4 5 7 0 0 0 5

QI_21 QI_Text 21 5 10 5 5 2 2 3 2 2 3 3 0 0 3 5 3 0 0 0 5

QI_22 QI_Text 22 5 8 5 5 1 1 3 3 3 3 3 0 0 3 5 1 0 0 0 5

QI_23 QI_Text 23 5 9 5 5 1 1 3 1 1 3 2 0 0 2 5 5 0 0 0 5

QI_24 QI_Text 24 5 7 5 5 1 1 3 2 2 3 2 0 0 2 5 2 0 0 0 5

QI_25 QI_Text 25 5 6 5 5 0 0 3 1 1 3 3 0 0 3 5 2 0 0 0 5

QI_26 QI_Text 26 5 11 5 5 5 5 3 3 3 3 2 0 0 2 5 1 0 0 0 5

QI_27 QI_Text 27 5 6 5 5 3 3 3 2 2 3 0 0 0 0 5 1 0 0 0 5

QI_28 QI_Text 28 5 9 5 5 3 3 3 3 3 3 3 0 0 3 5 0 0 0 0 5

QI_29 QI_Text 29 5 12 5 5 7 7 3 2 2 3 2 0 0 2 5 1 0 0 0 5

QI_30 QI_Text 30 5 10 5 5 3 3 3 3 3 3 0 0 0 0 5 4 0 0 0 5

QI_31 QI_Text 31 5 7 5 5 3 3 3 2 2 3 1 0 0 1 5 1 0 0 0 5

QI_32 QI_Text 32 5 10 5 5 5 5 3 3 3 3 1 0 0 1 5 1 0 0 0 5

QI_33 QI_Text 33 5 12 5 5 4 4 3 3 3 3 2 0 0 2 5 3 0 0 0 5

QI_34 QI_Text 34 5 10 5 5 2 2 3 4 4 3 3 0 0 3 5 1 0 0 0 5

QI_35 QI_Text 35 5 8 5 5 1 1 3 3 3 3 1 0 0 1 5 3 0 0 0 5

QI_36 QI_Text 36 5 13 5 5 0 0 3 1 1 3 3 0 0 3 5 9 0 0 0 5

QI_37 QI_Text 37 5 14 5 5 3 3 3 3 3 3 3 0 0 3 5 5 0 0 0 5

QI_38 QI_Text 38 5 7 5 5 1 1 3 1 1 3 5 0 0 5 5 0 0 0 0 5

QI_39 QI_Text 39 5 7 5 5 2 2 3 0 0 3 3 0 0 3 5 2 0 0 0 5

QI_40 QI_Text 40 5 12 5 5 2 2 3 2 2 3 3 0 0 3 5 5 0 0 0 5

QI_41 QI_Text 41 5 12 5 5 2 2 3 1 1 3 4 0 0 4 5 5 0 0 0 5

QI_42 QI_Text 42 5 11 5 5 1 1 3 1 1 3 3 0 0 3 5 6 0 0 0 5

QI_43 QI_Text 43 5 10 5 5 1 1 3 1 1 3 5 0 0 5 5 3 0 0 0 5

QI_44 QI_Text 44 5 13 5 5 0 0 3 2 2 3 2 0 0 2 5 9 0 0 0 5

QI_45 QI_Text 45 5 11 5 5 2 2 3 2 2 3 4 0 0 4 5 3 0 0 0 5

QI_46 QI_Text 46 5 12 5 5 1 1 3 3 3 3 4 0 0 4 5 4 0 0 0 5

QI_47 QI_Text 47 5 8 5 5 0 0 3 1 1 3 3 0 0 3 5 4 0 0 0 5

QI_48 QI_Text 48 5 7 5 5 0 0 3 2 2 3 2 0 0 2 5 3 0 0 0 5

QI_49 QI_Text 49 5 11 5 5 0 0 3 4 4 3 4 0 0 4 5 3 0 0 0 5

QI_50 QI_Text 50 5 8 5 5 2 2 3 1 1 3 3 0 0 3 5 2 0 0 0 5

QI Coding

Overview:

This sheet derives a potential final scoring for Quality Indicators. This approach is intend to provide and example of how the two types of Quality Indicators currently included (Boolean (Yes/No) and Range (1,2,3) )

can be combined into a final Quality Indicator Score that ranges between 1 to 5. However, this is an ordinal and not cardinal scale. The values represent relative ranks tieds to specific definitions, but do not represent points on a continuum.

Background

This illustration current considers two types of Quality Markers as described in the Overview sheet. These are discussed below.

Boolean or Yes/No Quality Markers

This type of Quality Marker measures if a grantee does, or does not, have a certain system, approach, qualification, etc. In other words, they capture existance and have a value of Yes (having) and No (not having).

For clarity, we refer to these as Boolean QMs throughout the rest of the spreadsheet..

A Quality Indicators may have an unlimited number of Boolean QMs. However, regardless of number, all Boolean QMs can combined into a multi-point scale such as that represented in the following table (Table QI-1).

Table QI-1a: Combining Multiple Compliance Boolean QMsBoolean Ratings Scale Value

Multiple No's 1

One No 2All Yes 3

Range Quality Indicators

Quality Markers that attempt to guage a grantee attribute on a more continuous scale are referred to as 'Range QMs'.

Range QMs, whose final scale is currently under development, are assumed rated on the 3-point scale (Fails to Meet, Meets, Exceeds) in this spreadsheet/illustration. However, that is not a requirement, and the approach can be extended to other scales.

A Quality Indicators may have an unlimited number of Range QMs. However, like Boolean QMs, Range QMs can be combined into a single multi-point scale like that presented in Table QI-2.

Table QI-2: Combining Multiple Range QMsRange Ratings Scale Value

All 1's 1

At least one 1 2

All 2's 3

No 1's, at lease one 2 4All 3's 5

Combining Boolean and Range Indicators

At this point, it is important to note that while this example is limited to two types of Quality Markers, that is not a limitation of this approach.

The approach discussed below can be scaled to include additional types of Quality Markers that may be either measured differently or considered differently during aggregation to the Quality Indicator level.

Two key things are needed to add additional types of Quality Markers:

1) A clear definition of how that new type of Quality Marker will be measured, and

2) A clear definition of how multiple quality markers of that new type can be combined to contribute to a Quality Indicator's score.

The following illustration a potenital way a mixed set of Boolean and Range Quality Markers can be combined to arrive at a final Quality Indicator score.

This process starts with clearly defining how each point on the Quality Indicators five-point scale is defined. These values and corresponding definitions are presented in the two left-most collumns of Table QI-3.

The lower value of 1 indicates severe and systematic issues, the highest excellance in all areas, and a value of 3 compliance in all areas with clear potential for improvement.

These are translated into combinations of the Boolean and Range Criteria scores (Tables QI-1 and 2) by first considering possible values from the Boolean QMs and then from the Range QMs.

The set labeled, Boolean Driven, focuses upon combinations mainly defined by the Boolean criteria. Failing to meet multiple of these QMs means multiple issues and the corresponding QI should not have value of one regardless of the Range Qis values.

Similarly, if only one Boolean is not met, then at most one Range QM can be found to be below expectations. Finally, if all Booleans are met, then the QI's value is driven by its Range QMs.

When multiple Range QMs are below expectations, multiple issues are present and the QI is scored as a one. If only one Range QM is below expectations, a score of 2 is possible as long of only one Boolean is not met.

Finally, QI scores of 3, 4 and 5 are determined be the mix of Range QMs with values of 2 and 3 assuming all Booleans are met.

Table QI-3: Quality Marker Scores and Associated Combinatorial Sets of Boolean and Range QisQuality Marker Scores Logical

Final Value Interpretation Boolean Criteria Value Logical Range Criteria Value OR Boolean

Criteria

Value Logical Range Criteria Value

1 Indicators show either severe or systemic issues and/or deficiencies Multiple No's 1 AND Any 1 - 5 OR Any 1 - 3 AND Multiple 1's 1

2 Indicators identify one or more area(s) of noncompliance and/or

noted underperformance

One No 2 AND At most one 1 2 - 5 OR One No

All Yes

2 or 3 AND At least one 1 2

3 Indicators show compliance and/or expected levels of performance

are achieved; however there are one or more indicators indicator

areas of improvement

OR All Yes 3 AND All 2's 3

4 All indicators show compliance and/or expected levels of

performance with possibly one or more indicators indicating

requirements are exceeded

OR All Yes 3 AND No 1's, at

lease one 2

4

5 All indicators note compliance and/or that expectations are

exceeded indicating innovations and/or promising practices.

All Yes 3 AND All 3's 5 OR All Yes 3 AND All 3's 5

All Boolan and Range combinations and Final PM Scores.

Reference 6 lists all possible combinations of Boolean and Range scales along the the final associated value. In the table, the right-most collumns is a reference code to pull the final value for further calculation.

Note 1: This approach, in general, and this crosswalk, in particular, can be expanded to include additional types of Quality Meatrics. Their addition would simply result in additional collumns and permutations.

Note 2: AS indicated by the Final Value collumn of Reference 6, the method for combining the two types of Quality Markers shown in Table QI-3 is rather strict in that it results in many ways for a QI to be scored a 1, but fewer and fewer to earn a 2,3,4, and 5

Reference 6: Quality Measures Reference CrossWalk*QM Boolean/Range Reference Bolean Range Final Value

11 1 1 1

12 1 2 1

13 1 3 1

14 1 4 1

15 1 5 1

21 2 1 1

22 2 2 1

23 2 3 1

24 2 4 1

25 2 5 2

31 3 1 1

32 3 2 2

33 3 3 3

34 3 4 435 3 5 5

SET ONE: Boolean Driven SET TWO: Range Driven

See Range Driven

See Range Driven

QI Coding

This sheet derives a potential final scoring for Quality Indicators. This approach is intend to provide and example of how the two types of Quality Indicators currently included (Boolean (Yes/No) and Range (1,2,3) )

can be combined into a final Quality Indicator Score that ranges between 1 to 5. However, this is an ordinal and not cardinal scale. The values represent relative ranks tieds to specific definitions, but do not represent points on a continuum.

Background

This illustration current considers two types of Quality Markers as described in the Overview sheet. These are discussed below.

Boolean or Yes/No Quality Markers

This type of Quality Marker measures if a grantee does, or does not, have a certain system, approach, qualification, etc. In other words, they capture existance and have a value of Yes (having) and No (not having).

For clarity, we refer to these as Boolean QMs throughout the rest of the spreadsheet..

A Quality Indicators may have an unlimited number of Boolean QMs. However, regardless of number, all Boolean QMs can combined into a multi-point scale such as that represented in the following table (Table QI-1).

Table QI-1a: Combining Multiple Compliance Boolean QMsBoolean Ratings Scale Value

Multiple No's 1

One No 2All Yes 3

Proposed Definition Scale Value Required+* Preferred*

1 1

1 2

1 3

2 1

2 2

2 3

All Required QMs are met, but multiple preferred QMs are not 3 3 1

All Required QMs are met, but at least one preferred QM is not 4 3 2

All QMs are met 5 3 3

Reference 3: QM_Scoring_RangeScale Rating Rating Scale

Fails to Meet Requirements 1

Meets Requirements 2

Exceeds Requirements 3

Table QI-2a: Combining Multiple Range QMsRange Ratings Scale Value

All 1's 1

At least one 1 2

All 2's 3

No 1's, at lease one 3 4All 3's 5

Proposed Definition Scale Value Required+* Preferred*

1 1

1 2

1 3

1 4

1 5

2 1

2 2

2 3

2 4

2 5

All required QMs meet requirements, but no Preferred QMs do 3 1

All Required QMs meet requirements, but at least one Preferred QMs does not 3 2

All QMs meet requirements 3 3

All required QMs meet requirements, and at least one Preferred QMs exceeds requirements3 4

All required QMs meet requirements, and all Preferred QMs exceeds requirements** 3 5

All required QMs meet and at least one exceeds requirements, but no Preferred QMs do 4 1

All required QMs meet and at least one exceeds requirements, but at least one Preferred QMs

does not4 2

All Required QMs exceed requirements, but no Preferred QMs meet requirements 5 1

All required QMs meet and at least one exceeds requirements, and all Preferred QMs meet

requirements4 3

All required QMs meet and at least one exceeds requirements, and at least one Preferred QM

exceeds requirements4 4

All required QMs meet and at least one exceeds requirements, and all Preferred QMs exceed

requirements4 5

All Required QMs exceed requirements, but at least one Preferred QMs does not requirements5 2

All Required QMs exceed requirements, and all Preferred QMs meet requirements 5 3

All Required QMs exceed requirements, and at least one Preferred QMs exceeds requirements5 4

All QMs exceed requirements 5 5

Proposed Definition Scale Value+ Boolean* Range* Average

1 1 1

1 2 1.5

2 1 1.5

1 3 2

3 1 2

2 2 2

3 2 2.5

2 3 2.5

1 4 2.5

Overview:

No

Co

mp

lian

ce Is

sues

3

4

5

Table QI-3: Combining Boolean and Range QM for final QI Score

No

n-C

om

plia

nce

Ind

ica

ted

Multiple required QMs fail to meet requirements

(The grantee has multiple points of non-compliance and no markers of high quality)1

At least one required QM fails to meet requirements

(The grantee has at least one point of non-compliance while other markers may indicate high

quality)

2

No

Co

mp

lian

ce

Issu

es

+For this illustrations, Required PMs are assumed tied to an area of compliance (i.e. a standard)

*In the case where the Quality Indicator has no QMs of that type, a score of 3 for that type is assumed.

Table QI-2b: Combining Range QMs

No

n-C

om

plia

nce

Ind

ica

ted

All Required QMs fail to meet requirements 1

At least one required QM fails to meet requirements 2

+For this illustrations, Required Quality Markers are assumed tied to an area of compliance (i.e. a standard)

*In the case where the Quality Indicator has no QMs of that type, a score of 5 for that type is assumed.

**Value set to three to allow complete range when no Preferred QMs exist for the QI

Table QI-1b: Boolean Type Cross-Walk

No

n-C

om

plia

nce

Ind

ica

ted Multiple Required QMs are missing 1

One Required QM is Missing 2

4 1 2.5

1 5 3

2 4 3

4 2 3

5 1 3

2 5 3.5

5 2 3.5

3 3 3

3 4 3.5

4 3 3.5

4 4 4

3 5 4

5 3 4

4 5 4.5

5 4 4.5

5 5 5

+Within this illustration, required Quality Markers indicate a standard and a QI score of 3 cannot be achieved unless the grantee is compliance with all required QMs

*In the case where the Quality Indicator has no QMs of that type, that type (Boolean and/or Range) is ignored and the value of the included type is used

No

Co

mp

lian

ce Is

sues

All QMs meet requirements and one may exceed them

(The grantee is compliant with all requirements with, at most, one QM indicating high quality)3

All QMs requirements and multiple QMs exceed them

(The grantee has multiple markers of high quality)4

All QMs meets or exceed requirements

(The grantee shows excellence with nearly universal indications of high quality)5

No

n-C

om

plia

nce

Ind

ica

ted

At least one required QM fails to meet requirements

(The grantee has at least one point of non-compliance while other markers may indicate high

quality)

2

PMPM Text

QI QI Text2 QI_ValueQI_Level for PM Final Weight for PM

QI Contribution to PM

(alt aggregation method)

ECD 1

School readiness efforts align with the expectations of receiving schools*, the Head Start Early

Learning Outcomes Framework (HSELOF), and state early learning standards. QI_1 QI_Text 1 5 High 1 5

FCE 3

The grantee’s education and child development services recognize parents’ roles as children’s lifelong

educators and encourage parents to engage in their child’s education. QI_1 QI_Text 1 5 High 1 5

PDM 1 The grantee’s program structure and design is informed by the community’s strengths and needs. QI_1 QI_Text 1 5 High 1 5

ECD 1


Learning Outcomes Framework (HSELOF), and state early learning standards. QI_1 QI_Text 1 5 Moderate 1 5

ERSEA 3 The grantee maintains and tracks enrollment for all participants, including pregnant women. QI_1 QI_Text 1 5 Moderate 1 5

HEA 5

For programs serving pregnant women, the grantee provides and monitors for quality services that

facilitate enrolled pregnant women’s access to health care and provide information to support

prenatal, postpartum, maternal, and infant health and emotional well-being. QI_1 QI_Text 1 5 Moderate 1 5

ECD 4

The grantee ensures home-based program services provide home visits and group socialization

activities that provide high-quality learning experiences. QI_10 QI_Text 10 5 High 1 5


PDM 1 The grantee’s program structure and design is informed by the community’s strengths and needs. QI_10 QI_Text 10 5 Low 1 5

Sys_1 A working System wide PM QI_10 QI_Text 10 5 Low 1 5

HEA 5




PDM 1 The grantee’s program structure and design is informed by the community’s strengths and needs. QI_10 QI_Text 10 5 Moderate 1 5

FIS 2

The grantee plans and implements a fiscal management system that supports the organization’s

ongoing capacity to execute its budget over time and meet the needs of its organization. QI_11 QI_Text 11 5 High 1 5

PMQ 1

The grantee establishes a management structure that consists of staff, consultants, or contractors

who ensure high-quality service delivery; have sufficient knowledge, training, experience, and

competencies to fulfill the roles and responsibilities of their positions; and provide regular supervision

and support to staff. QI_11 QI_Text 11 5 High 1 5

ECD 3

The grantee ensures teachers are prepared to implement the curriculum and support children’s

progress toward school readiness. QI_11 QI_Text 11 5 Low 1 5

PMQ 1




and support to staff. QI_11 QI_Text 11 5 Low 1 5

PMQ 1




and support to staff. QI_11 QI_Text 11 5 Moderate 1 5

PMQ 2

The grantee uses data to identify program strengths, needs, and areas needing improvement; to


standards; and to assess the effectiveness of professional development. QI_12 QI_Text 12 5 High 1 5

Sys_2 A second working System-wide PM QI_12 QI_Text 12 5 High 1 5

PMQ 2



standards; and to assess the effectiveness of professional development. QI_12 QI_Text 12 5 Moderate 1 5

FCE 2

The grantee provides services that strengthen parent-child relationships and support parents in

strengthening parenting skills. QI_13 QI_Text 13 5 High 1 5

HEA 2

The grantee supports a program-wide culture that promotes mental health and social and emotional

well-being, and uses mental health consultation to support staff. QI_13 QI_Text 13 5 High 1 5

PMQ 3

The grantee maintains a formal structure of program governance to oversee the quality of services for

children and families, and to make decisions related to program design and implementation. QI_13 QI_Text 13 5 High 1 5

FCE 1

The grantee collaborates with families to support family well-being, parents’ aspirations, and parents’

life goals. QI_13 QI_Text 13 5 Low 1 5

ECD 3


progress toward school readiness. QI_13 QI_Text 13 5 Moderate 1 5

PMQ 3


children and families, and to make decisions related to program design and implementation. QI_13 QI_Text 13 5 Moderate 1 5

FIS 3

The grantee’s financial management system provides for effective control over and accountability for

all funds, property, and other assets. QI_14 QI_Text 14 5 High 1 5

FCE 3


educators and encourage parents to engage in their child’s education. QI_14 QI_Text 14 5 Low 1 5

FIS 1

The grantee develops and implements its budget to sustain management, staffing structures, and the

delivery of services that support the needs of enrolled children and families. This entails relating

financial data to accomplishments of the grant award and an awareness of program progress, lessons

learned, and needed improvements. QI_14 QI_Text 14 5 Low 1 5

FIS 1




learned, and needed improvements. QI_14 QI_Text 14 5 Moderate 1 5

FIS 2



ECD 3



FIS 2


ongoing capacity to execute its budget over time and meet the needs of its organization. QI_15 QI_Text 15 5 Low 1 5

FIS 1




learned, and needed improvements. QI_16 QI_Text 16 5 High 1 5

FIS 3



Sys_2 A second working System-wide PM QI_16 QI_Text 16 5 High 1 5

ECD 4


activities that provide high-quality learning experiences. QI_16 QI_Text 16 5 Moderate 1 5

FIS 3


all funds, property, and other assets. QI_16 QI_Text 16 5 Moderate 1 5

Sys_1 A working System wide PM QI_16 QI_Text 16 5 Moderate 1 5

Sys_2 A second working System-wide PM QI_16 QI_Text 16 5 Moderate 1 5

FIS 4

The grantee complies with application, prior approval, and reporting requirements for facilities

purchased, constructed, or renovated with Head Start funds. QI_17 QI_Text 17 5 High 1 5

ERSEA 1

The grantee enrolls children or pregnant women who are categorically eligible or who meet defined

income-eligibility requirements. QI_17 QI_Text 17 5 Low 1 5

FIS 4


purchased, constructed, or renovated with Head Start funds. QI_17 QI_Text 17 5 Low 1 5


FCE 1


life goals. QI_17 QI_Text 17 5 Moderate 1 5


CLS 2

Teaching practices intentionally promote progress toward school readiness and provide high-quality

learning experiences for children. QI_18 QI_Text 18 5 High 1 5

ERSEA 3 The grantee maintains and tracks enrollment for all participants, including pregnant women. QI_18 QI_Text 18 5 High 1 5

FCE 1


life goals. QI_18 QI_Text 18 5 High 1 5

FCE 1



FCE 2



FIS 3



HEA 1 The grantee has an approach to ensuring delivery of high-quality health services. QI_18 QI_Text 18 5 High 1 5

FCE 3


educators and encourage parents to engage in their child’s education. QI_18 QI_Text 18 5 Moderate 1 5

FIS 3



PMQ 2




FCE 2



FIS 4



PMQ 3



PMQ 3



PMQ 3


children and families, and to make decisions related to program design and implementation. QI_19 QI_Text 19 5 Low 1 5

FCE 2


strengthening parenting skills. QI_19 QI_Text 19 5 Moderate 1 5

HEA 3

The grantee maintains and monitors for effective oral health practices and nutrition services that meet

the nutritional needs and accommodate feeding requirements and allergies. QI_19 QI_Text 19 5 Moderate 1 5

ERSEA 1


income-eligibility requirements. QI_2 QI_Text 2 5 High 1 5

CLS 2


learning experiences for children. QI_2 QI_Text 2 5 Low 1 5

PMQ 3



CLS 2


learning experiences for children. QI_2 QI_Text 2 5 Moderate 1 5

HEA 2


well-being, and uses mental health consultation to support staff. QI_2 QI_Text 2 5 Moderate 1 5

ECD 4



FCE 3



FCE 3


educators and encourage parents to engage in their child’s education. QI_20 QI_Text 20 5 Low 1 5

ERSEA 1


income-eligibility requirements. QI_20 QI_Text 20 5 Moderate 1 5

ERSEA 1



ERSEA 1




ERSEA 2

At least 10% of the grantee’s total funded enrollment is filled by children eligible for services under

IDEA or the grantee has received a waiver. QI_22 QI_Text 22 5 Low 1 5

ERSEA 2


IDEA or the grantee has received a waiver. QI_22 QI_Text 22 5 Moderate 1 5

ECD 3


progress toward school readiness. QI_23 QI_Text 23 5 High 1 5


ERSEA 3 The grantee maintains and tracks enrollment for all participants, including pregnant women. QI_23 QI_Text 23 5 Low 1 5

HEA 3


the nutritional needs and accommodate feeding requirements and allergies. QI_23 QI_Text 23 5 Low 1 5

ERSEA 1



HEA 3


the nutritional needs and accommodate feeding requirements and allergies. QI_24 QI_Text 24 5 Low 1 5

ECD 3



FCE 2



Sys_1 A working System wide PM QI_25 QI_Text 25 5 High 1 5

HEA 2


well-being, and uses mental health consultation to support staff. QI_25 QI_Text 25 5 Low 1 5

FCE 1



Sys_1 A working System wide PM QI_25 QI_Text 25 5 Moderate 1 5

ECD 1



ECD 1



FIS 1





HEA 2



CLS 2



CLS 2


learning experiences for children. QI_27 QI_Text 27 5 Moderate 1 5

FCE 3


educators and encourage parents to engage in their child’s education. QI_27 QI_Text 27 5 Moderate 1 5

ECD 3



ECD 3



ERSEA 2


IDEA or the grantee has received a waiver. QI_28 QI_Text 28 5 High 1 5

ECD 3



ECD 4



ECD 1


Learning Outcomes Framework (HSELOF), and state early learning standards. QI_29 QI_Text 29 5 Low 1 5

ERSEA 1


income-eligibility requirements. QI_29 QI_Text 29 5 Low 1 5

ECD 3



ECD 4



FIS 1





ECD 3




ECD 3



HEA 5



prenatal, postpartum, maternal, and infant health and emotional well-being. QI_3 QI_Text 3 5 Low 1 5

PMQ 2



standards; and to assess the effectiveness of professional development. QI_3 QI_Text 3 5 Low 1 5

HEA 2


well-being, and uses mental health consultation to support staff. QI_3 QI_Text 3 5 Moderate 1 5

PMQ 1





HEA 1 The grantee has an approach to ensuring delivery of high-quality health services. QI_30 QI_Text 30 5 Low 1 5

HEA 1 The grantee has an approach to ensuring delivery of high-quality health services. QI_30 QI_Text 30 5 Moderate 1 5

HEA 2



FCE 2


strengthening parenting skills. QI_31 QI_Text 31 5 Low 1 5

FIS 4



HEA 2



HEA 2




HEA 3


the nutritional needs and accommodate feeding requirements and allergies. QI_32 QI_Text 32 5 High 1 5

FCE 1


life goals. QI_32 QI_Text 32 5 Low 1 5

Sys_2 A second working System-wide PM QI_32 QI_Text 32 5 Low 1 5

HEA 3



HEA 4

The grantee has implemented a process for monitoring and maintaining healthy and safe

environments. This includes ensuring all staff have complete background checks. QI_33 QI_Text 33 5 Low 1 5


HEA 4


environments. This includes ensuring all staff have complete background checks. QI_33 QI_Text 33 5 Moderate 1 5

PMQ 1




and support to staff. QI_33 QI_Text 33 5 Moderate 1 5

PMQ 2




ERSEA 2



HEA 5



prenatal, postpartum, maternal, and infant health and emotional well-being. QI_34 QI_Text 34 5 High 1 5

HEA 5




ECD 1



FIS 3



FIS 4


purchased, constructed, or renovated with Head Start funds. QI_34 QI_Text 34 5 Moderate 1 5

ECD 1





ERSEA 1



ERSEA 1



ERSEA 2



HEA 2



PMQ 1





PMQ 1





HEA 3



PMQ 2




FCE 2



PMQ 2




CLS 2



ECD 1



ECD 4



FIS 2


ongoing capacity to execute its budget over time and meet the needs of its organization. QI_38 QI_Text 38 5 Moderate 1 5

HEA 5




PMQ 3




ERSEA 1



FIS 1





CLS 2




FIS 1





ECD 4



ECD 4



ERSEA 2



FIS 2



Sys_2 A second working System-wide PM QI_40 QI_Text 40 5 Low 1 5

FIS 2



PMQ 1





FIS 3


all funds, property, and other assets. QI_41 QI_Text 41 5 Low 1 5

PMQ 1





FIS 3



FIS 4



PMQ 1





CLS 2



FIS 4



FCE 2


strengthening parenting skills. QI_42 QI_Text 42 5 Moderate 1 5

FCE 1



FIS 2



HEA 4


environments. This includes ensuring all staff have complete background checks. QI_43 QI_Text 43 5 High 1 5

FCE 1



FCE 2



FIS 3



FCE 2




CLS 2



FCE 3



FCE 3



HEA 4



ECD 3




FIS 4


purchased, constructed, or renovated with Head Start funds. QI_45 QI_Text 45 5 Moderate 1 5

FIS 1





HEA 3



PMQ 3



ECD 1



ERSEA 1



ERSEA 1



FCE 3



HEA 4



HEA 4



ERSEA 2



ERSEA 2



ERSEA 2


IDEA or the grantee has received a waiver. QI_47 QI_Text 47 5 Moderate 1 5



FCE 1



PMQ 2




FIS 3



ECD 1



ECD 3



FCE 1



HEA 1 The grantee has an approach to ensuring delivery of high-quality health services. QI_5 QI_Text 5 5 Low 1 5


PMQ 2




ERSEA 2



FIS 2



HEA 2



FIS 1





HEA 2




FIS 2



HEA 2



HEA 4



HEA 4




HEA 3



ERSEA 3 The grantee maintains and tracks enrollment for all participants, including pregnant women. QI_7 QI_Text 7 5 Low 1 5

FIS 2



HEA 3




HEA 2



HEA 4




HEA 4



FIS 4



HEA 5



prenatal, postpartum, maternal, and infant health and emotional well-being. QI_9 QI_Text 9 5 High 1 5


FIS 3



HEA 5




QM ID QM Text QI Scoring Rule QM_Value QM_Num

QM_1 This is test 1 Yes/No Yes 1










QM_11 This is test 11 Range (1-3) Exceeds Requirements 3


























































































QI QI Text Relationship QM QM Text2 QM_Scoring Rule QM_Value QM_Num

QI_1 QI_Text 1 Preferred QM_1 This is test 1 Yes/No Yes 1


QI_3 QI_Text 3 Required QM_3 This is test 3 Yes/No Yes 1








QI_11 QI_Text 11 Preferred QM_11 This is test 11 Range (1-3) Exceeds Requirements 3




QI_15 QI_Text 15 Required QM_15 This is test 15 Range (1-3) Exceeds Requirements 3






































































































































































































































































































































































































































































































Content

Area

# of

PMs Grantee Max MinAverage

PM Wt

Content

Area %

Weighted

PM Scores

% of Wgted

Sum

Scaled to

CA %

% of Final

Score

% of

Possible

Max Wtd PM

Scores

Max Scaled

to CA %

% of Final

Score

Max Scaled

to CA %

% of Final

Score

Quality Marker Simulation Overall Rating 25 101 101 -101 40.93 100.0% 40.93 100.0% 100.0% 40.93 40.93 100% 36.83 100%

Quality Indicator Weights ECD PMs 4 12.00 12.00 -12.00 22% 20% 4.32 10.5% 8.19 20.0% 100.0% 4.32 8.19 20% 8.19 22%

Baseline or Rand Findings

Impacts PAM Program Design PMs 4 33.00 33.00 -33.00 74% 15% 14.70 35.9% 6.14 15.0% 100.0% 14.70 6.14 15% 6.14 17%

Importance or Prevalence

Impacts PAM

Family and Community

Engagement PMs3 13.00 13.00 -13.00 21% 15% 3.17 7.7% 6.14 15.0% 100.0% 3.17 6.14 15% 6.14 17%

Baseline or Rand CA %'s

Impacts PM Weights Health and Safety PMs 5 16.00 16.00 -16.00 19% 15% 4.85 11.8% 6.14 15.0% 100.0% 4.85 6.14 15% 6.14 17%

Fiscal 4 23.00 23.00 -23.00 32% 15% 6.49 15.9% 6.14 15.0% 100.0% 6.49 6.14 15% 6.14 17%

ERSEA PMs 3 4.00 4.00 -4.00 16% 10% 2.41 5.9% 4.09 10.0% 100.0% 2.41 4.09 10% 4.09 11%

System PMs 2 18.00 18.00 -18.00 50% 10% 5.00 12.2% 4.09 10.0% 100.0% 5.00 4.09 10% 4.09 11%

ScorePAM

ScoreMax Min Score Weight

PM # 12.00 12.00 -12.00 4.32 0.20

CLS 2 5 6.00 6.00 -6.00 5.00 0.29

ECD 1 5 1.00 1.00 -1.00 5.00 0.20

ECD 3 5 4.00 4.00 -4.00 5.00 0.25

ECD 4 5 1.00 1.00 -1.00 5.00 0.12

ScorePAM


PM # 33.00 33.00 -33.00 14.70 0.15

PDM 1 5 9.00 9.00 -9.00 5.00 0.61

PMQ 1 5 9.00 9.00 -9.00 5.00 1.00

PMQ 2 5 7.00 7.00 -7.00 5.00 0.99

PMQ 3 5 8.00 8.00 -8.00 5.00 0.34

ScorePAM


PM # 13.00 13.00 -13.00 3.17 0.15

FCE 1 5 5.00 5.00 -5.00 5.00 0.21

FCE 2 5 4.00 4.00 -4.00 5.00 0.20

FCE 3 5 4.00 4.00 -4.00 5.00 0.22

ScorePAM


PM # 16.00 16.00 -16.00 4.85 0.15

HEA 1 5 1.00 1.00 -1.00 5.00 0.20

HEA 2 5 4.00 4.00 -4.00 5.00 0.22

HEA 3 5 2.00 2.00 -2.00 5.00 0.16

HEA 4 5 6.00 6.00 -6.00 5.00 0.26

HEA 5 5 3.00 3.00 -3.00 5.00 0.13

ScorePAM


PM # 23.00 23.00 -23.00 6.49 0.15

FIS 1 5 8.00 8.00 -8.00 5.00 0.56

FIS 2 5 8.00 8.00 -8.00 5.00 0.36

FIS 3 5 4.00 4.00 -4.00 5.00 0.29

FIS 4 5 3.00 3.00 -3.00 5.00 0.09

ScorePAM


PM # 4.00 4.00 -4.00 2.41 0.10

ERSEA 1 5 1.00 1.00 -1.00 5.00 0.19

ERSEA 2 5 1.00 1.00 -1.00 5.00 0.13

ERSEA 3 5 2.00 2.00 -2.00 5.00 0.16

ScorePAM


PM # 18.00 18.00 -18.00 5.00 0.10

Sys_1 5 9.00 9.00 -9.00 5.00 0.50

Sys_2 5 9.00 9.00 -9.00 5.00 0.50

Baseline

Simulation/QM Information

Max_Score

Baseline

Importance

Equal

PAM PM Weighting

Early Childhood Development

Performance Measure

School readiness efforts align with the expectations of receiving

schools*, the Head Start Early Learning Outcomes Framework

(HSELOF), and state early learning standards.

Teaching practices intentionally promote progress toward school

readiness and provide high-quality learning experiences for

children.

The grantee ensures teachers are prepared to implement the

curriculum and support children’s progress toward school

readiness.

The grantee ensures home-based program services provide

home visits and group socialization activities that provide high-

quality learning experiences.

PAM PM Weighting

Program Design

Performance Measure

The grantee’s program structure and design is informed by the

community’s strengths and needs.

The grantee establishes a management structure that consists

of staff, consultants, or contractors who ensure high-quality

service delivery; have sufficient knowledge, training, experience,

and competencies to fulfill the roles and responsibilities of their

positions; and provide regular supervision and support to staff.

The grantee uses data to identify program strengths, needs, and

areas needing improvement; to evaluate progress toward

achieving program goals and compliance with program

performance standards; and to assess the effectiveness of

professional development.The grantee maintains a formal structure of program

governance to oversee the quality of services for children and

families, and to make decisions related to program design and

PAM PM Weighting

Family and Community Engagement

Performance Measure

The grantee collaborates with families to support family well-

being, parents’ aspirations, and parents’ life goals.

The grantee provides services that strengthen parent-child

relationships and support parents in strengthening parenting

skills.

The grantee’s education and child development services

recognize parents’ roles as children’s lifelong educators and

encourage parents to engage in their child’s education.

PAM PM Weighting

PAM PM Weighting

Health & Safety

Performance Measure

The grantee has an approach to ensuring delivery of high-quality

health services.

The grantee supports a program-wide culture that promotes

mental health and social and emotional well-being, and uses

mental health consultation to support staff.

The grantee maintains and monitors for effective oral health

practices and nutrition services that meet the nutritional needs

and accommodate feeding requirements and allergies.

The grantee develops and implements its budget to sustain

management, staffing structures, and the delivery of services

that support the needs of enrolled children and families. This

entails relating financial data to accomplishments of the grant

award and an awareness of program progress, lessons learned,

The grantee plans and implements a fiscal management system

that supports the organization’s ongoing capacity to execute its

budget over time and meet the needs of its organization.

The grantee’s financial management system provides for

effective control over and accountability for all funds, property,

and other assets.

The grantee complies with application, prior approval, and

The grantee has implemented a process for monitoring and

maintaining healthy and safe environments. This includes

ensuring all staff have complete background checks.

For programs serving pregnant women, the grantee provides

and monitors for quality services that facilitate enrolled pregnant

Fiscal

PM Weighting

Performance Measure

A working System wide PM

A second working System-wide PM

PAM

At least 10% of the grantee’s total funded enrollment is filled by

children eligible for services under IDEA or the grantee has

received a waiver.

The grantee maintains and tracks enrollment for all participants,

including pregnant women.

PAM PM Weighting

System

PAM PM Weighting

ERSEA

Performance Measure

The grantee enrolls children or pregnant women who are

categorically eligible or who meet defined income-eligibility

requirements.

Performance Measure

-40.00

-30.00

-20.00

-10.00

0.00

10.00

20.00

30.00

40.00

ECD PMs Program Design PMs Family andCommunity

Engagement PMs

Health and SafetyPMs

Fiscal ERSEA PMs System PMs

PAM Grantee Performance Score

ECD PMs, 20%

Program Design PMs, 15%

Family and Community Engagement PMs, 15%

Health and Safety PMs, 15%

Fiscal, 15%

ERSEA PMs, 10%

System PMs, 10%

PERCENT OF GRANTEE'S SCORE

0.00

1.00

2.00

3.00

4.00

5.00

6.00

7.00

8.00

9.00

ECD PMs Program Design PMs Family andCommunity

Engagement PMs

Health and SafetyPMs

Fiscal ERSEA PMs System PMs

PM Weighting Domain Specific Grantee Performance Score

ECD PMs

Program Design PMs

Family and Community Engagement PMs

Health and Safety PMsFiscal

ERSEA PMs

System PMs

ECD PMs

Program Design PMs

Family and Community EngagementPMs

Health and Safety PMsFiscal

ERSEA PMs

System PMs

Decomposition of PM Scoring ApproachRelationahip of PM Weights, Content Area Wieghts, and Final Scores

Average PM Wt

% of Wgted Sum

% of Final Score

PM Weighting Scoring Calculator SimulationOverview:

This describes the simulated example of the PM Weighting Scoring Approach

The PM Weighting Scoring Approach is based upon the following principles:

1. Each performance measure (PM) contributes to overall Grantee performance..

2. Grantee PM Score that rated on a scale of 1-5.

3. When determining grantee performance, different PMs contribute differently

NOTE: These different contributions are expressed in terms of weighted PM scores

NOTE: The initial set of PM weights were determined by a ranking of PMs by key OHS stakeholders

NOTE: The value of these initial PM weights are based upon the median rank of each PM across all surveyed stakeholders

4. Different content areas may also contribute differently to grantee performance.

Note: The relative importance of each content area is expressed in terms of what percentage of a grantee's overall score is comprised of that content area.

Note: Each content area's percentage is used to scale the PM scores within that content area

Note: This creates of two-tier weighting system (PM weights and content areas). Mathematically, this means that PM weights are relative within each content area and do not apply across content areas.

Note: The percentages for each content area currently arbitrarily set in the Baseline version, based upon surveys PMs weighted in the Compound version, equal in the Equal version or randomly assigned in the Random version.

NOTE: Content area percentages should all sum to 1.

Combined, these principles result in a final score that is a weighted sum of weighted PM scores. Specifically, the PM weights determine each PM's contribution with its content area

Then, the content area's percentage determine how much of a grantee's final score is comprised of that content areas weighted sum.

NOTE: Mathematically, this means a PM's weight influences with but not across content areas. Specifically, the greater the PM's weight relative to other PM weights within its content area, the greater that content area's

percentage contribution is comprised of that PM. However, the maximum possible contribution of the PM is bound by the content area's percentage. This is illustrated in the Direct Weight sheet.

Methodology applied in this document for Calculating a score

This numerical example applies the approach discussed above and limits calculation to only the use of PM values. These values are contained in Table 1 of the Input_Tables page.

This example is based upon the example originally developed in August intended to illustrate how grantee PMs could be used to developed a 5-star rating system

The key calculation of that illustration have been retained on the Direct Weight sheet, but not carried forward.

Grantee Scorecard

This sheet illustrates grantee performance could be presented across different content areas and accumulate in an overall score..

This presentation presents this information as both an absolute score as well as a percentage of the total possible score within each content area and overall.

To illustrate overall and with content area grantee performance, two radar plots are provided. This upper plot uses to absolute scores. The lower plot shows scores as a percentage of the total possible score.

One is able to see a variety of potential grantee scores by changing the sample grantee data used. This is done by the drop down menu in cell d5. The baseline is setting all PMs to 3.

If interested, one can also see the impact of adjusting either PM scores (weights) and/or content area percentages by changing values in in the Input_Tables tab.

Code GPMS PMs Wieght Code Content Area Percentage Baseline Even Rand Compound Generator

ECD 1School readiness efforts align with the expectations of receiving schools*, the Head Start Early Learning

Outcomes Framework (HSELOF), and state early learning standards. 0.2020 1 ECD Early Childhood Development20.0% 20.0% 14.3% 15.8% 10.5% 49.00

CLS 2Teaching practices intentionally promote progress toward school readiness and provide high-quality

learning experiences for children. 0.2890 2 HEA Health and Safety15.0% 20.0% 14.3% 9.6% 11.8% 30.00

ECD 3The grantee ensures teachers are prepared to implement the curriculum and support children’s progress

toward school readiness. 0.2520 3 PDM Program Design15.0% 15.0% 14.3% 9.3% 35.9% 29.00

ECD 4The grantee ensures home-based program services provide home visits and group socialization

activities that provide high-quality learning experiences. 0.1200 4 FIS Fiscal15.0% 15.0% 14.3% 9.6% 15.9% 30.00

HEA 1 The grantee has an approach to ensuring delivery of high-quality health services. 0.2000 5 FCE Family Community Engagement 15.0% 15.0% 14.3% 20.3% 7.7% 63.00

HEA 2The grantee supports a program-wide culture that promotes mental health and social and emotional well-

being, and uses mental health consultation to support staff. 0.2240 6 ERSEA ERSEA10.0% 15.0% 14.3% 9.3% 5.9% 29.00

HEA 3The grantee maintains and monitors for effective oral health practices and nutrition services that meet

the nutritional needs and accommodate feeding requirements and allergies. 0.1550 7 SYS System10.0% 0.0% 14.3% 26.0% 9.6% 81.00

HEA 4The grantee has implemented a process for monitoring and maintaining healthy and safe environments.

This includes ensuring all staff have complete background checks. 0.2600 100.0% 100.0% 100.0% 100.0% 97.4% 311.00

HEA 5


facilitate enrolled pregnant women’s access to health care and provide information to support prenatal,

postpartum, maternal, and infant health and emotional well-being. 0.1300

PDM 1 The grantee’s program structure and design is informed by the community’s strengths and needs. 0.6120

PMQ 1

The grantee establishes a management structure that consists of staff, consultants, or contractors who

ensure high-quality service delivery; have sufficient knowledge, training, experience, and competencies

to fulfill the roles and responsibilities of their positions; and provide regular supervision and support to

staff. 1.0000

PMQ 2



standards; and to assess the effectiveness of professional development. 0.9910

PMQ 3The grantee maintains a formal structure of program governance to oversee the quality of services for

children and families, and to make decisions related to program design and implementation.0.3370

FIS 1


delivery of services that support the needs of enrolled children and families. This entails relating financial

data to accomplishments of the grant award and an awareness of program progress, lessons learned,

and needed improvements. 0.5640

FIS 2The grantee plans and implements a fiscal management system that supports the organization’s ongoing

capacity to execute its budget over time and meet the needs of its organization.0.3550

FIS 3The grantee’s financial management system provides for effective control over and accountability for all

funds, property, and other assets. 0.2850

FIS 4The grantee complies with application, prior approval, and reporting requirements for facilities

purchased, constructed, or renovated with Head Start funds. 0.0940

FCE 1The grantee collaborates with families to support family well-being, parents’ aspirations, and parents’ life

goals. 0.2130

FCE 2The grantee provides services that strengthen parent-child relationships and support parents in

strengthening parenting skills. 0.2040

FCE 3The grantee’s education and child development services recognize parents’ roles as children’s lifelong

educators and encourage parents to engage in their child’s education. 0.2160

ERSEA 1The grantee enrolls children or pregnant women who are categorically eligible or who meet defined

income-eligibility requirements. 0.1930

ERSEA 2At least 10% of the grantee’s total funded enrollment is filled by children eligible for services under IDEA

or the grantee has received a waiver. 0.1260

ERSEA 3 The grantee maintains and tracks enrollment for all participants, including pregnant women.0.1630

Sys_1 Emotional Support 0.5000

Sys_2 Classroom Organization 0.5000

Input Table 1: PM Weights by Content Area Input Table 2: Content Area Percentages

Total (Check)

QM ID QM Text QM Scoring Rule Max_Score Min_Score Mid_Score Mixed All Boolean; Meets Range No Boolean; Meets Range No Boolean; Exceeds Range All Boolean; Fails Range All Boolean; Mixed Range Mixed Boolean; Exceeds Range

QM_1 This is test 1 Yes/No Yes No Yes Yes Yes No No Yes Yes Yes


QM_3 This is test 3 Yes/No Yes No Yes No Yes No No Yes Yes No








QM_11 This is test 11 Range (1-3) Exceeds Requirements Fails to Meet Requirements Meets Requirements Fails to Meet Requirements Meets Requirements Meets Requirements Exceeds Requirements Fails to Meet Requirements Fails to Meet Requirements Exceeds Requirements


QM_13 This is test 13 Range (1-3) Exceeds Requirements Fails to Meet Requirements Meets Requirements Meets Requirements Meets Requirements Meets Requirements Exceeds Requirements Fails to Meet Requirements Meets Requirements Exceeds Requirements








QM_21 This is test 21 Range (1-3) Exceeds Requirements Fails to Meet Requirements Meets Requirements Exceeds Requirements Meets Requirements Meets Requirements Exceeds Requirements Fails to Meet Requirements Exceeds Requirements Exceeds Requirements

















QM_38 This is test 38 Range (1-3) Exceeds Requirements Fails to Meet Requirements Meets Requirements Exceeds Requirements Meets Requirements Meets Requirements Exceeds Requirements Fails to Meet Requirements Meets Requirements Exceeds Requirements































































Reference 1: QM_Scoring_TypesYes/No QI_Scoring_Boolean

Range (1-3) QI_Scoring_Range

Reference 2: QM_Scoring_BooleanNo 0

Yes 1

Reference 3: QM_Scoring_RangeFails to Meet Requirements 1

Meets Requirements 2

Exceeds Requirements 3

Reference 7: QI_QM_Relationship_TypeRequired 1

Preferred 2

Reference 8: Combined Boolean QM RatingRef_Code Rating_Required Rating_Preferred Final_Rating

1_1 1 1 1

1_2 1 2 1

1_3 1 3 1

2_1 2 1 2

2_2 2 2 2

2_3 2 3 2

3_1 3 1 3

3_2 3 2 4

3_3 3 3 5

Reference 9: Combined Range QM RatingRef_Code Rating_Required Rating_Preferred Final_Rating

1_1 1 1 1

1_2 1 2 1

1_3 1 3 1

1_4 1 4 1

1_5 1 5 1

2_1 2 1 2

2_2 2 2 2

2_3 2 3 2

2_4 2 4 2

2_5 2 5 2

3_1 3 1 3

3_2 3 2 3

3_3 3 3 3

3_4 3 4 3

3_5 3 5 3

4_1 4 1 3

4_2 4 2 3

4_3 4 3 4

4_4 4 4 4

4_5 4 5 4

5_1 5 1 3

5_2 5 2 4

5_3 5 3 4

5_4 5 4 5

5_5 5 5 5

Reference 10: Final QI RatingRef_Code Boolean_Rating Range_Rating Final_Rating

11 1 1 1

12 1 2 1

21 2 1 1

13 1 3 1

31 3 1 1

22 2 2 2

32 3 2 2

23 2 3 2

14 1 4 2

41 4 1 2

15 1 5 2

24 2 4 2

42 4 2 2

51 5 1 2

25 2 5 2

52 5 2 2

33 3 3 3

34 3 4 3

43 4 3 3

44 4 4 4

35 3 5 4

53 5 3 4

45 4 5 5

54 5 4 5

55 5 5 5

Reference 4: Quality Indicator Scoring Table*QM_Scoring_Range Final Value Boolean Criteria Range Criteria

Deficient 1 More than One No Multiple 1's

Area of Concern 2 One No At least one 1

Meets with Improvement 3 All Yes All 2's

Meets All 4 All Yes One 2

Meets or Exceeds All 5 All Yes All 3's

*See Sheet QM_Coding_Range for Description of Derivation of Relationships

Reference 5: Performance Measures Scoring Table*PM_Scoring_Range Final Value

Deficient 1

Area of Concern 2

Meets with Improvement 3

Meets All 4

Meets or Exceeds All 5

*See Sheet PM_Coding_Range for Description of Derivation of Relationships

Reference 6: QI Levels and WeightsPriority Level Equal Essential/Non-Essential Linear Doubling

Low 1 1 1 1

Moderate 1 1 2 2

High 1 2 3 4

Total 3 4 6 7

Date post:	10-Jan-2022
Category:	Documents
Upload:	others
View:	12 times
Download:	0 times

GPMS Scoring Algorithm Comparison (Alpha Version 2.2)

Documents