Date post: | 03-Jan-2016 |
Category: |
Documents |
Upload: | donovan-dease |
View: | 23 times |
Download: | 3 times |
Applying a Life-Applying a Life-Cycle Approach to Cycle Approach to
Information QualityInformation QualityARRA Performance Data ARRA Performance Data
Office of the Chief Financial OfficerOffice of the Chief Financial Officerand theand the
Office of Environmental Office of Environmental InformationInformation
May 2009May 2009
What are performance What are performance data?data?
Data collected/used to monitor and Data collected/used to monitor and report a program’s accomplishments report a program’s accomplishments (its activities, outputs, outcomes) (its activities, outputs, outcomes) relative to a target relative to a target
Performance data can be Performance data can be environmental, administrative environmental, administrative (count), activity, survey, you name it(count), activity, survey, you name it
ARRA of 2009ARRA of 2009
OMB’s guidance regarding quality OMB’s guidance regarding quality assuring ARRA-performance assuring ARRA-performance
information:information: “…“…agencies are responsible for pre-agencies are responsible for pre-
dissemination review of all information that dissemination review of all information that will appear on Recovery.gov. All agencies will appear on Recovery.gov. All agencies must ensure all reporting related to ARRA must ensure all reporting related to ARRA funding is complete and accurate and funding is complete and accurate and complies with the agency's Information complies with the agency's Information Quality Act guidelines. Each agency will Quality Act guidelines. Each agency will provide its point-of-contact for quality provide its point-of-contact for quality information on its Recovery.gov page.”information on its Recovery.gov page.”
Other accountability Other accountability objectives in ARRA:objectives in ARRA:
Public benefits of funds reported Public benefits of funds reported clearly, accurately, in timely mannerclearly, accurately, in timely manner
Funds used for authorized purposeFunds used for authorized purpose No unnecessary project delays and No unnecessary project delays and
cost overrunscost overruns Programs meet their goals and Programs meet their goals and
targets, and contribute to targets, and contribute to improvement of broad economic improvement of broad economic indicatorsindicators
OMB Guidance on OMB Guidance on Performance Measures Performance Measures
under ARRA under ARRA Performance measures are expected quantifiable Performance measures are expected quantifiable
outcomes…with each outcome supported by a outcomes…with each outcome supported by a corresponding quantifiable output(s)—to indicate corresponding quantifiable output(s)—to indicate incremental change against present level of incremental change against present level of performanceperformance
Programs must specify results reporting period Programs must specify results reporting period (e.g., monthly, quarterly), measurement (e.g., monthly, quarterly), measurement methodology, and how the results will be made methodology, and how the results will be made accessible to public. accessible to public.
Use of existing measures will allow the public to Use of existing measures will allow the public to see the marginal performance impact of the see the marginal performance impact of the Recovery Act investments and will reduce burden.Recovery Act investments and will reduce burden.
ARRA Performance Measures—ARRA Performance Measures—Available in Draft:Available in Draft:
Developed by Programs receiving ARRA fundsDeveloped by Programs receiving ARRA funds Mix of Outcomes/OutputsMix of Outcomes/Outputs[1][1] Many are existing planning and annual commitment Many are existing planning and annual commitment
measuresmeasures ““Green” measures and economic efficiency measures (e.g., Green” measures and economic efficiency measures (e.g.,
jobs created/preserved)jobs created/preserved) Measures developed by OIG to detect waste, abuse and Measures developed by OIG to detect waste, abuse and
fraudfraud
[1][1] Number of existing heavy duty diesel engines (including Number of existing heavy duty diesel engines (including school bus engines) that have been retrofitted, replaced or school bus engines) that have been retrofitted, replaced or retired (DERA); Number of states that have awarded all of retired (DERA); Number of states that have awarded all of the 20% green project reserve (CW SRF); Number of jobs the 20% green project reserve (CW SRF); Number of jobs leveraged (Brownfields); Number of criminal, civil and leveraged (Brownfields); Number of criminal, civil and administrative actions (OIG)administrative actions (OIG)
Performance Measure/Data Quality Performance Measure/Data Quality
PlanPerformance Measures Development•Program performance questions•Data/information sources & definitions •Acceptance criteria/data quality objectives (award conditions)
Data/Info QA Plan•Sampling & lab analysis plans/protocols•Reporting & record keeping•Database design and operation•Data input protocol•Statistical analysis & presentation
Roles & responsibilities
CheckCheck quality processes for adequacy
Review oversight results for compliance with QA plans
Document issues, problems & deviations
Characterize validity of a priori assumptions about measure/data quality
Assess implications for intended use of/goals for performance measure/data
Identify options to improve performance measure/data quality
ActEvaluate measure/data quality options
Assess costs & benefits of options
Revise measures & QA plans as warranted
DoData/info collection & analysis•Note issues/deviations
Data recording, storage, access, aggregation, etc.
Oversight, corrective actions & implications
Analyze & interpret
IQG pre-dissemination review
Present/communicate
Management Action Plan IManagement Action Plan I
Explains EPA’s Quality Program’s role under Explains EPA’s Quality Program’s role under ARRAARRA
References existing grants, contracts, and References existing grants, contracts, and IAs receiving ARRA funds and specifies IAs receiving ARRA funds and specifies which have which have environmentalenvironmental data collection data collection components with Quality Management Plans components with Quality Management Plans and/or Quality Assurance Project Plansand/or Quality Assurance Project Plans
Good check for ARRA environmental Good check for ARRA environmental measures that have robust Plan/Do quality measures that have robust Plan/Do quality assuranceassurance
Management Action Plan IIManagement Action Plan II Broadens definition of performance data to include Broadens definition of performance data to include
administrative data and other typesadministrative data and other types Lays out “State of QA” for each performance measureLays out “State of QA” for each performance measure Inventory data standards and definitions, management Inventory data standards and definitions, management
controls for reporting and oversight, data flowcontrols for reporting and oversight, data flow Identifies gaps in QA processes & responsibilitiesIdentifies gaps in QA processes & responsibilities Requires programs to self-certify completeness of data Requires programs to self-certify completeness of data
quality processes through a signed Pre-dissemination quality processes through a signed Pre-dissemination ReviewReview
Bottom line purpose (of PDCA, too): Bottom line purpose (of PDCA, too): To minimize To minimize end-of-pipe errors in reporting performance end-of-pipe errors in reporting performance results!results!
Pat’s topicsPat’s topics
Where we are/were headed when Where we are/were headed when ARRA intervened – larger ARRA intervened – larger vision/contextvision/context
What MAP II represents compared to What MAP II represents compared to the whole effort: limited scope but the whole effort: limited scope but progress in involving the PM progress in involving the PM communitycommunity
Progress outside of MAP IIProgress outside of MAP II Opportunities for participationOpportunities for participation
Performance Performance MeasurementMeasurement
Goal – to further QA in all OMB-related Goal – to further QA in all OMB-related performance measures for EPAperformance measures for EPA
Beyond performance data reporting Beyond performance data reporting from ARRA-funded programsfrom ARRA-funded programs
Broaden to an Agency-wide QA contextBroaden to an Agency-wide QA context Plan, Do, Check, Act – quality life cyclePlan, Do, Check, Act – quality life cycle Information Quality Guidelines (IQGs)Information Quality Guidelines (IQGs) CIO 2106 – new Quality PolicyCIO 2106 – new Quality Policy
ARRA ProgressARRA Progress
MAP I QA of environmental data in MAP I QA of environmental data in ARRA financial instrumentsARRA financial instruments
MAP II QA adds MAP II QA adds performance data performance data and informationand information for ARRA measures for ARRA measures
Communication with and input from:Communication with and input from: NPO & Regional QA communitiesNPO & Regional QA communities IT/IM community (some affected by IT/IM community (some affected by
ARRA)ARRA) Stimulus Steering Committee - planners Stimulus Steering Committee - planners
from 5 program officesfrom 5 program offices
What’s NOT in MAP IIWhat’s NOT in MAP II QA of performance QA of performance measuresmeasures in the life cycle in the life cycle
framework (ARRA measures are framework (ARRA measures are setset, ICR , ICR issue)issue)
IT/IM QA for IT/IM QA for allall types of data bases types of data bases Enterprise systems (CERCLIS, SDWIS)Enterprise systems (CERCLIS, SDWIS) End-user systems (spreadsheets)End-user systems (spreadsheets)
Guidance from IT/IM QA standards referencesGuidance from IT/IM QA standards references GAO auditing protocols (FISCAM)GAO auditing protocols (FISCAM) Corporate accounting firmsCorporate accounting firms OEI policyOEI policy
QA review/approval of the proceduresQA review/approval of the procedures
Performance Measurement Performance Measurement QAQA
Plan with stakeholders, internal & Plan with stakeholders, internal & externalexternal
Apply the QA graded approach to assureApply the QA graded approach to assure Adequate QA resources for oversightAdequate QA resources for oversight Performance measure & data are adequate Performance measure & data are adequate
for intended usefor intended use Document, validate & verify data flow Document, validate & verify data flow
processesprocesses Use life cycle “Check” and “Act” for Use life cycle “Check” and “Act” for
evolution to environmental outcomesevolution to environmental outcomes
Opportunities for InputOpportunities for Input IT/IM Managers: Performance data quality IT/IM Managers: Performance data quality
expectations forexpectations for Large systemsLarge systems Small spreadsheet scale systemsSmall spreadsheet scale systems
QA Managers: QA adequacy ofQA Managers: QA adequacy of Non-environmental data and information used Non-environmental data and information used
to support the budget processto support the budget process Environmental data with additional intended Environmental data with additional intended
uses in performance reportinguses in performance reporting Performance reporting processPerformance reporting process
More Opportunities for More Opportunities for Management InputManagement Input
Senior managers: views on certificationSenior managers: views on certification Program managersProgram managers
Consult with IT/IM & QA managers on PDCA Consult with IT/IM & QA managers on PDCA application to design performance measures application to design performance measures more efficiently & apply graded approachmore efficiently & apply graded approach
To ensure performance data are adequateTo ensure performance data are adequate To ensure data flow procedures are effectiveTo ensure data flow procedures are effective To ensure verification & validation are To ensure verification & validation are
transparent and documentedtransparent and documented