Quality Review of Key Indicators at Statistics Canada

ICES-III, June 2007Claude Julien and Don Royce

Outline

Quality Management at Statistics Canada

Quality review process

Recommendations

Observations on the review process

Conclusion

Quality Management at STC

1997 Quality Audit

Quality Assurance Framework (QAF)

Audit on 4 key surveys

Office of the Auditor-General recommended to examine a wider application of enhanced self-assessment

Quality Assurance Framework (QAF)

Quality definition Fitness-for-use: relevance, accuracy, timeliness, coherence, accessibility, interpretability

Management of each dimensionPolicies, standards, guidelines and practices

Five main systems: users liaison, planning, methods and standards, dissemination and reporting

(Brackstone 1999, Survey Methodology)

Integrated Program Reports (IPR)

Quadrennial report and biennial follow-up

Quality, costs, management of HR and strategic direction

Self-assessment and user survey or consultation

Focus on quality of outputs

Review and feedback from Policy Committee

Quality Review Process

Initiating the review

Some serious errors in the preceding 18 monthsInternational tradeProductivity estimatesConsumer Price Index

Questions from Finance, other key users and mediaPolicy Committee created Steering Committee of DGs to review quality assurance practices

Headed by DG-Methodology and assisted by head of Quality Secretariat

Objectives of the review

Identify “best practices” that should be promoted to other programs

Identify specific weaknesses where action is needed

Programs Reviewed

Monthly programsLabour Force SurveyRetail Trade SurveyManufacturingInternational TradeConsumer Price IndexGDP by industry

Quarterly programsIncome and Expenditure AccountsLabour ProductivityBalance of Payments

Challenges – Types of programs

Labour Force SurveyRetail Trade SurveyManufacturingInternational TradeConsumer Price IndexGDP by industryIncome and Expenditure AccountsLabour ProductivityBalance of Payments

Traditional

Administrative data

Mixture of data

Derived data

Mixture of types

Challenges – Program process

Process = complex series of stepsSet objectives, define population, create frame, select sample, collect data, etc.

Identify a reasonable number of recognizable steps

Focus on one or two

Challenges – Aspects of a step

Design

Implementation

Execution

Evaluation

Challenges – Dimensions of quality

Relevance

Accuracy

Timeliness

Coherence

Interpretability

Accessibility

Mandate of the review

Review the practices during production (execution) to prevent the release of erroneous data (accuracy)Particular focus on the certification and data dissemination of data

Review of Dissemination and Communications Program

Identify the underlying risk factors and their importance.

Review Teams

Identify and “volunteer” 10 reviewers

10 review teams

3 reviewers + member of Steering Committee per team

Lead reviewer

Organization

Policy Committee

Directors General

Assistant Directors

Directors

Mandate

Recruitment Program

Introductory meeting

Program director, lead reviewer and head of QS

Introduce the review process

Introduce the program

Identify main steps of program process

Main steps of program process

Preparation for certification

Data collection

Editing and transformation

Imputation and estimation

Certification

Data release

Post-release

Gather information

Semi-structured questionnaireCover the topics of the mandate

Not to be completed

Interviews with review team and program director and managers

Available documentationMeta-data

Program reports

Assessment

Individual reportsReport by lead reviewerReviewed by Steering CommitteeVetted by program director

One-day debriefing session with all reviewersSummary report

Report by Steering CommitteePresentation to Policy Committee

Taking action

Disposition of recommendations by Policy Committee

Funding for corrective actions

Follow-up in Integrated Program Reports

Recommendations

Main recommendations

Program managers are well aware of the risks factors

The programs are well-positioned to identify the improvements

Some programs would benefit more than others from additional investment

Human resources concerns dominate all other risk factors.

Main recommendations

The existence of a research and analysis capacity separate from the production operation is a key factor in assuring quality.There are numerous “best practices” in all programs that can usefully be shared within the Agency.Statistics Canada should develop a formal Quality Incident Response Plan Proposals to further increase the timeliness of the programs should be regarded with extreme caution Statistics Canada should establish an ongoing program of Quality Assurance Reviews

Observations on the review process

Aspects to bring forward

Clear mandate from Policy Committee to focus the reviewParticipation at the DG level to assure legitimacyStrong cooperation from program directors and managersConcurrent reviews to assure attention and team workKnowledge and experience of reviewersComposition of review teams to assure some standardizationExcellent learning, development and networking opportunity for reviewersOverall positive experience for the reviewersCollection approach and toolsOne-day debriefing session

Aspects to improve

Better assessment and reporting of level of various risk factorsTimeliness

10 reviews in five months (Oct-Mar)Reviewers spent around 15 daysMore interviews than expectedTime to write, review and translate reports

More timely process and disseminationMore time to schedule meetings around production

More standard program reports

Conclusion

The review was developed and carried out very quicklyThe review was a success thanks to all participantsThe review was not thorough; it complemented other key quality management mechanismsMandate to propose an on-going review program

Build on recent reviewCriteria for selecting surveysTool box / Flexible approachPart of learning and development program

Merci

For more information, or

to obtain a French copy

of this presentation,

please contact:

Pour de plus amples

informations ou pour obtenir

une copie en français du

document, veuillez

contacter :

Claude JulienEmail / Courriel: claude.julien@statcan.caPhone number / Téléphone: 613-951-6937