Statistics Portugal | Planning, Control and Quality Unit
Magda Ribeiro | [email protected] Vera Morais | [email protected]
«
July 2008
«
Measuring Relevanceof Statistical Information
European Conference on Quality in Official Statistics 2008
• The official statistical information is an essential tool for the knowledge of any society
• The statistical information should allow a “real” quality picture on the economic and social situation
• The permanent society development is a major demand for the statistical activity
• Keeping the statistics relevant implies that they must closely follow the continuous societal changes, identifying at each moment what is important to measure
• Statistics should be evaluated according to international standards, assuring its quality and comparability among other national or international data
• Moreover, statistics are to be released to all users at the same time
Statistical information
Relevance means
The decision to keep or start a statistical
project is directly associated with the
relevance of that project. Those decisions
are taken under the national and
international statistical systems.
Relevance Pertinence, Utility, Importance …
The relevance of the disseminated statistical information is defined by the users recognising the utility and importance of that information.
Multi-annual statistical work programmes
Annual statistical work programmes
Strategical
Domain
Operacional
Domain
Relevance - Standard Quality Report
Relevance is one of 6 dimensions of the definition on Quality in Statistics and the Standard Quality Report
Relevanceaccuracy
timeliness and punctuality
accessibility and clarity
comparability
and coherence
“Relevance is the degree to which
statistics meet current and
potential users’ needs. It refers to
whether all statistics that are
needed are produced and the
extent to which concepts used
(definitions, classifications etc.)
reflect user needs” [Eurostat
2003a].
«
Degree of relevance
Accuracy, timeliness and punctuality, accessibility and clarity, comparability and coherence
(depends)
Example:
If a statistical data is not released on time, it could loose its utility and consequently become not relevant.
Suppose that:
The announcement of the unemployment rate for the 1st quarter 2008 is done in 2010. Is this information useful for any decision on the labour market?
Relevance - Standard Quality Report
• A description and classification of users
• A description of the variety of users’ needs (by class of users, mainly), the possible contradictory expectations among them and priorities made. If a class of users is given strategic importance, a more thorough description of their needs can prove useful
• Reference of specific documents where the description of more comprehensive needs could be found, if any
• Main results regarding the satisfaction of users. In particular, appraisals by the most important class of users, if any. Main reasons for lack of relevance
• The number or percentage of unavailable results, compared to what should be available.
• Reasons for incompleteness as well as the prospects for future solutions
• Follow-up of the user satisfaction assessment, i.e. the measures and actions taken to improve user satisfaction
• Circulation and /or readership of publications (paper of electronic)
• Number of Web hits for the relevant web-pages and/or number of downloads of specific products
Relevance - Guidelines
Relevance - Standard Quality Report
Source: Doc. Eurostat/A4/Quality/03/General/Standard_Report
«Relevance - Standard Quality
Indicators
User Satisfaction Index
A recommended methodology does not exist.
Normally it is evaluated through user satisfaction surveys
Rate of available statistics
It is calculated by dividing the number of values provided in a concrete data set divided by the total number of fields for which data has to be provided:
Rate of available statistics = Number of values provided / Number of fields applicable
Source: Doc. ESTAT/02/Quality/2005/9/Quality Indicators
«
European Statistics Code of Practice
PRINCIPLE 11: RELEVANCE
European statistics must meet the needs of users
Relevance – Code of practice
Indicators:
Processes are in place to consult users, monitor the relevance and practical utility of existing statistics in meeting their needs, and advise on their emerging needs and priorities.
Priority needs are being met and reflected in the work programme.
User satisfaction surveys are undertaken periodically.
«
Reflections on relevance
1. Conception of the statistical project: What to do and how to do it?
2. Users of the statistical informatios: For whom?
3. Results of the statistical project: How to disseminate them and how will they be understood by the users?
Aspects related to the relevance of the statistical information :
Life cycle – Statistical project
I. Conception IV. EvaluationIII. DisseminationII. Operation
II.2. Data collection
II.3. Data treatment and analysis
II.1. Planing and preparation of data collection
I.1. Viability Study
I.2. Methodological Study
I.3. Technical approval
III.1. Dissemination IV.1. Evaluation
Statistical Production Handbook Procedures
«Life cycle – Statistical project
The phases directly related with the relevance are:
Conception
Dissemination
Evaluation
I. Conception IV. EvaluationIII. DisseminationII. Operation
«Life cycle – Conception
Conception:
1st phase of a statistical operation, when the methodological study is developed
I.3. Survey approval
I.2. Methodological study
I.1. Feasibility studySub-processes related to relevance
Life cycle - Conception
Feasibility study – Document /Template
• Objectives of the survey
• Identify user's needs
• General characterization of the survey
Main tasks related with relevance
- Communitarian Legal Acts
- Main documents about users needs
- Document with the generic
characterization of the survey
Documentation:
Life cycle - Conception
Methodological study – Document /Methodological Document
Main points related with relevance
Purpose
• Summarize the objectives of a statistical operation
Users information
• Users description by type and strategical importance
• Description of information needs by users type
Products
Quality standards – desirable release calendar
Products to make available – associated information for each product: Name; Product type; Periodicity; Geographic level of disaggregation; Availability type; Strategic users
Information available for each variable
Name, Measure unit, Classification variables
I. Conception IV. EvaluationIII. DisseminationII. Operation
«Life cycle - Dissemination
Dissemination: selection, adaptation, promotion and publishing of the information resulting from statistical operations. Its major purpose is to put available to the community statistical data with quality, to support decisions and further research.
• Prepare the dissemination
• Execute the dissemination
• Promote
Main tasks related with relevance:
• Adapt contents and products to users needs
• Adapt the content’s promotion strategy
• Adapt communication strategy to the users
I. Conception IV. EvaluationIII. DisseminationII. Operation
«Life cycle - Evaluation
• Survey quality evaluation a posteriori
• Products and dissemination services quality evaluation
• Audits
Main tasks related with relevance:
Evaluation: life cycle evaluation for a statistical operation; strong points and improvement opportunities to implement or adapt to other projects.
Quality Report, by statistical project
Example: Peer review
Example: User satisfaction surveys
Evaluation - Peer Review – Principle 11
Iceland
Statistics Iceland benefits from a very close cooperation with users.
Intensive cooperation with users and their frequent consultation has been put on an institutional level. Two user groups, on price statistics and national accounts, and one advisory group on wages have been set up. The user and advisory groups unite both users and cooperative partners in the production of statistics. The meetings are organised in an open manner, with participants raising subject to be discussed. Minutes are recorded and published. This creates a very open style of managing statistics. Liechtenstein
Users' recommendations are integrated in publications whenever possible.
Lithuania
User satisfaction surveys.
The user satisfaction surveys cover general users, web users and specific user groups. The surveys – some outsourced to private opinion institutes - ask for opinions on the following subjects: visibility and image perception, quality of official statistics, internet accessibility, statistical publications, monitoring of user-requests, alert-me services, library-bookshop in Statistics Lithuania head-office and visitors’ corners in Regional statistical offices.
Source: Summary of good practises identified during the European Statistics Code of Practice peer reviews carried out during 2006-
2008 - Version 1.0 of 10 June 2008
«Evaluation – Peer Review – Principle 11
Peer Review Reports associated to each Member State and Eurostat - principle 11: RELEVANCE
Without references /
No improvement actions proposedImprovement actions
8 20
Most frequent initiatives:
User satisfaction surveys
Contact/communications with users
Media
As a result of the analysis of a user satisfaction survey to the media, Statistics Portugal has promoted several workshops.
The purpose of this workshops was to inform the media about the statistical activity along the life cycle of each project (information availability, release calendars, methodologies, etc.)
«Evaluation – Customer satisfaction surveys
The results analysis of a user satisfaction survey to researchers has showed that the satisfaction levels were different among them, according to the statistical area considered.
This analysis useful the definition of improvements for different statistical projects and users.
Examples:
Researchers
«How can we measure the
relevance of statistical information
Measuring
Relevance • Evaluation of the relevance of statistical data on decision making
• Analysis of users (strategic or non strategic)
• Analysis of user data requests by statistical project and user type
• Website statistics analysis by statistical project and service type
• User satisfaction surveys by statistical project and user type
• Promotion of focus groups during product definition and set-up
Know users and users needs:
Users type – users classification
By project
Relevance level
Conception:
Expectations on relevance level are high
Low high
After data dissemination
Relevance level depends on user perception about the information
It can be increased with promotion activities
1
2
Continuous information on users
Adapt contents to users requests
Improvement actions3
Decrease the distance between 1 and 2
«Relevance level
• Efficient communication between NSI’s and their users
• Meet users’ needs according to a users typology
• Understand and act on users perception - users interaction
• Oriented promotion of contents and products, according to users typology
The relevance level of a statistical project can be increased
Communication
Even assuming that the relevance level of a statistical project is high, that might not be perceivable or understood by users.Relevance is a dynamic attribute!
Promotionstatistical literacy
Explaining
«Promotion
Promotion efficiency
Examples:
1. Statistics Portugal has conducted a promotional campaign for the Demographic Studies Review, through a mailing to potential users. The results were:
• 76% of sales were a direct result of this mailing
• Among other products with similar promotion, this Studies Review had the highest number of downloads (3449)
2. A mailing about the publication “Tourism Statistics” was responsible for an additional number of new users (33%).
«Communication/Perception
Example: Portuguese media
When searching by the keywords: “Employment”, “Unemployment” and “Labour Market”, 862 results were obtained referring to news published on Portuguese media mentioning statistical data produced by Statistics Portugal, during 1st quarter 2008.
Some user types are active actors on the promotion of statistical information, contributing to increase the relevance level of statistical information
Example: Media
During 2005 news were published in the Portuguese media suggesting that the data disseminated in “Study on the Local Purchasing Power” were manipulated due to political pressure.
Statistics Portugal then invited journalist to a press conference in order to clarify the methodology of the referred study.
The statistical information relevance might be compromised by the user perception
«
• Official statistical producers can have a key role on increasing the recognised relevance of statistical products through their adequacy to different users’ needs.
• Implies knowing how to classify users: groups of users with common needs.
• Specialized users of statistics are normally request more detailed data whereas non specialized users are interested in general and more aggregated data.
• Satisfied users tend to recognize greater relevance to statistics.
• The users literacy level is directly related with their recognition of statistical information relevance.
Relevance - Users
Considerations on the users
Users oriented activity
Rel
eva
nce
lev
el
«Conclusions
Relevance
The process of measuring relevance must be harmonized among countries and statistical projects.
We can measure relevance in advance:
In the conception phase of a survey
We can measure relevance afterwards (à posteriori):
Using the results of Customer Satisfaction Surveys;
Using customer’s Database information – Customer profile, Typology of information requested, Customers level of experience, Demand indicators
Relevance
Demand statistical information Index
User Satisfaction Index
«Conclusions
Producers and disseminators of statistical information have the option to increase statistical relevance level by:
- Studying the relevance: using quantitative, qualitative and descriptive indicators
- Improving processes
- Communicating with usersBy statistical project
Relevance level
«Future developments
Cross analysis of quality reports
A case study of a statistical project - What is the relevance level of the Labour Force Survey?
Statistics Portugal | Planning, Control and Quality Unit
Magda Ribeiro | [email protected] Morais | [email protected]
July 2008
«
Thank you
