Post on 28-Mar-2015
transcript
Census Microdata: findings and futures
Manchester 1 – 3 September 2008
The future of microdata
Professor Denise Lievesley UN African Centre for Statistics
Addis Ababa, Ethiopia and President, International Statistical Institute
Graduating from being data producers to generators of information and knowledge
attention to data collection at expense of generation of information and knowledge– collection costly and difficult– importance of quality of data
mountains of data – insufficiently processed and analysed
most people not adept at understanding data– even more important for statisticians to get
involved in interpretation and use of information
Partnerships for data use
with– subject experts– data analysts– researchers in government, universities,
private sector and civil society
who can contribute to development of data production systems in countries
this requires making census and survey data accessible to these stakeholders
The importance of international guidelines on ensuring data access
The UN Principles and Recommendations for the 2010 Population and Housing Census urge countries to create census databases as part of the process of census data management.– “in order to expand the life and usability of
data, and as a complement to the standard production of tables, NSOs are encouraged to store census data in various computerized database forms” including macro and micro- databases.
These recommendations give one of the main advantages of micro databases as “permitting the retrieval of data at least in principle, at any level of detail”.
Additional advantages:– to broaden data use and reuse;– to foster diversity and deepen the quality of
data analysis thereby extracting more information from the data;
– to add value to data by bringing subject-matter knowledge to data analysis;
– to improve data quality (Data analysts can and often do detect errors in data and when they provide feedback to statistical agencies, this can lead to improvements in future data collection.)
According to the International Household Survey Network established as one of the six action points of the Marrakech Action Plan for Statistics, national and international micro databases should be established to:– promote the acquisition, documentation,
dissemination and preservation of microdata essential for the production of national statistics, for research and for instruction in the social sciences,
– promote the effective use of existing survey and census data,
– ensure the continued viability and usability of microdata now and in the future, and,
– promote equitable access to these data within the framework of the national statistical legislation.
Obstacles
Despite these international guidelines and exhortations for data access, obstacles do exist and we must be sensitive to them. They include:– legal obstacles– technical and financial obstacles including in-
house capacity to handle the complex aspects of micro-data dissemination such as data anonymization
– political obstacles– psychological obstacles: the tendency to
control access perhaps because of concerns over its mis-interpretation or because ‘data is power’
Legal constraints
in many countries the statistical legislation in use is out dated and does not recognise the dissemination of electronic data particularly micro-data
some legislation, or the interpretation of it, actually prevents such dissemination on account of confidentiality
new legislation needed– the African Centre for Statistics is working with
countries to help them to prepare both legislation and professional frameworks relevant to today’s era of electronic information.
Data are collected using valuable resources – both financial but also the time of survey and census respondents – and are increasingly seen as being a vital part of democratic systems, since people are empowered though information.
Official data are a “public good”: part of the enabling environment for national and international development, which all stakeholders in society should have access to and benefit from.
Conclusion - I
It is the responsibility of statisticians to ensure that the widest possible use is made of data; consistent of course with the legal constraints and ethical undertakings.
Preservation is essential Having collected data at some cost to the
taxpayer, it behoves official statisticians to manage them well.
Alongside dissemination, this entails data preservation.
Due to poor data management, human error as well as technical change and inadequate use of technology, many data sets including critical census data are no longer readable.
Thus all that remains of this important legacy are the, often quite superficial, reports that were produced at the time.
To this extent an important part of our heritage is lost and we will be severely limited in our analysis of change.
Long term preservation of electronic material is not a straightforward task, especially in resource-poor and technology-weak developing country statistical offices.
It can be hard to persuade financial authorities to spend money on the preservation of data for historians and statisticians of the future, when there are so many pressing problems today.
Partnerships for data preservation
To this end, partnership - for both technical work and advocacy – across the data archiving, data librarian, statistical and research communities is to be encouraged.
Welcome the formation of new organisations such as the African Association of Statistical Data Archivists
Value of international networks and support systems
Metadata It is necessary not only to preserve
data but also to create and preserve metadata and contextual information. This is essential to ensure that the interpretation of the data will be informed.
The documentation should include– data collection instruments and forms– instruction manuals– definitions and concepts– descriptions of scope and coverage and
other aspects of quality– codebooks– basic tables– records of validation and post-enumeration
checks
Documentation
The documentation should be clear and easy to understand and should help users to:– identify and find the data they are
interested in– understand what the data are measuring
and how they have been created– assess the quality of data and fitness for
their purpose
International standards for metadata creation such as the DDI should be implemented
Conclusion - II
The leadership of the national statistical agencies should ensure that census and survey macro- and micro-data are well documented and archived.
Case made for data dissemination and preservation.
Data alone have no “intrinsic value”.– Their value is “extrinsic”: derived from the fact that
they can be disseminated to those who need them, are understood and are used for a variety of purposes in a timely fashion.
– So one way to justify the high cost of data production is to disseminate and mine them for information
– especially for evidence-based policy and comparative analysis
Data dissemination in a variety of ways (through reports, compilations of data, electronic extracts etc) is, therefore, important not only to complete the data cycle but also to enhance data relevance and usability.
Data grow in value the more they are used, unlike most commodities which are diminished with use.
Evidence-based policy
What is evidence-based policy? Need for use evidence at all stages in the
policy cycle Why is evidence-based policy important? Evidence but one input into policy process What is evidence? Evidence initiatives Challenges for statisticians Examples from the developing world
What is evidence-based policy ?
Helping people to make well-informed decisions about policies, programmes and projects, by putting the best available evidence from research at the heart of policy development and implementation
Enlightening through making explicit what is known through scientific evidence and importantly what is not known
Better statistics, better decisions, better outcomes Vision of the African Centre for Statistics
In contrast to opinion based policy
which relies heavily on – either the selective use of information– or the untested views of individuals or
groups often inspired by ideological standpoints, prejudices or speculative conjecture.
Need an evidence base at all stages in the policy cycle
in shaping agendas, in defining issues, in identifying options, in making choices of action, in delivering them and in monitoring their impact and outcomes.
Context in the UK
Modernising Government White Paper commits to policy making that is – strategic – outcome-focussed – joined up (works across organisational boundaries) – inclusive (is fair and takes account of the interests of
all – gender sensitive, promotes equity) – flexible – innovative (means taking risks – and so risks must be
identified, monitored and managed)– robust (works and continues to work)
Tackles causes not symptoms. Is acceptable to the public.
Why is evidence-based policy important?
Sherlock Holmes “it is a capital mistake to theorise before you have all the evidence. It biases the judgement”
Policy makers may be well-intentioned but misguided
Results of a meta-analysis
Collation of the results of many studies contradict this advice
Extract from publicity prepared for the UK ‘Reduce the Risk’ Campaign (early 1990s)
“The risk of cot death is reduced if babies are not put on the tummy to sleep. Place your baby on the back to sleep. ….Healthy babies placed on their backs are not more likely to choke.”
Iain Chalmers
“No doubt like millions of his other readers, I passed on and acted on this apparently rational and authoritative advice.”
“We now know that the advice promulgated so successfully in Spock's book led to thousands, if not tens of thousands, of avoidable cot deaths.” (Letter to BMJ)
Judging what works
Outcomes often defined too narrowly eg performance management whereby targets are often set ‘top down’ and may have unintended consequences.– Hitting the target but missing the point– The good, the bad and the ugly report of the
Royal Statistical Society– Difficulty is that sometimes different things work
in different circumstances and this is interpreted as nothing works.
(Ref Robert Martinson’s 1974 review of American offender rehabilitation programmes- He concluded that there was no single approach which worked consistently – different things worked in different circumstances but this was misread as ‘nothing works’.)
The policy making processPolicy making is the process by which governments translate their political vision into programmes and actions to deliver desired changes in the real world.
Evidence but one input into policy process
Ideology
Lobbies
Values/beliefs
Tradition
Self interest
Judgement
Campaign promises
Expert viewsExperience Resource
constraints
Acceptabilityto public
“ There is nothing a government hates more than to be well-informed: for it makes the process of arriving at decisions much more complicated and difficult. ”
John Maynard Keynes
What is evidence?
Expert knowledge; published research; existing statistics; stakeholder consultations; previous policy evaluations; the Internet; outcomes from consultations; costings of policy output from economic and statistical modelling. UK Cabinet Office 1999
Scientific, rigorous, critically appraised, well documented
Fit for purpose Beware policy-based evidence? Unbiased (note problems of publication
bias) But data are never value-free
All scientific evidence is imperfect.
“The absence of excellent evidence does not make evidence-based decision making impossible: what is required is the best evidence available not the best evidence possible”
Muir Gray 1997
“ Evidence rarely provides neat and tidy prescriptions to decision makers as to what they should do. Often it generates more questions to be resolved ”
Petrosino et al 2001
Challenge : is it essential that evidence is based on experiments?
Adrian Smith ‘Mad cows and ecstasy : choice or chance in evidence-based society’
Experiments used too little in social and economic research
Rich data from different sources Science is often misrepresented as the body
of knowledge acquired by performing replicated controlled experiments – it is much broader – the acquisition of reliable knowledge about the world
Jared Diamond Collapse
Challenges for analysts
increase understanding of the policy process, where and how evidence can feed into it (eg SPATS)
improve interpretation and communication of data particularly in relation to uncertainty
speak the language of policy makers forge strong relationships of official
statisticians with policy analysts, increase number of policy analysts
combine humility and confidence improve training for analysts
– (International Research Forum on statistical reasoning, thinking and
literacy http://www.srtl.stat.auckland.ac.nz)
Particular challenges with respect to timing
demand for quick fixes means that policy makers often do not have time for in-depth research – “Demands by Ministers are short term, quick fix
solutions rather than major carefully considered strategic initiatives” (Bullock et al)
collection of quality data takes time – hence the existence of public use samples can accelerate research
to be effective the research strategy needs to look beyond the timescales of one government
many issues are chronic – beware “unwarranted impatience” (Boruch)
Evidence compilations – example initiatives
Micro-data based on samples from Censuses
Data archives Cochrane collaboration Campbell collaboration National Library of Health Communities of practice
Communities of practice
European social survey CROP - the Comparative Research
Programme on Poverty whose major aim is to produce sound and reliable knowledge, which can serve as a basis for poverty reduction
RENCORE - encourage and enhance comparative empirical research of individual, national and institutional level data from the states of western, central and eastern Europe
Cleveland conference on education research
African Programme on Rethinking Development Economics
Evidence base urgently needed in developing countries
Problems are severe and urgent Need ownership by countries… and to empower them Develop policies which are relevant to
their needs… and ensure effective implementation Need to counteract corruption Paucity of data is a major obstacle Lack of recognition /acceptance of
much data
Joint Marrakech Memorandum
Affirm a commitment to fostering a global partnership on managing for development results. Awareness is growing that getting better development results requires management systems and capacities that put results at the center of planning, implementation and evaluation. We need to align cooperation programs with desired country results, define the expected contribution of our support to country outcomes and rely on and strengthen countries’ monitoring and evaluation systems to track progress and assess outcomes … better distil the lessons of countries’ experiences and disseminate knowledge about what gets results in different country contexts.
C.Scott for Paris 21Use of good statistics having a positive effect on policy
Uganda– Poor public service delivery caused by
government’s failure to ensure that budgeted funds reached front line agencies
Brazil and Mexico– Tackled child poverty and education by a
programme to give child benefits to mothers according to the attendance of their children in school
C.Scott for Paris 21Absence of data or failure to use available information has negative effect on policy making
Malawi– Data from Save the Children on child malnutrition
disregarded by government because of conflict with crop data
Botswana– Deficiencies in data on HIV/AIDS (from sentinel
surveillance systems) legitimised the rejection of the message on the scale of the problem
Current challenge in policy development in poor countries Development literature of the 1990s dominated by
the view that growth is central to any strategy aimed at poverty reduction.
Since modified: not growth per se but structure of growth that matters.
Further recognised that income inequality matters when making progress on poverty reduction.
Example Nigeria– Aigbokhan asks why the rate of poverty is so high in
Nigeria despite the strong growth performance and concludes it is because of a lack of explicit concern with inequality in public policy. He analyses data on Nigeria and on other countries for comparative purposes to argue that if growth occurs in the sectors that require skills that the poor of Nigeria do not possess although the economy grows it has little impact on poverty reduction.
– Essential that access is provided to researchers which allows them to examine the distribution of resources within countries
The gross national product does not allow for the health of our children, the quality of their education, or the joy of their play; it does not include the beauty of our poetry or the strength of our marriages, the intelligence of our public debate or the integrity of our public officials. It measures neither our wit nor our courage, neither our wisdom nor our learning, neither our compassion nor our devotion to our country, it measures everything in short except that which makes life worthwhile.
Robert Kennedy
Comparative research – building of cross-national resources
Knowledge of the self is gained through knowledge of others
Tension between the value and importance of cross-national data and its fragility
To what extent are differences artefacts of the measurement ?
Benefits of cross-national research
Contacts, enabling researchers to capitalise on their experience and knowledge of different intellectual traditions and to compare and evaluate a variety of conceptual approaches.
Comparisons can lead to fresh, exciting insights and a deeper understanding of issues that are of central concern in different countries - the identification of gaps in knowledge and sharpen the focus of analysis of the subject under study by suggesting new perspectives.
Cross-national projects give researchers a means of confronting findings in an attempt to identify and illuminate similarities and differences, not only in the observed characteristics of particular institutions, systems or practices, but also in the search for possible explanations in terms of national likeness and unlikeness.
Cross-national comparativists are forced to attempt to adopt a different cultural perspective, to learn to understand the thought processes of another culture and to see it from the native's viewpoint, while also reconsidering their own country from the perspective of a skilled, external observer.
Cross-national data is a special case of comparative research
Spatially comparative data – within countries– between countries
Collecting data over time (the past is a foreign country ) Many parallel issues in spatially and time
related comparative data such as achieving a balance between
specificity versus generalisability, and coping with differences/changes.
Purposes of cross-national data
To learn from one another (contrast and similarity) For purposes of national accountability (the
indicator movement) To build a greater global understanding through
comparison and through multiple instances of the same phenomena (Jared Diamond)
To aggregate across national boundaries for a regional or global picture– for advocacy– for resource mobilisation
To accelerate progress through sharing resources To make research more credible/ defensible
recognising that research which displeases is attacked rather than accepted
To distance the research from the political process (tension – policy relevance v. autonomy)
Challenges to comparability
Language Culture Social systems and structure Administrative systems Ideology and politics Economics and resources History Context – events Different methodologies and types of
methodological expertise (often deeply ingrained)
Different research climate
Over-interpretation
Jowell “Good science should not turn a blind eye to known imperfections – nor should these be concealed from users”– Concerns about lower standards being applied
for cross-national studies than for national studies
Goldstein (“Surprising naivety”) cross-national educational studies– Concerns about validity of cross-national
measures, inappropriate conclusions about causation, over-simplification of complex relationships
– Overall Goldstein argues for celebrating diversity and encouraging transparency
The UN model
Attempting to achieve comparable data across the world
Assumptions about universality of aims The survey ‘hammer’ is owned by richer
countries Under-resourced at UN and at country level Countries with resources play a bigger role (city
groups) Fragmentation within the UN decentralised
system (regionally and by substantive area) Concentrates on ‘facts’ and often assumes that
comparison is unproblematic Exhorted to “Think globally, act locally”
Problems with the UN model
One size fits all– Difficulty of accommodating national
differences– Cutting edge and trailing tail countries’
needs are very different– Context is critical but is often ignored
The nation is too often the unit of analysis– variation within countries ignored
Need to encourage country ownership and recognition of their data
Accountability of countries v. their autonomy
Combining administrative with survey and census data
Adds value to dataProvides contextEnables better understandingImproves communication – tells a better storyExcites and interests usersEngages the media
Summary
Case made for data preservation and dissemination
Data resources are critical for evidence based policy making
and for cross-national, comparative research
Welcome this meeting where we will discuss the value of this work, the challenges which face us and ways in which we can redouble our efforts to build these resources