Nordisk Statistikermøde i København 11.-14. august 2010
The archive statistical method 1960-201050 years - A Summary
bySvein Nordbotten
8/11/2010 1Svein Nordbotten
8/11/2010 Svein Nordbotten 2
The situation 50 years ago
• Separate statistical organizational units for limited subject matter domains• Few possibilities to link data within and between established domains• Reuse of micro data expensive and time consuming• Creating longitudinal micro data sets rarely possible
• Published statistics followed a narrow and restricted tabulation plan• Analysis limited to aggregated macro data
• Statisticians were experts in their subject matter domains• Little exchange of experience and information among different groups
8/11/2010 Svein Nordbotten 3
Two important events in the 1950s
Electronic, programmed computers made processing of large data sets more effective for administrative and statistical purposes.
Electronic recording made establishing, maintaining and sharing comprehensive population/object registers possible.
8/11/2010 Svein Nordbotten 4
The information technology
The UNIVAC 1 computer entered the official statistical scene in the US Census Bureau in the early 1950’s, and when the processing of the 1960 censuses were due, a number of NSIs were prepared with their own computers.
When the information technology later merged with communication technology, it changed the time and space dimensions for statistical processing.
The new technical potensial inspired to think about all activities of an NSI asinterrelated parts of a production system in contrast to the traditional separate statistical processes, and the term statistical production became common.
8/11/2010 Svein Nordbotten 5
Administrative registers
Local object identifiers had been used for a long time in many admistrative processes. These specialized systems were expensive to maintain , represented a vast duplication of efforts and were of limited usability as statistical data sources.
The efficiency of shared, public object identifiers were obvious. About 1950 the first steps to establish general, permanent, unique object identifier registers for the economic units as well as for the population were taken in the Nordic countries with active participation by statisticians.
8/11/2010 Svein Nordbotten 6
Statistical ideas in the 1950s
Unique and permanent identifiers permitted compilation of the ’life lines’ of statistical objects including interobject events by data linkage.
Models for statistical micro systems named statistical archive systems, statistical file systems, register-based statistical systems, etc. were discussed in international fora from about 1960.
The next slides illustrate the popular version of ideas discussed. Ten years later the generic version became known as data base managementsystems.
8/11/2010 Svein Nordbotten 8
Objects
Attributes
Time
Censusattributes
Censusday
Humanpopulation
Archive data container
8/11/2010 Svein Nordbotten 9Time
Objects
Timeplane
Object plane
Attribute plane
Attributes
The 3-dimensional space model
Object:BTime: t-1
Object type Atype ATime tTime tObject type ATime t
Object: CTime: t-1
Object: CTime: tOOOOO
Object A:Time: t-2
Object: CTime: t-2
Object : ATime: t
Object: BTime: t
Object: BTime: t-2
Object: ATime: t-1
Link H
Link HLink H
Link H
Link H
Link H
Link V
Link V
Link V
Link V
Link V
Link V
8/11/2010 10Svein Nordbotten
Latitudinal and longitudinal integration
8/11/2010 Svein Nordbotten 12
Summary of the 50 year old visions
Comprehensive official register systemsGeneral utilization of systems
Fast and high capacity ITExtensive and userfriendly communicationSoftware for data processing, saving, retrieval and communication
Data collection independent of specific usageUse of communication network for collectionFunctional organization
Wider statistical coverage by more intensive reuse of dataStatistics on-demandAnalytical interest in and tools for exploiting the new possibilities
Legislation regulating the use of micro data
8/11/2010 Svein Nordbotten 13
Utilization of public identification systems
• Population register systems in general use• Economic unit register systems in general use by administrative organizations• Register systems for geographical location of properties established and in use
8/11/2010 Svein Nordbotten 14
Sample of present applications
Censuses (population, location registers)
Demographics (demographic registration)
Foreign trade (custom data)
Income (taxation data)
Social statistics (registration of services)
Employment (unemployment registration)
Education (registration of students)
Health (medical registrations)
Crime ( Police, court, prison registrations)
Business (Business registers)
8/11/2010 Svein Nordbotten 15
Methodological challenges
The present intensive utilization of administrative data in officialstatistics has also required work on methods in several fields:
1.Electronic processes for collecting administrative data2.Quality control of data received from external administrative sources3.Integrating data from different statistical and administrative sources4.Evaluating quality of integrated data sets5.Editing data using associated administrative background data6.Estimating population parameters using administrative background data7.Confidentiality and security in systems based on administrative data8.Statistics on demand from systems based on administrative data9.Evaluation of costs and benefits from using administrative data
8/11/2010 Svein Nordbotten 16
Privacy issues
•Statistical legislation•Privacy legislation•Misuse of information
•Value of knowledge•Regulatory use and research
•Solutions? A political question
AD1 ADM 1
AD2 ADM 2
SD SPNSI
Information privacy law
Statistical law
Dat
a co
llecti
onD
ata products
8/11/2010 17Svein Nordbotten
8/11/2010 Svein Nordbotten 18
Anticipations/realizations
Comprehensive official register systems……………………………………………... . YesGeneral utilization of systems………………………………………………………………. Yes
Fast and high capacity IT………………………………………………………………………. YesExtensive and userfriendly communication…………………………………………… YesSoftware for data processing, saving, retrieval and communication……… Yes
Data collection independent of specific usage…………………………………….. ? Use of communication network for collection......................................... YesFunctional organization……………………………………………………………………….. Yes?
Wider statistical coverage by more intensive reuse of data…………………. YesStatistics on-demand…………………………………………………………………………… ?Analytical interest in and tools for exploiting the new possibilities……… Yes?
Legislation regulating the use of micro data………………………………………… Yes?
8/11/2010 Svein Nordbotten 19
What next?
1. Electronically observed data2. Mobile use of statistics
Require legislative, technical and methodoligical preparations.