Data Archiving and Networked Services

&

Peter Doorn (DANS)Ruben Dood (CBS)

a long-standing cooperation to serve research

REGIONAL WORKSHOP16th & 17th October 2014, Athens

Contents

Past:• Long-standing collaboration dating back to 1960s

– Steinmetz Archives acquires punch-card collections– Scientific Statistical Agency founded in 1994– Digitization of historical censuses

Present: • Covenant governing the collaboration

– De-anonymized microdata sets of samples archived at DANS– Remote & On-Site Access: separate contracts with CBS

Future:• Linked data, big data, more data…

Problems of comparing and accessing the Housing Survey and the Labour Force Sample Survey of the CBS (1985)

One of my first publications

Tape of the Labour Force Sample Survey 1981

Digitization of historical censuses, 1996 - …

CBS-archive & library: analog data warehouseFormer CBS building in Voorburg

Joint websites

1999

2004

www.volkstellingen.nl

2015

Archiving and digital restoration of the 1960 census

Program code by Rinus Deurloo merged with data records:

1115100421115 6302120001000995581111405057126086200 B"("N3=")"5ZD,10B 17601115100421110 1306363301000075-81111718035817732405 SC2+NSC3); 17701115100421116 1305352202000900521111205041728284204 ",/")"); 17801115100421119 4303430001000930521111203038829276500 B"("N3=")"5ZD,10B 1790

Punch errors or bitrot?121586010 3855413012060 3 52701322981010 106012158W000 3755113010010 2 52801322981010 0061121586001 3406713012050 00 0152701322981010 1860

Missing and double records:

Persons Missing persons

Double persons

Men 183.970 254.100Women 182.755 7.661Total 366.725 261.761

• 11 milion anonymized inhabitants on 11 million punch cards

• 2000 punch cards per box, 5500 boxes

• Stored in two locations in the late 60s, early seventies

• Punch cards read in 1973, stored on tape

• Further inventory of the census data in 1982

• Analysed and cleaned in 1994

• Re-analysis and digital restoration in 2004

7

datasets andadministrations

survey andadministations

Companies

(government) agencies

administrations

Internetsurvey

Callcenter

Interview

Statistics Netherlands process

Individuals

CBS=Statistics Netherlands (SN)

Linking ‘anonymous’ datasets

RIN

PersonCompanyHouseholdLocation

RINRIN

RIN

Connecting with previous years

RIN

20132012

20112010

Kinds of data

Secure Use Files• For SN employees only. Data at individual level.

Datasets can be linked.• Scientific research under very strict conditions.

Scientific Use Files• Also: Microdata Under Contract, Controlled

Circulation Files.• Data at individual level, but not as detailed as

secure use• Datasets can not be linked

Public Use Files• Can be freely distributed.

Risk management

The risk is that a single person, household or company can be recognized in the dataset.

With Secure Use Files this is unavoidable. Hence very strict conditions.

With Scientific Use Files this is still possible, therefore we provide extra security measurements.

Costs concerning types of data

Secure UseFiles

Scientific Use Files

Public Use Files

Production Covered by SN-budget

Publication

Costs concerning types of data

Secure UseFiles

Scientific Use Files

Public Use Files

Production Covered by SN-budget Not covered open data

Publication Not covered Not covered open data

Microdataservices

MicroDataServices (MiDaS)

Reserved for:• Institutions mentioned in the Statistics

Netherlands Act: universities, planning agencies and Eurostat.

• Institutions/projects admitted by the Central Committee for Statistics.

• no direct administrative authority,

• explicit research aim,

• publish for public use and

• a good reputation

Conditions for access tot secure use files

• Access is limited to the datasets that are required for the research at hand.

• Proof of employment for each researcher.• Confidentiality declaration by each

researcher.• IT-security: Citrix• Output screening by SN.• Additional bio-metric identification

Biometric identification

16

Current use

• Over 2000 well-documented datasets.• Over 100 customer organisations (mainly

universities, planning agencies and statistical research agencies).

• Over 500 registered researchers.• 8 On Site, 100 Remote Access work stations.

Over 15 of the RA-sites are not in the Netherlands.

• Over 10 new projects per month. • Over 280 active projects at this moment.

17

Scientific Use Files

Can be part of a contract between DANS and Statistics Netherlands. Can also be funded by a third party.

Are published in DANS EASY format and published on the DANS website.

Present: Covenant CBS-DANS• Period: 2011-2015• CBS makes available (micro-) data for scientific users entitled or

authorized by CBS, according to a yearly agreed list of data files. The files are delivered with sufficient documentation and a code book

• DANS archives these files in a durable way and makes them available to those entitled or authorized by CBS

• DANS pays CBS a yearly sum (116 Keuro since 2013)• CBS en DANS jointly propose better data access for research to

ministries

CBS Datasets in DANS EASY archive

of which 87 protected microdata files

Joint publications

[Including related projects:Historical GIS, HASH, HDNG]

Joint seminars & conferences

DANS at CBS Volleyball tournament

Present & future: Linked DataChallenge: link information from tables across time and to information in the outside world• 3 Types of Censuses [Population/Occupation/Housing]• 17 census years • 2288 census tables• 33283 annotations• 17 million characters

Tables

Linked data from tables (RDF)

Linked Open Data cloud

Harmonization Harmonization

T1 T2 T3

Census 1869 Province of Noord-Brabant

• 110,585,567 total triples• 10,272,862 marked cells triples• 389,132 hierarchical row headers• 7,960,911 data cells• 61,110 column headers• 3,609 row properties• 2,150 titles• 1,581,546 row headers• 274,404 metadata cells

http://www.cedar-project.nl/ http://lod.cedar-project.nl

original paper table

digital spreadsheet

Additional DANS wishes for future and further cooperation

1. More data (e.g. on health, economy, etc.)2. More data3. More data4. More user groups entitled/authorized to access CBS data

(and data produced in collaboration with CBS)5. Use of CBS register data for sample drawing for research

projects6. Joint project proposal on e.g. linked data and big data7. Archiving of historical CBS datasets by DANS8. CBS certification (as trustworthy digital repository)

Data Archiving and Networked Services

&Thank you for your attention

www.dans.knaw.nlwww.narcis.nlwww.cbs.nl

peter.doorn@dans.knaw.nlTwitter: @pkdoorn @DANSKNAW