Open Data Trentino presented at the European Commission (JRC)

transcript

10/04/231 Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it

http://dati.trentino.it/

Open Government Data*

*Part of this presentation is taken from the “Open Government Data Tutorial” gave at CLEI2013 Conference by Lorenzino Vaccari and Juan Pane (Universidad Nacional de Asuncion, Paraguay)

Lorenzino Vaccari

Autonomous Province of Trento, Trento, Italy lorenzino.vaccari@provincia.tn.it

In this presentation…• Introduce Open Government Data

• Intro (Part 1)• Issues (Part 2)

• If you need it, how can you organize it?• Real experience (Part 3)

• Reusing open data• Applications (Part 4)• Semantic layer (Part 5)

10/04/233 Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it 15/10/2013Juan Pane, Lorenzino Vaccari3http://www.point-fort.com/index.php?2012/01/25/805-why-how-what

http://www.point-fort.com/index.php?2012/01/25/805-why-how-what

“is data that can be freely used, reused and redistributed by anyone – subject only, at most, to the requirement to attribute and

sharealike.” *

*(Source: )

http://www.opendefinition.org

usereuse

“open” = redistributioncommercial reusederivative works

BUT, may require:- attribution- share alike

http://myfbcovers.com/uploads/covers/2012/06/09/16628a1094aa012f7c6e0025902480d2/watermarked_cover.jpg

J. Gray (OKF): http://www.slideshare.net/jwyg/open-government-data-what-why-how

The value is in its use

Maurizio Napolitano: http://www.youtube.com/watch?v=YlkjrVAW43Q

New visualizations

http://wheredoesmymoneygo.org/

New visualizations

13/11/20139J. Gray (OKF): http://www.slideshare.net/jwyg/open-government-data-what-why-how

http://openspending.org

Why The Open data are the knowledge base to:

Improve the economic grow and the entrepreneurship based on the development of digital services reusing Public Sector Information

Answer to social needs through the publication of innovative services and applications

Aims at reducing the cost of the public administrative activities within Public – Private Partnerships (PPP)

Improve the transparency of the activities of the public institutions and the participation of the citizens to these activities

How - PrinciplesTim Berners-Lee (5-Stars of Linked Open Data)vs.Tim Davis (5-Stars of Open Data Engagement)vs.OGD: Ten principles for opening up government information…

http://sunlightfoundation.com/policy/documents/ten-open-data-principles/

http://5stardata.info/

http://www.timdavies.org.uk/2012/01/21/5-stars-of-open-data-engagement/

5 Stars Linked Open DataTim Berners-Lee

http://5stardata.info

Create Communityhttp://msnbcmedia.msn.com/j/MSNBC/Components/Photo/_new/pb-121007-spain-tarragona-pyramid-nj-02.photoblog900.jpg

5-Stars of Open Data Engagement

Tim Davis

Open Government Data: Ten principles for opening up government information1. Completeness

2. Primacy (primary source)

3. Timeliness

4. Ease of Physical and Electronic

Access

5. Machine readability

6. Non-discrimination

7. Use of Commonly Owned

Standards

8. Licensing

9. Permanence

10. Usage Costs

State of the ArtWhat is happening around us?• Globally• Europe• Italy

Open Data Charter - G8The principles are: Open Data by Default Quality and Quantity Useable by All Releasing Data for Improved Governance Releasing Data for Innovation

http://opensource.com/government/13/7/open-data-charter-g8

https://www.gov.uk/government/publications/open-data-charter/g8-open-data-charter-and-technical-annex

http://census.okfn.org/

Open Data Census (OKF)

http://census.okfn.org/country/

http://census.okfn.org/

Open Data Barometer (ODI)

OGD in Europe

screenshots

http://epsiplatform.eu/content/european-psi-scoreboard

OGD in EuropeInsert table

http://epsiplatform.eu/content/european-psi-scoreboard http://epsiplatform.eu/content/psi-scoreboard-indicator-list

http://open-data.europa.eu/

OGD in Italy

http://www.dati.gov.it/content/infografica

OGD: Part 2 - Issues

10/04/2325 Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it 08/10/2013Juan Pane, Lorenzino Vaccari25http://evian-thesource.com/kids-having-fun/http://evian-thesource.com/kids-having-fun/

Open Data. Oh ohh

08/10/2013Juan Pane, Lorenzino Vaccari26

LegalLegalOrganizationalOrganizational TechnicalTechnicalAdoptionAdoptionBarriersBarriers

ContextualContextual

http://www.wallpapermania.eu/wallpaper/trick-or-treat-cute-pumpkins-lanterns-halloween-wallpaper

http://de.straba.us/wp-content/uploads/2012/08/barrieres_for_implementation_of_ogd.png

Organizational Barriers

• Not ready• Lack of resources

• IT• Human

• Don’t want to be ready

http://montcomediation.org/images/MCMC_MyWayYourWay.jpg

Legal barriersOpen the Data

All the data that was produced using public money has to be made publicly available (with exceptions)

vs PrivacyYou cannot open data that could allow

correlation of private personal data

Or the complete lack of legislation!

Adoption barriersData is not contextualizedPeople are not informedOpening data is a complex task, opening cleaned

data is even more complex.Unclear licenses

Technical BarriersAccess to data:

OrganizationalTechnical, Downtimes, logins, Payment fees

Fragmentation, incomplete data, scattered

FormatCataloging, indexing, searchLack of explicit semantics, metadataData is not reliableConflicting standards, models,

ontologies

BarriersZuiderwijk et al 2010

Listed 118 socio-technical impediments for opening data in the literature.FindabilityUsabilityUnderstandablityQualityLinkingComparability and compatibilityMetadata….

http://www.ejeg.com/issue/download.html?idArticle=255

Context Barriers

Privileged access to dataOther companies what to avoid legislation of

privacy.Transparency is bad for fraudulent business

http://img.gawkerassets.com/img/182n8vzdlg1iojpg/original.jpg

http://netdna.webdesignerdepot.com/uploads/photo_manipulation/manipulation-9.jpg

Part 3 - Real Experience

Our story started with GeoData…

http://www.territorio.provincia.tn.it

5 Stars Linked Geo Data Catalog

DBpedia TrentinoGeoData Freebase

The “Open Data in Trentino” project

• The “Open Data in Trentino” project is a 3 years initiative finalized to develop an open data infrastructure to enhance Service Innovation for Trentino following the PAT strategy for services innovation enabled by ICT. The project will be developed within a partnership between Trento RISE and the Autonomous Province of Trento (PAT) according to the innovation PAT model

• Goals• Improved quality of life for citizens• Open Data and local businesses• Transparency• Improved efficiency and productivity

Workplan – Best practices Not only a Project, but also a “Change management process”

Best Practices:- Guidelines (metadata, formats, licences)- Point of contact (domain, operator)- ONE dataset each provider- Community Building- Distributed catalog- Clear Licences- Enterprises- Courses- Contest

Nome (Acronimo) Descrizione

Tipo di Dato Estensione del file

Comma Separated Value (CSV) Formato testuale per l'interscambio testuale di tabelle, le cui righe corrispondono a linee e i cui valori delle singole colonne sono separati da una virgola (o punto e virgola)

Dato tabellare .csv

Geographic Markup Language (GML) Formato XML utile allo scambio di dati territoriali di tipo vettoriale

Dato geografico vettoriale

Keyhole Markup Language (KML) Formato basato su XML creato per gestire dati territoriali in tre dimensioni nei programmi Google Earth, Google Maps

Open Document Format (ODF) Formato per l'archiviazione e lo scambio di documenti di testo, fogli di calcolo, diagrammi e presentazioni

Dato tabellare .odc

Resource Description Framework (RDF) Basato su XML, e' lo strumento base proposto da World Wide Web Consortium (W3C) per la codifica, lo scambio e il riutilizzo di metadati strutturati e consente l'interoperabilità tra applicazioni che si scambiano informazioni sul Web

Dato strutturato .rdf

ESRI Shapefile (SHP) Lo Shapefile ESRI è un popolare formato vettoriale per sistemi informativi geografici. Il dato geografico viene distribuito normalmente attraverso tre o quattro files (se indicato il sistema di riferimento delle coordinate). Il formato è stato rilasciato da ESRI come formato (quasi) aperto

.shp, .shx, .dbf,

Extensible Markup Language (XML) E' un formato di markup, ovvero basato su un meccanismo che consente di definire e controllare il significato degli elementi contenuti in un documento o in un testo attraverso delle etichette (markup)

Dato strutturato .xml

Guidelines

…MeteoMeteo GeoDatiGeoDati StatisticaStatistica Comune

TrentoComuneTrento TrasportiTrasporti Etc…Etc……

Tecnological platform

Data Sources

Data Sources Plan

Novembre

Dicembre Gennaio

Catasto # 18

SGC CSW #9

10 20 30 302010 10 20 30

Attività Culturalii #59

Servizio Istruzione #57Attività Form #58

Dati Energia #30

Dati Progettone #63

Dati Motorizzazione #72

Elettorali #35

Gestione Strade #16

Bilancio PAT #37, 38

PersonalePAT #41

Turismo STU #53Idrometr

ici#26

Trentino Cultura #32

Ufficio Rifiuti #34

Servzio Europa #56

Aff FinanziariConsuienze #36

Min. Linguistiche #48

Pub. Esercizi #49

Imp Funivie #50

Immigrazione #52

Sovr. Beni Arch #60Dati Scuola #61

Agenzie Forestali #64Incendi #65

Cinformi Stranieri #66

Imp Depurazione #68Opere Civili #69

Dati Traffico Stra #70

Gestioni Patrimonialii #71

Dati SAT #28

Dati Cons. Prov #3

Trasporti 2.0 #6

OsservatorioLavori Pubb #17

Comune Trento update

Dati Cons. Prov #3

Dati SAT #28

Dati Cons. Prov #3

Dati SAT #28

Dati Cons. Prov #3

Catalog

The Open Knowledge Foundation (OKF) is a non-profit organisation founded in 2004 and dedicated to promoting open data and open content in all their forms – including government data, publicly funded research and public domain cultural content.

(2004)

http://okfn.org

http://dati.trentino.it*

Analysis: http://dati.trentino.it/stats Admin: http://dati.trentino.it/admin Harvesting: http://dati.trentino.it/harvest

* Available for all the data providers of Trentino

Services

Legal Issues

Permissions: share, create, adapt

Actual interoperability!

Constraints: nothing!

http://www.hoax-slayer.com/images/privacy.jpghttp://www.destateparks.com/images/general_info/privacy_policy.jpg

Permissions: share, create, adapt

Actual interoperability!

Constraints: nothing!

Organizational Issues - Macro

Organizational Issues - Micro

Community buildingMunicipalities“Consorzio dei

Comuni”

Municipalities“Consorzio dei

Comuni”

“Comunità di Valle”

of Trentino

“Comunità di Valle”

of Trentino

Private Companies

Private Companies CitizensCitizens

Educational Institutes

Research InstitutesResearch Institutes

International Community

Also Trentino is going to launch a challenge to build software applications and creative products (multimedia, audiovisual products, posters, illustrations) based on the datasets published on the http://dati.trentino.it open data catalog.

#ODTChallenge will be the official hashtag for our first open data challenge in Trentino!

8 months until now68.555 visits 7.988 unique visits2.516 downloads

37,36% returning visitors

62,64% new visitors

NOW- ALL the departmnets demand to be involved- Plus other local actors

AgricultureCultureGeographical DataWelfareWeather ForecastSocial policiesStatisticsTransports…MUNICIPALITY OF TRENTO, and

INFORMATICA TRENTINA

580 datasetsprovided by 10 departments of PAT…

20 reporting errors15 asking for new data10 new suggestions6 OD Applications

100% ENTHUSIASTIC REACTIONS

Want to Know more? A couple of links

http://www.theodi.org/

http://schoolofdata.org/

http://schoolofdata.org/online-resources/

OGD: Part 4 - Applications

Apps4Italy

Best Application: http://parlamento17.openpolis.it/

Open Bilancio

Best Idea: http://opendata.comune.fi.it/open_bilancio/

http://limaio.innovacion.pe/ http://www.limaio.com/demo

Open Source, Open Data, Open Hardware

http://www.mysociety.org/2007/more-travel-maps/morehousing

Johann MITTHEISZ (CIO der Stadt Wien)

http://www.slideshare.net/BrigitteLutz/keynote-mittheisz-cio-stadt-wien/16

Total hours to develop 38 applications:around 2.600

City of Wien saved around 208.000 Euro

Beyond Data (The OpenStreetMap Case)

OpenStreetMap

OpenStreetMap project creates and provides geographical data, such as road maps, freely available to anyone. Behind the establishment and growth of the project have been restrictions on use or availability of map information across much of the world and the advent of inexpensive portable satellite navigation devices.

OpenStreetMap is a free map of theworld, created by someone like you

http://tools.geofabrik.de/mc/?mt0=mapnik&mt1=googlemap&lon=11.12042&lat=46.07224&zoom=18

Watercolor maps

http://content.stamen.com/files/cartography/index_watercolor.html#18.00/46.07204/11.12097

From maps to blankets…

http://softcities.net

Sharing Data Globally(the eHabitat example)

The Group of Earth Observation

GEOSS Data Sharing Principles • Full and Open

Exchange of Data, recognizing Relevant International Instruments and National Policies

• Data and Products at Minimum Time delay and Minimum Cost

• Free of Charge or minimal Cost for Research and Education

http://www.geoportal.org/web/guest/geo_home

GEOSS for biodiversity

http://www.eurogeoss-broker.eu/

The eHabitat Model

http://ehabitat-wps.jrc.ec.europa.eu/ehabitat/

OGD: Part 5 – Semantic Layer

Available

Structured

Open formats

Redefenceable

Linked

Linked Open Data

The best data is an open data

All data must be perfect

Lack of explicit semanticsThe real meaning of the data was kept in the developers mind when creating the data

78http://goo.gl/npEHKr

Lack of explicit semanticsCan lead to things like:

Semantic heterogeneityDifference in the meaning of local data

Available

Structured

Open formats

Redefenceable

Linked

Data Catalog

Entity centric

Importing tool

Entity centric

Importing tool

Entity centric: Added valueAggregated dataAccurate data, manually curatedUnique identifiers, distributed perspectives

Re-think identifiersSemantified values

name Juan Pane

nationality italian

lives in Trento

affiliation Univ. Trento

name Ignacio P. F.

born in Paraguay

date of birth 1980

affiliation PF-UNA

EntitiesReal world: is something that has a distinct,

separate existence, although it need not be a material (physical) existence. Has a set of properties, which evolve over time. Example:

Mental: personal (local) model created and maintained by a person that references and describes a real world entity.

Digital: capture the semantics of real world entities, provided by people.

Entity based Semantic Layer:• Address the integration problems due to

semantic heterogeneity:• Different formats• Different identifiers• Implicit semantics• Homonyms, synonyms, aliases• Partial knowledge• Knowledge evolution

http://www.webfoundation.org/2011/11/5-star-open-data-initiatives/

ImportingTool

The semantic Layer: why?

ImportingTool

REST/HTTPREST/HTTP

i i+1v0

Applications use entities instead of raw data

Importing steps

Selection

Schema Matching

Data Validation

Semantic Enrichment

Reconciliation

Exporting

Publishing

Visualization

Take raw data from dati.trentino.it

Cleanse data

Map to an EntityType

Link data to entities/concepts

Update/insert entities

Export to Entitypedia

Publish to dati.trentino.it

Get insights about entities

1. Source SelectionImport one data file at a time

2. Schema MatchingSelect a target type of entity -> correspondences between the input columns and the output attributes

nome provincia descrizione funivie lat long

Andalo (1047) Provincia di Trento

Sorge su un'ampia sella prativa al centro...

3 654463 712857

Canazei (1450) Trento Prov. Situato all'estremità settentrionale della...

2 511504 147444

3. Data ValidationApplies format and structure validation and possible automatic transformations needed to have the input data in the expected format.

4. Semantic Enrichment (1/2)Entity disambiguation: Transform text references into links to existing entities.

4. Semantic Enrichment (2/2)Natural Language Processing: Extract concepts and entity references from free-text.

5. ReconciliationRun Identity Management Algorithms to identify each row as a new or existing entity.

Result•No Match•Match•Multiple Matches

Action:•Use ID•New ID•Ignore Row

6. ExportingAt this point:We know what to export.All values for target attributes conform to the expected format.All text has been semantified (NLP).All textual references to entities are converted to linksEach row has an identifier

i i+1v0

7. PublishingPut back the semantified entities into CKAN so that

the entities can be Open Data and can be found in the same catalog as the original data.

Developers and find the data files of the cleaned, aggregated entities

But can also interact with the entities via the Entitypedia APIs

8. VisualizationSearch and Navigation

Our Goal

http://www.youtube.com/watch?v=Bq_ZWl1ZXA0

BEYOND

Thanks to all the Open Data in Trentino Team and in particular to:Juan Pane, Maurizio Napolitano, Marco Combetto, Moaz Reyad and Luca Paolazzi

Open Data Trentino presented at the European Commission (JRC)

Technology