Open local data: challenges and opportunities

transcript

WMRO Open Data Event, Birmingham, July 15, 2010

Open Local Data Challenges & opportunities

It’s been a good year for open government datadata.gov.uk, Ordnance Survey, MPs expenses...

But what about local data?

• Frankly it’s a mess

• Sporadically published by central government

• Inaccessible & impenetrable council websites

• Opaque local public bodies and quangos (goodbye RDAs, hello LEPs)

• At best unclear & at worst unusable legal situation

• Start with the basics. Who are the councillors, where do they represent and what committees do they sit on?

• How easy it that information to find & reuse?

Simple questions, hard to answer

• How does the budget of my council compare with similar ones?

• What’s the background of the chief executive of my council?

• Is my route to work going to affected by roadworks next week?

• What are the connections between the companies awarded contracts and the councillors & senior management?

• How will cuts in services be decided and what can I do to influence the decisions?

TransparencyA brief example.

• A story in Private Eye

• An investigation by the District Auditor

• A report buried in the nether recesses of the website...

• ...with no relevant heading

• Accessible only as a PDF of a scan of a document

• Is it any wonder suspicions are raised?

Engagement

• Local Authorities now have a duty to engage. This means• Reducing barriers to understanding

and involvement

Engagement

and involvement

• Accessible for all

Engagement

and involvement

• Available to use when, where & how we want to

Engagement

and involvement

• Available to use when, where & how we want to

• Publishing as data means easy to be repurposed – for mobile, for mashups, for offline use (e.g. Postcode Paper)

Equality

• At the moment, all this information is available... at a cost

Equality

£££

Equality

• Huge asymmetry of information (and thus power)

£££

Equality

• Huge asymmetry of information (and thus power)

• Raises the barriers to involvement, and, critically, to challenge

£££

EfficiencyNow: cumbersome, wasteful, opaque, error prone

Typical local data flows

Council

Govt Dept A

Govt Dept B

Govt Dept C

upload

Council

Govt Dept A

Govt Dept B

Govt Dept C

upload

public (maybe)

Enter OpenlyLocal...

• Inspiration was a Manchester project, MCC Work For You. Doing something concrete to solve a problem.

• Screen-scrapes council websites, to turn web pages into structured data. Now over 150 councils done, with basic information on all 434

• Pulls info from about local councils, plus ONS, OS, NPIA, CLG...

• Increasingly working with local authorities to help them publish their information as open data

• All open data – free for reuse, including commercial reuse

• 100% accessible. Data first; bells & whistles later

RDFJSON

RDFJSONXML

OpenlyLocal: where next?

• More data.

• More data. • Election data

• More data. • Election data✓

• More data. • Election data• Spending data

✓✓

• More data. • Election data• Spending data• Where I live...

✓✓

• More data. • Election data• Spending data• Where I live...

✓✓✓

• More data. • Election data• Spending data• Where I live...• Cabinet members

✓✓✓

• More data. • Election data• Spending data• Where I live...• Cabinet members• Mayors

✓✓✓

• More data. • Election data• Spending data• Where I live...• Cabinet members• Mayors• Police authorities

✓✓✓

• More councils.

✓✓✓

• More councils.

• More connections.

✓✓✓

• More councils.

• More ways for councils to give us data.

✓✓✓

• More councils.

• More ways for councils to give us data.

• More tie-ups with hyperlocal news/community sites.

✓✓✓

The OpenElectionData project

• Tackling the open local data problem, one set at a time & learning lessons on the way

• Succeed, or fail forward

• No public database of local election results, only a commercial one (subsidized by the Electoral Commission)

• Allows even those with no prior knowledge of linked data/RDF/semantic web (choose fave buzzword here) to take part

• Just need HTML competence

• Over 20 councils took part, and invaluable lessons learned (see report)

<tr rel='openelection:candidacy'>  <th scope="row" rel="openelection:candidate">Keith William Cockroft</th> <td rel="openelection:party" resource="http://openelectiondata.org/id/parties/6">Labour</td>  <td property="openelection:candidateVoteCount" datatype="xsd:integer">330</td> <td>16.6%</td> <td property="openelection:elected" datatype="xsd:boolean" content="false">No</td> </tr> <tr rel='openelection:candidacy'> <th scope="row" rel="openelection:candidate"> Brenda Lilian Constable 37 Morley Road, Burntwood, Staffordshire WS7 2DE </th>

The future for local data? Local government for local people

Publish once, consume many times

Council open dataGovt Dept A

Govt Dept B

Govt Dept C

publicpublic

interest/media/

community/commercial

Open data opportunities

✤ Massive efficiency savings (somewhat important when the cuts are so severe) – reducing duplication, management layers, automating regular routine tasks

✤ Reduction in bureaucracy (and associated frustration of frontline staff and citizens)

✤ Improved online services for citizens at zero cost (provided by websites/apps that work with any council)

✤ Increased transparency

✤ More freedom from central government targets, auditing

Challenges

✤ Lack of ‘corporate’ awareness/understanding of open data issues

Challenges

✤ There is a lack of even basic web skills at some councils

Challenges

✤ Many councils lack web publishing resources, never mind the resources to implement open, linked data publishing

Challenges

✤ The understanding of even the basics of linked data and the steps to publishing public data in this way is very, very limited, even outside of the public sector

Challenges

✤ Getting councils to publish data in open, linked formats requires dedication, and a range of resources and a mix of skills

Challenges

✤ Getting councils to publish data in open, linked formats requires dedication, and a range of resources and a mix of skills

✤ Outsourced web publishing contracts often make it difficult and expensive to publish open, linked data

More Challenges

• IDs – councils don’t generally use them; central government uses many... inconsistently

More Challenges

• Data tied up in PDFs (will we ever get this back?)

More Challenges

• Legacy systems. That nobody now understands

More Challenges

• Private companies/JVs. No FoI. No access to data

More Challenges

• Outsourcing (councils have outsourced the skills & knowledge and now are often dumb consumers)

More Challenges

• Outsourcing (councils have outsourced the skills & knowledge and now are often dumb consumers)

• Silos, especially of budgets. How do you get money from one budget to another (even if the whole benefits)

Yet More Challenges

✤ The size of the cuts mean they will be blunt, eliminating useful services, sparing wasteful ones. Who & what will be left & what will it mean for local public services?

Yet More Challenges

✤ Few resources to invest in doing this, and even less understanding of where to invest it. Danger of using usual procedures, suppliers (procurement, Capita, Serco etc)

Yet More Challenges

✤ Few resources to invest in doing this, and even less understanding of where to invest it. Danger of using usual procedures, suppliers (procurement, Capita, Serco etc)

✤ Little incentive for those organisations and people threatened by open data – for them it is a Darwinian fight for survival

An example dataset: local spending data (deadline jan 2011)

An example dataset: local spending data (deadline jan 2011)✤ Technical challenge: how to extract the data from the accounts system

without including non-suppliers (e.g. foster parents, children)✤ How many councils are working on this (hopefully all of them)

✤ How many have blogged, posted the steps/issues/solutions – zero?

✤ What are the key fields that MUST be included, and do these need to be listed somewhere?

✤ Organizational challenge✤ What is this going show about our spending/processes/contracts?

✤ How will we compare with other similar bodies?

✤ What will be the consequences (FoI requests, changed relationship with suppliers, unpopular services)

✤ Organizational challenge✤ What is this going show about our spending/processes/contracts?

✤ How will we compare with other similar bodies?

✤ What will be the consequences (FoI requests, changed relationship with suppliers, unpopular services)

✤ Systemic challenge✤ How will this affect the relationship with the citizens?

✤ What are the implications for our structure, processes, power...?

Power in the public sector was...

✤ Access to people

✤ Access to information

✤ Understanding structure/how things worked

undermined by social networking

democratized by open data

crowdsourced? bypassed?

All this is a proxy for the real battle: relevance

✤ How relevant is your local government to you?

✤ How well do the ‘proper channels’ work for you?

✤ What is your ability to achieve things outside the existing structures?

✤ What difference does your council make to your life?

Open data is local government’s chance to stay relevant to their

citizens/economy(but it’ll have its work cut out)

Open local data: challenges and opportunities

News & Politics