Data journalism: Data rules, while data rule

transcript

IntroductionData: a new era of journalism (?)

Lying with dataSummary and conclusion

References

Data rules, while data rule

George Giannakopoulos 1,2

(ggianna@iit.demokritos.gr)

1SKEL Lab, NCSR “Demokritos”, Greece2SciFY PNPC, Greece

George Giannakopoulos (ggianna@iit.demokritos.gr) Data rules, while data rule

References

MotivationWhat is data?From data to big, linked, open dataA potential future: Nice projects and future tools

Why are we here?

Learn the basics of data

Understand value and risks of using dataAvoid pitfalls when reporting data-basedfindingsIdentify the role of the journalist with respectto data

References

Why are we here?

Learn the basics of dataUnderstand value and risks of using data

Avoid pitfalls when reporting data-basedfindingsIdentify the role of the journalist with respectto data

References

Why are we here?

Learn the basics of dataUnderstand value and risks of using dataAvoid pitfalls when reporting data-basedfindings

Identify the role of the journalist with respectto data

References

Why are we here?

Learn the basics of dataUnderstand value and risks of using dataAvoid pitfalls when reporting data-basedfindingsIdentify the role of the journalist with respectto data

References

Why can I help?

Not a journalist

Not a visualization expertData mining over 10 yearsCommunicating findings over 10 yearsReviewing over 10 years

References

Why can I help?

Not a journalistNot a visualization expert

Data mining over 10 yearsCommunicating findings over 10 yearsReviewing over 10 years

References

Why can I help?

Not a journalistNot a visualization expertData mining over 10 years

Communicating findings over 10 yearsReviewing over 10 years

References

Why can I help?

Not a journalistNot a visualization expertData mining over 10 yearsCommunicating findings over 10 years

Reviewing over 10 years

References

Why can I help?

Not a journalistNot a visualization expertData mining over 10 yearsCommunicating findings over 10 yearsReviewing over 10 years

References

Enough about me

A little more aboutyou?

References

Data and friends

Data measured or calculated value ofproperty

Information data extended with interpretationKnowledge contextualized, interpreted, validated

data, supporting action (e.g. prediction)

See also: Βλαχάβας et al. 2002, Chapter 3, Boisot and Canals 2004, Tuomi 1999

References

Data and friends

Information data extended with interpretation

Knowledge contextualized, interpreted, validateddata, supporting action (e.g. prediction)

See also: Βλαχάβας et al. 2002, Chapter 3, Boisot and Canals 2004, Tuomi 1999

References

Data and friends

Information data extended with interpretationKnowledge contextualized, interpreted, validated

data, supporting action (e.g. prediction)See also: Βλαχάβας et al. 2002, Chapter 3, Boisot and Canals 2004, Tuomi 1999

References

Kinds of data (based on structure)

Unstructured free text, blog entries

Fully structured databases, data tables, spreadsheetsSemi-structured eXtensible Markup Language (XML), Resource

Description Framework (RDF)

References

Unstructured free text, blog entriesFully structured databases, data tables, spreadsheets

Semi-structured eXtensible Markup Language (XML), ResourceDescription Framework (RDF)

References

Unstructured free text, blog entriesFully structured databases, data tables, spreadsheetsSemi-structured eXtensible Markup Language (XML), Resource

Description Framework (RDF)

References

Where can we find data?

Finance index values, yearly reports

Health health record, lab measurementsMore? What do you think?

References

Finance index values, yearly reportsHealth health record, lab measurements

More? What do you think?

References

Finance index values, yearly reportsHealth health record, lab measurementsMore? What do you think?

References

Do we generate data?

Social media posts, comments

Mobile device use position, usage of applicationsMore?

References

Social media posts, commentsMobile device use position, usage of applications

References

Social media posts, commentsMobile device use position, usage of applications

References

Who uses data?

State policy making, international relations,public image

Businesses marketing, due diligence, riskassessment, business policy

Researchers build hypotheses, confirm findings,discover relations

Citizens understand everyday changes,understand choices, plan ahead

References

Who uses data?

References

Who uses data?

References

Who uses data?

References

Why are data useful?

Problem indicators (e.g. deviations)

Common basis for discussionMeasurableConsidered unbiasedConsidered verifiableOftentimes interlinkable

References

Problem indicators (e.g. deviations)Common basis for discussion

MeasurableConsidered unbiasedConsidered verifiableOftentimes interlinkable

References

Problem indicators (e.g. deviations)Common basis for discussionMeasurable

Considered unbiasedConsidered verifiableOftentimes interlinkable

References

Problem indicators (e.g. deviations)Common basis for discussionMeasurableConsidered unbiased

Considered verifiableOftentimes interlinkable

References

Problem indicators (e.g. deviations)Common basis for discussionMeasurableConsidered unbiasedConsidered verifiable

Oftentimes interlinkable

References

Problem indicators (e.g. deviations)Common basis for discussionMeasurableConsidered unbiasedConsidered verifiableOftentimes interlinkable

References

Let us summarize

Data can be the basis for knowledgeData have various levels of structureAll generate and consume dataData can have value

References

Data flavours

Big data Large scale, volatile, challenging to follow

Linked data Inherent interconnectivityOpen data Available in a meaningful manner

References

Data flavours

Big data Large scale, volatile, challenging to followLinked data Inherent interconnectivity

Open data Available in a meaningful manner

References

Data flavours

Big data Large scale, volatile, challenging to followLinked data Inherent interconnectivityOpen data Available in a meaningful manner

References

What is Big Data?

Datain a new scale...and more...

References

Examples of Big Data

Social media posts per day: Gbytes per day on Twitter (50 milliontweets / day)

Wind turbine sensors: Tbytes per day per turbine (KHz samplingof values)Bioinformatics data: Tbytes per day per machine (up to 3 billionreads per run)Overall: 2.5 exabytes of data generated per day (and going up)

References

Social media posts per day: Gbytes per day on Twitter (50 milliontweets / day)Wind turbine sensors: Tbytes per day per turbine (KHz samplingof values)

Bioinformatics data: Tbytes per day per machine (up to 3 billionreads per run)Overall: 2.5 exabytes of data generated per day (and going up)

References

Social media posts per day: Gbytes per day on Twitter (50 milliontweets / day)Wind turbine sensors: Tbytes per day per turbine (KHz samplingof values)Bioinformatics data: Tbytes per day per machine (up to 3 billionreads per run)

Overall: 2.5 exabytes of data generated per day (and going up)

References

Social media posts per day: Gbytes per day on Twitter (50 milliontweets / day)Wind turbine sensors: Tbytes per day per turbine (KHz samplingof values)Bioinformatics data: Tbytes per day per machine (up to 3 billionreads per run)Overall: 2.5 exabytes of data generated per day (and going up)

References

Features of Big Data

Volume size

Velocity speed of changeVariety variety and difficulty to read and combineVeracity uncertainty and difficulty of validationValue difficulty to turn into actionable knowledge

References

Volume sizeVelocity speed of change

Variety variety and difficulty to read and combineVeracity uncertainty and difficulty of validationValue difficulty to turn into actionable knowledge

References

Volume sizeVelocity speed of changeVariety variety and difficulty to read and combine

Veracity uncertainty and difficulty of validationValue difficulty to turn into actionable knowledge

References

Volume sizeVelocity speed of changeVariety variety and difficulty to read and combineVeracity uncertainty and difficulty of validation

Value difficulty to turn into actionable knowledge

References

Volume sizeVelocity speed of changeVariety variety and difficulty to read and combineVeracity uncertainty and difficulty of validationValue difficulty to turn into actionable knowledge

References

What is Linked Data?

Small statements

Fragments of knowledgePowerful when combined

<George, loves, Georgia>

References

Small statementsFragments of knowledge

Powerful when combined

<Georgia,listensTo,folkRock><Dylan,plays,folkRock>

References

Small statementsFragments of knowledgePowerful when combined

Suggest to George a Dylan CD asa gift for Georgia

References

Examples of Linked Data

Web annotations hidden in web pages, blogs, social media (e.g.Facebook Graph)

DBPedia linked-data view of WikipediaNano-publications scientific results and data as statements

References

DBPedia linked-data view of Wikipedia

Nano-publications scientific results and data as statements

References

DBPedia linked-data view of WikipediaNano-publications scientific results and data as statements

References

Features of Linked Data

Usually automatically generatedComputer-friendlyNo meaning

References

What is Open Data?

Openly available data

...usable

...reusable under well-specified licencesUsually public data

References

What is Open Data?

Openly available data...usable

...reusable under well-specified licencesUsually public data

References

What is Open Data?

Openly available data...usable...reusable under well-specified licences

Usually public data

References

What is Open Data?

Openly available data...usable...reusable under well-specified licencesUsually public data

References

Examples of Open Data

Country-wide statistics and demographicsGeo-locations of monumentsPublic consultation results

References

Let us summarize

Big data: many, volatile, difficult to handleLinked data: fragments of knowledge, computerfriendlyOpen data: publicly available, reusable

References

Your Data Stories

Easily consume dataCombine and integrate dataEmpower with social mediafeedback

yourdatastories.eu

References

DemocracIT

Build on consultationsQuantify problems and biasesCrowdsourcing

democracit.org

References

Other projects

References

More to come?

Support editing in real-timeFind related dataSuggest visualization

Stay tuned.

References

An informal break

Teaser: Privacy, Security, Anonymity and Data

References

The value of dataOpportunities and risks in using dataData analysis and visualization: mistakes and pitfalls

What data can do

Data contain storiesData support storiesData verify stories

References

Case 1: Money trails

Find Mr Smith payments (public money) from Diavgeia

Claim that researchers get too much money!

References

Case 1: Money trails

Find Mr Smith payments (public money) from DiavgeiaClaim that researchers get too much money!

References

Case 1: Risks

Source idiosyncracies: Diavgeia problems

Limited (?) documentationLack of cross-check method

References

Case 1: Risks

Source idiosyncracies: Diavgeia problemsLimited (?) documentation

Lack of cross-check method

References

Case 1: Risks

Source idiosyncracies: Diavgeia problemsLimited (?) documentationLack of cross-check method

References

Case 2: Leaked documents

Thousands of leaked documents from a government

Apply sentiment analysis to understand sentimentShow that all is gloomy!

References

Thousands of leaked documents from a governmentApply sentiment analysis to understand sentiment

Show that all is gloomy!

References

Thousands of leaked documents from a governmentApply sentiment analysis to understand sentimentShow that all is gloomy!

References

Case 2: Risks

Tool idiosyncracies: sentiment analysis approx. 75% accurate

Lack of cross-check method

References

Case 2: Risks

Tool idiosyncracies: sentiment analysis approx. 75% accurateLack of cross-check method

References

Data analysis

know what you look forunderstand what you readuse your intuition (but not too much!)double check (through a different path!)

References

Visualization: keep it simple

keep it simplekeep it focusedverifydocument and support (through references)tell a storythink of your audience

References

Why you shouldn’t trust scientists (too much)Hiding BiasReporting research

Lines and hypotheses

References

Self-fulfilled prophecy

A leader claims that the market will go down in a day

The market will go down

References

Self-fulfilled prophecy

A leader claims that the market will go down in a dayThe market will go down

References

Correlation vs causality

References

Beware of the index

An index of poverty...based on the percentage of people that take less than 5% of themaximum incomeIs it a good idea?

References

Axes: Normalization

References

Axes: Units and scales

References

Axes: Units and scales

References

Ratios not kept

References

Other cases:

Ordering: Greek electionsLying with percentages: increase from 10% to 12%; 20%increase?Question bias: important answers omittedQuestion bias: generic question

References

Pragmatics

References

What we should look for

Experimental setting

Confidence vs statistical confidenceExact conclusionFunding sourcesConfirmation by experts

References

Experimental settingConfidence vs statistical confidence

Exact conclusionFunding sourcesConfirmation by experts

References

Experimental settingConfidence vs statistical confidenceExact conclusion

Funding sourcesConfirmation by experts

References

Experimental settingConfidence vs statistical confidenceExact conclusionFunding sources

Confirmation by experts

References

Experimental settingConfidence vs statistical confidenceExact conclusionFunding sourcesConfirmation by experts

References

Another informal break

Teaser: Changing our lives through reporting

References

How can I apply data analysis (if I am not an expert)?Summary: things to remember

Locate sources

https://www.diavgeia.gov.gr/http://data.gov.gr/http://geodata.gov.gr/http://open-data.okfn.gr/https://delicious.com/http://www.linkedopendata.gr/https://data.europa.eu/euodp/https://www.constituteproject.org/http://www.findthedata.com/

References

Next steps

Study more

Form a networkCollaborateFocus

References

Next steps

Study moreForm a network

CollaborateFocus

References

Next steps

Study moreForm a networkCollaborate

References

Next steps

Study moreForm a networkCollaborateFocus

References

Who should I work with?

Visualization experts

Data mining expertsDomain expertsDevelopers... with a twist

References

Visualization expertsData mining experts

Domain expertsDevelopers... with a twist

References

Visualization expertsData mining expertsDomain experts

Developers... with a twist

References

Visualization expertsData mining expertsDomain expertsDevelopers... with a twist

References

Data is useful, but can be misinterpreted

Data contain, support, confirm stories...but they cannot tell the storyAnalysis of data needs expertsVisualization of data needs expertsJournalism needs expertsWorking with data needs a team

References

Data is useful, but can be misinterpretedData contain, support, confirm stories

...but they cannot tell the storyAnalysis of data needs expertsVisualization of data needs expertsJournalism needs expertsWorking with data needs a team

References

Data is useful, but can be misinterpretedData contain, support, confirm stories...but they cannot tell the story

Analysis of data needs expertsVisualization of data needs expertsJournalism needs expertsWorking with data needs a team

References

Data is useful, but can be misinterpretedData contain, support, confirm stories...but they cannot tell the storyAnalysis of data needs experts

Visualization of data needs expertsJournalism needs expertsWorking with data needs a team

References

Data is useful, but can be misinterpretedData contain, support, confirm stories...but they cannot tell the storyAnalysis of data needs expertsVisualization of data needs experts

Journalism needs expertsWorking with data needs a team

References

Data is useful, but can be misinterpretedData contain, support, confirm stories...but they cannot tell the storyAnalysis of data needs expertsVisualization of data needs expertsJournalism needs experts

Working with data needs a team

References

Data is useful, but can be misinterpretedData contain, support, confirm stories...but they cannot tell the storyAnalysis of data needs expertsVisualization of data needs expertsJournalism needs expertsWorking with data needs a team

References

Thank you

George Giannakopoulos 1,2

(ggianna@iit.demokritos.gr)

1SKEL Lab, NCSR “Demokritos”, Greece2SciFY PNPC, Greece

References

References I

Boisot, Max and Agustí Canals (2004). “Data, information andknowledge: have we got it right?” In: Journal of EvolutionaryEconomics 14.1, pp. 43–67. : 0936-9937, 1432-1386. :10.1007/s00191-003-0181-9 (cit. on pp. 12–14).

Tuomi, I. (1999). “Data is more than knowledge: implications of thereversed knowledge hierarchy for knowledge management andorganizational memory”. In: IEEE Comput. Soc, p. 12. :978-0-7695-0001-0. : 10.1109/HICSS.1999.772795. :http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=772795 (cit. on pp. 12–14).

Βλαχάβας, Ι. et al. (2002). Τεχνητή Νοημοσύνη. ΕκδόσειςΓαρταγάνη, Θεσσαλονίκη (cit. on pp. 12–14).

Data journalism: Data rules, while data rule

Data & Analytics