Data Journalism 2: cleaning, combining, communicating

transcript

Monday, 5 March 2012

Watch: West Wing on maps www.youtube.com/watch?v=n8zBC2dvERM

Online JournalismCity UniversityPaul Bradshaw

Data 2: clean, combine, communicate

5 things you need to know about eachData journalism in actionWalkthrough

Themes

How clean is the data?

Watch: The Wire: http://www.youtube.com/watch?v=xH_6_8NOfwI

http://onlinejournalismblog.com/2010/10/12/statistical-analysis-as-journalism-benfords-law/Monday, 5 March 2012

“With the help of just Benford’s law and data sets to compare he’s able to demonstrate how the police are systematically hiding over a thousand murders a year in a single state, and that’s just in one small part of the article”

- Pete WardenMonday, 5 March 2012

http://delicious.com/paulb/benfordslawMonday, 5 March 2012

1. Data always needs cleaning up2. Treat the ‘source’ like a source3. Use the right ‘average’ and percentage4. Watch for changing context: inflation, boundaries, classification5. Always work on copies of raw data

5 things you need to know about cleaning data

“What the Independent have done is confuse the UK’s deficit with our debt [making] the debt problem look around eight times worse than it is. And it used the whole of its front page to do so.”

- James BallMonday, 5 March 2012

A town has two hospitals. Hospital A is bigger than hospital B. One of them has a birth rate of 60% boys. Which one is it more likely to be?

Question?

The smaller hospital is more likely to have a 60% birth rate - larger samples are more stable.

Question?

16http://blog.ouseful.info/2011/10/31/power-tools-for-aspiring-data-journalists-r/

Measurement doesn't answer anything if there's only one variableStatistical significanceSample size and selectionControls and the placebo effectRegression to the meanRead up.

What is the data worth?

Data > Text to columns or =SPLITFind & replace=IF(condition, if met, if not)=TRIM, =CONCATENATE=RIGHT, =LEFT, =MID=REPLACE, =SUBSTITUTE=LEN

Getting data ready to answer questions

Edit cells > common transformsEdit cells > split multi-valued cellsFacet > text facetExport...

Walkthrough: cleaning data in Google Refine

Communicating data stories

1. Choose the chart for the purpose2. For answers or for story?3. Good design is when there’s nothing more to take away4. It should be self-contained & have refs5. Be careful with scales and classes

5 things you need to know about visualising data

or http://chartchooser.juiceanalytics.com/Monday, 5 March 2012

http://junkcharts.typepad.com/junk_charts/trifecta-checkup/

What is wrong with this picture?

http://simplecomplexity.net/statistics-without-context/

ManyEyes, Tableau, Number PictureWordle, TagxedoBatchGeo, FusionTablesGephiDelicious.com/paulb/vis+tools

Visualisation tools

Publish embed code & link to dataHave or join a Flickr group for visualisations, comment on othersTumblr blog Digg, Reddit, StumbleuponBuzzdata

Distribution: getting social

Mashing data

1. It is what a journalist does best2. Look for a point of connection: place? Person? Company? Date? Code?3. Mashups can be live, updated or static 4. What an API can do5. What APIs there are

5 things you need to know about mashing data

Yahoo! Pipes, xFruitsOpenHeatMapMapalist, Maptube, FusionTablesScraperwikiGoogle Refine

Mashup tools

Edit column > Add column by fetching URLsUse GREL (Google Refine Expression Language)Search web for help & examples

Walkthrough: grabbing geo data with Google Refine

Questions?

OnlineJournalismClasses.tumblr.comDelicious.com/paulb/cityoj09Delicious.com/paulb/datajournalismDelicious.com/paulb/visualisationDelicious.com/paulb/statistics Delicious.com/paulb/mashups

Before the lab: play with these techniques yourself, have problems, find solutions, raise questions. Install Google Refine and Tableau on your laptop to use.- Visualise, interrogate or mash data

Kaiser Fung - Numbers Rule Your WorldBen Goldacre - Bad ScienceDonna Wong - The WSJ Guide to Information GraphicsBrian Suda - A Practical Guide to Designing with Data

Data Journalism 2: cleaning, combining, communicating

Education