From Publisher To Platform: How The Guardian Used Content, Search, and Open Source To Build a...

21 May 2010Apache Lucene EuroCon

1

From publisher to platformHow the guardian used content, search, and open source to build a powerful new business modelStephen Dunn, Guardian News and Media

http://mail.google.com/mail/?ui=1&view=att&th=11e8ab72613d00d3&attid=0.1&disp=attd&zw
















21 May 2010Apache Lucene EuroCon 2

The publishing era


















We started a long time ago:


















To secure the financial and editorial independence of the Guardian in perpetuity. To promote freedom in the press and liberal journalism globally.

To become the world's leading liberal voice.

“To secure the financial and editorial independence of The Guardian in perpetuity.”

“To promote freedom in the press and liberal journalism globally.”


















2010


















Swine flu

Keyword page

Twitter updates

Content partnerships

Audio

Video Data API

Live blogs

Comment

Mobile siteiPhone app

Newspapers

2010


















1996















1999

7


















1999

7


















01-> 06


















2009★ 1.5M pages

and counting

★ 250M+ pages/month

★ 30M visitors/month

★ 4x Webby award winner (best newspaper site)


















2009★ 1.5M pages

and counting





















2009★ 1.5M pages

and counting





















2009★ 1.5M pages

and counting





















Part of the Web


















• “A cool URI is one that does not change” Tim Berners-Lee 1998• 1.5 million resources redirected to new scheme

11

1. Permanent

http://www.flickr.com/photos/fstorr/


















2. Addressable★ Resources are “about” something - ready for the

social web.

★ We live in “the age of point-at-things” (Coates 2005)


















★ Multiple routes to content

★ Tagging drives discovery

3. Discoverable




















3. Discoverable




















3. Discoverable




















3. Discoverable



































The hackable guardian.co.ukhttp://www.guardian.co.uk/....

















http://www.guardian.co.uk



/technology/internet

/technology/all

/environment/climatechange


















http://www.guardian.co.uk/technology/all










/technology/all



+business/globaleconomy





























/technology/all



+business/globaleconomy





























/technology/all



/rss

/rss

+business/globaleconomy/rss































Results...

16


















First release

Final ReleaseSite traffic growthUnique Users


















3,750,000

7,500,000

11,250,000

15,000,000

18,750,000

22,500,000

26,250,000

30,000,000

Sep 2005 Feb 2006 Jul 2006 Dec 2006 May 2007 Oct 2007 Mar 2008 Aug 2008 Jan 2009

Uni

que

Use

rs

First release



















3,750,000

7,500,000

11,250,000

15,000,000

18,750,000

22,500,000

26,250,000

30,000,000


Uni

que

Use

rs Pre - project

First release



















3,750,000

7,500,000

11,250,000

15,000,000

18,750,000

22,500,000

26,250,000

30,000,000


Uni

que

Use

rs Pre - project

First release


36M


















However...

18


















1 Billion+Internet Users!





































































....”How I stopped worrying about my website and learned to love the whole Internet.”

Matt McAlister


















OPEN IN

Bring in data and apps from the Internet

OPEN OUT

Enable partners to build applications using Guardian content and services for other digital platforms

The Open Strategy





































































"Our most interesting experiments lie in combining what we know with the experience, opinions and expertise of the people who want to participate rather than passively receive.”


















BETA

The Open Platform


















OPEN OUT

Allow partners to build applications using Guardian content and services for other digital platforms

OPEN IN


BETA

OPEN IN



















OPEN OUT


OPEN IN


BETA


















The suite of services enabling partners to build

applications with the Guardian

BETA


















32

BETA


















32

CONTENT API

A service for selecting and

collecting content from the Guardian

for re-use

DATA STORE

A directory of useful data curated by Guardian editors

POLITICS API

Open database of candidates, voting

records, constituencies, election results,

live data on election day

BETA


















Guardian database

CMSSearch engine

REST API

Your App Here!BETA

CONTENT APIA service for selecting and collecting content from the Guardian for

re-use


















BETA

















21 May 2010Apache Lucene EuroCon 35• Stamen Design - APIMaps.org


















Text


















BETA

DATA STOREA directory of

useful data curated by Guardian

editors


















BETA

POLITICS APIOpen database of candidates, voting

records, constituencies, election results, live data on election day



















records, constituencies, election results, live data on election day

BETA


















Open for Business

BETA


















Open for Business


















3 Tiers of access, 3 Revenue models

BESPOKE: Take, reformat, augment our content. Same access as Guardian. Revenue model to be negotiated. Combination of Media, Fees, Downloads.

APPROVED: Take our full article content, with an advert. Guardian keeps ad revenue, you keep rest-of-page revenue

KEYLESS: Take our headlines. You keep associated revenues

1



































OPEN OUT: Developers can now access our full content APIs on demand with keys post-approved.

We are now positioning the platform as a place to do business with us.

So, rapid scalability, reliability, performance, are now core requirements

What this means


















44


re-use



editors


records, constituencies,

election results, live data on election day

2 Open In


















44


re-use



editors


records, constituencies,

election results, live data on election day

MICROAPPSA framework for

integrating 3rd party applications into guardian.co.uk.

2 Open In


















OPEN OUT


OPEN IN





















































App showcase


















What this meansOpen In: Partners can now more easily integrate into our core

The Open Platform will become key to our commercial future.


















Evolving the architecture


















From Publisher to Platform

★Seeking massive growth, but no longer only broadcasting content

★User/partner engagement & contribution on★journalism★data★software★applications★revenue and ads

★ Support developers and partners with data and APIs, need scalability, reliability, speed


















App server App server App server

Web server Web server Web server

CMS

Oracle

Memcached


















App server App server App server

Web server Web server Web server

CMS Data feeds

Oracle

Memcached

Why RDBMS?

5 years ago, fewer alternatives

Understand operations procedures

Can easily recruit DBAs / devs

Developer/ops tools

Business critical system: a safe choice


















Scaling


















Unique Users


















3,750,000

7,500,000

11,250,000

15,000,000

18,750,000

22,500,000

26,250,000

30,000,000


Uni

que

Use

rs

Unique Users


















Unique Users


















12,250,00014,500,00016,750,00019,000,00021,250,00023,500,00025,750,00028,000,000

May 2008 Jul 2008 Sep 2008 Nov 2008 Jan 200956

Unique Users


















Whatʼs going on?

57

★We tag our content (multifaceted)

★Guardian.co.uk is a faceted browse through our tag-space, with editorial teams “spotlighting” key resources on selected nodes.

★Can apply multiple facets in queries faster in a search-like architecture, than an RDBMS


















Whatʼs going on?

57





















Whatʼs going on?

57





















“Related content” from search engine


















59


















Guardian database

CMSSearch engine

REST API

Your App Here!


re-use



































We used Solr/LuceneCan perform complex queries, including full text search

We can change the schema with no downtime.

On our dataset most queries are of a similar cost

Scales very well horizontally

Replication makes it easy to work in the cloud

62


















App server

Web servers

CMS

Memcached

Core

rdbms

63

















Solr

Content API

Cloud, EC2


App server

Web servers

CMS

Memcached

Core

Solr

Solr

Solr

Solr

Solr

rdbms

63




















Simple REST/ HTTP framework allows lightweight development

Applications proxied for performance

Apps generally hosted in the cloud, hot deployment into production

Open in?




















Simple REST/ HTTP framework allows lightweight development

Applications proxied for performance

Apps generally hosted in the cloud, hot deployment into production

Open in?


















App server

Web servers

CMS

Memcached

Core

App

App

App

App

App

App

Apps

Proxy

external hostingapp engine etc

rdbms

65

























App servers

Web servers

CMS

Memcached

Solr

Solr

Solr

Solr

Solr

Solr

Cloud, EC2

App

App

App

App

App

App

Proxyexternal hostingapp engine etc

rdbms

OPEN IN OPEN OUT


















C

Clo

OI

external r

C

Clo

OI

external r

CONTENT

???????


















Thank you

http://www.guardian.co.uk/open-platform

Twitter: @openplatform @cuica (Stephen Dunn)





















Date post:	09-May-2015
Category:	Technology
Upload:	the-guardian-open-platform
View:	9,614 times
Download:	0 times

From Publisher To Platform: How The Guardian Used Content, Search, and Open Source To Build a...

Technology