8/8/2019 Scaling the Grammys

1/94

Scaling the

Grammyswith Drupal

Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

2/94

Nathan Haug

"quicksketch" on d.o

Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

3/94

Nathan Haug

"quicksketch" on d.o

Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

4/94

Nathan Haug

"quicksketch" on d.o

Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

5/94

Nathan Haug

"quicksketch" on d.o

Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

6/94

Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

7/94

The Recording Academy wanted more

"social" features and interaction.

Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

8/94

The Recording Academy wanted more

"social" features and interaction.

Easier maintenance of site andfunctionality.

Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

9/94

The Recording Academy wanted more

"social" features and interaction.

Easier maintenance of site andfunctionality.

Single place for Nominee and Winnerhistory.

Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

10/94

Some Numbers

Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

11/94

Some Numbers

Accurate

^

Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

12/94

214 million page views on Grammy

Award Show day, most over the 6 hourwindow of the broadcast.

Some Numbers

Accurate

^

Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

13/94

214 million page views on Grammy

Award Show day, most over the 6 hourwindow of the broadcast.

50,000 new registered users over the

same period.

Some Numbers

Accurate

^

Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

14/94

214 million page views on Grammy

Award Show day, most over the 6 hourwindow of the broadcast.

50,000 new registered users over the

same period.

5% Database load during show.

Some Numbers

Accurate

^

Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

15/94

214 million page views on Grammy

Award Show day, most over the 6 hourwindow of the broadcast.

50,000 new registered users over the

same period.

5% Database load during show.

1% Web server load during show.

Some Numbers

Accurate

^

Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

16/94

Hardware

Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

17/94

Hardware

Web Servers

Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

18/94

Hardware

Web Servers

Load Balancer

Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

19/94

Hardware

Web Servers

Load Balancer

Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

20/94

Hardware

Web Servers

Load Balancer

MySQL Server

Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

21/94

Hardware

Web Servers

Load Balancer

MySQL Server

Solr Server

Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

22/94

Hardware

Web Servers

Load Balancer

MySQL Server

Solr Server

NAS (files directory)Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

23/94

Hardware

Web Servers

Load Balancer

MySQL Server

Solr Server

NAS (files directory)Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

24/94

Hardware

Web Servers

Load Balancer

MySQL Server

Solr Server

NAS (files directory)Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

25/94

Hardware

Web Servers

Load Balancer

MySQL Server

Solr Server

NAS (files directory)Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

26/94

Hardware

Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

27/94

Software

Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

28/94

Software

Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

29/94

Use caching mechanisms and software

Software

Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

30/94

Use caching mechanisms and software

Configure Drupal to utilize this caching

Software

Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

31/94

Use caching mechanisms and software

Configure Drupal to utilize this caching

Give servers multiple roles

Software

Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

32/94

Use caching mechanisms and software

Configure Drupal to utilize this caching

Give servers multiple roles

Dedicate servers for "expensive"operations (i.e. Search)

Software

Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

33/94

Web Server

Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

34/94

Web Server

PressFlow

Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

35/94

Web Server

PressFlow

APC Op-Code Cache for PHP

Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

36/94

Web Server

PressFlow

APC Op-Code Cache for PHP

Database Slave

Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

37/94

Web Server

PressFlow

APC Op-Code Cache for PHP

Database Slave

Varnish in front of Apache

Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

38/94

Web Server

Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

39/94

Web Server

Distribution of Drupal (100% API

Compatible with existing modules)

Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

40/94

Web Server

Distribution of Drupal (100% API

Compatible with existing modules)

Enables use of CDNs, ReverseProxies, and Slave database servers

Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

41/94

Web Server

Distribution of Drupal (100% API

Compatible with existing modules)

Enables use of CDNs, ReverseProxies, and Slave database servers

Does not create sessions for

Anonymous users

Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

42/94

Web Server

Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

43/94

Web Server

APC (Another PHP Cache) reduces

disk access when parsing PHP files.

Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

44/94

Web Server

APC (Another PHP Cache) reduces

disk access when parsing PHP files.

Keeps fully parsed PHP files inmemory.

Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

45/94

Web Server

Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

46/94

Web Server

SQL Slave

Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

47/94

Web Server

SQL Slave

All "Pagered" and Limit queries hit

the slave server.

Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

48/94

Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

49/94

Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

50/94

Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

51/94

Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

52/94

Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

53/94

Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

54/94

Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

55/94

Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

56/94

Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

57/94

Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

58/94

Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

59/94

Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

60/94

Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

61/94

Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

62/94

Web Server

Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

63/94

Web Server

The MOST important cache on our

infrastructure.

Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

64/94

Web Server

The MOST important cache on our

infrastructure.

Serves up images, static files, andanonymous pages from memory.

Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

65/94

Web Server

The MOST important cache on our

infrastructure.

Serves up images, static files, andanonymous pages from memory.

Cookie-based skipping of Varnish

lets users fall through to Apache.

Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

66/94

Master Database

Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

67/94

Master Database

MySQL

Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

68/94

Master Database

MySQL

Doubles as Memcache server

Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

69/94

Master Database

MySQL

Doubles as Memcache server

Mirrored as a backup, NOT a Master-Master configuration.

Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

70/94

Master Database

Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

71/94

MySQL 5.2

Master Database

Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

72/94

MySQL 5.2

Turn on query caching in my.cnf:

Master Database

Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

73/94

MySQL 5.2

Turn on query caching in my.cnf:

Master Database

query_cache_limit = 1M

query_cache_size = 16M

Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

74/94

Master Database

MEMCACHED

Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

75/94

All Cache tables and Sessions

stored in memory.

Master Database

MEMCACHED

Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

76/94

All Cache tables and Sessions

stored in memory.

16 Cache tables on Grammy.com

Master Database

MEMCACHED

Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

77/94

All Cache tables and Sessions

stored in memory.

16 Cache tables on Grammy.com

Anonymous pages served entirely

from memory with page cache

enabled.

Master Database

MEMCACHED

Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

78/94

Solr Server

Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

79/94

Outsources search to dedicated box.

Solr Server

Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

80/94

Outsources search to dedicated box.

Java server. Not a PHP technology.

Solr Server

Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

81/94

Outsources search to dedicated box.

Java server. Not a PHP technology.

TomCat as host.

Solr Server

Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

82/94

All Together Now

MEMCACHED

Pure

Anonymous

Anonymous

w/ NO_CACHE Authenticated

Slave

Master

Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

83/94

All Together Now

MEMCACHED

Pure

Anonymous

Anonymous

w/ NO_CACHE Authenticated

Slave

Master

Peak Traffic Times

Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

84/94

Off-Peak

All Together Now

MEMCACHED

Pure

Anonymous

Anonymous

w/ NO_CACHE Authenticated

Slave

Master

Peak Traffic Times

Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

85/94

Off-Peak

All Together Now

MEMCACHED

Pure

Anonymous

Anonymous

w/ NO_CACHE Authenticated

Slave

Master

Peak Traffic Times

Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

86/94

Off-Peak

All Together Now

MEMCACHED

Pure

Anonymous

Anonymous

w/ NO_CACHE Authenticated

Slave

Master

Peak Traffic Times

Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

87/94

Off-Peak

All Together Now

MEMCACHED

Pure

Anonymous

Anonymous

w/ NO_CACHE Authenticated

Slave

Master

Peak Traffic Times

Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

88/94

Drupal Integration

Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

89/94

Drupal Integration

Most all configuration in settings.php

Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

90/94

Drupal Integration

Most all configuration in settings.php

Slave database setup

Sunday, June 27, 2010

8/8/2019 Scaling the Grammys

91/94

Drupal Integration

Most all configuration in settings.php

Slave database setup

Memcache server setup

Sunday, June 27, 2010

D l I i

8/8/2019 Scaling the Grammys

92/94

Drupal Integration

Most all configuration in settings.php

Slave database setup

Memcache server setup

Varnish Reverse Proxy setup

Sunday, June 27, 2010

D l I i

8/8/2019 Scaling the Grammys

93/94

Drupal Integration

Most all configuration in settings.php

Slave database setup

Memcache server setup

Varnish Reverse Proxy setup

NO_CACHE cookie checking

Sunday, June 27, 2010

D l I i

8/8/2019 Scaling the Grammys

94/94

Drupal Integration

Most all configuration in settings.php

Slave database setup

Memcache server setup

Varnish Reverse Proxy setup

NO_CACHE cookie checking

One Custom module to handle all cacheedge-cases and set the NO_CACHE

ki