PerformanceTuningandOptimizationforhightrafficDrupal

sitesKhalidBaheyeldin

DrupalCamp,TorontoMay1112,2007

Agenda Introduction TheLAMPStack

Linux,Apache,MySQL,PHP Drupal

Databasequeries Modules Caching

Measurementandmonitoringtools Whatcangowrong? Questions,discussion

About2bits

BasedinWaterloo,Ontario ActivememberoftheDrupalcommunitysince2003 Memberofsecurityandinfrastructureteams 24+modulesondrupal.org ListedonDrupal.org'sserviceproviderssection Maintainmodulesthatrunondrupal.org(donations,feature,

lists) GoogleSummerofCodementoring(2005,2006,2007)

2bitsServices ClientsmainlyinUSAandCanada Subcontractingdevelopmentprojects Customizationofexistingmodules Developmentofnewmodules Installation,upgrades Automatedbackups Performancetuningandoptimization

AboutKhalid Developingforcomputersforwaytoo

long(22years),Drupalsince2003

Corecontributions

Sitemaintenancefeature LoggingandalertsinDrupal6 Severalpatches

Memberof

Drupalsecurityteam webmastersteam infrastructureteam

Cofounderof2bits Blogathttp://baheyeldin.com

Contributedmodules Adsense

Userpoints

Nodevote

Jobsearch

Favoritenodes

Flagcontent

StockAPIandmodule

CustomError

Currency

Imagewatermark

Sitemenu

Emailloggingandalerts

SecondLife

Technorati

Clickthru

Referral

TheIron Physicalservermatters

Dedicated VPS

Notapplicabletosharedhosting DualOpteronskickass LotsofRAM(cachingthefilesystemandthedatabase,asmuchas

possible) Multipledisksifyoucan Alwaysmirrored!

MultipleServers Onedatabaseserver+multiplewebservers CanuseDNSroundrobin Orproperloadbalancers(commercial,free) Evenareverseproxy(squid) Doitonlyifyouhavethebudget

Complexityisarunningcost Tuningasystemcanavoid(ordelay)thesplit

TheLAMPstack MostcommonlyusedstackforhostingDrupalandsimilarapplications Linux Apache MySQL PHP

Mostofthispresentationappliesto*BSDaswell.PartsapplytoWindows.

Linux

Useaprovenstabledistro(Debian,Ubuntu) Userecentversions(noFedoraCore4please) Beaminimalist Installonlywhatyouneed

(e.g.NoX11,nodesktop,NoPostgreSQLifyouareonlyusingMySQL,...etc.)

Balancecompileyourownvs.upgrades

Apache Mostpopular,supportedandfeaturerich Otherwebservers

lighttpd(lighty) PopularwithRuby 1MBperprocess Recentmemoryleaks

nginx Morestablethanlighty(noleaks)

Apache Cutthefat

Enableonlymod_phpandmod_rewrite Disableeverythingelse(java,python) MayneedextendedstatusforMunin

TuneMaxClients Toolow:youcan'tserveatrafficspike(Digg,Slashdot)

Toohigh:yourmemorycannotkeepupwiththeload,andyoustartswapping(serverdies!)

Apache(cont'd) KeepAlive

5to10secondsOK Morethanthat,ittiesupprocesess

Allowoverrides SettoNone MoveDrupal's.htaccesscontentstovhosts

mod_gzip/mod_deflate CompromiseofCPUusagevs.Bandwidthusage

MySQL

MostpopulardatabaseforDrupal Notthebestdatabasefromthetechnologypointofview(ACID,transactions,concurrency),butstilladequateforthejob

Variouspluggableengines

MySQLEngines MyISAM

Fasterforreads Lessoverhead Poorconcurrency(tablelocking)

InnoDB Transactional Slowerinsomecases Betterconcurrency Oracleownstheenginenow...

MySQLEngines

Twonewengines,ownedbyMySQLAB Falcon.NotmatureenoughtomatchInnoDB,benchmarksshowitisstillslow

SolidDB. PBXT

PrimeBaseXT

MySQLtuning

Querycache Probablythemostimportantthingtotune

Tablecache Alsoimportant

Keybuffer

PHP

Usearecentversion InstallanOpcodecache/Accelerator

eAccelerator APC Xcache Zend(commerical)

APCvs.eAcceleratorbenchmarkon2bits.

Opcodecaches Benefits

Dramaticspeedupofapplications,speciallycomplexoneslikeDrupal

SignificantdecreaseinCPUutilization Considerabledecreaseinmemoryutilization Thebiggestimpactonabusysite

Drawbacks Maycrashoften UselogwatchertoautorestartApache

mod_php

Normally,Apachemod_phpisthemostcommonlyusedconfiguration

Sharednothing Nostateretainedbetweenrequests Lessissues

Staywithmod_phpifyoucan. Canbeaslowas1012MBperprocess Sawitashighasmid20s+

PHPasCGI

CGIistheoldestmethodfromtheearly90s. Forksaprocessforeachrequest,andhenceveryinefficient.

Somehostsofferitbydefault(security)orasanoption(e.g.runningaspecificPHPversion).

Don'tuseit!

FastCGI FCGIisfasterthanCGI(usesasockettothePHPprocess,notforking)

MostlywithLighttpdandnginx,sinceitistheonlywaytorunPHPforthoseservers,butalsowithApache

Therearesomecases(e.g.drupal.orgitself) Betterseparationofpermissions(e.g.Sharedhosting) IfyouhaveoneserverandoneLinuxuser,permissionsmaynotbeanissue.

Drupal

Mainlydatabasebottlenecks Bottlenecksareworkedonastheyarefoundbythecommunity

Somemodulesknowntobeslow Notallsitesaffectedbyallbottlenecks

Watchdog

Avoiderrors(404songraphics,favicon)TIME STATE INFO

24 updating DELETE FROM watchdog WHERE timestamp < 1176392718

24 Locked INSERT INTO watchdog (uid, type, message, severit

19 Locked INSERT INTO watchdog (uid, type, message, severit

14 Locked INSERT INTO watchdog (uid, type, message, severit

11 Locked INSERT INTO watchdog (uid, type, message, severit

6 Locked INSERT INTO watchdog (uid, type, message, severit

OptionalinDrupal6(syslogasanoption)

Sessions Heavilyusedinhightrafficsites

TIME STATE INFO

28 Locked UPDATE sessions SET uid = 0, hostname = '212.154.

28 Copying to t SELECT ... FROM sessions WHERE timestamp >= 11776

28 Locked SELECT ... FROM users u INNER JOIN sessions s ON

27 Locked UPDATE sessions SET uid = 0, hostname = '222.124.

27 Locked UPDATE sessions SET uid = 0, hostname = '201.230.

27 Locked SELECT ... FROM users u INNER JOIN sessions s ON

27 Locked SELECT ... FROM users u INNER JOIN sessions s ON

27 Locked SELECT ... FROM users u INNER JOIN sessions s ON

27 Locked SELECT ... FROM users u INNER JOIN sessions s ON

27 Locked SELECT ... FROM users u INNER JOIN sessions s ON

Drupal(cont'd) Disablemodulesthatyoudonotneed. Enablepagecaching

Mayexpiretoooftenonabusysite,causingslowdowns! Considercachingmodules

FSFastpath boost

Makesurecronrunsregulary Enablethrottle

Bewaryaboutthrottleandcache

Puggablecaching

Using$confvariableinsettings.php 'cache_include'=>'./includes/yourcache.inc'

Allowsyoutohaveacustomcachingmodule Cachingusingmemcachedisbeingworkedon Tip:canbeusedtodisablecachefordevelopment(stubfunctions)

Slowmodules Statisticsmodule

Addsextraqueries EvensloweronInnoDB(COUNT(*)slow) DisablePopularContentblock

gsitemap(XMLsitemap)

Hadanextrajoin,patchaccepted Aggregator2

Abandoned! Manymore...

MeasureandMonitor

Howdoyouknowyouhaveaproblem? Userscomplain(siteissluggish,timeouts)? Losingyouraudience?Lossofinterestfromvisitors?

Toolsforvarioustasks

Top ClassicUNIX/Linuxprogram Realtimemonitoring(i.e.WhatthesystemisdoingNOW)

Loadaverage CPUutilization(user,system,nice,idle,waitI/O) Memoryutilization Listofprocesses,sorted,withCPUandmemory Canchangeorderofsorting,aswellastimeinterval,andmanyotherthings

vmstat

FromBSD/Linux Showsaggregateforthesystem(noindividualprocesses)

Showssnapshotorincremental Processesintherunqueueandblocked Swapping CPUuser,system,idleandiowait

netstat

Showsactivenetworkconnections(allandESTABLISHED)

netstatanp netstatanp|grepEST

mtop,mytop

mtop Liketop,butforMySQL Realtimemonitoring(nohistory) Showsslowqueries

mytop Similartomtop

SHOWFULLPROCESSLIST

mysqlreport/dbtuning Mysqlreport

Perlshellscript Displaysstatistics Norecommendations

Dbtuning AshellscriptthatreadsvariablesfromMySQL Annoyinguseofcolors Usefulrecommendations

GraphmonitoringMunin

Niceeasytounderstandgraphs. Historyoveraday,week,monthandyear CPU,memory,network,Apache,MySQL,andmuchmore

CanaddyourownmonitoringscriptsCacti

Similarfeatures

Drupaltools Develmodule

Totalpageexecution Queryexecutiontime Querylog Memoryutilization

Tracemodule Morefordebugging,butalsousefulinknowingwhatgoesonunderthehood

Whatcangowrong?

CPUusageistoohighMemoryoverutilizationToomuchdiskI/OToomuchnetworktraffic

CPU

FindoutwhoisusingtheCPU?Findoutwhichtype(user,system,waitI/O)

CPU IfitisanApacheprocess,theopcodecachewillhelp,unless

youhaveabug. IfitisMySQL,thensomeofthatisnormal(intensivequeries),

otherwise tunetheindexes splittheservertotwoboxes. Tunethequerycache

Ifitissomethingelse,andconsistent,thenconsiderremovingit.

CPU100%

OutputfromToptop - 10:16:58 up 75 days, 59 min, 3 users, load average: 152.70, 87.20, 46.98

Tasks: 239 total, 157 running, 81 sleeping, 0 stopped, 1 zombie

Cpu(s):100.0%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st

Mem: 2075932k total, 1558016k used, 517916k free, 13212k buffers

Swap: 1574360k total, 49672k used, 1524688k free, 442868k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

659 www-data 21 0 61948 14m 4060 R 3 0.7 0:14.35 apache2

960 www-data 20 0 62084 14m 4076 R 3 0.7 0:10.51 apache2

989 www-data 20 0 62036 14m 4052 R 3 0.7 0:09.95 apache2

.... hundreds of them

CPU100%

Vmstatoutput# vmstat 15

procs -----------memory---------- ----cpu----

r b swpd free buff cache us sy id wa

152 0 40868 1190640 13740 465004 22 6 71 2

153 0 40868 1190268 13748 464996 100 0 0 0 155 0 40868 1189740 13756 464988 100 0 0 0 154 0 40868 1189540 13768 465044 100 0 0 0

CPU100% Whatwasit? eAccelerator(svn303+PHP

5) AttempttogetoverPHP

crashes NoteCPUutilization(100%,

thenhigh,thendroppedlowwhengoodversionused)

Memory

Swappingmeansyoudon'thaveenoughRAM Excessiveswapping(thrashing)isserverhell! ReducethesizeofApacheprocesses ReducethenumberofApacheprocesses(MaxClients) Turnoffprocessesthatarenotused(e.g.Java,extracopiesofemailservers,otherdatabases)

Buymorememory!Costeffectiveandworthit.

Memory

Impactonmemoryusagewhenthereisnoopcodecachevs.withanopcodecache

DiskI/O

Firsteliminateswappingifgethitbyit. Getthefastestdisksyoucan.7200RPMataminimum. TurnoffPHPerrorloggingto/var/log/*/error.log Considerdisablingwatchdogmoduleinfavorofsyslog(Drupal6willhavethatoption),orhackthecode

OptimizeMySQLonceaweek,oronceaday

Network

Normallynotanissue Occasionallyyouwillhaveastubborncrawlerthough

OrevenaDdoS Orworse,extortion Caneatupresources,includingnetwork

Diggfrontpage?

OnGoodFriday,adsoftheworld.comwasonDigg'sfrontpage. Thefounderwroteaboutit

http://creativebits.org/webdev/surviving_the_digg_effect Survivedthediggwell. Anotherserver(untuned)gotdiggedtwiceanddied

ResourcesandLinks

General http://2bits.com/articles/drupalperformancetuningandoptimizationforlargewebsites.html http://www.lullabot.com/articles/performance_and_scalability_seminar_slides

Apache http://httpd.apache.org/docs/2.0/misc/perftuning.html

MySQL http://www.mysqlperformanceblog.com/

http://dev.civicactions.net/moin/CodeSprint/SanFransiscoMarch2007/PerformanceAndScalabilitySeminar

Conclusion

Questions? Comments? Discussions?