BridgingtheDeveloperandtheDatacenter

June2009

BriefHistory•  8yearsinmanagedservicesindustry

•  Foundedfirstcompanyin2001,soldin2004•  FoundedReduxin2004,soleowner•  Managing200servers,1200VMsacrosstwofaciliCesin

Portland

•  StrongOSSAdvocates–Adium,ImageMagick,Pidgin,SimpleMachines,CaminoBrowser

PropertyofNetworkRedux,LLC

Whyshouldwebuildabridge?•  HostedapplicaConsrequirecommunicaConbetween

datacentersanddevelopers

•  Providedeveloperswiththetoolstopowerdecisionmaking

•  BecomelessreliantonhosCngprovidersfortroubleshooCng

•  UnderstandthehosCngservicestack

PropertyofNetworkRedux,LLC

TheNetwork

•  Ethernetisthestandard•  10Mb,100Mb,1Gb,10Gbandnow

100Gb

•  1Mbpssustainedequals~320GB

•  BursCngisimportant,knowyourlimits•  >70%sustainedusageisdangerous

PropertyofNetworkRedux,LLC

Data Center / Bandwidth

BursCngExample:Twi]erVoteReport

PropertyofNetworkRedux,LLC

DataTransfer

•  Core,DistribuCon,Access•  BorderGatewayProtocol(BGP)•  VirtualRouterRedundancy

Protocol(VRRP)•  Bandwidthisacommodity•  Redundancyhastohappenat

alllayers•  QualityoverquanCty•  Carrierneutraliskey•  ethtool,traceroute,tcpdump

PropertyofNetworkRedux,LLC

DataBackups

•  Donotrelyonthehostfordisasterrecovery•  SLA’sarecarefultoindemnifyyourprovider•  Ifyouarestoringclientdata,youneedtobackitup

andindemnifyyourself

•  RAIDisNOTabackupstrategy•  Understandtheretrievalprocess•  Verifytheintegrityofyourbackuparchives

PropertyofNetworkRedux,LLC

TwoCommonApproaches

•  Agentresidesonserver•  Masterpullservers•  Expensive•  Proprietary•  VeryReliable•  Manycommercial

vendors

PropertyofNetworkRedux,LLC

Pull – Agent / Host Push – server to server

• Serverpushesdata• Rsync/rsnapshot• Inexpensive• Easytodeploy• Asreliableasyoudesignit• MorehomegrownsoluCons

TheAgent/HostModel

•  Commonlydeployedinhostedenvironments

•  Worksatthechassislevel•  Moreodenthannotakernelmodule

•  CanprovideCDP(ConCnuousDataProtecCon)•  EncrypCon,DiskVerificaCon•  CDPfromR1Sodishiengthesweetspot

PropertyofNetworkRedux,LLC

CommonIssues

•  Don’tassumeyourproviderisbackingup

•  Don’tassumethebackupdatahasbeenverified•  DataisulCmatelythedevelopersresponsibility

•  Becarefulwithprovidersnotwillingtobackupyourdata•  Haveyourowndisasterrecoveryplan

PropertyofNetworkRedux,LLC

Power

•  UnbelievablycriCcal,mostcommonpointoffailureinafacility

•  Powercreatesheat,andheatcauseshardwaretofail

•  SystemAdministratorsarenotElectricians

PropertyofNetworkRedux,LLC

KeyPointsonPower

•  Redundantpowersuppliesneedtoroutethroughdiversepowerpaths

•  AutomatedTransferSwitches(ATS)canbeasinglepointoffailure

PropertyofNetworkRedux,LLC

Servers–ResourcesandMetrics•  CPU,MemoryandDiskIO•  DiskIOisthemostcommonly

overlookedandleastunderstoodbo]leneck

•  WerarelyseeCPU’sasthebo]lenecksforapplicaConhosCng

•  Memoryisaneasytounderstand,easytoinflate/deflatecommodity

•  ThingsodenfindtheirwaybacktoslowdiskaccessCmes

PropertyofNetworkRedux,LLC

NarrowinginonDiskI/O•  Disksaretheslowestcomponent

•  SATAandSASarewidelyinuse•  RAID–Balancingperformanceandcost.

•  Don’tconfusememoryshoriallswithdiskaccessproblems•  Disksarethemostcommoncomponenttofail

•  sarandiostatarevaluabletools•  Profileyourenvironmenttobestgaugeyourrequirements

PropertyofNetworkRedux,LLC

Profilingyourrequirements•  SystemmetricsarecriCcal

•  Eachservicelayerisitsowndiscussion

•  Cachingwillsavetheday

•  Logfilesareunderrated

•  sarandvmstatarefantasCc

•  ProfilingyourapplicaConrequirementswillensureefficientspendingforservices

PropertyofNetworkRedux,LLC

Sometimes performance tuning must be “out of the box.” Analysis on the microscopic level should regularly be retired to more macroscopic views. This multiresolutioned problem analysis can turn a question such as “How can I merge all these log files faster?” into “Why do I have all these log files to merge and is there a better way?” Or a question such as “How can I make this set of problematic database queries faster?” becomes “Why am I putting this information in a database?”

Theo Schlossnagle Scalable Internet Architectures

networkredux.com