Date post: | 06-Jul-2015 |
Category: |
Technology |
Upload: | spil-engineering |
View: | 859 times |
Download: | 6 times |
Total Cost of Ownership Zsolt Fabian Database Engineering
2
1. Who is Spil Games? 2. What is TCO? 3. What drives the costs? 4. Possible improvements?
Outline of the talk
Who are we? Who is Spil Games?
4
• Company founded in 2001 • 350+ employees world wide • 200M+ unique visitors per month • 45 portals in 19 languages
• Casual games • Social games • Real Kme mulKplayer games • Mobile games
• 35+ MySQL clusters
Facts
TCO Total Cost of Ownership
6
• 1980 – “Life Cycle CosKng” • 1987 – Gartner Group • Total Cost of Ownership • Analysis of enterprise spending, vendor evaluaKon • Many inter-‐related direct and indirect costs • Problems with TCO
What is TCO and why it is important?
CAPEX – Capital Expenses OPEX – OperaKng Expenses
Relevant Costs
7
Infrastructure life-cycle
8
• CollecKng data • Different base systems • 4,3 TB of RAM • 402 drives wriKng 3 ExaBytes • 3 DBA • 12,3 kW power consumpKon • Asset value exceeds € 500,000 • Cables are available in all colors of the spectrum J
A quick inventory
9
Aging servers
• Component failures • Power consumpKon • Volume of deployment
10
• The most obvious • Base server + CPU + Mem + Disk + Controller • Replacement parts and Warranty • NPV – Net Present Value • Mining invoices
CAPEX: Purchase costs
AFR – Annual Failure Rate ALE – Annual Loss Expectancy
11
• Business need • Cost of downKme • Levels of HA and Fault Tolerance
• Server downKme • CPU failure • MySQLd crash • Solar acKvity
• Combined availability formula:
CAPEX: Purchase costs: HA
A = 1 – (1 – Ax) ^ n
12
• Giving back J • MySQL Standard EdiKon: $ 2,000
• Enterprise: $ 5,000 • Cluster Carrier Grade: $ 10,000
• Microsoj SQL Server: $ 6k+ / core • Oracle 10/11g: $ 40k+ • AddiKonal: Technology extensions, Dashboards
CAPEX: License costs
13
• Work hours spent… • SLA, Responsibility, Stress, Insider Threat • MySQL annual subscripKon: $ 2,000 • Specialist support from $ 1,500 / server / year
• Unlimited servers for $ 30,000 / year • Depends on complexity
• € 2,700 / server / year
OPEX: MySQL Professional support
It's cheaper to add more servers than hire more engineers.
14
• Cage – Rack – Server
• Networking • Deployment (+ fuel + Kme) • Power density constraints
• Let’s say € 60 / server / month
OPEX: Hosting: Colocation
15
• Wao = Ampere * Volt • € 0.2 / kWh
• 12,31 kWh (~ 16% of total) • € 21,300 annually
• Measurements • IPMI • Power meter J
• Contributors • HDD (25%) • Roles (5-‐10W)
OPEX: Hosting: Power
16
OPEX: Hosting: Power
Putting it together
Pu@ng it together The model
Cost Item CAPEX OPEX TCO 5y
Base System ?
Extra memory ?
Extra disks ?
Licence fees ?
Replacement parts / Warranty
? ?
Co-‐locaKon ?
Power ?
Professional support ?
TOTAL ? ? ?
Pu@ng it together Cluster of 6 nodes
Cost Item (6x) CAPEX OPEX TCO 5y
Purchase: HP DL380 G7 € 25,000
Purchase: Extra memory € 6,000
Professional support € 16,200
ColocaKon cost € 4,320
Power costs @200W € 2,100
Replacement parts
€ 1,500
TOTAL € 32,500 € 22,620 € 145,600
Pu@ng it together Cluster of 6 nodes
Purchase costs 18%
Professional support 58%
ColocaKon 15%
Power 8%
Replacement parts 1%
TCO 5 years
Improvements “Insanity: doing the same thing over and over again and expecSng different results” -‐ Albert Einstein
22
Purchase prioritization
Extend warranty
MTBF
Obsolete, Not Urgent
Limited reparability
Repairable
New
23
• SSD reduces the cost of IOPS • SSD is fast for reads, but not for writes • Has limited lifespan • NAND Flash and RAM based SSDs • MLC (MulK Level Cell) vs. SLC (Single Level Cell) • R/W in pages, but always erased in blocks • The main advantage that it doesn’t have to spin
HDD vs. SSD: Inner workings
24
• Dropping over Kme • Facts
• Tsunami • Rare-‐earth minerals
• Cost per GB is cheaper at HDD • Some predicts that in 5 to 10 years, SSD will replace HDD completely
HDD vs. SSD: Pricing
25
HDD vs. SSD: Power consumption Type Model Idle Power AcSve Power Annual cost
contribuSon
SSD Intel 520 600 mW 850 mW € 1,49
SSD Intel X25-‐M 60 mW 150 mW € 0,26
SSD Intel X25-‐E 60 mW 4,2 W € 7,35
HDD ST9146852S 5 W 7 W € 12,2
26
• MTBF: ~ 200 years • HDD
• AFR: 0,55% (-‐ 1,5% -‐ 2% -‐ …) • ALE!
• SSD have LIMITED write cycles • SLC: 100,000 writes / block • MLC: 10,000 writes / block • Typical MLC: 5,000 writes / block • Future developments
HDD vs. SSD: Endurance
27
• Intel 520 SSD: 5 years (with normal desktop usage) • 35 TB write endurance (TBW) • Official endurance is overprovisioned (~ 50 %) • SaturaKon
• Filesystem overhead (0.25 btrfs-‐> 0.015 (ext4)) • Readout
• S.M.A.R.T.: Percentage of rated lifeKme used • UKliKes • RAID controllers
HDD vs. SSD: Endurance
28
• Determined during the design • The equaKon is about: Buffers x ConnecKons • DIMM populaKon guide • DRAM market: OVERSUPPLY
Memory configuration
29
• Extending Linux block-‐cache with SSD • Block-‐cache works by caching accessed blocks • Perfect sizing -‐> performance similar to SSD only • Failures
• Writethrough cache || storage • Writearound –(w)–> storage –(r)–> cache –> • Writeback –(w)–> cache –(w)–> storage –>
Flashcache
30
• 6 x 300 GB RAID10 -‐> 2 x 100 GB SSD + 2 x 1 TB RAID1 • ConsideraKons
• I/O rates • Write savings
• Overprovisioning • ParKKoning
• How long to cache • What to cache
• Less HDD purchases, More SSD purchases (High ALE)
Flashcache
31
• 4 nodes of 6 x 146 GB HDD • CAPEX: € 4,700 • OPEX: € 293 • TCO 5y: € 6,165
• 4 nodes of 2 x 600 HDD + 2 x Intel 520 SSD, 30% opt. writes • Replacing them every year (TBWy: 100 TB) • CAPEX: € 11,680 • OPEX: € 110 • TCO: € 12,230
Flashcache: Case study
32
• TCO is a very handy (someKmes misleading) tool • RCO -‐ Relevant Cost of Ownership • CAPEX:OPEX raKo • Giving back is OPEX • Keep eye on your power consumpKon • Buy DRAM • Consider ALE when you switch to SSD
Summary
Questions?
34
• PresentaKon can be found at: hop://spil.com/tcomeetup2013
• You can contact me at: [email protected]
Thank you!