Date post: | 20-Jun-2015 |
Category: |
Technology |
Upload: | continuent |
View: | 528 times |
Download: | 1 times |
©Continuent 2013
Marketing Automation at Scale: How
Marketo Solved Key Data
Management Challenges with
Continuent Tungsten
Nick Bonfiglio – VP Operations, Marketo
Robert Hodges – CEO, Continuent
About Us
• Nick Bonfiglio - Marketo VP of Operations / Responsible for Cloud Services and Security at
Marketo
/ Extensive background in scaling SaaS products and services
• Robert Hodges - Continuent CEO / Tungsten Replicator designer
/ DBMS internals and applications for three decades
/ Author of Scale-Out Blog (http://scale-out-blog.blogspot.com)
Marketo is the global leader in Revenue Performance
Management.
Marketo’s powerful, yet easy-to-use marketing automation
and sales effectiveness solutions, transform how
marketing and sales teams of all sizes work — and work
together — to drive dramatically increased revenue
performance and fuel business growth.
The company’s proven technology, comprehensive
services, and expert guidance are helping corporations
around the world to turn marketing from a cost center to a
business-building revenue driver.
About Marketo
/ The leading provider of data replication and
clustering for open source databases
/ Our Product: Continuent Tungsten
• Clustering – Commercial-grade HA, performance
scaling, and data management for MySQL
• Replication – Flexible, high-performance data
movement
About Continuent
4
Customers from hot startups to Fortune 500 companies
©Continuent 2013
Marketo’s Data
Management Challenge
Nick Bonfiglio
Marketo Architecture
• Marketo uses a mixed stack architecture to support its customers’ subscriptions, leveraging the best technologies for the solution.
• Marketo is a platform consisting of a collection of systems that provide user interface, landing pages/forms, fast secondary search (solr), big data (hadoop) and archiving (cassandra).
• Our systems employ several custom queue-based background processing, which are key to a successful marketing automation platform. And, custom email assembly, scripting and tracking facilities.
Systems Architecture
Marketo SaaS Platform
File Services Database Services
Batch
Processes
User Interface
Munchkin (tracking)
API Landing Pages/Forms
Email Services
Infrastructure Architecture
Marketo DB Design Goals
• Deliver increased availability by providing redundancy and real-time resiliency at all layers.
• Deliver an architecture that allows us to upgrade software and routine database schema changes without incurring downtime to customers.
• Improve overall performance, scalability and management, including backups and disaster recovery of large 7TB+ DB servers.
Marketo DB Architecture
DB on
FlexVol DB on
FlexVol
DB on
FlexVol
Replicator Replicator Replicator
Master Slave Slave
Marketo Web Cluster Marketo Queue Cluster
MySQL Client Library MySQL Client Library
Manager Manager Manager
Pod Data
DBMS Vendor Client Library DBMS Vendor Client Library Tungsten Connector Tungsten Connector
Mo
nito
ring an
d co
ntro
l
Mo
nito
ring an
d co
ntro
l
Marketo DB Hardware Architecture
• Leveraging NetApp FlexVols for snapshot, near-line and DR replication
• Symmetric DB servers so failover yields full performance
• Aggregates are 300GB drives for increased spindles
• Each DB Server ~6-8T
• Leverage flexcache for certain volumes
Performance Metrics
• Over 600 Million MySQL Transactions per day
• Over 600TB of data
• Processing 25 Million Campaigns per day
• Sending over 500M Emails per month
• Serving 15M Landing Page Visits per day
• A few proof points: – Several failovers which would have caused an
outage
– Parallel replication allows is to keep lag to <300sec
– Several roll rotations for maintenance (schema changes, dropping large dbs, db pruning, etc.)
©Continuent 2013
So… How Does Tungsten
Do That?
Robert Hodges
Challenges to Hosting Critical Data on MySQL
/ Get back online quickly after DBMS failures
/ Perform DBMS maintenance and SQL upgrades
without stopping applications
/ Load balance SQL across replicas to use
hardware efficiently
/ Ensure slaves keep up with high master
transaction loads
/ Address all of this without migrating or rewriting
applications
Tungsten Database-as-a-Service for MySQL
Replicator Replicator
• Cluster constructed from
off-the-shelf MySQL
servers
• Automatic failover
• Online maintenance
• Query Load balancing for
speed and h/w utilization
• No migration of data or
application changes
Manager
db2
Manager
Replicator
db1
Manager
db3
Slave Slave Master
Application Stack
Tungsten Connector
Application Stack
Tungsten Connector
Replicator Replicator
Transparent connectivity enables illusion of single
DBMS to applications
• Locate appropriate
server for transactions
• Select most up-to-date
server for reads
• Shift connections from
offline nodes
• Hide offline databases
• Supports SSL
Send writes to master Load balance
reads over slaves
Application Stack
Application Stack Application
Stack
Tungsten Connector
db2 db1 db3
©Continuent 2013
Demo:
Introducing a Tungsten
Data Service
What Causes Slave Lag?
Application Application
Application
Master
InnoDB Buffer Pool
NetApp Storage
Slave
InnoDB Buffer Pool
NetApp Storage
Typical SaaS Application: 1.) Data size >> buffer pool 2.) Persistent stored on disk
SaaS Applications Can Replicate Tenants in
Parallel!
• Sharded replication divides updates into
independent streams (“shards”)
tenant_1
tenant_2
tenant_3
tenant_4
Tenants independent from each other
Tenant database updates applied in parallel
tenant_1
tenant_2
tenant_3
tenant_4
High-Speed Parallel Replication Lets Slaves Keep
Pace with Busy Masters
• Shard transactions by
schema on master
• Transfer log serially
• Apply shard updates
over “channels”
• 5x or more speed boost
in best cases
• Works on all MySQL
versions from 5.0 to 5.6
Replicator
db1
Master
db2
Slave
Replicator
Application Stack
Application Stack Application
Stack
Problem: Uneven SaaS Tenant Size
Problem: “Hot” Tables within Single
Schemas that Block Other Tenants
Tungsten’s Innovative Caching Minimizes
Effects of Lumpy SaaS Workload
Slave
InnoDB Buffer Pool
NetApp Storage
Tungsten Replication Service
NetApp Storage
Fetch from Master
Write to Log
Apply to Slave
Linux Page Cache
Read log from page cache
(No reads from storage)
Write log to page cache
Fsync to storage
Shards apply Independently!
Innovative Features of Tungsten Parallel
Replication Support
• Tungsten leans on OS page cache to keep
long updates from blocking progress
• Use any number of apply channels
• Support for clean failover and backup
• Completely crash-safe for InnoDB
• Configuration checks to prevent accidental
slave corruption
• Sharding algorithm is configurable!
©Continuent 2013
Demo:
Maintenance on MySQL
master under load
What Else Can Tungsten Do?
• Cross-site data services
– Primary/disaster recovery sites
– Multi-master replication between sites
• Fast data warehouse/NoSQL loading from
MySQL/Oracle
• Oracle-to-Oracle replication
What’s Next? Push-Button Cloud Operation
Configure DBMS type and
resource levels for servers
Define backup and
recovery policies
Confirm configuration
and launch
Choose Service Type Manage Deployment
1
2 3 4
Making it Better – Visual Management
Making it Better – Visual Management
Failure detected
and failover
completed within 15
seconds
Making it Better – Visual Management
Failed db instance
is isolated from the
service
Making it Better – Visual Management
©Continuent 2013
Conclusion and Q&A
In Summary…
33
• Marketo performs 600M transactions daily
on MySQL running over NetApp
• Tungsten delivers fast failover, rolling
maintenance, and parallel replication to
make architecture work
• Marketo and Continuent together have
enabled an architecture that supports
economical scaling to 1000s of customers
Contact Us!
Continuent, Inc.
560 S. Winchester Blvd
Suite 500
San Jose, CA 95128
Tel +1 (866) 998-3642
Fax +1 (408) 668-1009
e-mail: [email protected]
www.continuent.com
Marketo, Inc.
901 Mariners Island Blvd
Suite 200
San Mateo, CA 94404
Tel +1 (877) 260-MKTO
(6586)
Fax +1 (650) 376-2331
e-mail: [email protected]
www.marketo.com