Post on 12-Apr-2017
transcript
1
Vertica The Convertro way
Zvika GutkinBig data dbOpszvika.gutkin@convertro.com
Agenda
WhyVertica
How weLoad
How wequery
Not allgood
Vertica The Convertro wayConvertro
WhoWe are
Tips
0%
0%
0% 0%
0%
100%
Last click model
Convertro Vertica The Convertro way
100%
15%
10%
5% 10%
20%
40%
Multi attribution
Convertro Vertica The Convertro way
Convertro
Reports
AnalyticsDashboard
Batch Dashboard
Aggregation
Regression R
Vertica The Convertro way
Convertro
Batch Dashboard
Active active
Copy aggregated data
Predictable customer experience
Vertica The Convertro way
Client facing
WhyVertica
How weLoad
How wequery
Not allgood
Convertro
WhoWe are
Tips
Vertica The Convertro way
Pro Tip
HDFS Vs. Hadoop Vs.Application
Loader
Big Bulks
Bring your files to your
cluster
Load from Several Nodes
Convertro
Pro Tip
Pro Tip
Vertica The Convertro way
Unified TempTable
Target Table/Partition
Stream COPY
Number of parallel loads1Number of parallel nodes2
Chunk size per loads3
TEMP
TEMP
TEMP
MOVE PARTITIONS
MOVE PARTITIONS
Vertica The Convertro wayConvertro
WhyVertica
How weLoad
How wequery
Not allgood
WhoWe are
Tips
Vertica The Convertro wayConvertro
Denormalize
Use DBD for encoding
Events DBD
Check query_events system table
Vertica The Convertro wayConvertro
MMM HydroImprovements
Out of the box improvements
Conversion Table ~2B/Month
Oracle (raw store) 418GB / 147GB
Vertica (column store) 21 GB
Vertica The Convertro wayConvertro
70% Reduction
MMM HydroImprovements
Measure, Measure, Measure!
QueriesLocksResources……
Inpu
t
SizeResourcesErrorsQuery…
Tren
dsA
nom
alie
s
Measure, Measure, Measure!
Select M1,.. From… Where filter1 = ‘a’ and filter2 = ‘b’
Select M1,.. From… Where filter1 = ‘c’ andfilter 2 = ‘b’
Select M1,.. From… Where filter1 = “” and filter2 = “”
MMM HydroImprovements
MMM HydroImprovements
Real Time ETR
select A from B where C=‘D’
Business Logic Topology
Sampling
Lookup
Aggregate
Hyd
ro
Web Service
WhyVertica
How weLoad
How wequery
Not allgood
WhoWe are
Tips
Vertica The Convertro wayConvertro
High concurrency
Deletes Updates
Short queries with
high concurrency.
(Routable queries)
Vertica The Convertro wayConvertro
WhyVertica
How weLoad
How wequery
WhoWe are
Vertica The Convertro wayConvertro
Tips
Not allgood
ROS CONTAINERSROS Pushback
Vertica The Convertro wayConvertro
ROS CONTAINERS
Vertica The Convertro wayConvertro
ROS Pushback
ROS CONTAINERS
Vertica The Convertro wayConvertro
ROS Pushback
ROS CONTAINERS
Convertro Vertica The Convertro way
ROS Pushback
ROS CONTAINERS
Convertro Vertica The Convertro way
ROS Pushback
ROS CONTAINERS
Convertro Vertica The Convertro way
ROS Pushback
ROS CONTAINERS
New Table
Convertro Vertica The Convertro way
ROS CONTAINERS
New Table
Convertro Vertica The Convertro way
ROS CONTAINERS
Convertro Vertica The Convertro way
WhyVertica
How weLoad
How wequery
WhoWe are
Vertica The Convertro wayConvertro
Tips
Not allgood
Convertro
NodeCrash
Slow RecoverProcess
Checking RecoveryStatus
Incremental recoveryreplay-delete
Vertica The Convertro way
Many Deletes / Updates
Convertro
NodeCrash
Slow RecoverProcess
Checking RecoveryStatus
Incremental recoveryreplay-delete
Solution 1 Wait
Vertica The Convertro way
Many Deletes / Updates
Convertro
NodeCrash
Slow RecoverProcess
Checking RecoveryStatus
Incremental recoveryreplay-delete
Solution 2Set make_ahm_now
Incremental By Containers
Vertica The Convertro way
Many Deletes / Updates
Convertro
Many Deletes / Updates
NodeCrash
Slow RecoverProcess
Checking RecoveryStatus
Incremental recoveryreplay-delete
Solution 3Delete only
one file
Incremental By Containers
Vertica The Convertro way
Great Database Even Steph Curry can’t Do it allKeep It Simple
Convertro
Great Database Even Lebron can’t Do it allKeep It Simple
Convertro
Convertro
Yap, Can’t do it all
Great Database Even Lebron can’t Do it allKeep It Simple
Thank You
zvika.gutkin@convertro.comhttps://github.com/Convertro/Hydrohttp://www.meetup.com/Tech-Talk-Teach/