Post on 15-Apr-2017
transcript
HazelcastStriimHotCachePoweredByStriimCDC
Your Presenters Today
Steve WilkesCTO / Founder
Striim
Viktor GamovSenior Solutions Architect
Hazelcast
The Cache Inconsistency Issue
App ServerApp ServerApp Server
YouHaveAnApplicationThatUsesaDatabaseToStoreandRetrieveInformation
Application
The Cache Inconsistency Issue
App ServerApp ServerApp Server
BuyStuff CurrentInventory
InThisExampletheAppisanOnlineStoreandPartoftheDataIncludesProductInventory
Application
The Cache Inconsistency Issue
App ServerApp ServerApp Server
BuyStuff CurrentInventory
ToSpeedUpPageResponsesYouUseanIn-MemoryHazelcastCache
Application
HazelcastHazelcast
App
The Cache Inconsistency Issue
App ServerApp ServerApp Server
BuyStuff CurrentInventory
ButOtherApplicationsorBatchUpdatesAlsoModifytheDatabaseOutsidetheCache
Batch
Add/RemoveInventory
Application
HazelcastHazelcast
App
The Cache Inconsistency Issue
App ServerApp ServerApp Server
BuyStuff CurrentInventory
LeadingtotheCacheBeingOut-of-SyncOrInconsistentwiththeDatabase
HazelcastHazelcast Batch
Add/RemoveInventory
Application
App
The Cache Inconsistency Issue
App ServerApp ServerApp Server
BuyStuff CurrentInventory
AndReflectedThroughInaccurateValuesBeingPresentedinTheApplication
HazelcastHazelcast Batch
Add/RemoveInventory
Application
App
The Cache Inconsistency Issue
App ServerApp ServerApp Server
BuyStuff CurrentInventory
TheCacheEvictionorRefreshPolicyDictatestheWindowofInconsistency
HazelcastHazelcast Batch
Add/RemoveInventory
EvictAfter10Minutes
Application
App
The Cache Inconsistency Issue
App ServerApp ServerApp Server
BuyStuff CurrentInventory
WhatYouNeedIsaWaytoUpdatetheCacheWhenevertheDatabaseChanges
HazelcastHazelcast Batch
Add/RemoveInventory
Application
?
That’s Where Striim Comes In
Striim provides
Streaming Integration and Intelligence
enabling companies to Make data useful the instant it’s born.
Striim Handles the Hard Bits
CONTINUOUSCOLLECTION
SUPPORTING MULTIPLE SOURCES
ADDRESSING SCALE,
FAILURES
DISTRIBUTED GRID
/PERSISTENCEMANAGEMENT
SHARDING /SCALING
OVER LARGE STREAMING
DATA VOLUMES
DISTRIBUTED RESULTS CACHE
HIGH READ THROUGHPUT
INTEGRATION INTO MULTIPLE TARGETS AND
DATA FORMATS
REALTIME VISUALIZATION
REALTIME ALERTS
SIMPLE DECLARATIVE
INTERFACE TO DELIVER DATA
DRIVEN APPS✔ ✔ ✔ ✔ ✔
Collect Process Deliver
App Developersfocus on business logic
CONTINUOUS
✔ ✔ ✔ ✔ ✔
Databases &Data Warehouses
Messaging
Big Data &NOSQL
Cloud
FilesDatabases
Log files
Sensors
Messaging
ExternalContext
Filtering Enrichment
Aggregation
Transformation
Windowing
ContinuousQueries
STRE
AMIN
G IN
TEG
RATI
ON
Streaming CDC
Parallel LogCollection
Edge Processing
Continuous Event Collection
Streaming Integration
Databases &Data Warehouses
Messaging
Big Data &NOSQL
Cloud
FilesDatabases
Log files
Sensors
Messaging
ExternalContext
Filtering Enrichment
Aggregation
Transformation
Windowing
ContinuousQueries
STRE
AMIN
G IN
TEG
RATI
ON
Streaming CDC
Parallel LogCollection
Edge Processing
Continuous Event Collection
STRE
AMIN
G
INTE
LLIG
ENCE
Alerts
ResultsReal
-tim
eD
ashb
oard
s
Correlation Detection MatchingTriggers
Streaming Intelligence
Design Flows Analyze Deploy
Visualize Monitor
One Consistent Easy-to-Use Distributed Platform
App
Cache Inconsistency Solved
App ServerApp ServerApp Server
BuyStuff CurrentInventory
TheHazelcastStriimHotCacheSolutionUsesCDCtoSynchronizetheCacheinReal-Time
Batch
Add/RemoveInventory
Application
HotCacheUpdatesInms
HazelcastHazelcast
CDC
DatabasesWriteAllOperationstoaSetofTransactionLogsforRecovery/OtherPurposes
CDC
TransactionLogs
DB
How Change Data Capture (CDC) Works
CDC
TransactionLogs
DB
How Change Data Capture (CDC) Works
InsteadofUsingTimestampsorTriggersCDCReadsTheseLogsandCollectsOperations
Delete Update
CDCDMLInsertUpdate
Delete
TransactionLogs
110100
001101
010011
InsertDB
How Change Data Capture (CDC) Works
WhenDMLOperationsOccurTheyAreWrittentotheLogandCapturedasEventsbyCDC
Delete Update
CDCDMLInsertUpdate
Delete
TransactionLogs
110100
001101
010011
InsertDB
How Change Data Capture (CDC) Works
TheseEventsCanThenBeUsedtoUpdatetheCacheandSynchronizeitinReal-Time
TABLE PRODUCT_INV
SKU INTEGER (PK)
STOCK DECIMAL
NAME VARCHAR(20)
LAST_UPDATE TIMESTAMP
PIC BLOB
public classProductInv {public long sku;public double stock;public Stringname;public DatelastUpdated;public byte[]picture;
}
Example Hot Cache Implementation
DatainthePRODUCT_INVDatabaseTableisRepresentedintheCacheasProductInv Class
HazelcastHazelcast
TABLE PRODUCT_INV
SKU INTEGER (PK)
STOCK DECIMAL
NAME VARCHAR(20)
LAST_UPDATE TIMESTAMP
PIC BLOB
public classProductInv {public long sku;public double stock;public Stringname;public DatelastUpdated;public byte[]picture;
}
Example Hot Cache Implementation
TheMappingBetweentheTwoisRepresentedThroughanORMXMLFile
HazelcastHazelcast
<entity name="prodInv" class="com.customer.vo.ProductInv"><table name="test.PRODUCT_INV"/><attributes>
<id name ="sku" attribute-type="long"><column nullable="false" name="SKU"/>
</id><basic name="stock" attribute-type="double">
<column nullable="false" name="STOCK"/></basic><basic name="name" attribute-type="String">
<column name="NAME"/></basic><basic name="lastUpdated" attribute-type="java.util.Date">
<column name="LAST_UPDATED"/></basic><basic name="picture" attribute-type="binary">
<column name="PIC"/></basic>
</attributes></entity>
Example Hot Cache Implementation
UsingStriimYouCreateaNewApplicationUsingaCDCtoHazelcastTemplate
Example Hot Cache Implementation
StriimStepsYouThroughSettingUpYourDBConnectionandChecksCDCisWorkingProperly
Example Hot Cache Implementation
YouThenSelecttheTablesYouWantResultinginaChangeDataStream
Example Hot Cache Implementation
NextYouConfiguretheHazelcastConnectionandORMFile– WhichWeValidateForYou
Example Hot Cache Implementation
ThisisUsedtoGenerateaHazelcastTargetwithintheStriimApplication
Example Hot Cache Implementation
TheResultingStriimDataFlowHasEverythingNeededtoKeepHazelcastSynchronized
Custom Monitoring of Hot Cache Solution
AdditionalAnalyticsFlowsinStriimCanProvideReal-TimeMonitoringDashboardsandMore
Live Demo of Hazelcast Striim Hot Cache
Questions?
Want To Know More?