Post on 25-May-2020
transcript
Presenters
Phillipa AverySenior Software Engineer
Project Technical Lead and EngineerDownloads License Accounting
pavery@Netflix.com / @PhillipaAvery
Robert RetaSenior Software Engineer
Event Sourcing System ArchitectDownloads License Accounting
rreta@netflix.com / @rreta04
Phillipa
Robert
Overview
1. Why do we need a Downloads License Accounting Service?
2. Event Sourcing overview.3. Deep dive into the Event Sourcing Architecture.4. What’s it like working with the License Accounting
Service after release?
LICENSE
CREATE PLAYBACK CONTEXT
SESSION EVENTS(START, PAUSE,
RESUME, KEEPALIVE)
Streaming Playback Lifecycle
RELEASELICENSE
SESSION EVENT(STOP)
STOP
Streaming Playback Lifecycle
LICENSE
CREATE PLAYBACK CONTEXT
DOWNLOADED ENCODED MOVIE
Downloads Playback Lifecycle
~ 1 year
SESSION EVENTS(START, PAUSE, RESUME,
KEEPALIVE, STOP)
RENEWLICENSE?
X
RELEASELICENSE
Downloads Playback Lifecycle
Download Business Requirements
● Devices with downloads
Download Business Requirements
● Devices with downloads● Downloads per studio
Download Business Requirements
● Devices with downloads● Downloads per studio● Movie downloads or playback over year
~ 1 year
License Accounting Service Requirements
❏ Flexible
❏ Debuggable
❏ Reliable
❏ Scalable
Flexibility
Debuggability
Scalability
Reliability
License Accounting Service Requirements...
✓ Flexible: data model can be changed❏ RDBMS❏ Document Model
License Accounting Service Requirements...
✓ Flexible: data model can be changed❏ RDBMS❏ Document Model
✓ Debuggable❏ Event Sourcing
License Accounting Service Requirements...
✓ Flexible: data model can be changed❏ RDBMS❏ Document Model
✓ Debuggable❏ Event Sourcing
✓ Reliable❏ Fallbacks
License Accounting Service Requirements...
✓ Flexible: data model can be changed❏ RDBMS❏ Document Model
✓ Debuggable❏ Event Sourcing
✓ Reliable❏ Fallbacks
✓ Scalable
Event Sourcing
Domain Model
Make Purple
Command
Commands
Commands
Make Purple
Command
Command Handler
Added Red Event
Added Blue Event
Make Purple
Command
Command Handler
Commands
Events
Added Red Event
Added Blue Event
Event Handler
Events
Added Blue Event
Event Handler
Domain Model
Aggregate
Commands
Make Red Command
Command Handler
Command Handler
Commands
Make Red Command
Aggregate ServiceREST
Endpoint
Aggregate Repository
Event StoreQuery Query Query
Event Store
Row ID Events
Event Store
Row ID Events
Row ID Events
e1 e2 e3 e5 e6 e7e4
Event Store
Aggregate id 1 Aggregate id 2
Aggregate Repository
Query Event Store
Aggregate Repository
Event StoreQuery
select * from eventswhere rowId =
Aggregate Repository Event StoreResponse
e1 e2 e3 e5 e6 e7e4
Aggregate Repository
e1 e2 e3 e5 e6 e7e4
Aggregate id 1 Aggregate id 2
e1 e2 e5 e6 e3 e4 e7
Aggregate Repository
Aggregate id 2
Aggregate id 1
e1 e2 e5 e6
e3 e4 e7
Aggregate 1
Aggregate 2
Aggregate Repository
Aggregate 2
Aggregate 1
Aggregate id 2
Aggregate id 1
e1e2 e5 e6
e3 e4 e7
Aggregate Repository
Aggregate 2
Aggregate 1
Aggregate id 2
Aggregate id 1
e2e5 e6
e3 e7
Aggregate Repository
Aggregate 2
Aggregate 1
Aggregate id 2
Aggregate id 1
e5e6
e7
Aggregate Repository
Aggregate 2
Aggregate 1
Aggregate id 2
Aggregate id 1
e6
Aggregate Repository
Aggregate 2
Aggregate 1
Aggregate Repository
Command
Aggregate 1
Command Handler
Aggregate Repository
Command Handler
e8 e9
Aggregate Repository
e8 e9Aggregate 1
Aggregate Repository
Aggregate 1e8 e9
Aggregate Repository
Aggregate 1e9
Aggregate Repository
Aggregate 1
Aggregate Repository Event StoreUpdate
append events
where rowId =
e8 e9
Event Store
Aggregate id 1 Aggregate id 2
Row ID Events
e1 e2 e3 e5 e6 e7e4 e8 e9
Aggregate RepositoryQuery
Aggregate Service
Get all aggregates for a customer
Aggregate Repository
Aggregate Service Response
Aggregate 2Aggregate 1
Aggregate
DownloadedAggregate
License Aggregate
License Service
Acquire License
EndpointQuery
customerId: BayleytitleId: GlowS01E01
License Service
Query
customerId: BayleytitleId: GlowS01E01
Downloaded Service
isAllowed?
Query
customerId: BayleytitleId: GlowS01E01since: 6/27/2016
Downloaded Service
Downloaded Repository
getAggregates
Downloaded Service
Downloaded Repository
Response
DownloadedAggregate
customerId: BayleytitleId: GlowS01E01date: 2/15/2017
DownloadedAggregate
customerId: BayleytitleId: GlowS01E01date: 5/25/2017
License Service
Downloaded Service
Response
isAllowed?
aggregates.size() < yearly limit
License Service
Downloaded Service
Response
isAllowed?
Trueaggregates.size() < yearly limit
(2) < (3)
License Service
Update
License Repository
Create Aggregate
Create License Command
customerId: BayleytitleId: GlowS01E01date: 6/27/2017
License Aggregate
customerId:titleId:expires:released:
License Repository
Command HandlerCreate License
CommandcustomerId: BayleytitleId: GlowS01E01date: 6/27/2017
License Aggregate
customerId:titleId:expires:released:
License Repository
Command HandlerCreate License
CommandcustomerId: BayleytitleId: GlowS01E01date: 6/27/2017
License Aggregate
customerId:titleId:expires:released: License Created
Event
customerId: BayleytitleId: GlowS01E01date: 6/27/2017
License Repository
Event Handler
License Aggregate
customerId:titleId:expires:released:
License Created Event
customerId: BayleytitleId: GlowS01E01date: 6/27/2017
License Aggregate
customerId: BayleytitleId: GlowS01E01expires: 7/27/2017released: false
License Repository Event StoreSave
License Created Event
customerId: BayleytitleId: GlowS01E01date: 6/27/2017
License Service
License Repository
Create Aggregate
CreatedAcquire License
EndpointResponse
License Service License Repository
Response
isAllowed?
aggregates.filter(GlowS01E01) .size()
< yearly limit
Get all licenses
Table
Partition Keys Clustering Columns Columns
Event Table
rowIdtext
Event Timetimestamp
Event Datablob
Aggregate Idtext
Event MapperString
● A fast and efficient object graph serialization framework for Java
○ https://github.com/EsotericSoftware/kryo
● Able to preconfigure the library to your data models for extra compaction
● Custom serializers
AggregateIdtext
Event Datablob
rowIdtext
Event Timetimestamp
House of Cards LicenseCreatedMatt:LicenseAggregate 0
Buddy Thunderstruck LicenseCreated1
The Hardy Boys LicenseCreatedJeff:LicenseAggregate 2
AggregateIdtext
Event Datablob
rowIdtext
Event Timetimestamp
House of Cards LicenseCreatedMatt:LicenseAggregate 0
House of Cards LicenseReleased3
Buddy Thunderstruck LicenseCreated1
The Hardy Boys LicenseCreatedJeff:LicenseAggregate 2
AggregateIdtext
Event Datablob
rowIdtext
Event Timetimestamp
House of Cards LicenseCreatedMatt:LicenseAggregate 0
House of Cards LicenseReleased3
Buddy Thunderstruck LicenseCreated1
The Hardy Boys LicenseCreatedJeff:LicenseAggregate 2
AggregateIdtext
Event Datablob
rowIdtext
Event Timetimestamp
House of Cards LicenseCreatedMatt:LicenseAggregate 0
House of Cards LicenseReleased3
Buddy Thunderstruck LicenseCreated1
The Hardy Boys LicenseCreatedJeff:LicenseAggregate 2
Snapshotting
House of Cards LicenseCreatedMatt:LicenseAggregate 0
House of Cards LicenseReleased3
Buddy Thunderstruck LicenseCreated1
Master of None LicenseCreated0
Master of None LicenseReleased3
Bojack LicenseCreated1
Lucha Underground LicenseCreated0
Lucha Underground LicenseReleased3
Grace & Frankie LicenseCreated1
House of Cards LicenseCreated0
Snapshot Table
rowIdtext
versionint
Snapshot Datablob
Versionint
Event Datablob
rowIdtext
1 binaryDataMatt:LicenseAggregate
AggregateIdtext
Event Datablob
rowIdtext
Event Timetimestamp
House of Cards LicenseCreatedMatt:LicenseAggregate:0 0
House of Cards LicenseReleasedMatt:LicenseAggregate:1 3
Jessica Jones LicenseCreated1
The Hardy Boys LicenseCreatedJeff:LicenseAggregate:0 2
Working With Event Sourcing
Flexibility in Practice
● Changes to the Data Model are trivial!○ Device deactivation
New Device Deactivation Requirement
Deactivate Device
EndpointDevice Service
Device Repository
Device Aggregate
Device Deactivated
EventDeactivate
Device Command
New Device Deactivation Requirement...
Deactivate Device
Endpoint
Device ServiceDeactivate Device
Can Deactivate?
DeviceRepository
Response
DeviceAggregate
CustomerId: BayleyDeviceId: iPhoneDeactivations: 1
New Device Deactivation Requirement...
Deactivate Device
Endpoint
Device ServiceDeactivate Device
Can Deactivate
License ServiceRelease Licenses
Device Repository
Command Handler
Deactivate Device CommandcustomerId: BayleydeviceId: iPhone
Device Deactivated Event
customerId: BayleydeviceId: iPhone
DeviceAggregate
CustomerId: BayleyDeviceId: iPhoneDeactivations: 1
New Device Deactivation Requirement...
Debugging
● Possibly the biggest win
● Current tooling is rudimentary
{ "Event" : "LicenseAcquiredEvent", "Event Time" : "2017-06-25 05:23:00 PM", "aggregateId" : "aggregateId1", "committed" : true, "eventTimeStamp" : 1498411380903, "eventOrderNumber" : 0}
Debugging...
Acquire License Event
customerId: BayleydeviceId: iPhonedate: 01/05/2017
Release License Event
customerId: BayleydeviceId: iPhonedate: 01/05/2017
Renew License Event
customerId: BayleydeviceId: iPhonedate: 01/06/2017
License Aggregate
customerId:Bayleyexpires: 01/07/2017released: True
Reliability: Fallbacks
Edge API Layer
License Accounting Service Client
License Accounting
Service CassandraFallback Response
How Did it Scale?
Predictive Scaling
Scalability with Cassandra SSD (I2) Nodes
Scalability with Cassandra SSD Nodes...
● Could handle much higher loads● Storage use ramping up very quickly
Storage Optimizations
● TTLs● HDD (D2) clusters:
○ More storage○ Higher latency (up to 1 second)
Storage Optimizations...
● Partitioned approach○ Snapshot and subsequent events saved to i2 (SSD) cluster○ Archive events to D2 (HDD) cluster
Snapshot DataVersion 3
Event DataVersion 3
Event DataVersion <= 2
i2
D2
Storage Optimizations...Partitioned approach using CQRS (Command Query Responsibility Segregation)
● Write segregation○ Event handlers that determine partitioned usage
● Query segregation○ Query SSD or HDD (or both)○ Uninterrupted event stream
Event Data
i2
D2
Snapshot + Events
Event Handler
Key Event Sourcing Take-Aways
● Flexible: Adapting to change can be simple
Key Event Sourcing Take-Aways
● Flexible: Adapting to change can be simple
● Debugging: Debugging data / state transitions greatly simplified
Key Event Sourcing Take-Aways
● Flexible: Adapting to change can be simple
● Debugging: Debugging data / state transitions greatly simplified
● Reliable: Fallbacks provide service reliability
Key Event Sourcing Take-Aways
● Flexible: Adapting to change can be simple
● Debugging: Debugging data / state transitions greatly simplified
● Reliable: Fallbacks provide service reliability
● Scalable: Service scaled well, but good architectural solutions for data storage solutions should be considered.
Questions?
Phillipa AverySenior Software Engineer
Project Technical Lead and EngineerDownloads License Accounting
pippa@netflix.com / @PhillipaAvery
Robert RetaSenior Software Engineer
Event Sourcing System ArchitectDownloads License Accounting
rreta@netflix.com / @rreta04