CQRS & EVS with
• My name is Lluis Fernández.
• I’m involved in a tech group of freelancers named B-Team.
• We offer consultancy on distributed system architectures.
• Contact: [email protected]
• What’s this about.
• The patterns: CQRS, EVS and ESB.
• All together: An example of distributed enterprise application.
CQRS & EVS with
CQRS & EVS with
• NoSQL databases, like MongoDB, have contributed to the adoption and implementation of archetypes and patterns to develop distributed enterprise applications.
• High read/write rates, huge data storesmanagement, highly scalable read-only replication and schema-less document orientation, are well suited for those kind of patterns and techniques.
CQRS & EVS with
• This presentation is about how some of these patterns and archetypes had been applied during the implementation of real-word use case: a Shipment Management application for one of the most important Messaging and Packages Delivery companies in Spain.
CQRS & EVS with
• CQRS stands for Command-Query Responsibility Segregation.
• Commands are expected to do writes of smallportions of data.
• Queries are expected to do reads of big portions of data.
• CQRS encourages the separation of these different workloads.
CQRS & EVS with
CQRS facade
Commands
Queries
Shipment management core services
View data synchronization
Regular operations on Master data
Master DB
(R/W)
Views DB
(Readonly)
Replication
Corporateshipment
management web app.
Packages
Add package
Add destination
Add origin
Add delivery
Assign courier
Pending deriveries
Lost packages
Packages by shipment
Packages by shipment
CQRS & EVS with
• Reduces contention and collision between queries and commands persistence-level operations.
• Allows the scaling-out of queries, usually the most common and expensive operations.
• Allows the use of heterogeneous view data stores.
CQRS & EVS with
Shipment management core services
Simple automatic synchronization(replication)
Queries
Commands
Datasets optimized for expected queries (grouping, filtering, joins, etc.)
Infrastructure-provided synchronization mechanism.No optimization:Master data = View data
Regular datasets(packages, deliveries, etc.)
These synchronization processes introduce staleness of data provided to the queries.
Views DB (Readonly) Master DB (R/W)
View data synchronization
Regular operations on Master data
Packages
DeliveriesCustomers
Packages by shipment
Package tracking
CQRS & EVS with
Queries
Commands Shipment management core services
Queries
Views DB (Readonly)
View data synchronization
View data synchronization
Regular operations on Master data
Master DB (R/W)
More copies of view data or more different views.Also, copies of view data can be heterogeneous.
Views DB (Readonly)
Packages by shipment
Package tracking
Packages by shipment
Package tracking
Packages
Deliveries
Customers
CQRS & EVS with
• MongoDB replica sets and ReadPreference can be used to segregate master and view data:Commands write on the PRIMARY.
View data is also written on the PRIMARY.
Queries read from SECONDARIES.
• Separate master and view data collections in different databases (MongoDb write locking).
• Schema optimization and indexing in collections storing view data also improve performance.
CQRS & EVS with Shipment management core services
Queries are launched against SECONDARIES (SECONDARY or SECONDARY_PREFERRED read preference)
MongoDB replication
Commands
Queries
View data synchronization
Packages per shipment view synchronizer
Data optimized for views: Schema design. Indexing.
Multiple SECONDARIES allows queries scaling.
Regular operations on Master data
Commands are launched against PRIMARIES (PRIMARY or PRIMARY_PREFERRED read preference)
View data ALSO goes to PRIMARIES
Master data (affected by commands) goes toPRIMARIES
PRIMARY(Master and Views data)
SECONDARIES(View data)
Packages by shipment
Package tracking
Packages
DeliveriesCustomers
Packages by shipment
Package tracking
Packages
DeliveriesCustomers
CQRS & EVS with
PRIMARY(Master and View data)
SECONDARY(View data)
Packages
MongoDB replication
Packages per shipment
QueriesPackages per
shipment
Shipment managementcore services
Master and View data are stored in PRIMARY, in different databases.
View data is replicated to SECONDARY
CommandsPackages per shipment
view synchronizer
View data synchronization
CQRS & EVS with
• Caveats:Staleness of data read by queries.
Availability loss and overloads of PRIMARY in case of SECONDARIES failures.
Balancer in sharding environments can introduce additional staleness and duplication in data being read.
CQRS & EVS with
• EVS stands for EVent Sourcing.
• Closely related to DDD (Domain Driven Design) and CQRS.
• Events are changes to the state of a domain model subset of an application.
• Events occurred in the past, and are also referred as Domain events.
• EVS consists in expressing the state of a domain model subset as a sequence of events.
CQRS & EVS with
Shipment managment core service
EVS engine
Add package
Add package
Create
Add package
Assign courier
Add delivery
Add delivery
Add delivery
Last event
Store
Incoming Domain Events Last event
Delivery
Package
Customer
Courier
Promo
Rebuild
Retrieve
Event store
Apply
ALL retrieved events are replayed to rebuild domain model subset state
Last event is aplied after domain model subset rebuild, and stored in the Event Store.
CQRS & EVS with
• Events are stored in a Event Store, that can be huge (ideally infinite), depending on the domain subset’s lifetime.
• To keep growth of the Event Store under control:Store a point-in-time state of the domain model
subset (take a snapshot of it).
Store only events occurred after the last snapshot.
CQRS & EVS with Shipment management core service
EVS engine
Add package
Add package
Create
Add package
Assign courier
Add delivery
Add delivery
Add delivery
Last event
Store
Incoming Domain Events Last event
Delivery
Package
Customer
Courier
Promo
Rebuild
Retrieve
Event store
Snapshot
Apply
Store snapshot
ONLY events after snapshot are used to rebuild domain model subset state.
A snapshot is taken after replaying some events.
Remaining events (events after th snapshot) are used to rebuild the state of the domain model subset and remain stored in the Event Store.
CQRS & EVS with
• Single collection event store:Both snapshots and events conform a single
document => Single collection.
findAndModify can be well suited to add events and update or create snapshots in a single atomic operation.
Usable in scenarios with low frequency rate of events.
CQRS & EVS with
Event
Event
Event
Event
Event
Event
Event
Event
Storesnapshot
AggregateId
AggregateId
Root
Entity
Valueobject
Valueobject
Entity
Root
Entity
Valueobject
Valueobject
Entity
Component / service
EVS engine
StoreRetrieve
Event store
Event
Event
Documents include both snapshot and events pending to apply.
Events are added with a $push operation in a findAndModify atomic operation.
Due to growth of events array, documents can be moved often, impacting performance
CQRS & EVS with
• Multiple collection event store:Snapshots and events are persisted in different
collections.
May require some locking and operations isolationmanagement in multithreaded environments.
CQRS & EVS with
Event
Event
Event
Event
Event
Event
Event
Event
Storesnapshot
AggregateId
AggregateId
AggregateId
AggregateId
AggregateId
AggregateId
AggregateId
AggregateId
AggregateId
AggregateId
Root
Entity
Valueobject
Valueobject
Entity
Root
Entity
Valueobject
Valueobject
Entity
Component / service
EVS engine
StoreRetrieve
Event store
Snapshots and events are stored in their respective collection, separated from each other.
Snapshots and events are stored in a single atomic update operation.
CQRS & EVS with
• ESB = Enterprise Service Bus. It is an infrastructure piece of software than a pattern itself.
• Implements different messaging patterns:Publish / Subscribe
Request / Reply
Dead letter channel.
Others.
CQRS & EVS with
Publisher Subscriber
ServiceService Service Service
Subscriber Subcriber
ESB config DB
Message published and delivered to all subscribers.
ESB configuration data persistence: subsciptions database.
Requester DB Service DB Service DB
Request message, delivered to replier. Replier info in message header.
Reply message delivered to requester.
CQRS & EVS with
• MongoDB is applicable as a persistence layer for subscriptions store.
• In complex applications, with many services interconnected:Persistence layer for ESB usage statistics and
metrics (latency, msg/s, queue length, etc.).
Persistence layer for Dead Letter Channel subscribers (logger for all services connected to ESB).
CQRS & EVS with
Publisher Subscriber
ServiceMonitor Service Service
Subscriber Subcriber
ESB config DB
Metrics, heartbeat, statistics sample message.
Statistics DB Service DB Service DB
Dead letter channel message (error on component captured)
Statistics database.Dead letter channel database (logger).
CQRS & EVS with
• Two main messaging streams: data and control.
• Control messages: Monitoring metrics samples.
Start / pause / stop service processing.
Heartbeat messages (ping to services).
• Message traffic is lead by a transport tier.
• To prevent contention and service saturation, a buffering or queuing system is recommended.
CQRS & EVS with
Service 2 Service 3
Control ControlProcess Process
Service 1
ControlProcess
Bus service
Process (Message Routing) Control
Action handling Action handling Action handling
Datamessages
Controlmessages
Control handling
Control handling
Control handling
Durable direct RabbitMQ exchange (Control)
Durable direct RabbitMQ exchange (Command, Events, Documents)
Service contracts serialized as JSON messages
Bus config DB
Service 3
Service 2
Mirrored RabbitMQ queuesMirrored RabbitMQ queuesMirrored RabbitMQ queues
CQRS & EVS with
• Main functional requirements:Allow management of shipments generated online
by agencies.
Allow management of agency and corporate customers.
Allow offline loading of shipments generated by large customers.
Integrate with other line-of-business systems (CRM, legacy applications, etc.).
CQRS & EVS with
• Main QoS requirements:Nearly 550 agencies.
3K online users during office time.
Several thousands of customers.
Storage needs based on 50K shipments / day (online and offline workloads) and 45 days of operational shipment lifetime.
Estimation of 20 state changes per shipment during its lifetime.
CQRS & EVS with
• Solution components:Frontend: MVC UI.
A layer of CQRS WS interfaces frontend and backend.
Backend: services (some implementing EVS) interconnected by ESB.
A MongoDB Sharding Cluster as the persistence tier for them all.
CQRS & EVS with
•Technologies stack:UI: ASP.NET MVC + JQuery.
CQRS + REST facade: ASP.NET MVC
ESB + Services: .NET Framework based Windows Services.
Messaging transport: RabbitMQ cluster.
Persistence: MongoDb sharding cluster.
CQRS & EVS with
Serv
ices
CommandsEvents
Documents
CommandsEvents
Documents
CommandsEvents
Documents
CommandsEvents
Documents
MongoDb Sharding Cluster
DDDServices
Publishers SubscribersEVS
Services
Queries
CommandsCQRS REST
WS
Domain data: Events. Aggregates. Snapshots.
Domain data: Entities. Value objects. Aggregates.
Domain data: Entities. Value objects. Aggregates. Specifications.
Master dataViews
Service contracts: Events. Commands. Documents.
CQRS & EVS with
Web FrontendWeb applications
CQRS WSLoad balancing
RabbitMQ clientMongoDB
IISNLB
.NET FrameworkASP.NET MVC
MongoDb Router
Services tierBackend and core servicesApplication servicesIntegration services.NET FrameworkSQL Server ClientCliente RabbitMQMongoDb Router (mongos)
Cluster config
Config 1
Config 2
Config 3
Shard 2
Node 1 Node 2Arbiter
Shard 1
Node 1 Node 2Arbiter
Persistence tierCQRS view data.
Services Domain models persistenceESB subscriptions DB persistence
MongoDB sharded cluster(2 shards)
MongoDb Router (mongos).A router per appserver.They cache a local copy of the cluster config DB.
Two-phase commit replication servers.If one or two config servers fail, cluster configuration DB goes readonly (chuncks cannot be moved between shards, no balancing), but data remains writable and available.
Data partition and distributionSharded collections: distribution is done in a per-collection basis. Data is partitioned and distributed across shards, attending to a sharding (partition) key.Each partition unit is called a chunk.EVS collections can be partitioned by a hash on document _id.Sharding keys for collections persisting view data must be choosed for each single case, to guarantee queries performance.