CQRS & EVS withBuilding of a modern distributed enterprise application

CQRS & EVS with

• My name is Lluis Fernández.

• I’m involved in a tech group of freelancers named B-Team.

• We offer consultancy on distributed system architectures.

• Contact: llffernandez@gmail.com

b-team_b-team@googlegroups.com

• What’s this about.

• The patterns: CQRS, EVS and ESB.

• All together: An example of distributed enterprise application.

CQRS & EVS with

CQRS & EVS with

What’s this about?

CQRS & EVS with

• NoSQL databases, like MongoDB, have contributed to the adoption and implementation of archetypes and patterns to develop distributed enterprise applications.

• High read/write rates, huge data storesmanagement, highly scalable read-only replication and schema-less document orientation, are well suited for those kind of patterns and techniques.

CQRS & EVS with

• This presentation is about how some of these patterns and archetypes had been applied during the implementation of real-word use case: a Shipment Management application for one of the most important Messaging and Packages Delivery companies in Spain.

CQRS & EVS with

CQRS.

CQRS & EVS with

• CQRS stands for Command-Query Responsibility Segregation.

• Commands are expected to do writes of smallportions of data.

• Queries are expected to do reads of big portions of data.

• CQRS encourages the separation of these different workloads.

CQRS & EVS with

CQRS facade

Commands

Queries

Shipment management core services

View data synchronization

Regular operations on Master data

Master DB

(R/W)

Views DB

(Readonly)

Replication

Corporateshipment

management web app.

Packages

Add package

Add destination

Add origin

Add delivery

Assign courier

Pending deriveries

Lost packages

Packages by shipment

Packages by shipment

CQRS & EVS with

• Reduces contention and collision between queries and commands persistence-level operations.

• Allows the scaling-out of queries, usually the most common and expensive operations.

• Allows the use of heterogeneous view data stores.

CQRS & EVS with

Shipment management core services

Simple automatic synchronization(replication)

Queries

Commands

Datasets optimized for expected queries (grouping, filtering, joins, etc.)

Infrastructure-provided synchronization mechanism.No optimization:Master data = View data

Regular datasets(packages, deliveries, etc.)

These synchronization processes introduce staleness of data provided to the queries.

Views DB (Readonly) Master DB (R/W)

View data synchronization

Regular operations on Master data

Packages

DeliveriesCustomers

Packages by shipment

Package tracking

CQRS & EVS with

Queries

Commands Shipment management core services

Queries

Views DB (Readonly)

View data synchronization

View data synchronization

Regular operations on Master data

Master DB (R/W)

More copies of view data or more different views.Also, copies of view data can be heterogeneous.

Views DB (Readonly)

Packages by shipment

Package tracking

Packages by shipment

Package tracking

Packages

Deliveries

Customers

CQRS & EVS with

• MongoDB replica sets and ReadPreference can be used to segregate master and view data:Commands write on the PRIMARY.

View data is also written on the PRIMARY.

Queries read from SECONDARIES.

• Separate master and view data collections in different databases (MongoDb write locking).

• Schema optimization and indexing in collections storing view data also improve performance.

CQRS & EVS with Shipment management core services

Queries are launched against SECONDARIES (SECONDARY or SECONDARY_PREFERRED read preference)

MongoDB replication

Commands

Queries

View data synchronization

Packages per shipment view synchronizer

Data optimized for views: Schema design. Indexing.

Multiple SECONDARIES allows queries scaling.

Regular operations on Master data

Commands are launched against PRIMARIES (PRIMARY or PRIMARY_PREFERRED read preference)

View data ALSO goes to PRIMARIES

Master data (affected by commands) goes toPRIMARIES

PRIMARY(Master and Views data)

SECONDARIES(View data)

Packages by shipment

Package tracking

Packages

DeliveriesCustomers

Packages by shipment

Package tracking

Packages

DeliveriesCustomers

CQRS & EVS with

PRIMARY(Master and View data)

SECONDARY(View data)

Packages

MongoDB replication

Packages per shipment

QueriesPackages per

shipment

Shipment managementcore services

Master and View data are stored in PRIMARY, in different databases.

View data is replicated to SECONDARY

CommandsPackages per shipment

view synchronizer

View data synchronization

CQRS & EVS with

• Caveats:Staleness of data read by queries.

Availability loss and overloads of PRIMARY in case of SECONDARIES failures.

Balancer in sharding environments can introduce additional staleness and duplication in data being read.

CQRS & EVS with

EVS.

CQRS & EVS with

• EVS stands for EVent Sourcing.

• Closely related to DDD (Domain Driven Design) and CQRS.

• Events are changes to the state of a domain model subset of an application.

• Events occurred in the past, and are also referred as Domain events.

• EVS consists in expressing the state of a domain model subset as a sequence of events.

CQRS & EVS with

Shipment managment core service

EVS engine

Add package

Add package

Create

Add package

Assign courier

Add delivery

Add delivery

Add delivery

Last event

Store

Incoming Domain Events Last event

Delivery

Package

Customer

Courier

Promo

Rebuild

Retrieve

Event store

Apply

ALL retrieved events are replayed to rebuild domain model subset state

Last event is aplied after domain model subset rebuild, and stored in the Event Store.

CQRS & EVS with

• Events are stored in a Event Store, that can be huge (ideally infinite), depending on the domain subset’s lifetime.

• To keep growth of the Event Store under control:Store a point-in-time state of the domain model

subset (take a snapshot of it).

Store only events occurred after the last snapshot.

CQRS & EVS with Shipment management core service

EVS engine

Add package

Add package

Create

Add package

Assign courier

Add delivery

Add delivery

Add delivery

Last event

Store

Incoming Domain Events Last event

Delivery

Package

Customer

Courier

Promo

Rebuild

Retrieve

Event store

Snapshot

Apply

Store snapshot

ONLY events after snapshot are used to rebuild domain model subset state.

A snapshot is taken after replaying some events.

Remaining events (events after th snapshot) are used to rebuild the state of the domain model subset and remain stored in the Event Store.

CQRS & EVS with

• Single collection event store:Both snapshots and events conform a single

document => Single collection.

findAndModify can be well suited to add events and update or create snapshots in a single atomic operation.

Usable in scenarios with low frequency rate of events.

CQRS & EVS with

Event

Event

Event

Event

Event

Event

Event

Event

Storesnapshot

AggregateId

AggregateId

Root

Entity

Valueobject

Valueobject

Entity

Root

Entity

Valueobject

Valueobject

Entity

Component / service

EVS engine

StoreRetrieve

Event store

Event

Event

Documents include both snapshot and events pending to apply.

Events are added with a $push operation in a findAndModify atomic operation.

Due to growth of events array, documents can be moved often, impacting performance

CQRS & EVS with

• Multiple collection event store:Snapshots and events are persisted in different

collections.

May require some locking and operations isolationmanagement in multithreaded environments.

CQRS & EVS with

Event

Event

Event

Event

Event

Event

Event

Event

Storesnapshot

AggregateId

AggregateId

AggregateId

AggregateId

AggregateId

AggregateId

AggregateId

AggregateId

AggregateId

AggregateId

Root

Entity

Valueobject

Valueobject

Entity

Root

Entity

Valueobject

Valueobject

Entity

Component / service

EVS engine

StoreRetrieve

Event store

Snapshots and events are stored in their respective collection, separated from each other.

Snapshots and events are stored in a single atomic update operation.

CQRS & EVS with

ESB.

CQRS & EVS with

• ESB = Enterprise Service Bus. It is an infrastructure piece of software than a pattern itself.

• Implements different messaging patterns:Publish / Subscribe

Request / Reply

Dead letter channel.

Others.

CQRS & EVS with

Publisher Subscriber

ServiceService Service Service

Subscriber Subcriber

ESB config DB

Message published and delivered to all subscribers.

ESB configuration data persistence: subsciptions database.

Requester DB Service DB Service DB

Request message, delivered to replier. Replier info in message header.

Reply message delivered to requester.

CQRS & EVS with

• MongoDB is applicable as a persistence layer for subscriptions store.

• In complex applications, with many services interconnected:Persistence layer for ESB usage statistics and

metrics (latency, msg/s, queue length, etc.).

Persistence layer for Dead Letter Channel subscribers (logger for all services connected to ESB).

CQRS & EVS with

Publisher Subscriber

ServiceMonitor Service Service

Subscriber Subcriber

ESB config DB

Metrics, heartbeat, statistics sample message.

Statistics DB Service DB Service DB

Dead letter channel message (error on component captured)

Statistics database.Dead letter channel database (logger).

CQRS & EVS with

• Two main messaging streams: data and control.

• Control messages: Monitoring metrics samples.

Start / pause / stop service processing.

Heartbeat messages (ping to services).

• Message traffic is lead by a transport tier.

• To prevent contention and service saturation, a buffering or queuing system is recommended.

CQRS & EVS with

Service 2 Service 3

Control ControlProcess Process

Service 1

ControlProcess

Bus service

Process (Message Routing) Control

Action handling Action handling Action handling

Datamessages

Controlmessages

Control handling

Control handling

Control handling

Durable direct RabbitMQ exchange (Control)

Durable direct RabbitMQ exchange (Command, Events, Documents)

Service contracts serialized as JSON messages

Bus config DB

Service 3

Service 2

Mirrored RabbitMQ queuesMirrored RabbitMQ queuesMirrored RabbitMQ queues

CQRS & EVS with

All together: An example of distributed application.

CQRS & EVS with

• Main functional requirements:Allow management of shipments generated online

by agencies.

Allow management of agency and corporate customers.

Allow offline loading of shipments generated by large customers.

Integrate with other line-of-business systems (CRM, legacy applications, etc.).

CQRS & EVS with

• Main QoS requirements:Nearly 550 agencies.

3K online users during office time.

Several thousands of customers.

Storage needs based on 50K shipments / day (online and offline workloads) and 45 days of operational shipment lifetime.

Estimation of 20 state changes per shipment during its lifetime.

CQRS & EVS with

• Solution components:Frontend: MVC UI.

A layer of CQRS WS interfaces frontend and backend.

Backend: services (some implementing EVS) interconnected by ESB.

A MongoDB Sharding Cluster as the persistence tier for them all.

CQRS & EVS with

•Technologies stack:UI: ASP.NET MVC + JQuery.

CQRS + REST facade: ASP.NET MVC

ESB + Services: .NET Framework based Windows Services.

Messaging transport: RabbitMQ cluster.

Persistence: MongoDb sharding cluster.

CQRS & EVS with

Serv

ices

CommandsEvents

Documents

CommandsEvents

Documents

CommandsEvents

Documents

CommandsEvents

Documents

MongoDb Sharding Cluster

DDDServices

Publishers SubscribersEVS

Services

Queries

CommandsCQRS REST

WS

Domain data: Events. Aggregates. Snapshots.

Domain data: Entities. Value objects. Aggregates.

Domain data: Entities. Value objects. Aggregates. Specifications.

Master dataViews

Service contracts: Events. Commands. Documents.

CQRS & EVS with

Web FrontendWeb applications

CQRS WSLoad balancing

RabbitMQ clientMongoDB

IISNLB

.NET FrameworkASP.NET MVC

MongoDb Router

Services tierBackend and core servicesApplication servicesIntegration services.NET FrameworkSQL Server ClientCliente RabbitMQMongoDb Router (mongos)

Cluster config

Config 1

Config 2

Config 3

Shard 2

Node 1 Node 2Arbiter

Shard 1

Node 1 Node 2Arbiter

Persistence tierCQRS view data.

Services Domain models persistenceESB subscriptions DB persistence

MongoDB sharded cluster(2 shards)

MongoDb Router (mongos).A router per appserver.They cache a local copy of the cluster config DB.

Two-phase commit replication servers.If one or two config servers fail, cluster configuration DB goes readonly (chuncks cannot be moved between shards, no balancing), but data remains writable and available.

Data partition and distributionSharded collections: distribution is done in a per-collection basis. Data is partitioned and distributed across shards, attending to a sharding (partition) key.Each partition unit is called a chunk.EVS collections can be partitioned by a hash on document _id.Sharding keys for collections persisting view data must be choosed for each single case, to guarantee queries performance.

CQRS & EVS with

Than you for your attention.