Architectural Overview of Distributed Parcel Tracking and Management System

Mark Cheesemanmcheeseman@visaglobal.com.auhttp://hombredequeso.id.au/

Public Health Warning (Required by Law…)The following presentation includes graphic scenes of UI torture and SQL Server abuse that may disturb some viewers.

Usable User Interface design and relational use of relational databases were both harmed in the making of this product.

Viewers who suspect they may find such scenes disturbing may wish to confirm the presence of a sick bag in the back of the seat before you.

Website/API

Business Services

Process BookingsProduce Labels

CustomsBilling

Parcel Scan In/Out...

Tracking Website

Customs

CustomersParcel

Recipients

AusPost

NZ Post

Whip Operators

Conceptual Diagram

...

Website API *Tracking Website *

API Update ServiceTracking Update

Service

Parcel Audit Service *

AusPost Service *

Billing Service *

Booking Service *

Cyberfreight Service *

MAWB Service

NZ Post Service * NZ Post API Service

Reporting Service *

Scan Processing Service*

Email Processing Service *

File Transfer Service

Customers/Operators

(Web Browser)

Scan/Print(Application)

Parcel Recipients(Web Browser)

Cyberfreight(Application)

NZ Post (API)

CyberfreightCustoms (SMTP)

AusPost (SFTP)

Logical Diagram

Bus

* = has own database

Azure VM

Azure Service Bus

NSB MessageTransport

Physical Diagram

Azure SQL Server

· API· Booking· AusPost· Cyberfreight· ...

Azure SQL Server

· Tracking Website

Azure Web AppWebsite/API

(ASP.NET MVC API)

Azure Web AppTracking Website(ASP.NET MVC API)

RavenDb Server

· Booking NSB EP· AusPost NSB EP· NZPost NSB EP· ...

Service: Booking NSB Endpoint

Service: AusPost NSB Endpoint

Service: NZPost NSB Endpoint

Service: Cyberfreight NSB Endpoint

...

REST API: Siren

Specification: https://github.com/kevinswiber/siren

{ "class": ["booking"], "properties": { "bookingNumber": "BREDH10000910", "merchant": "Red Hare", "merchantCode": "REDH", "parcelCount": 1000, "created": "2015-05-24T11:47:37.4721121Z", "processes": "Labelling, Customs", "parcelsRecordedForCustomsCount": 1000, "parcelsRecordedForCustomsTimestamp": "2015-05-24T11:47:41.2653291Z", "ausPostLabelsGenerated": "2015-05-24T11:47:48.9757701Z", "ausPostLabelsRecordedForCustomsCount": 1000, "ausPostLabelsRecordedForCustomsTimestamp": "2015-05-24T11:48:00.2774165Z", "labelledSpreadsheetCreated": "2015-05-24T11:48:02.6825541Z" }, "entities": [ { "class": ["bookingSubmission"], "rel": ["/rel/booking-bookingsubmission"], "links": [ { "rel": ["self"], "href": "http://localhost:54747/api/Merchant/BookingSubmissions/92be90a5-710d-40ef-bb6c-a4a201672282" } ] } ], "links": [ { "rel": ["self"], "href": "http://localhost:54747/api/Merchant/Bookings/BREDH10000910" }, { "rel": ["file","xlsx","booking/labelledContent"], "href": "http://localhost:54747/api/Merchant/Bookings/BREDH10000910/labelledContent" } ], "actions": [ { "name": "post-bookingDespatchManifestRequest", "title": "Generate Despatch Manifest For Booking", "method": "POST", "href": "http://localhost:54747/api/Merchant/Bookings/BREDH10000910/bookingDespatchManifestRequests", "type": "application/x-www-form-urlencoded", "fields": [] } ]}

GET http://localhost:54747/api/Merchant/Bookings/BREDH10000910

{ "class": ["booking"], "properties": { "bookingNumber": "BREDH10000910", "merchant": "Red Hare", "merchantCode": "REDH", "parcelCount": 1000, "created": "2015-05-24T11:47:37.4721121Z", "processes": "Labelling, Customs", "parcelsRecordedForCustomsCount": 1000, "parcelsRecordedForCustomsTimestamp": "2015-05-24T11:47:41.2653291Z", "ausPostLabelsGenerated": "2015-05-24T11:47:48.9757701Z", "ausPostLabelsRecordedForCustomsCount": 1000, "ausPostLabelsRecordedForCustomsTimestamp": "2015-05-24T11:48:00.2774165Z", "labelledSpreadsheetCreated": "2015-05-24T11:48:02.6825541Z" }, "entities": [ { "class": ["bookingSubmission"], "rel": ["/rel/booking-bookingsubmission"], "links": [ { "rel": ["self"], "href": "http://localhost:54747/api/Merchant/BookingSubmissions/92...82" } ] } ], "links": [ { "rel": ["self"], "href": "http://localhost:54747/api/Merchant/Bookings/BREDH10000910" }, { "rel": ["file","xlsx","booking/labelledContent"], "href": "http://localhost:54747/api/Merchant/Bookings/BREDH10000910/labelledContent" } ], "actions": [ { "name": "post-bookingDespatchManifestRequest", "title": "Generate Despatch Manifest For Booking", "method": "POST", "href": "http://localhost:54747/api/Merchant/Bookings/BREDH10000910/bookingDespatchManifestRequests", "type": "application/x-www-form-urlencoded", "fields": [] } ]}

LogicalView

{ "class": ["booking"], "properties": { "bookingNumber": "BREDH10000910", "merchant": "Red Hare", "merchantCode": "REDH", "parcelCount": 1000, "created": "2015-05-24T11:47:37.4721121Z", "processes": "Labelling, Customs", "parcelsRecordedForCustomsCount": 1000, "parcelsRecordedForCustomsTimestamp": "2015-05-24T11:47:41.2653291Z", "ausPostLabelsGenerated": "2015-05-24T11:47:48.9757701Z", "ausPostLabelsRecordedForCustomsCount": 1000, "ausPostLabelsRecordedForCustomsTimestamp": "2015-05-24T11:48:00.2774165Z", "labelledSpreadsheetCreated": "2015-05-24T11:48:02.6825541Z" }, "entities": [ { "class": ["bookingSubmission"], "rel": ["/rel/booking-bookingsubmission"], "links": [ { "rel": ["self"], "href": "http://localhost:54747/api/Merchant/BookingSubmissions/92be90a5-710d-40ef-bb6c-a4a201672282" } ] } ], "links": [ { "rel": ["self"], "href": "http://localhost:54747/api/Merchant/Bookings/BREDH10000910" }, { "rel": ["file","xlsx","booking/labelledContent"], "href": "http://localhost:54747/api/Merchant/Bookings

/BREDH10000910/labelledContent" } ], "actions": [ { "name": "post-bookingDespatchManifestRequest", "title": "Generate Despatch Manifest For Booking", "method": "POST", "href": "http://localhost:54747/api/Merchant/Bookings

/BREDH10000910/bookingDespatchManifestRequests", "type": "application/x-www-form-urlencoded", "fields": [] } ]}

API Event Store

Bus (NSB)

API Update Service

WEB API

Event Stream to Siren Entity Transform

Event Store

Http Request (GET)

NSB Events from various endpoints (Booking, AusPost, Customs)

“Event Store” Events

NSB Events

Siren Entity

Business Processes Involving Multiple Services

(Long Running Business Processes)

Process Manager: Saga as Process Controller

Booking Endpoint*

Saga

AusPost Endpoint*

Customs Endpoint*

7. Booking “fully” accepted

Saga State

Fastway Endpoint*

1. BookingReceivedEvent

2. CreateApLabelsCommand

3. CreateFwLabelsCommand

6. CreateConsolidationsCommand

5. ApLabelsCreatedEvent

4. FwLabelsCreatedEvent

Process Manager: Routing Slip

Booking Endpoint*

AusPost Endpoint*

Customs Endpoint*

Fastway Endpoint*

ProcessBookingSubmissionCommand

Routing Slip (on Booking Message):1. AusPostEndpoint2. FastwayEndpoint3. CustomsEndpoint

Booking Booking Booking

Booking(“fully” accepted)

Process Manager: Event Driven Booking Processes on Event

Booking Endpoint*

AusPost Endpoint*

Customs Endpoint*

Fastway Endpoint*

1. BookingCreatedEvent

2. AusPostLabelsCreatedEvent

3. FastwayLabelsCreatedEvent

Booki ng Repr esent at i on on al l Event s:{

Gui d Booki ngI dBooki ngPr ocess[ ] Pr ocesses

}

publ i c enum Booki ngPr ocess{

Label l i ng = 1,Cust oms = 2,AusPost Pr eLabel l ed = 3

}

4. Booking “fully” accepted

Solving Big(ish) Data on the Bus

AusPost Service *

Booking Service *

Cyberfreight Service *

NZ Post Service * NZ Post API Service

Email Processing Service *

File Transfer Service

The Idealistic Vision...

(Logical) Bus

AusPost Service *

Booking Service *

Cyberfreight Service *

NZ Post Service * NZ Post API Service

Email Processing Service *

File Transfer Service

Bus: Message Bus + Data Bus

OHS

ACL

ACL

ACL

OHS

OHS

ACL

ACL

(Message) Bus(Data) Bus

AusPost Service *

Booking Service *

Cyberfreight Service *

NZ Post Service * NZ Post API Service

Email Processing Service *

File Transfer Service

Data Bus Implementation

SQL Database

NSB (Physical) Transport:Azure Service Bus

Data Bus: Implemention viaSQL queries

Not All Transports are Created Equal…

References

Siren Specification

https://github.com/kevinswiber/siren

Richardson Maturity Model (for API's)

http://martinfowler.com/articles/richardsonMaturityModel.html

Recommended books on REST

Richardson and Amundsen, RESTful Web APIs.

Webber et. al., REST in Practice: Hypermedia and Systems Architecture

Messaging in General

Hohpe and Woolf, Enterprise Integration Patterns

(for various forms of routing as process management see ch. 7)

Blog Series on NServiceBus and Process Management Patterns

https://lostechies.com/jimmybogard/2013/05/14/saga-patterns-wrap-up/

(Top of post has links to all post in series)

https://lostechies.com/jimmybogard/2014/10/02/nservicebus-5-0-behaviors-in-action-routing-slips/

Event Based Systems

Vaughn Vernon, Implementing Domain-Driven Design, ch. 8.

Dynamic Event Replay Mechanism

http://blogs.msdn.com/b/davidebb/archive/2010/01/18/use-c-4-0-dynamic-to-drastically-simplify-your-private-reflection-code.aspx