Clojure@NudayJosh Glover, Lead Backend Developer

Nuday Games

● Stockholm-based game company

● Maker of Rock Science, “The Rock Game of the Century”

Why Clojure?

● See “Hackers and Painters”● Small dev team can do a lot quickly● Easier to reason about concurrency● Engineers attracted to Clojure tend to be

good● Fits in well with all the battle-tested Java

machinery for service at scale

Architecture

● Frontend (iOS app for now)● Load balancers● HTTP servers● RESTful API● Databases

Users

DNS round robin

nginx nginx

Elastic Beanstalk

ELB

Tomcat

...

...

SNS

S3

Elastic Beanstalk

Worker

Tomcat

SQS

RDS DynamoDB

RESTful API

● Tomcat 7 on Elastic Beanstalk● Liberator / Compojure

Routes

(ns insurrection.routes.core (:require [insurrection.resource.get :as get] [insurrection.resource.put :as put]))

(defroutes rockscience-routes (GET "/health" [] get/health) (GET "/player/:playerId/profile" [] get/profile) (GET "/player/:playerId/games" [] get/games) (GET "/player/:playerId/events" [] get/events) (PUT "/player/:playerId/pushToken" [] put/push-token))

Resources(ns insurrection.resource.get)

(def events (get-resource :events view/events [:playerId :timestamp] :transform-fn {:playerId to-int :timestamp to-long} :validation-fn {:playerId (validate EntityId) :timestamp (validate Timestamp)}))

Resources under the hood(ns insurrection.routes.resources)

(defn get-resource [resource resource-fn args & {:as opts}] (-> (lib/resource :allowed-methods [:get] :available-media-types [json-media-type] :available-charsets ["UTF-8"] :authorized? authorized? :exists? (make-exists? resource resource-fn args opts) :handle-ok (make-response-handler resource :ok) :handle-not-found (make-not-found-handler resource args opts)) wrap-keyword-params wrap-json-params wrap-json-response))

http://clojure-liberator.github.io/liberator/assets/img/decision-graph.svg

Handling a GET

(defn make-exists? [resource-name resource-fn args & [opts]] (fn [ctx] (let [params-str (params->str ctx args opts)] (log/info (log-str "Getting %s" resource-name params-str)) (try (when-let [resource (apply resource-fn (params->values ctx args opts))] {resource-name resource}) (catch Exception e (log-error e (log-str "Failed to get %s" resource-name params-str)) {:exception e})))))

Response(defn make-response-handler [resource-name & [status]] {:pre [(s/validate s/Keyword resource-name) (or-validate HttpStatus status)]}

(let [status (or status :ok)] (fn [ctx] (let [exception (:exception ctx) resource (if exception {:status false, :message (.getMessage exception)} (resource-name ctx)) resp (make-response resource) http-status (if exception 500 (http/status status))]

(ring-response (-> resp (assoc :status http-status)))))))

Types, what are they good for?

● Dynamic typing is wonderful, but…● Prismatic Schema gives you validation and

documentation without fighting the compiler

Schemas(ns rockscience.models.entity (:require [rockscience.rules :as rockscience] [schema.core :as s])

(def EntityId (s/both s/Int (s/pred pos? 'pos?)))

(def Timestamp (s/both s/Int (s/pred #(>= % rockscience/epoch) 'ts?)))

Event sourcinghttp://martinfowler.com/eaaDev/EventSourcing.html

The fundamental idea of Event Sourcing is that of ensuring every change to the state of an application is captured in an event object, and that these event objects are themselves stored in the sequence they were applied for the same lifetime as the application state itself.

Querying event logGET /player/:playerId/events?timestamp=1413801431000

{ "timestamp": "2014-10-20T15:40:47Z", "events": [ { "eventType": "ChallengeReceived", "playerId": "123", "timestamp": "2014-10-20T11:05:40Z", "entityId": "912" }, { "eventType": "RoundAnswered", "playerId": "123", "timestamp":"2014-10-20T11:06:12Z", "entityId":"912" }, { "eventType":"GameEnded", "playerId":"123", "timestamp":"2014-10-20T11:09:05Z", "entityId":"912" } ]}

Event log view

(defn events [player-id timestamp] {:pre [(validate EntityId player-id) (validate Int timestamp)] :post [(validate ApiPlayerNotificationEvents %)]}

{:timestamp (->api-timestamp (to-long (now))) :events (->> (player/get-events player-id timestamp) (map ->api-player-notification-event))})

More interesting schemas(def ApiPlayerNotificationEvents {:timestamp ApiTimestamp :events [ApiPlayerNotificationEvent]})

(def ApiTimestamp (s/both s/Str (s/pred api-timestamp? 'api-timestamp?)))

(defn api-timestamp? [s] (try (parse (formatters :date-time-no-ms) s) (catch Exception _ false)))

Schemas built of schemas(def ApiPlayerNotificationEvent (->> (merge PlayerNotificationEvent {:event-type ApiPlayerNotificationEventType :timestamp ApiTimestamp}) (mmap #(vector (->camelCase %1) %2))))

(def ApiPlayerNotificationEventType (->> event-types (map ->api-event-type) (apply s/enum)))

(def event-types #{:challenge-received :round-completed :game-ended})

It’s turtles all the way down

(def PlayerNotificationEvent {:player-id s/Str :event-type PlayerNotificationEventType :entity-id s/Str :timestamp s/Int})

(def PlayerNotificationEventType (apply s/enum event-types))

Transmogrify turtles to elephants(defn ->api-player-notification-event [ev] {:pre [(s/validate PlayerNotificationEvent ev)] :post [(s/validate ApiPlayerNotificationEvent %)]}

(->> ev (mmap (fn [k v] [(->camelCase k) (case k :event-type (->api-event-type v) :timestamp (->api-timestamp v) v)]))))

(defn ->api-timestamp [ts] {:pre [(s/validate s/Int ts)] :post [(s/validate ApiTimestamp %)]} (unparse ts-formatter (from-long ts)))

Persisting events

● Events created on every mutation, so write fast

● Not really relational● Good fit for NoSQL!

Specifically, DynamoDB● Dynamo is scaled for

reads/sec and writes/sec per table

Events table(require '[amazonica.aws.dynamodbv2 :as dynamo])

(-> (dynamo/describe-table "events") :table pprint)

;=> {:table-size-bytes 28296, :item-count 418, :table-status "ACTIVE", :creation-date-time #<DateTime 2014-09-17T11:41:47.000+02:00>, :table-name "events", :attribute-definitions [{:attribute-type "S", :attribute-name "player-id"} {:attribute-type "N", :attribute-name "timestamp"}], :key-schema [{:attribute-name "player-id", :key-type "HASH"} {:attribute-name "timestamp", :key-type "RANGE"}], :provisioned-throughput {:number-of-decreases-today 0, :read-capacity-units 1, :write-capacity-units 1}}

Querying events(->> (dynamo/query :table-name "events" :select "ALL_ATTRIBUTES" :key-conditions {:player-id {:attribute-value-list ["123"] :comparison-operator "EQ"} :timestamp {:attribute-value-list [1413801431000] :comparison-operator "GE"}}) :items first pprint)

{:event-type "challenge-received", :player-id "123", :entity-id "912", :timestamp 1413803140000}

Writing events

(dynamo/put-item :table-name "events", :item {:event-type "challenge-received", :player-id "123", :entity-id "912", :timestamp 1413803140000}))

Replaying events

● Our events report on mutation, but don’t cause it

● Not enough data for replay :(

CQRS?http://martinfowler.com/bliki/CQRS.html

Ideal mutation flow?HTTP POST, PUT, or DELETE

Create mutation commandLog mutation command to DynamoDB, return ID

Publish mutation ID to SQS

Receive mutation ID from SQSApply mutation command

Update DynamoDB with mutation outcome

HTTP GET /mutation/:idUpdate client state

App

REST service

Worker

App

Challenges with this approach

● Our app assumes synchronous requests● Extra SQS puts and gets introduce more

latency● Implementing long polling after the fact is a

PITA● Web sockets to the rescue?

Compromise mutation flow

HTTP POST, PUT, or DELETE

Apply mutation commandLog mutation command to DynamoDB

Return result

Update client state

App

REST service

App

Replaying commands

Read command from DynamoDBPublish command to SQS

Receive mutation command from SQSApply mutation command

If result is different than it was the first time, fail?

Loader

Worker

No CQRS

HTTP POST, PUT, or DELETE

Log mutation model to DynamoDBPerform mutation

Return result

Update client state

App

REST service

App

Replaying without CQRS

Read mutation model from DynamoDBHTTP POST, PUT, or DELETE

Loader

Perform mutationReturn result

REST service

If result is different, fail? Loader

When event sourcing, do:

● Ensure that your mutations are idempotent● Log enough to replay