API Statistics – Scalable way

Sumedha Rubasinghe (@sumedha_ruba)Director, API Architecture – WSO2

September, 2014

API Data Model• API Publisher• API• Documentation, SDK, etc• Version• Backend Endpoint• Lifecycle, workflow• Tier/Quota/Usage Plan• API Subscriber/Developer• Application• End User• Access Token• Request/Response

– Geo Location, User Agent, Application, Request time, Version, resource, etc

API Statistics “Summarized outcome of collecting & linking both runtime and static data associated with

API eco system..”

Why?..

“Evaluation of API initiative’s objective”“Usage Patterns for further improvements”

“Understand scaling requirements”“Fraud detection”

“(Statistics) scalability can have many faces.”

Source : http://www.slideshare.net/jmusser/j-musser-apibizmodels2013

WSO2 API Management Platform

API Management - Core Platform

API Gateway

Handler Architecture

Offline Processing

Collecting Statistics from API Runtime

Scaling Statistics CollectionAbility to push to multiple receivers, receiver groupsLocal persistence

Multiple Data ReceiversWorking in a cluster

Communication over Thrift

Apache Cassandra – High volume storage

Apache Hadoop cluster being used for analytics.Apache Hive as the analyzer engine.

Summarized Data – written to relation DB

1

2

5

4 6

3

Dashboards

Using Google Analytics

Online Processing

Events + Events -> Complex Events

WSO2 CEP Engine - Architecture

Siddhi Query - Structure

define stream <event stream>(<attribute> <type>,<attribute>

<type>, ...);

from <event stream>select <attribute>,<attribute>, ...insert into <event stream> ;

Siddhi Query - Examplesdefine stream TempStream

(deviceID long, roomNo int, temp double);

from TempStreamselect roomNo,

tempinsert into

OutputStream ;from TempStream [temp > 30.0 and roomNo != 2043]select roomNo, tempinsert into HotRoomsStream ;

from TempStream#window.time(1 min)select roomNo, avg(temp) as avgTempinsert into HotRoomsStream ;

from TempStreamselect roomNo, tempinsert into RoomTempStream ;

from TempStreamselect deviceID, tempinsert into DeviceTempStream ;

Query ScalingVertical Scaling

Horizontal Scaling

Availability

Execution plan in “RedundantNode” based distributed processing mode<executionPlan name="RedundantNodeExecutionPlan" statistics="enable" trace="enable" xmlns="http://wso2.org/carbon/eventprocessor"> ... <siddhiConfiguration> <property name="siddhi.enable.distributed.processing">RedundantNode</property> <property name="siddhi.persistence.snapshot.time.interval.minutes">0</property> </siddhiConfiguration> ...</executionPlan>

Siddhi Performance• Same JVM Performance (Siddhi with Esper) 4 core machine

– Filters 8M Events/Sec vs Esper 2M – Window 2.5M Events/Sec vs. Esper 1M– Patterns 1.4M Events/Sec about 10X faster

than Esper • Over the Network Performance (Using thrift

based WSO2 event format) • - 8 core machine

– Filter 0.25M (or 250K) Event/Sec

Future Directions

• Storm Integration• WSO2 Machine Learning Project