"Introduction to Kx Technology", James Corcoran, Head of Engineering EMEA at First Derivatives

transcript

Kx Introduction

November 2016

2 | KX SYSTEMS

Agenda

• About

• Kx Technology

• Solutions

• Technical Overview

• Resources

• Q & A

3 | KX SYSTEMS

First Derivatives plc

People 1,600 FTEs

Locations

Ireland, London, New York, Singapore,

Sydney, Tokyo, Toronto, Zurich

History

Founded in 1996 Public listed Company

Always Profitable

Divisions

Big Fast Data

Capital Markets Consulting

Domain experts Technology experts

MRP/Prelytix Digital marketing

resources

Our Software

Streaming Analytics, In Memory Computing, - the market standard

Tools for Kx

Develop, Deploy, Visualize

Industry Solutions

Powered by Kx Implemented and supported by us

4 | KX SYSTEMS

Kx Technology Overview

• Integrated columnar database & programming system

• Streaming, real time and historical data

• Built for massive data volumes

• In-database analytics

• Parallelism

• Compression

• More

The kdb+ database Features

• Interpreted

• Event-driven

• Functional

• Array / Vector

• Query

• Time-series

q Programming Language

5 | KX SYSTEMS

Tools for Kx Solutions

Solutions Overview

• Dashboards for Kx

• Analyst for Kx

• Control for Kx

• Stream for Kx

• Kx for Algos

• Kx for Flow

• Kx for Monitoring

• Kx for Pharma

• Kx for Surveillance

• Kx for Sensors

• Kx for Telcos

Visualise &

Discover

Architect &

Manage

Enterprise Concepts

6 | KX SYSTEMS

Technical Overview

7 | KX SYSTEMS

• Integrated columnar database & programming system

• Streaming, real time and historical data

• 32 bit free version

• 64 bit licensed version

8 | KX SYSTEMS

kdb+ Features

• Standard OS & hardware

• In-database analytics

• Compression

• Parallelism

• Interfaces

9 | KX SYSTEMS

q programming language

• Functional

• Array/Vector

• Query

• Interpreted

• Time-series

q Language

Seamless database and data manipulation language • We are fast not only due to data architecture, our native

programming language, q, runs inside the database not in separate processes with costly data passing.

• We are one of the few fully 64 bit databases and unique in having time as a native type, with nanosecond resolution and a full set of operations over time.

Quick to Deliver • Unlike many compile-link and run approaches q is

dynamic allowing much shorter development and deployment cycles

Allows you to get more from less code • q is terse - this means a smaller code base to maintain

and debug (one line in q versus many of Java).

Other Key Features • Array/Vector Language • Parallelism • Native Compression

Project Euler Problem 1* in Java

Project Euler Problem 1* in q

*Find the Sum of all multiples of 3 or 5 below 1000. For more information on the Euler Programming Problems see - https://projecteuler.net

Sample q query

select open: first price, high: max price, low: min price, close: last price from trade where date = 2013.05.01, sym = `AAPL

q vs. standard SQL

• q supports order for faster execution

• q eliminates many joins

• Standard SQL does not enforce row order

• Built-in support for CSV, JSON & XML

• higher productivity & less maintenance

q vs. standard SQL

select rev wavg s.n=`BRAZIL by o.d.year from l where

o.c.n.r=`AMERICA, o.d.year in 1995 1996, p.t=`$"ECONOMY ANODIZED STEEL"

select year,sum(case when name='BRAZIL' then rev else 0 end)/sum(rev) from(

select extract(year from o.d)as year,l.x*(1-l.xd) as rev,n2.name

from p,s,l,o,c,n n1,n n2,r

where p.p=l.p and s.s=l.s and l.o=o.o and o.c=c.c and c.n=n1.n and n1.r=r.r and r.name='AMERICA'

and s.n=n2.n and o.d between date'1995-01-01' and date'1996-12-31' and

p.t='ECONOMY ANODIZED STEEL')t

group by year order by year

Parallel queries in q

Serial:

calcEvent each eventTypes

Parallel:

calcEvent peach eventTypes

Time-Series Analysis

• Native temporal data types

• high-precision (nanosecond)

• temporal arithmetic

• asof join/bi-temporal joins

• join time-series on-the-fly

Time-Series Analysis cont’d

select count i by userAgent,5 xbar time.minute from events where date within (.z.D - 7;.z.D), eventType=`login

aj[`time; select time,price from trade where date=last date,sym=`MSFT; select time,bid,ask from quote where date=last date,sym=`MSFT]

q v Hive ETL comparison

`product_dtl set (“ISFS”; enlist “,”) 0: `input.txt;

create table product_dtl ( product_id: int, product-name: string, product_price: float, product_category: string) rows format delimited fields terminated by ',' ;

describe product_dtl;

load data local inpath ’input.txt' into table product_dtl;

Hive –f /home/cloudera/sample.sql

Streaming, Real-Time and Historical Data

19 | KX SYSTEMS

Sample Architecture

20 | KX SYSTEMS

Kx Solutions

Technical Overview

21 | KX SYSTEMS

Dashboards for Kx

22 | KX SYSTEMS

Analyst for Kx

23 | KX SYSTEMS

Streams for Kx

Fast Data Management

• Clustered services

• Load-balancing

• Failover

• Automated recovery

• Query Manager

• Messaging Server

• Alert Framework

• File Watchers & Loaders

• Scheduling

24 | KX SYSTEMS

Resources

25 | KX SYSTEMS

Resources – Wiki Page

26 | KX SYSTEMS

Resources – Kx Community

27 | KX SYSTEMS

Resources – Forum & Google Group

28 | KX SYSTEMS

Resources – Github

29 | KX SYSTEMS

Resources – Q For Mortals

30 | KX SYSTEMS

Resources – Q Tips

31 | KX SYSTEMS

Resources – Free 32-bit download

32 | KX SYSTEMS

Resources – Kx Community Meetups

33 | KX SYSTEMS

Resources – Lecture Series

34 | KX SYSTEMS

Thank You.

kx.com @kxsystems

35 | KX SYSTEMS

Appendix

• Kx Wiki

• Kx Community

• Google Group

• Kx Github

• Q For Mortals

• Q Tips

• STAC benchmarks

• Free version

• Kx Meetups

• FD Lecture Series

"Introduction to Kx Technology", James Corcoran, Head of Engineering EMEA at First Derivatives

Data & Analytics