Date post: | 16-Apr-2017 |
Category: |
Data & Analytics |
Upload: | dataconomy-media |
View: | 174 times |
Download: | 5 times |
Kx Introduction
November 2016
2 | KX SYSTEMS
Agenda
• About
• Kx Technology
• Solutions
• Technical Overview
• Resources
• Q & A
3 | KX SYSTEMS
About
First Derivatives plc
People 1,600 FTEs
Locations
Ireland, London, New York, Singapore,
Sydney, Tokyo, Toronto, Zurich
History
Founded in 1996 Public listed Company
Always Profitable
Divisions
Kx
Big Fast Data
Capital Markets Consulting
Domain experts Technology experts
MRP/Prelytix Digital marketing
resources
Our Software
Kx
Streaming Analytics, In Memory Computing, - the market standard
Tools for Kx
Develop, Deploy, Visualize
Industry Solutions
Powered by Kx Implemented and supported by us
4 | KX SYSTEMS
Kx Technology Overview
• Integrated columnar database & programming system
• Streaming, real time and historical data
• Built for massive data volumes
• In-database analytics
• Parallelism
• Compression
• More
The kdb+ database Features
• Interpreted
• Event-driven
• Functional
• Array / Vector
• Query
• Time-series
q Programming Language
5 | KX SYSTEMS
Tools for Kx Solutions
Solutions Overview
• Dashboards for Kx
• Analyst for Kx
• Control for Kx
• Stream for Kx
• Kx for Algos
• Kx for Flow
• Kx for Monitoring
• Kx for Pharma
• Kx for Surveillance
• Kx for Sensors
• Kx for Telcos
Visualise &
Discover
Architect &
Manage
Enterprise Concepts
6 | KX SYSTEMS
kdb+
Technical Overview
7 | KX SYSTEMS
kdb+
• Integrated columnar database & programming system
• Streaming, real time and historical data
• 32 bit free version
• 64 bit licensed version
8 | KX SYSTEMS
kdb+ Features
• Standard OS & hardware
• In-database analytics
• Compression
• Parallelism
• Interfaces
9 | KX SYSTEMS
q programming language
• Functional
• Array/Vector
• Query
• Interpreted
• Time-series
10
q Language
Seamless database and data manipulation language • We are fast not only due to data architecture, our native
programming language, q, runs inside the database not in separate processes with costly data passing.
• We are one of the few fully 64 bit databases and unique in having time as a native type, with nanosecond resolution and a full set of operations over time.
Quick to Deliver • Unlike many compile-link and run approaches q is
dynamic allowing much shorter development and deployment cycles
Allows you to get more from less code • q is terse - this means a smaller code base to maintain
and debug (one line in q versus many of Java).
Other Key Features • Array/Vector Language • Parallelism • Native Compression
Java
Project Euler Problem 1* in Java
Project Euler Problem 1* in q
*Find the Sum of all multiples of 3 or 5 below 1000. For more information on the Euler Programming Problems see - https://projecteuler.net
11
Sample q query
select open: first price, high: max price, low: min price, close: last price from trade where date = 2013.05.01, sym = `AAPL
12
q vs. standard SQL
• q supports order for faster execution
• q eliminates many joins
• Standard SQL does not enforce row order
• Built-in support for CSV, JSON & XML
• higher productivity & less maintenance
13
q vs. standard SQL
q:
select rev wavg s.n=`BRAZIL by o.d.year from l where
o.c.n.r=`AMERICA, o.d.year in 1995 1996, p.t=`$"ECONOMY ANODIZED STEEL"
SQL:
select year,sum(case when name='BRAZIL' then rev else 0 end)/sum(rev) from(
select extract(year from o.d)as year,l.x*(1-l.xd) as rev,n2.name
from p,s,l,o,c,n n1,n n2,r
where p.p=l.p and s.s=l.s and l.o=o.o and o.c=c.c and c.n=n1.n and n1.r=r.r and r.name='AMERICA'
and s.n=n2.n and o.d between date'1995-01-01' and date'1996-12-31' and
p.t='ECONOMY ANODIZED STEEL')t
group by year order by year
14
Parallel queries in q
Serial:
calcEvent each eventTypes
Parallel:
calcEvent peach eventTypes
15
Time-Series Analysis
• Native temporal data types
• high-precision (nanosecond)
• temporal arithmetic
• asof join/bi-temporal joins
• join time-series on-the-fly
16
Time-Series Analysis cont’d
select count i by userAgent,5 xbar time.minute from events where date within (.z.D - 7;.z.D), eventType=`login
aj[`time; select time,price from trade where date=last date,sym=`MSFT; select time,bid,ask from quote where date=last date,sym=`MSFT]
17
q v Hive ETL comparison
q:
`product_dtl set (“ISFS”; enlist “,”) 0: `input.txt;
Hive:
create table product_dtl ( product_id: int, product-name: string, product_price: float, product_category: string) rows format delimited fields terminated by ',' ;
describe product_dtl;
load data local inpath ’input.txt' into table product_dtl;
Hive –f /home/cloudera/sample.sql
18
Streaming, Real-Time and Historical Data
19 | KX SYSTEMS
Sample Architecture
20 | KX SYSTEMS
Kx Solutions
Technical Overview
21 | KX SYSTEMS
Dashboards for Kx
22 | KX SYSTEMS
Analyst for Kx
23 | KX SYSTEMS
Streams for Kx
Fast Data Management
• Clustered services
• Load-balancing
• Failover
• Automated recovery
• Query Manager
• Messaging Server
• Alert Framework
• File Watchers & Loaders
• Scheduling
24 | KX SYSTEMS
Kx
Resources
25 | KX SYSTEMS
Resources – Wiki Page
26 | KX SYSTEMS
Resources – Kx Community
27 | KX SYSTEMS
Resources – Forum & Google Group
28 | KX SYSTEMS
Resources – Github
29 | KX SYSTEMS
Resources – Q For Mortals
30 | KX SYSTEMS
Resources – Q Tips
31 | KX SYSTEMS
Resources – Free 32-bit download
32 | KX SYSTEMS
Resources – Kx Community Meetups
33 | KX SYSTEMS
Resources – Lecture Series
34 | KX SYSTEMS
Thank You.
kx.com @kxsystems
35 | KX SYSTEMS
Appendix
• Kx Wiki
• Kx Community
• Google Group
• Kx Github
• Q For Mortals
• Q Tips
• STAC benchmarks
• Free version
• Kx Meetups
• FD Lecture Series