2
!�…
� O-I C LA? C C pfyW
E LA?C pfyW 0 ?F
B B F 0
CHDF D E 2D H 1 E K
? FDGD H 0
g La e w V W
3
!�Apache Kylin
� S Tt E
� R C D DFA: M w
cxM P DD A CGDF AD
x
� d bo
ApacheKylin� � IY�¨2�
<²=Y��-£}£¯�0�W�—TedDunning,ApacheI,:+lÅ
4
Apache Kylin;B
/2
y
1
h Eis
C D DFA: DGG? F:
P SQ B
QE
5432612A DO
/2 0
C D DFA: DGG? F:
P SQ B
/2
rQ 0l
3
5
Apache Kylin�r
6
!�Kyligence
• Kyligence’svisionistounleashbigdataproductivityforeveryone's
analyticsneeds.
• ThecompanywasfoundedbytheteamwhocreatedApache
Kylin™,atopopensourceOLAPenginebuiltforinteractive
analyticsatpetabyte-scaledataonHadoop.Kyligenceisthe
primarycontributortotheopensourceKylinprojectglobally.
• Kyligenceprovidesaleadingintelligentdataplatformtosimplify
bigdataanalyticsfromon-premisestocloud.
¬©�Hadoop���5�j3'/Ú�
Hadoop��ç
�Îk¿a
e�ß�ê
�FSv
�m�~�àé
�~A�
¤»ädQ
rj��ka
rj$��Û
l!w�p�ê
SQ N SQ K
k
SQL
kh
/
e Q
nL CDH
DC :DDE
Q R e
Kylin³mX�rGQ�z�á®��
e Q
nSQ
SQ NSQ K & H
I& HI
& HI
& HI
•e
• M w
l
e
• /
u m
9
Apache Kylin��~
MapReduce
Kylin
BITools,WebApp…
ANSISQL
V1.5�^·�ƾ
All rights reserved ©Kyligence Inc.http://kyligence.io
11
V1.5�StreamingCubing
MapReduce/Spark
Kylin
BITools,WebApp…
ANSISQL
StandaloneJVM
processtoatime
window
PuttoHBase
cron
V1.5:�rj(� –���áÕ�\s
All rights reserved ©Kyligence Inc.http://kyligence.io
P��¦¤DcpQSegment
All rights reserved ©Kyligence Inc.http://kyligence.io
�7Dc
�7Dc
pÁ=½â
All rights reserved ©Kyligence Inc.http://kyligence.io
• ¡% �u*y�àé • ~X���7�¼
• ¥��(�s��R�z
• ~X�6ä�¯{
• ÞÐaA;ä
• s�¤»p�ê
15
v1.6Streaming³lf�~��K�W¬
MapReduce/Spark
Kylin
BITools,WebApp…
ANSISQL
Adapter
Adapter
Tables
Topics
CubeSegmentioffset)(
All rights reserved ©Kyligence Inc.http://kyligence.io
• Segmentioffset)(,��yª5• [�Hiverj�²(-�á#segment�offset²�Ì�E4®*Z• [�Kafkarj�²4partition�offset�5²��segment�offset
• Segment�á&Ê��á"ª5• ��offset²�� rj��`9�Î,±k
1:00– 1:05 1:04– 1:11 1:08-1:40
Seg[0-100) Seg[100-400) Seg[400-2000)
Whenqueryfortime1:10²KylinwillscanallSegmentscontainthistime
K 2�¼
All rights reserved ©Kyligence Inc.http://kyligence.io
• C©(�°Û���eh|���»t
• �'.|*'�|rj²��>2Ò��e
• 2¬��$�mO�A�o�~�eèª
"JqØ
All rights reserved ©Kyligence Inc.http://kyligence.io
• �7\sfTG¸�®offset
• ogSD[JSON�b
• og�L��á�g
• �ÊPsegmentV�~X/5V
• §ÔRestAPI¢>
KylinStreaming�%|�
All rights reserved ©Kyligence Inc.http://kyligence.io
• Kafka�{0.10d��
• Kafka�b�JSON�[²_æ�� timestampG�
• �?��iÄ�}xP Topic�join²��og
topic�Hive �join±
KylinStreaming�%|�
All rights reserved ©Kyligence Inc.http://kyligence.io
��Kylin(�Twitter�b
All rights reserved ©Kyligence Inc.http://kyligence.io
• Demo«N� AWS 53vm�Hadoop�²Kafka 3broker
• ¢Í�z�Twitter samplerj�²��TwitterSDK1AÁ?�+(�J
�Kafkatopic;
• Topic“TwitterSample”H%<T£n�Topic“TWITTER_TAG_STREAM2”H
%�<T£n�u-®HASH_TAG�b´
Twitter�b
All rights reserved ©Kyligence Inc.http://kyligence.io
./bin/kafka-console-consumer.sh --zookeeper10.0.0.205:2181--bootstrap-server\
10.0.0.207:6667,10.0.0.208:6667,10.0.0.209:6667--topicTwitterSample
TwitterTag�b
All rights reserved ©Kyligence Inc.http://kyligence.io
./bin/kafka-console-consumer.sh --zookeeper10.0.0.205:2181--bootstrap-server\
10.0.0.207:6667,10.0.0.208:6667,10.0.0.209:6667--topicTWITTER_TAG_STREAM2
OJSON�bvNc ¸�
All rights reserved ©Kyligence Inc.http://kyligence.io
Ó(Kafka��b
All rights reserved ©Kyligence Inc.http://kyligence.io
Ó(£nparser�b
All rights reserved ©Kyligence Inc.http://kyligence.io
¸��]Up
All rights reserved ©Kyligence Inc.http://kyligence.io
×��i�¥�Kafka�TableR�c °�JYF1eModel9Cube
1e�L
All rights reserved ©Kyligence Inc.http://kyligence.io
�w§�?@{�yUZ
• æyV� �á/90�H�¡gK���á�b�Î��
E�(ݶ2��yVMINUTE_START��/90
1eCube
All rights reserved ©Kyligence Inc.http://kyligence.io
�w§Cube�yUZ´
hL��inmem��2�k¨cube~X
[�á0yVhierarchy`¶�[Ñ M�»dyVjoint²&Q·D�
~XCube
All rights reserved ©Kyligence Inc.http://kyligence.io
WebGUI�.Ç>�q
RESTful API¢>³
curl-XPUT--userADMIN:KYLIN-H"Content-Type:application/json;charset=utf-8"
-d'{"sourceOffsetStart":0,"sourceOffsetEnd":9223372036854775807,
"buildType":"BUILD"}'
http://localhost:7070/kylin/api/cubes/twitter_tag_cube5/build2
“sourceOffsetStart”:0 � ���~X�¸�¦fT�
“sourceOffsetEnd”:9223372036854775807� ~X*kafka]�xt��b���
Kylin��Kafka APIÁ?C partition�xtoffset�b²�c�6
~XCube- continued
All rights reserved ©Kyligence Inc.http://kyligence.io
• 87MR job²V���partitionÁ?�£n
• 7º�e^M�HDFS���bØÂ�Hiveµ
6
Twitter�b�(�K�
All rights reserved ©Kyligence Inc.http://kyligence.io
• ¸DTwitter�b�#M²ËÉ/�K�
• K�1:i�á�ÏÈ�/�£n¡Ü@8
• K�2:i�á�Xr³�/�¡Ü@8
• K�3:i�á�ÏÈ�/��§ß®Tag
• K�4:[¹VTag²�²)ã�á®èª@8
�j®�ËÉ3Cube
All rights reserved ©Kyligence Inc.http://kyligence.io
• Cube1:ogK�1&2²»d��á�ÏÈ�Xr³´�
W«�count(*)
• Cube2:ogK�3²»d��á�ÏÈ�dÜ�TopN
• Cube3:ogK�4²»d��á�Tag²W«�count(*)
Dashboard
All rights reserved ©Kyligence Inc.http://kyligence.io
StreamingÖUBQµ
All rights reserved ©Kyligence Inc.http://kyligence.io
z}T�
All rights reserved ©Kyligence Inc.http://kyligence.io
• �8MapReduce\n²k/~X¨W
• _Íxº�eh|�SparkStreaming/Flink
• ÃKW�À¦
ThanksSee you on our next meeting