PowerPoint



4


5
fast instant
freq blackl
freq
json
Xpage

Source Avro SourceThrift SourceExec SourceJMS SourceSpooling Directory SourceKafka SourceNetCat SourceSequence Generator Source Syslog SourceHTTP SourceStress SourceLegacy SourceCustom SourceScribe Source
Channel Source Memory ChannelJDBC ChannelKafka ChannelFile ChannelSpillable Memory ChannelPseudo Transaction ChannelCustom Channel
Sink Sink HDFS SinkHive SinkLogger SinkAvro Sink Thrift SinkIRC SinkFile Roll SinkNull SinkHBase SinkAsync HBase SinkMorphline Solr SinkElastic Search SinkKite Dataset SinkKafka SinkCustom Sink
7
Flume3

Channels.type=flume.channel.KafkaChannel Channels. brokerList=ip:port Channels. topic = zhihu-freq Channels. zookeeperConnect=ip:port/kafka
9
zhihu-kafka-hdfs-agent.channels.zhihu-kafka-channel-freq.type = org.apache.flume.channel.kafka.KafkaChannel
zhihu-kafka-hdfs-agent.sinks.zhihu-hdfs-sink-freq.hdfs.writeFormat = Text//sequenceText, Writable
//bytes0 zhihu-kafka-hdfs-agent.sinks.zhihu-hdfs-sink-freq.hdfs.callTimeout = 120000//HDFS zhihu-kafka-hdfs-agent.sinks.zhihu-hdfs-sink-freq.hdfs.batchSize = 10000//HDFSevents zhihu-kafka-hdfs-agent.sinks.zhihu-hdfs-sink-freq.channel = zhihu-kafka-channel-freq//sinkchannel
Channels.type=flume.channel.KafkaChannel Channels. brokerList=ip:port Channels. topic = zhihu-freq Channels. zookeeperConnect=ip:port/kafka
10

11

Broker1 Broker2
Group1 Group2
Kafka Cluster
Topic12partition 38 broker
16
kafka headertimestampdocidbasename
circlenumber url8pushtype freqfreq3instantfast
timestamp docid basename circlenumber
Broker2
C3
Broker1
T1-P1-R1T2-P1-R0-L
spark-submit \
driver memoryexecutor memorydriver memory DMdriver memoryEMexecutor
memory --class $mainClass \
$libDir/zhihu-kafka-to-hbase-assembly-1.0.jar $configFile
JobYarn- client


21