A Very Brief Introduction to Hadoop - Meetupfiles.meetup.com/18870621/OpenSOC_Platform_v3.pdf ·...

A Very Brief Introduction to Hadoop

© 2015 Cisco and/or its affiliates. All rights reserved. Presentation_ID:TECSEC-3900 Cisco Public

Hadoop – In the wild

2


Hadoop – What really matters

3

•  The tool that solves your problem.


Tools We Use in OpenSOC

4



5

•  Kafka – a distributed input queue


OpenSOC – The Architecture

6

AccessMessaging SystemData CollectionSource Systems StorageReal Time Processing

StormKafka

B Topic

N Topic

Elastic Search

Index

Web Services

Search

PCAP Reconstruction

HBase

PCAP Table

Analytic Tools

R / Python

Power Pivot

Tableau

Hive

Raw Data

ORC

Passive Tap

PCAP Topic

DPI Topic

A Topic

Telemetry Sources

Syslog

HTTP

File System

Other

Flume

Agent A

Agent B

Agent N

B Topology

N Topology

A Topology

PCAP

Traffic Replicator

PCAP Topology

DPI Topology



7

•  Kafka – a distributed input queue •  Flume – Listens for “syslog” style data and directs it into Kafka (or HDFS)



8

•  Kafka – a distributed input queue •  Flume – Listens for “syslog” style data and directs it into Kafka (or HDFS)

Syslog

Snort

Other

Flume Source

Flume Source

Flume Source

Flume Sink

Flume Sink

Flume Sink

Kafka



9


StormKafka

B Topic

N Topic

Elastic Search

Index

Web Services

Search

PCAP Reconstruction

HBase

PCAP Table

Analytic Tools

R / Python

Power Pivot

Tableau

Hive

Raw Data

ORC

Passive Tap

PCAP Topic

DPI Topic

A Topic

Telemetry Sources

Syslog

HTTP

File System

Other

Flume

Agent A

Agent B

Agent N

B Topology

N Topology

A Topology

PCAP

Traffic Replicator

PCAP Topology

DPI Topology



10

•  Kafka – a distributed input queue •  Flume – Listens for “syslog” style data and directs it into Kafka •  Storm – A stream processor



11


StormKafka

B Topic

N Topic

Elastic Search

Index

Web Services

Search

PCAP Reconstruction

HBase

PCAP Table

Analytic Tools

R / Python

Power Pivot

Tableau

Hive

Raw Data

ORC

Passive Tap

PCAP Topic

DPI Topic

A Topic

Telemetry Sources

Syslog

HTTP

File System

Other

Flume

Agent A

Agent B

Agent N

B Topology

N Topology

A Topology

PCAP

Traffic Replicator

PCAP Topology

DPI Topology



12

•  Kafka – a distributed input queue •  Flume – Listens for “syslog” style data and directs it into Kafka •  Storm – A stream processor •  Hbase – An In Memory Database (NoSQL)



13


StormKafka

B Topic

N Topic

Elastic Search

Index

Web Services

Search

PCAP Reconstruction

HBase

PCAP Table

Analytic Tools

R / Python

Power Pivot

Tableau

Hive

Raw Data

ORC

Passive Tap

PCAP Topic

DPI Topic

A Topic

Telemetry Sources

Syslog

HTTP

File System

Other

Flume

Agent A

Agent B

Agent N

B Topology

N Topology

A Topology

PCAP

Traffic Replicator

PCAP Topology

DPI Topology



14

•  Kafka – a distributed input queue •  Flume – Listens for “syslog” style data and directs it into Kafka •  Storm – A stream processor •  Hbase – An In Memory Database (NoSQL) •  Hive – SQL access to HDFS Data (Java, JDBC, ODBC)



15


StormKafka

B Topic

N Topic

Elastic Search

Index

Web Services

Search

PCAP Reconstruction

HBase

PCAP Table

Analytic Tools

R / Python

Power Pivot

Tableau

Hive

Raw Data

ORC

Passive Tap

PCAP Topic

DPI Topic

A Topic

Telemetry Sources

Syslog

HTTP

File System

Other

Flume

Agent A

Agent B

Agent N

B Topology

N Topology

A Topology

PCAP

Traffic Replicator

PCAP Topology

DPI Topology



16

•  Kafka – a distributed input queue •  Flume – Listens for “syslog” style data and directs it into Kafka •  Storm – A stream processor •  Hbase – An In Memory Database (NoSQL) •  Hive – SQL access to HDFS Data •  Elastic Search – For Indexing PCAP

PCAP_ID

Source IP

Source Port

Dest IP

Dest Port

TS_Micro



17


StormKafka

B Topic

N Topic

Elastic Search

Index

Web Services

Search

PCAP Reconstruction

HBase

PCAP Table

Analytic Tools

R / Python

Power Pivot

Tableau

Hive

Raw Data

ORC

Passive Tap

PCAP Topic

DPI Topic

A Topic

Telemetry Sources

Syslog

HTTP

File System

Other

Flume

Agent A

Agent B

Agent N

B Topology

N Topology

A Topology

PCAP

Traffic Replicator

PCAP Topology

DPI Topology



18


StormKafka

B Topic

N Topic

Elastic Search

Index

Web Services

Search

PCAP Reconstruction

HBase

PCAP Table

Analytic Tools

R / Python

Power Pivot

Tableau

Hive

Raw Data

ORC

Passive Tap

PCAP Topic

DPI Topic

A Topic

Telemetry Sources

Syslog

HTTP

File System

Other

Flume

Agent A

Agent B

Agent N

B Topology

N Topology

A Topology

PCAP

Traffic Replicator

PCAP Topology

DPI Topology



19


StormKafka

B Topic

N Topic

Elastic Search

Index

Web Services

Search

PCAP Reconstruction

HBase

PCAP Table

Analytic Tools

R / Python

Power Pivot

Tableau

Hive

Raw Data

ORC

Passive Tap

PCAP Topic

DPI Topic

A Topic

Telemetry Sources

Syslog

HTTP

File System

Other

Flume

Agent A

Agent B

Agent N

B Topology

N Topology

A Topology

PCAP

Traffic Replicator

PCAP Topology

DPI Topology

Building Your Own


Kittens!

21

-  Pets are given names like Fluffy -  They are unique and lovingly raised and

cared for -  If they get sick, you take them to the vet and

nurse them back to health -  You hope they’ll live forever


Cattle

22

-  Cattle are given numbers like dn01. -  They are almost identical to other cattle. -  When they get sick, you take “normal”

measures to cure. Cost/benefit model. -  Cattle have minimum life expectancy. -  To serve their purpose, they must be

“herded.”


Clusters Nodes are Cattle!

23


What are the implications?

24



25

•  Most of the “Enterprise Class” Server rules no longer apply!



26

•  Most of the “Enterprise Class” Server rules no longer apply! •  Automation is King



27

•  Most of the “Enterprise Class” Server rules no longer apply! •  Automation is King •  Cheaper is usually better



28

•  Most of the “Enterprise Class” Server rules no longer apply! •  Automation is King •  Cheaper is usually better •  No 3-5 Year refresh cycle

To-Do in 3 Years 1.  Buy new servers to replace the old

ones 2.  Re-install software 3.  Transfer Data


So what do I need to run OpenSOC?

29



30

•  A Cluster running Core Hadoop + Hive, Hbase, Kafka, Storm and Elastic Search

Elastic Search

Elastic Search

Elastic Search

Elastic Search

OK, but not required



31


•  3 Physical Servers for NN, Zookeeper, YARN and “Master” Nodes



32


•  3 Physical Servers for NN, Zookeeper, YARN and “Master” Nodes •  Data Nodes: As many as you can

•  Depending on Data Retention Requirements & Ingestion Rates


OpenSOC at Cisco (aka MTD)

33

Hardware footprint (40u): -  14 Hadoop Data Nodes (UCS C240 M3) -  3 Cluster Control Nodes (UCS C220 M3) -  2 ESX Hypervisor Hosts (UCS C220 M3) -  1 PCAP Processor (UCS C220 M3 + Napatech NIC) -  2 SourceFire Threat alert processors -  1 Anue Network Traffic splitter -  1 Router -  1 48 Port 10GE Switch Software Stack -  HDP 2.2 -  Kafka 0.8.1 -  Elastic Search 1.3.0 -  MySQL 5.5 (Hive Meta & GeoData)


OpenSOC at Cisco

34

Ctrl01 Zookeeper

NN1

ES Master

Ctrl02 Zookeeper

NN2

ES Master

Nimbus Server/UI

Hive Meta Hbase Master

Ctrl03 Zookeeper

YARN / History Server

ES Master

Hbase Master StdBy

Flume Agents

DataNode (10-14) YARN / HDFS

Hbase

Storm Client

Kafka*

*Dedicated disks Elastic Search Nodes (8) 3x Elastic Search


Installation Suggestions

35

•  Kickstart / Autoyast / etc. – Automated builds as much as possible



36

•  Kickstart / Autoyast / etc. – Automated builds as much as possible •  Hadoop Distro or Roll Your Own

Distribution

•  Fast Setup •  Vendor Support

•  Can cost more than nodes!

•  Potential Lock-in

Roll your own

•  Longer Startup •  Experience

required •  Lower Cost •  More control



37

•  Kickstart / Autoyast / etc. – Automated builds as much as possible •  Hadoop Distro or Roll Your Own •  Plus Configuration Management

•  We’re working on Ansible scripts that we expect to release

Date post:	23-Jun-2020
Category:	Documents
Upload:	others
View:	1 times
Download:	0 times

A Very Brief Introduction to Hadoop - Meetupfiles.meetup.com/18870621/OpenSOC_Platform_v3.pdf ·...

Documents