Processing Intelligence Processing Intelligence Feeds with Open Source Feeds with Open Source SoftwareSoftware

Chris Horsley, SC Leung, Tomas Lima, L. Aaron Chris Horsley, SC Leung, Tomas Lima, L. Aaron Kaplan, Raphael VinotKaplan, Raphael Vinot

OverviewOverview• Current topics in automatic incident handling for CERTsCurrent topics in automatic incident handling for CERTs• IFASIFAS• HKCERT , IFAS and use-casesHKCERT , IFAS and use-cases• IHAP projectIHAP project• ContactDB projectContactDB project• Current R&DCurrent R&D

IFASIFAS• Information Feed Analysis SystemInformation Feed Analysis System

Knowing whatKnowing what’’s going ons going on

How do national CSIRTs know whatHow do national CSIRTs know what’’s s happening?happening?

National CSIRTs need visibility on network in their economyNational CSIRTs need visibility on network in their economy

However, many national CSIRTs donHowever, many national CSIRTs don’’t operate networks t operate networks themselves, and normally donthemselves, and normally don’’t have global (or any) direct t have global (or any) direct visibilityvisibility

How does the CSIRT know whatHow does the CSIRT know what’’s going on in their country?s going on in their country?

The kindness of strangersThe kindness of strangersLuckily, there are a lot of network operators, research teams, Luckily, there are a lot of network operators, research teams, vendors, and other CSIRTs out there that collect information, vendors, and other CSIRTs out there that collect information, and will share it with national CSIRTs.and will share it with national CSIRTs.

And here comes the And here comes the ““butbut””......

So much data, so many So much data, so many formatsformats

There are many feeds, all with their own data formats and There are many feeds, all with their own data formats and mediums:mediums:

Formats: CSV, JSON, XML, STIX, IODEFFormats: CSV, JSON, XML, STIX, IODEF

Mediums: HTML, RSS, email, HTTP APIsMediums: HTML, RSS, email, HTTP APIs

While there are efforts to standardise data formats, this will While there are efforts to standardise data formats, this will take a long time, and will likely never cover 100% of feedstake a long time, and will likely never cover 100% of feeds

We canWe can’’t change the format of remote feeds - we can only t change the format of remote feeds - we can only change what we do with the data.change what we do with the data.

The need for standardsThe need for standardsDifferent feeds use many terms to mean the same thing:Different feeds use many terms to mean the same thing:

ip, source_ip, src_ip, endpoint, attacker_ip, cnc_ip...ip, source_ip, src_ip, endpoint, attacker_ip, cnc_ip...

If we receive events from many feeds, we need to normalise If we receive events from many feeds, we need to normalise so we can compare them together.so we can compare them together.

The need for storageThe need for storageAs a national CSIRT, weAs a national CSIRT, we’’re concerned with the health of re concerned with the health of national networks: which means measurement.national networks: which means measurement.

We can only measure longterm if we store events, enabling We can only measure longterm if we store events, enabling us to analyse them.us to analyse them.

We also want to search through events, like:We also want to search through events, like:

C&C servers in domestic networks in last weekC&C servers in domestic networks in last week

Bots infected with Trojan.abc on BigISPBots infected with Trojan.abc on BigISP

Defaced web sites targeting gov.zzDefaced web sites targeting gov.zz

Need for automationNeed for automationThereThere’’s way too much network event data out there to s way too much network event data out there to manually processmanually process

Options:Options:

a) use lots of analyst time doing tedious log processinga) use lots of analyst time doing tedious log processing

b) write lots of small, independent scriptsb) write lots of small, independent scripts

c) ignore inbound logs completelyc) ignore inbound logs completely

d) use an automated processing systemd) use an automated processing system

So what do we need?So what do we need?We need something which automatically:We need something which automatically:

Gathers many different types of feedsGathers many different types of feeds

Normalises the data in those feedsNormalises the data in those feeds

Stores that data somewhereStores that data somewhere

Allows search and performs statistical analysisAllows search and performs statistical analysis

IFASIFASIFAS = Information Feed Analysis SystemIFAS = Information Feed Analysis System

Project sponsored by HKCERT and developed by HKCERT and Project sponsored by HKCERT and developed by HKCERT and CSIRT FoundryCSIRT Foundry

An integration of open source tools, released as open source An integration of open source tools, released as open source for CSIRTsfor CSIRTs

ArchitectureArchitecture

ArchitectureArchitectureAbusehelper: gather, process, and enrich feeds, generate Abusehelper: gather, process, and enrich feeds, generate eventsevents

Logstash: process and normalise feedsLogstash: process and normalise feeds

Elasticsearch: store events in schema-free index serverElasticsearch: store events in schema-free index server

Kibana: search through eventsKibana: search through events

IFAS Reporter: get overall statistics, build realtime IFAS Reporter: get overall statistics, build realtime dashboardsdashboards

Kibana event searchesKibana event searches

Freeform statistical Freeform statistical reportingreporting

Nesting, filtering, Nesting, filtering, deduplicationdeduplication

IFAS – DashboardIFAS – Dashboard Visualize informationVisualize information

*Drill down right at the chart

What you need to startWhat you need to start

SoftwareSoftwareOpen source under Apache 2.0 LicenseOpen source under Apache 2.0 License

Only possible with the hard work released under open source Only possible with the hard work released under open source licenses from Abusehelper and Elasticsearch teamslicenses from Abusehelper and Elasticsearch teams

Contributions, bug reports, feature requests most welcome!Contributions, bug reports, feature requests most welcome!

HardwareHardwareProduction: 8-16GB memory machineProduction: 8-16GB memory machine

Dev: 4GB possibleDev: 4GB possible

Multi-core machine (4+ ideal)Multi-core machine (4+ ideal)

Runs in a VM no problemRuns in a VM no problem

Out of the box feedsOut of the box feedsOther developed Plugins Malc0de Malicious Domain List Arbor SRF Shadowserver Zone-H

Future … more, and your own

Out of Box Feed Plugins(4 publicly available)Abuse.chCleanMXMillersmilesPhishtank

Where to get itWhere to get itCurrently under closed pilot to trusted CSIRTsCurrently under closed pilot to trusted CSIRTs

Eventually public releaseEventually public release

Please contact Please contact contact@ifas.io for details for details

DemosDemos

IFAS and Use CasesIFAS and Use Cases

SC Leung, HKCERTSC Leung, HKCERT

Give a sense of Today’s Give a sense of Today’s EventsEvents

IFAS - Log SearchIFAS - Log Search Powerful search on all the information collectedPowerful search on all the information collected

Keywords here

Add columns of interests

Feed Details

IFAS - ReporterIFAS - Reporter Statistical analysis-Trends & DistributionsStatistical analysis-Trends & Distributions

Free form statistical reportsFree form statistical reports

1.1.

5.5. 2.2. 4.4.

6.6.

3.3.

Nesting, filtering, Nesting, filtering, deduplicationdeduplicationNumber of phishings in “.AU” in each ASN by brand

IFAS - AlertIFAS - Alert Set tracking criteria – get notify ASAPSet tracking criteria – get notify ASAP

domain: domain: *.gov.hk*.gov.hk Alert lists : educational institutions (Alert lists : educational institutions (hkeduhkedu), NGOs (), NGOs (hkorghkorg))

!!

DashboardDashboard Real-time situational awareness Real-time situational awareness for CERT management for CERT management

Public Situational Public Situational AwarenessAwareness

on on Compromised Servers / Compromised Servers /

PCsPCs

Hong Kong Security Watch ReportHong Kong Security Watch Report

• Correlate Cryptolocker 2013-Oct with ZeusCorrelate Cryptolocker 2013-Oct with Zeus

Analysis of Trend with Analysis of Trend with EventsEvents

Engage ISPs for large scale incident Engage ISPs for large scale incident handlinghandling• Data do help Data do help

HKCERT HKCERT engaging ISPs engaging ISPs (their sales (their sales team)team)

• Data do help a Data do help a server hosting server hosting SP understand SP understand their customers’ their customers’ security security problemsproblems

ISP

Converting security events into Converting security events into incident reportsincident reports• DefacementDefacement• PhishingPhishing Export to CSV for batch processing, with Export to CSV for batch processing, with

some other scriptssome other scripts

• Malware hosting – a bit difficultMalware hosting – a bit difficult• Large volume of incidents – need prioritisationLarge volume of incidents – need prioritisation

Future of IFAS - a Future of IFAS - a collaboration platformcollaboration platform• All you can useAll you can use

• All you can contributeAll you can contribute• Add input filters for new feedsAdd input filters for new feeds• Add new plug-in modulesAdd new plug-in modules• Add new chart and visualizationAdd new chart and visualization• Integrate with other systems, e.g. RTIRIntegrate with other systems, e.g. RTIR• ……

• Standard languageStandard language: STIX, taxonomy of ENISA: STIX, taxonomy of ENISA

• An ongoing project that turn security events into Actionable An ongoing project that turn security events into Actionable DataData• Set Priority, Choose Monitors, Consolidate ResultsSet Priority, Choose Monitors, Consolidate Results

DSMS DSMS (Decision Support & Monitoring (Decision Support & Monitoring System)System)

Decision Decision SupportSupport

Sub-systemSub-system

IFASIFAS

Interfaces to Interfaces to MonitorsMonitors

PrivatePrivate analysis sysanalysis sys

Public Public analysis sys analysis sys (VirusTotal, (VirusTotal,

ThreatExpert)ThreatExpert)

Web Web reputation reputation (D-Shield)(D-Shield)

Interface Interface ModulesModules

Interface Interface ModuleModule

Interface Interface Modules Modules

Request Request to to

monitormonitor

OutputOutputStoryStory

ProfileProfile

Input Input URLURL

TasksTasks

IncideIncidentnt

MgmtMgmt

Status CheckStatus Check(HTTP, DNS) (HTTP, DNS)

via proxyvia proxyStatus Status ??Interface Interface

ModuleModule (online /offline)(online /offline)

MonitoringMonitoringServicesServices

ConsolidateConsolidated Resultsd Results

IHAPIHAPIncident handling automation projectIncident handling automation project

IHAPIHAP• Very similar to IFAS, developed in parallel by CERT.pt, Very similar to IFAS, developed in parallel by CERT.pt,

CERT.atCERT.at• Also uses Logstash, Elastic Search and AbusehelperAlso uses Logstash, Elastic Search and Abusehelper• Less work on the Webinterface, more work on Ontology, Less work on the Webinterface, more work on Ontology,

„Data harmonisation document“„Data harmonisation document“

IHAP - HistoryIHAP - History• Discussions about CERT.AT developments/documentsDiscussions about CERT.AT developments/documents• Discussions about cooperation between CERTsDiscussions about cooperation between CERTs• ENISA supportENISA support

IHAP - GoalsIHAP - Goals• Open SourceOpen Source • MaintainableMaintainable • Flexible and Modular Flexible and Modular - must be possible to integrate existing - must be possible to integrate existing

software and modules (Pastemon, AbuseHelper, etc..)software and modules (Pastemon, AbuseHelper, etc..)• ReusableReusable • Easily ExtendableEasily Extendable - should require little knowledge and basic - should require little knowledge and basic

programming skillsprogramming skills• Easily DeployableEasily Deployable • Easily Updatable Easily Updatable – easy to share new developments with other – easy to share new developments with other

CERTs and update the system with that new codeCERTs and update the system with that new code• Easily Configurable Easily Configurable - config files that can be easily modified to fit - config files that can be easily modified to fit

CERT‘s needsCERT‘s needs• DocumentedDocumented - must be well documented - must be well documented

Links & CodeLinks & Code

http://www.enisa.europa.eu/activities/cert/support/incident-http://www.enisa.europa.eu/activities/cert/support/incident-handling-automationhandling-automation

Common field names for Common field names for AHAH• https://bitbucket.org/clarifiednetworks/abusehelper/wiki/Data

%20Harmonization%20Ontology• A standard set of well defined field names within A standard set of well defined field names within

Abusehelper (AH)Abusehelper (AH)• Allows CERTs to:Allows CERTs to:

• Write bots which are interoperable within AHWrite bots which are interoperable within AH• Measure in identical waysMeasure in identical ways• Easier to parse different feeds („generic santizer bot“) : Easier to parse different feeds („generic santizer bot“) :

you just have to define the mappingsyou just have to define the mappings

contactDBcontactDB

Background/ problemBackground/ problem• abuse@ lookups suck (IRT object not in use, no standard; abuse@ lookups suck (IRT object not in use, no standard;

Just now RIPE DB is changing with abuse-c:)Just now RIPE DB is changing with abuse-c:)• Getting the right lookup is non-trivial, complex Getting the right lookup is non-trivial, complex • Many (national) CERTs create their own abuse contact Many (national) CERTs create their own abuse contact

lookup DBs.lookup DBs.• National CERT DB, TI directory, FIRST data can not be looked National CERT DB, TI directory, FIRST data can not be looked

up automatically via scripts.up automatically via scripts.

IdeaIdea• A caching contact database with more specific internal dataA caching contact database with more specific internal data• Some of this data (tel nos, etc) will never be in the public Some of this data (tel nos, etc) will never be in the public

whoiswhois• Unify with TI, FIRST etc dataUnify with TI, FIRST etc data• Make it query-able by scriptsMake it query-able by scripts

Abuse contact lookup - flowAbuse contact lookup - flowWhat databases exist? What can we query?What databases exist? What can we query?

Number based Number based resource:resource:

IP addr, netblock, IP addr, netblock, ASNASN

Name based Name based resource:resource:

domain name, domain name, hostnamehostname

MaxmindMaxmindRIPE DBRIPE DBCymru, ..Cymru, ..

..

Get Get country()country()

National National CERT DBCERT DBCERT.orgCERT.org

Email AddressEmail Address

Whois Whois DB DB

(RIPE, (RIPE, ARIN, ..)ARIN, ..)

IRT object, abuse-c, IRT object, abuse-c, ......

Country codeCountry code

TI, FIRST, TI, FIRST, CERT.org CERT.org

DBsDBs

Whois DB Whois DB (registrant(registrant, registrar), registrar)

IANA IANA ccTLD listccTLD list

Extract ccTLDExtract ccTLD

National CERT for countryNational CERT for country

Country codeCountry code

Gethostbyname()Gethostbyname()

What exists now?What exists now?• Public code repo ;-)Public code repo ;-)• Whois server (thx Mauro)Whois server (thx Mauro)• RESTful API (Mauro, Rafiot)RESTful API (Mauro, Rafiot)• Some scripts to import TI data (Aaron, David)Some scripts to import TI data (Aaron, David)• Still some bugs ;-)Still some bugs ;-)

Code & document with Code & document with RIPERIPE• Document (WIP):Document (WIP):• https://github.com/certtools/contactdb/blob/master/doc/cont

act-databases-for-abuse-handling.mkd

• Codebase:Codebase:https://github.com/certtools/contactdb

• (thx Rafiot, David, Mauro!)(thx Rafiot, David, Mauro!)

SummarySummary

SummarySummary• The CERT community has limited ressources for The CERT community has limited ressources for

developmentdevelopment• We re-implement the same thing all the timeWe re-implement the same thing all the time• Let‘s share code or at least exchange ideas on how to Let‘s share code or at least exchange ideas on how to

automate incident handling!automate incident handling!• Let‘s share on how to measure successLet‘s share on how to measure success• Thanks HKCERT, ENISA, CERT.at, CERT.pt, CIRCL, etc..Thanks HKCERT, ENISA, CERT.at, CERT.pt, CIRCL, etc..

• Mailinglist: Mailinglist: https://tiss.trusted-introducer.org/mailman/listinfo/ihap

Thanks!Thanks!