(c) 2017 DomainTools LLC 1 DomainTools App for QRadar App Startup Guide for Version 1.0.480 Updated November 1, 2017 Table of Contents DomainTools App for QRadar ......................................................................................................... 1 App Features ............................................................................................................................... 2 Prerequisites ............................................................................................................................... 3 Data Source Identification ....................................................................................................... 3 Data Source FQDN Field .......................................................................................................... 3 App Configuration ....................................................................................................................... 4 QRadar User Account .............................................................................................................. 4 App Settings ............................................................................................................................ 4 Log Sources ............................................................................................................................. 5 App Log.................................................................................................................................... 5 Reference Data............................................................................................................................ 6 Managing Reference Data ....................................................................................................... 6 DomainTools Reference Data Collections ............................................................................... 7 Sample AQL ................................................................................................................................. 9 DomainTools App Area.............................................................................................................. 10
Transcript
(c) 2017 DomainTools LLC 1
DomainTools App for QRadar App Startup Guide for Version 1.0.480 Updated November 1, 2017 Table of Contents DomainTools App for QRadar ......................................................................................................... 1
App Features ............................................................................................................................... 2 Prerequisites ............................................................................................................................... 3
Data Source Identification ....................................................................................................... 3 Data Source FQDN Field .......................................................................................................... 3
Reference Data ............................................................................................................................ 6 Managing Reference Data ....................................................................................................... 6 DomainTools Reference Data Collections ............................................................................... 7
Sample AQL ................................................................................................................................. 9 DomainTools App Area .............................................................................................................. 10
(c) 2017 DomainTools LLC 2
App Features
The DomainTools App for QRadar populates reference data with DomainTools domain profile and risk scores for domain names observed in QRadar events. It also provides a DomainTools app area to research a single domain name to uncover domain ownership profiles, risk scores, and more. Key capabilities enabled by the app include:
• Create offenses using DomainTools proprietary proximity-‐based domain risk scores • Investigate domain names in-‐context, without leaving QRadar • Target threat hunting at key aspects of a domain name’s registration profile
(c) 2017 DomainTools LLC 3
Prerequisites
Data Source Identification
Before installing the app, first identify which data source(s) in your QRadar instance contain domain names. DomainTools data works best with web proxy log data, because the domain names are easy to extract, and the web traffic captures most of the interactions between end-‐user workstations on your network and potentially malicious domain names. Other less common but still effective log sources include DNS logs or logs from next-‐generation, layer 7 firewalls that also contain domain name data. Once you locate the list of data sources, take note of the log source names in QRadar. You will use it later when setting up the DomainTools app.
Data Source FQDN Field
For the DomainTools app to function optimally, your log source should provide a field that contains only a fully-‐qualified domain name, and if possible, it should be labeled “FQDN”. This documentation will assume the field name is “FQDN” unless otherwise noted. Here’s why this is important. DomainTools provides Whois and risk scoring data on second-‐level domain names. Examples of a second-‐level domain names include “domaintools.com”, “google.com”, and “bbc.co.uk”. Most traffic on a network does not reference these second-‐level domains directly – instead, logs will contain fully-‐qualified domain names (also known as FQDNs or hostnames) or even complete URLs. Examples of FQDNs include “research.domaintools.com”, “www.google.com” or “www.bbc.co.uk”. Those FQDNs must first be collapsed to only their domain name before a query is made to the DomainTools API to avoid making unnecessary requests. In most networks, this results in a 10x reduction in the volume of API queries, and it also improves performance by enabling effective caching. The task of extracting a second-‐level domain name from an FQDN or a complete URL is non-‐trivial, and cannot be performed effectively with regular expression matching. The optimal solution requires a list of domain extensions, and there are code libraries dedicated to solving the problem efficiently. QRadar does not provide a built-‐in mechanism to make that conversion, so the DomainTools app handles that for you. You may find it necessary to add a custom field to your data source to extract the FQDN from a URL or other unparsed field. Adding a custom field to a log source in QRadar is out of the scope of this documentation.
(c) 2017 DomainTools LLC 4
App Configuration
QRadar User Account
The DomainTools app runs a process that queries your QRadar event logs for new events, finds domain names, and then populates reference sets with Whois and Risk Score data from DomainTools APIs. For this to work, the app needs a QRadar user account to sign in with and read those events. Create that account in QRadar, and then note the username and password so you can set that in the app settings page.
App Settings
Access the DomainTools App configuration page by first visiting the Admin settings page in QRadar, then scroll down to the DomainTools Configuration option. Click the DomainTools icon to open the settings page and enter the correct values for your environment. DomainTools application user name User name of a QRadar user the app will use to
read events and store reference data. Password Password for the QRadar user account. DomainTools host name Must be set to api.domaintools.com API user name DomainTools API username (contact your eval
point of contact if you do not have an API username and API key)
API user token DomainTools API key. Use HTTPS protocol to invoke DomainTools APIs
Whether to use SSL when accessing the DomainTools APIs. We strongly recommend setting this to “false” to get the most throughput and fastest response times from the server. API keys are still protected with HMAC signatures even when SSL is disabled.
Verify SSL certificate is used to invoke DomainTools APIs
Some environments with SSL filtering require accepting an organization’s CA, but that CA may not be loaded into the QRadar instance. Again, disable HTTPS queries whenever possible to avoid problems and improve throughput.
Max number of records to fetch from log source at a time.
Start with a value of 200 and adjust as needed.
Max threshold value of reputation score Domain names with a score higher than this threshold will be added to a special reference set. The score ranges from 0 to 100 with higher numbers indicating a riskier domain.
(c) 2017 DomainTools LLC 5
DomainTools recommends starting with a minimum value of 70.
Time interval to invoke the scheduler in minutes.
Set how frequently the job will run that extracts log data. Start with 10 minutes and adjust as needed.
After how many cycles the settings to be refreshed
App settings are cached between successive runs of the enrichment job and are periodically refereshed. Start with a value of 1 while you are adjusting the settings, then increase to at least 4 for best performance.
No. of records to be displayed in a page Adjust pagination for pivot data returned on the domain profile page. Start with 50 and adjust accordingly.
Log Sources
Access the DomainTools app configuration page, then click on “Delete Log Source”. The app installs with an example log source that you should remove once you familiarize yourself with the expected values for the log source name and domain column name. Next, click on “Add Log Source” to add one or more log sources that contain domain names (see Prerequisites above). Ensure the values in the fields match the data source name and column name, then click the “Submit” button. Repeat for as many data sources as you need.
App Log
Once the app is configured, the DomainTools App will run a job at the interval specified in the settings, query the logs, and fetch DomainTools data to populate in reference sets. A QRadar administrator can access application logs on the QRadar server to monitor this process and provide debugging information to DomainTools if problems arise. The logs are stored in one of these folders:
The container_id portion of the path is not a predicable value, so it will require visiting each directory to find the one with the DomainTools log files. The correct folder will have a “dtstore.db” file and a “log” directory – navigate to the “log” directory to find the app.log file. If you have command line access to the server, this command can help you locate the folder more quickly than trial-‐and-‐error: find /store -maxdepth 4 -name "dtstore.db"
(c) 2017 DomainTools LLC 6
Reference Data
Managing Reference Data
QRadar supports several reference data collection types, but it only provides a UI to manage the contents of reference sets. There is no option in the QRadar admin interface to view reference maps or reference tables, both of which are used extensively by the DomainTools app. The only way to confirm these reference data were created properly, and to view their contents, is to use the API. Fortunately, QRadar provides interactive API documentation under the “Help” menu. To view a list of reference maps:
• Go to "Help" > "Interactive API for Developers" • Navigate to the 7.0 tree, down to /reference_data • Click on /maps • Scroll down through the page that appears on right and click "Try it now" • The Response Body will list details on each active reference map
To view the contents of a reference map:
• Go to "Help" > "Interactive API for Developers" • Navigate to the 7.0 tree, down to /reference_data • Expand the /maps node and click /{name} • Scroll down through the page that appears on right and locate the parameters section • Enter the name of the reference map in the name field and click "Try it now" • The Response Body will list details on each active reference map
(c) 2017 DomainTools LLC 7
DomainTools Reference Data Collections
Name Type Usage dt_fqdn_to_domain Reference
Set Contains key / value pairs mapping fully-‐qualified domain names (FQDNs) to their second-‐level domain name. Provide a FQDN as the key to obtain a domain name. This reference set is also used to manage caching in the DomainTools app. Log entries that already have an entry in this reference set for the value in their FQDN field will be excluded from the enrichment job. Use this field in a custom AQL query to create a domain name column that can be used to lookup risk score and Whois data. For example: SELECT REFERENCESET('dt_fqdn_to_domain',FQDN) AS domain_name
dt_domains_risk_score Reference
Set Contains key / value pairs mapping second-‐level domain names to a DomainTools risk score. Provide a domain name as the key. Use this field in a rule with custom AQL to create offenses when domain names exceed a threshold. For example: REFERENCESET('dt_domains_risk_score', REFERENCESET('dt_fqdn_to_domain',FQDN)) >= 70
dt_whois_details Reference Table
Contains a set of columns with parsed Whois data, indexed by the second-‐level domain name. Columns names include:
• Registrant Country • Registrant Name • Registrant Org • Registrant Phone • Registrar Name • Created Date • Expired Date • Updated Date
Use this data to enrich log searches or to create custom AQL rules based on attributes in the Whois record of a domain name. For example, this rule could alert on domains registered at a specific registrar:
This AQL may be used to enrich a log source that contains an FQDN in the ‘FQDN’ column. Adjust the ‘LOG_SOURCE_NAME’ value to match the name of your log source. SELECT starttime, LOGSOURCENAME(logsourceid), FQDN, REFERENCEMAP('dt_fqdn_to_domain',FQDN) AS domain, REFERENCEMAP('dt_domains_risk_score',domain) AS dt_risk_score, REFERENCETABLE('dt_whois_details','Registrant Country',domain) AS dt_reg_country, REFERENCETABLE('dt_whois_details','Registrant Name',domain) AS dt_reg_name, REFERENCETABLE('dt_whois_details','Registrant Org',domain) AS dt_reg_org, REFERENCETABLE('dt_whois_details','Registrant Email',domain) AS dt_reg_email, REFERENCETABLE('dt_whois_details','Registrar Name',domain) AS dt_registrar, REFERENCETABLE('dt_whois_details','Created Date',domain) AS dt_create_date FROM events WHERE LOGSOURCENAME(logsourceid)= 'LOG_SOURCE_NAME' AND domain IS NOT NULL
(c) 2017 DomainTools LLC 10
DomainTools App Area
When the app is installed, a new tab will appear on the QRadar navigation menu labeled “DomainTools”. Access that tab to view a dashboard focused on key threat hunting and risk metrics. You can adjust threshold and parameters for the dashboard panels by clicking the pencil icon next to the panel titles.
(c) 2017 DomainTools LLC 11
To investigate a specific domain name, click the "Search" tab near the top of the dashboard and enter a domain name in the search box. The app loads risk score and Whois information on a single domain name from the DomainTools API. You may also click these elements to view additional related domains using DomainTools Reverse Whois, Reverse IP and Reverse Name Server datasets:
• Registrant, abuse and admin email addresses • Registrant name on the Domain Profile tab • IP address • Name servers
