MacGyver Guide: Installing and Integrating EB CEB and Syslog …€¦ · o Vanilla install of ArcMC...

Installing and Integrating EB CEB and Syslog Collector - Install CEB on EB server - Install Syslog Collector on DNS server - Incorporate demo Logger 6.61 (with EB receiver) Last updated 9/24/18

MacGyver Guide: Installing and Integrating EB CEB and Syslog Collector

2

Contents Incorporating Existing ArcSight VMs ............................................................................................................. 4

Boot DNS, ArcMC, EB, and Logger VMs to be Used .................................................................................. 4

Deploying CEB on EB ..................................................................................................................................... 5

CEB Options Selected / Set ....................................................................................................................... 5

Verify CEB Details in ArcMC UI ................................................................................................................. 7

Re-Verify CEB Details in ArcMC UI ............................................................................................................ 9

Verify CEB Deploy on EB Server via Command Line ................................................................................. 9

Shutdown All Servers and Take Snapshots ............................................................................................. 10

Restart Lab Servers ................................................................................................................................. 10

Re-Verify CEB Details in ArcMC UI Post Re-Start .................................................................................... 10

Re-verify CEB Deploy on EB Server via Command Line Post Re-start .................................................... 11

Setting Logger Receiver to Consume from eb-cef Topic ............................................................................. 12

Logger Configuration – Step 1: Authentication ...................................................................................... 12

Logger Configuration – Step 2: Receiver Configuration ......................................................................... 13

Configuring Receiver .......................................................................................................................... 13

Verifying Receiver Active in Logger .................................................................................................... 13

Verifying Receiver Active in ArcMC .................................................................................................... 15


Restart All Servers and Verify All Services Running ................................................................................ 17

Use the DNS VM for a Syslog Collector Sending to CEB .............................................................................. 20

Installing a Collector to Forward Syslog to CEB ...................................................................................... 20

Upload the Latest Collector installer File to ArcMC (Linux) ............................................................... 20

Create a Deployment Template ......................................................................................................... 20

Create a Destination Template ........................................................................................................... 22

Deploying the Syslog Collector ........................................................................................................... 22

Updating the Connectivity to the Syslog Collector ............................................................................. 26

Updating the Connectivity to the Syslog Collector ............................................................................. 27

Sending Sample Syslog Events to the Collector .......................................................................................... 28

Using Kiwi sysloggen utility (Windows utility) ........................................................................................ 28


Appendix A: Troubleshooting and Status Checking Information ............................................................ 31

3

Eb-Diag Script (automated log collection for EB) ................................................................................... 31

Collecting Logs from ArcMC with Snapshot Feature .............................................................................. 31

Remove and Re-add EB Host in ArcMC................................................................................................... 33

If Re-Add of EB Node Required, Shutdown All Servers and Take Snapshots ..................................... 36

Restart Lab Servers ............................................................................................................................. 36

Checking Ansible Log Files for Deployment Details (Collector install) ................................................... 37

Kubectl Command Examples .................................................................................................................. 37

Appendix B: Logger 2-way Authentication Update Process .................................................................... 40

Micro Focus Trademark Information .......................................................................................................... 43

Company Details ......................................................................................................................................... 43

4

Incorporating Existing ArcSight VMs With a default CentOS 7.4 build of Event Broker (and a corresponding BIND server), using the 172.16.100.x configuration, some existing demo VMs from Technical Enablement can be extracted and integrated into the Event Broker environment (without having to build each of the other demo 172.16.100.x systems from the ground up).

• In this guide, the Logger 6.61 demo VM is used

o Existing Logger instances can also be used instead – doesn’t have to be a pre-built demo VM

• Future versions of guide will test using ESM 7.0 P1 (compact mode) demo VM

NOTE 1: In this guide, I make use of multiple snapshots during the build and configuration process. These are not “required”, just done to have multiple snapshots at each major step in the installation / configuration.

NOTE 2: By default, the Tech Enablement VMs make use of the 172.16.100.x subnet.

Regarding this IP block from the EB Deployment Guide:

• The Installer will use the following network ranges by default:

o 172.16.0.0/16: Subnetwork of 65,536 addresses for operation of Kubernetes pods with containers running in them. Each pod will operate with /24 subnetwork from the following range.

o 172.30.78.0/24: Subnetwork of 256 addresses for operation of Kubernetes services, including internal Kubernetes DNS service, located on pod 172.30.78.78.

• For best results, make sure your network is conflict-free with the /16 and /24 ranges of addresses.

o For this version of this guide, I’m leaving the default IPs for demo VM systems in the 172.16.100.x range, however may change this in future versions.

Boot DNS, ArcMC, EB, and Logger VMs to be Used

The following previously built VMs were prepared using other guides. For this guide, an instance of each of these systems is required (but they don’t have to be Micro Focus defaults):

• DNS VM (dns.example.com)

o Vanilla install of BIND built using details provided in MacGyver Guide: CentOS 7.4 VM guide

• ArcMC VM (arcmcvan.example.com)

o Vanilla install of ArcMC 2.81 built using details provided in MacGyver Guide: CentOS 7.4 VM guide

• EB VM (eb1.example.com)

o Single node master and worker, built with Micro Focus Technical Enablement “ground-up” guide

• Logger VM (logger.example.com)

o In this guide, using a MF-built demo VM of Logger 6.61.

5

Deploying CEB on EB For a CEB deployment, there are 3 key stages covered in this doc:

1. In this stage, CEB will be deployed in EB 2. Next, Logger will be configured to receive data from the EB topic to which CEB writes 3. Later in this doc, a Collector will be installed on a Linux system (for this guide, using dns.example.com as

the location for the Collector), and that Collector will be configured to forward events to the CEB installed on EB

Sources for info from these sections:

• Followed instructions outlined in ArcMC 2.81 Admin Guide o Starting on page 69

CEB Options Selected / Set

1. In ArcMC, click Dashboard > Deployment View. 2. In the Event Broker column, once EB icon displays green, click in the Event Broker image

3. In the Event Broker column, click the + icon next to eb-con-syslog (the topic to which Collectors send by

default)

4. On the Deploy CEB dialog, in CEB Name, enter a name for the CEB. The name must be smaller than 256

characters.

CEB Name: vmCEB1

5. Under Acknowledgment mode, click the down arrow, then select the Acknowledgment mode for this CEB. (none/leader/all)

a. The mode you select affects the performance of your system as well as the safety of stored events in case of immediate system failure.

Acknowledgement mode: all

6. Under Source Topic, view default (can’t be changed): eb-con-syslog

6

7. Under Destination Topics, click the down arrow, then select one or more destination topics (CEF {eb-cef} or binary {eb-esm}) for the CEB

a. For this guide, using eb-cef to start since test events to be consumed by Logger

Destination Topic: eb-cef

8. Click Deploy a. Following status window displays:

9. Click Ok

a. The CEB deployment job status can be viewed in Job Manager. b. Click on the small arrow to the right of the job column to see more details

i. Wait for all tasks to complete – during initial stages, multiple “Failed to connect to CEB” status messages are normal

7

• Once deployment status shows as Complete, click Refresh to view the job status

Verify CEB Details in ArcMC UI • Click Node Management > View All Nodes

o Click Containers tab Verify Container 1 displays (new container on eb1.example.com)

• Click on Connectors tab o Verify the name of the new CEB connector displays (in this example, used vmCEB1)

8

• From Connectors tab, click on CEB connector name • Review details from the connector deployment

o Verify source topic set as: eb-con-syslog o Verify destination topic set as: eb-cef

• Click Dashboard > Topology View • Click in the Event Broker image

• Verify Event Broker displays the vmCEB1 entry

o Also shows default routing of eb-con-syslog to eb-cef topic

9

Re-Verify CEB Details in ArcMC UI

Use the following steps to re-verify the CEB details have persisted in ArcMC.

If, after these steps, the Container/Connector does not display, one option is to remove and re-add the EB node. Steps to do this are outlined in the Troubleshooting Appendix of this document. Verify the Collector and Container display in ArcMC prior to proceeding to further steps.

• Log off ArcMC UI, and re-log on • Click Node Management > View All Nodes



Verify CEB Deploy on EB Server via Command Line • Logon to EB as root • Get list of EB-CEB’s deployed on Event Broker

kubectl get pods --all-namespaces -o wide | grep eb-ceb

For example: [root@eb1 bin]# kubectl get pods --all-namespaces -o wide | grep eb-ceb

eventbroker1 eb-ceb-0 1/1 Running 0 21h 172.16.46.13 eb1.example.com

10

Shutdown All Servers and Take Snapshots

Even though the DNS and Logger servers not touched in this stage, snapshot taken of all 4 servers to have a consistent starting point for the next sections (Logger receiver configuration and Collector install).

• Shutdown eb1.example.com (then take snapshot) o With EB 2.21, prior needed commands (like sync, etc.) have been rolled into the default

shutdown process, so no extra steps needed

shutdown -h now

• Shutdown dns.example.com (then take snapshot) • Shutdown arcmc.example.com (then take snapshot) • Shutdown logger.example.com (then take snapshot)

Restart Lab Servers

• When all snapshots complete, restart all servers o Started servers in following order:

DNS ArcMC

• Logon to ArcMC UI when bootup complete EB (wait for all EB pods to get to “running” status)

• As root, from ssh window, track with following command once EB processes boot up enough to allow connection:

watch -n 15 kubectl get pods --all-namespaces -o wide

• Note: Even after all pods show as running, it will take a few minutes for ArcMC to update the status of EB to “green”

Logger (wait until able to logon to UI before proceeding to next steps)

Re-Verify CEB Details in ArcMC UI Post Re-Start

Use the following steps to re-verify the CEB details have persisted in ArcMC.

• If, after these steps, the Container/Connector does not display, one option is to remove and re-add the EB node. Steps to do this are outlined in the Troubleshooting Appendix of this document. Verify the Collector and Container display in ArcMC prior to proceeding to further steps.

• Log on ArcMC UI • Click Node Management > View All Nodes


11


Re-verify CEB Deploy on EB Server via Command Line Post Re-start • Logon to EB as root • Get list of EB-CEB’s deployed on Event Broker

kubectl get pods --all-namespaces -o wide | grep eb-ceb

For example: [root@eb1 bin]# kubectl get pods --all-namespaces -o wide | grep eb-ceb


12

Setting Logger Receiver to Consume from eb-cef Topic In this stage, with CEB installed, and eb-cef set as the destination topic, update the Logger 6.61 VM to consume (aka “subscribe” and pull) from the eb-cef topic in preparation for deploying a syslog Collector (on the DNS server) and sending raw syslog events to CEB (which will route automatically to the topic from which Logger will be subscribed).

• Prior to proceeding, ensure ArcMC shows the CEB connector and container in the Dashboard > Monitoring Summary. If not, complete that step first.

o Node Management > View All Nodes (check both Containers and Connectors tabs)

Logger Configuration – Step 1: Authentication

With EB and Logger, two-way communication is required for status to display correctly in ArcMC.

• On a new install (of ArcMC) where a Logger has not been added before, proceed to the Receiver Configuration step

• If a same-named Logger had been used with the ArcMC before, certificate updates may be required between ArcMC and that Logger. Please refer to Appendix B: Logger 2-way Authentication to update certificates

o NOTE: A demo VM Logger receiver can be configured without the following certification steps, however in ArcMC, post receiver setup the Logger may be listed as “unreachable” and display as follows in the Topology View

This may be due to a prior logger.example.com system being used in the demo ArcMC and Logger VMs, and there being a mis-match in hostnames and/or certificates.

• If, under Dashboard > Topology View, both Event Broker and the Logger receiver show a “green” state, proceed to the next section.

13

Logger Configuration – Step 2: Receiver Configuration

Configuring Receiver • Log on to Logger UI • Click Configuration > Receivers • Click Add • Set name of receiver to: vmCEB1_receiver • Set Type to: Event Broker Receiver • Click Next • Set default values for Event Broker connection:

o Name (leave default) o Event broker host(s) and port: 172.16.100.191:9092

If a host name is used, the UI will display an error that the entry is not an IP

• In my /etc/hosts on Logger, eb1 is the first alias in list and eb1.example.com is

listed as second alias, however Logger didn’t want to resolve to either hostname (at the Logger UI level)

• eb1 and eb1.example.com both respond to nslookups at the OS level (Logger OS set to 172.16.100.53 for DNS).

o Event Topic list: eb-cef o Retrieve event from earliest offset: true o Consumer Group (Logger Pool): ebconsumer1

Note: You do not need to actually create a Consumer Group anywhere. The Consumer Group is simply a logical grouping of consumers, specified by this field. It must be the same on every Logger in the pool.

For the demo example, just using “ebconsumer1” as a placeholder o Use SSL/TLS: false <leave default> o Use Client Authentication: false <leave default> o Enable <leave default check box enabled>

• Click Save o The Logger receiver list will re-display automatically

Verifying Receiver Active in Logger

• From Configuration > Receivers, verify the vmCEB1_receiver displays as enabled in the receiver list

14

• From the Logger Summary page, in Event Summary by Receiver, verify vmCEB1_receiver is listed o It may take a couple mins (2-5 mins on VM) for the receiver entry to display. Can also log off

and back on Logger and re-check

o Click on the vmCEB1_receiver o Click on All Fields, set field set to “Base Event Fields”, and click Go

Verify “Connector Raw Event Statistics” events (“name” field) from agent:050 events (“deviceEventClassId” field) are displayed in the search page

15

Verifying Receiver Active in ArcMC

• In ArcMC, browse to Dashboard > Topology View

o Verify a line displays between Event Broker and the Logger destination

• Click in the Event Broker image

• Verify connectivity line from eb-cef topic under Event Broker to the logger.example.com Consumer

16

o If initially displayed, can ignore the red “Fatal” status of vmCEB1 – this can show because of the default rule that triggers on “AVG EPS_IN < 05: Last 2min”

After a few minutes, in demo systems, status changed back to green:


Even though the DNS server not touched necessarily, snapshot taken of all 4 servers to have a consistent starting point for the next sections (Collector install).



shutdown -h now

o Wait until EB server completely down, then shutdown the following servers • Shutdown dns.example.com (then take snapshot) • Shutdown arcmc.example.com (then take snapshot) • Shutdown logger.example.com (then take snapshot)

17

Restart All Servers and Verify All Services Running


DNS • Wait until OS logon prompt displays

ArcMC • Logon to ArcMC UI when bootup complete

EB (wait for all EB pods to get to “running” status) • As root, from ssh window, track with following command once EB processes

boot up enough to allow connection:




• Log back in to the ArcMC UI o Click Dashboard > Topology View

Wait until EB icon shows in green status as initialized

18

Wait until Logger receiver icon under Destinations shows in green status as initialized • Will show red if Logger booting up, and consumer shows as unreachable • Will show as yellow due to default rule of “Rule Violation : ANY

RECEIVER_STATUS = DOWN : Last 5min” • Will show green once all communication re-established and all processes

running

Until Logger fully starts (and Logger node displayed as green), connectivity (the line) between EB and Logger consumers may not display

19

• Log back in to the Logger UI: o From the Summary window, click on the vmCEB1_receiver

Update the field set to Base Event Fields and click Go to see core fields up front, like Time, and name

NOTE: The IP address of the vmCEB1_receiver may automatically change after reboot (because it will re-deploy, and be given an updated IP address from within the cluster)

• In screen shot below from 172.16.51.24 to 172.16.51.9

• From the EB command line:

o Verify CEB on Event Broker shows as running

kubectl get pods --all-namespaces -o wide | grep ceb

or following command to automatically get refreshed status every 15 seconds


For example:

[root@eb1 bin]# kubectl get pods --all-namespaces -o wide | grep ceb


20

Use the DNS VM for a Syslog Collector Sending to CEB In this section, will be using the following systems for deploying a Linux-based Collector to route syslog to CEB:

• Will piggy-back on the DNS server (dns.example.com) to install an ArcSight Collector

Installing a Collector to Forward Syslog to CEB

Upload the Latest Collector installer File to ArcMC (Linux)

In this example, using Linux because Windows installations have multiple PowerShell install pre-reqs, and Linux enables using an existing Linux VM in the pool (DNS server) to minimize resources required.

• Download the connector or Collector installer file to a secure network location • In ArcMC, click Administration > Application > Repositories • In the navigation menu, click Upgrade Files • Click Upload • Under Upload Upgrade Repository, click Choose File. Then, browse to and select the installer file you

previously downloaded o For this guide, used Linux 64 Collector from SmartConnector release 7.9.0.8084.0: ArcSight-

7.9.0.8084.0-Collectors-Linux64.bin • Select file, and click Open • Click Submit

o The installer file is uploaded to ArcMC. • Status will update for repository:

Create a Deployment Template

In this stage, this template is to deploy a Collector (and define the “input” information for the Collector).

A later section will provide details for a “Destination Template”, which defines where collected inbound events need to go (aka the “output” information).

1. Click Configuration Management > Deployment Templates. 2. In the navigation menu, from the list of supported connectors or Collectors, select the type of

connector/Collector for which you wish to create a template. a. For this guide, use Collectors > Syslog > Syslog Collector Daemon

21

3. In the management panel, click New

a. By default, there are no templates in the ArcMC demo VM b. To clone a template from an existing template of the same type, click + New/Clone.

i. Select one from the Copy from dropdown and the values are populated based on the selected template instance.

4. Enter values for any required settings (marked with an asterisk *), as well as any settings you wish to apply to all connectors or Collectors of that type when using Instant Connector Deployment. (Note: Spaces in file or path names are not supported.)

a. Set Version to: 7.9.0.8084.0 i. Can pick the newest version that has been uploaded to the ArcMC repository

b. Set Template Name to: syslog_to_vmCEB1 c. Leave Fields defaults:

i. Network Port: 514 ii. IP Address: (ALL)

iii. Protocol: UDP d. Set Common Fields (connector and service naming):

i. Name: linux_syslog_collector1 ii. Service Internal Name: linux_sys_coll1

iii. Service Display Name: linux_sys_coll1

By default, the install will add an “arc_” as a prefix to the service automatically.

e. Set Global fields:

i. Collector Remote Management: 48080 (leave default) ii. Additional options:

1. Additional files <only displays for certain collectors>: a. If additional files are needed for operation, such as a Voltage server

certificate, under File Table Fields, enter values for file name, type, and any other required fields. If more than 1 additional file is needed, click Add Row, and then specify the details of the additional file. Repeat for additional files as needed.

2. Regular Expression Filtering (raw syslog filtering in or out) a. For raw regex filtering (in or out) of events, filter details can be set in

the Global Fields section: i. Field 1: true/false to do raw filtering

ii. Field 2: “include” regex iii. Field 3: “exclude” regex

22

5. Click Save a. Click OK at the Template addition window

Create a Destination Template

This is “the destination to which data flowing from the collector should be sent” configuration.

1. Browse to Configuration Management > Deployment Templates 2. In Nav panel on left, select Collector Destinations > Event Broker

3. Click +New / Clone 4. Set the parameters for a destination of Event Broker:

a. Version: <Select the latest SmartConnector version> b. Template Name: syslog_to_vmCEB1_dest1 c. Initial Host:Port(s): eb1.example.com:9092

i. 9092 is default port d. Topic: eb-con-syslog <leave default> e. Acknowledgement mode: leader <leave default> f. Use SSL/TLS: false <leave default>

5. Click Save a. Click OK on the template window

Deploying the Syslog Collector

1. Click Dashboard > Deployment View. 2. In the Connectors/Collectors column label, click + (at the top, next to the column label)

23

3. Select Add Collector a. On the Add Collector dialog, enter values for the syslog collector to be added. Any fields marked

with an asterisk (*) are required. b. Note that your selected deployment template may populate some fields automatically, but you

may overwrite the values in these fields, if needed, for a particular deployment. 1. Host:

a. Click on Host dropdown and select “Add Host” b. Type dns.example.com (and click check box to enable)

2. Host username: root 3. Host password: <root password for server> 4. Job Name: syslogcollector1 5. Operating System: Linux 64 bit (left default)

b. Exception: you may only use the latest version of the connector you have uploaded to the repository when you set up deployment templates. You can add multiple destinations for each connector if needed.

4. Set the Install location for the collector a. Filled in following for Install Location:

/opt/arcsight/syslogcollector1

5. Pick the Collector Template to use a. For this guide, select the “syslog_to_vmCEB1” template that was created in an earlier step

24

6. Fill in the password field to set the Collector Remote management password:

Arcs1ght!

i. Unlike a CONNector remote install, there is no default password. Set the password to be used here.

7. In the Destination Template dropdown, pick the destination template that was created in this guide in

earlier step a. For example: syslog_to_vmCEB1_dest1

b. The details from the template will display

8. Click Install

a. System prompt will display for checking on job status:

b. Click Ok c. Click on the Job Manager icon

25

d. While the collector is installing, status will show as “In Progress”

i. Once the deployment is successful, should see status like the following:

Can also expand the status for the job and display details as process runs:

ii. If the deployment fails, check the ansible log file on the ArcMC server to try and track down errors:

cd /opt/arcsight/userdata/logs/arcmc/ cat ansible.log.0

Can also tail the ansible log when doing a deployment to watch the status of the log as the installation is done:

tail –f ansible.log.0

26

Additional Notes:

• Note: Instant Connector Deployment (including Collectors) is not supported from RHEL/CentOS 6.9 to a remote Windows host.

• You can track and manage deployment jobs and issues using the Job Manager.

• Note: If you later connect to a host where connectors or Collectors were installed through Instant Deployment, and run the Connector setup wizard from the command line, you should run agent setup by setting the mode with option, -i, such as: ./runagentsetup.sh -i console, where options are swing, console, silent, and so on.

• For more information on options, see the SmartConnector User's Guide.

Updating the Connectivity to the Syslog Collector

By default with the demo VMs, although the Syslog Collector will be successfully deployed, the collector may not display in Dashboard > Topology View (in the Connectors / Collectors column).

After installation, click on the Job Status icon, and reviewing the Collector install status in the Details column.

• The following status should display when a collector successfully deploys:

• If there are issues with configurations, the following status about the collector not being added as a managed node may display:

or

If “node failure” errors are displayed, to update the connectivity, the host (dns.example.com) can be added manually.

1. Click Node Management > View All Nodes 2. Click on the Default location (in the vanilla ArcMC VM) 3. Click Add Host 4. Fill in following values:

a. Hostname/IP: dns.example.com b. Type: Collector c. Collector Credentials: collector <leave default> d. In password field: Arcs1ght! <password set during deploy> e. Port: 48080 <use default>

27

5. Click Add

6. Click Continue a. Host should display in the list at the bottom of the screen

Updating the Connectivity to the Syslog Collector

1. Browse to Dashboard > Topology View

a. Click on the Event Broker icon i. The linux_syslog_collector1 entry should display in the Connectors/ Collectors column,

with a destination linking to the eb-con-syslog topic on Event Broker

2. Browse to Node Management > View All Nodes b. Click on the Collectors tab

i. Verify the newly deployed syslog-collector is displayed

28

Sending Sample Syslog Events to the Collector Based on the setup in this guide:

• Raw syslog events will be sent to the syslog Collector (installed on dns.example.com) • Those events will be routed from the Collector to the eb-con-syslog topic on EB (eb1.example.com) • The CEB deployed on EB (vmCEB1) will read from the eb-con-syslog topic, normalize and categorize

events (same capabilities as a syslog SmartConnector), and publish events to the eb-cef topic on EB • The receiver on logger.example.com will pull those events from eb-cef and display them in searches and

dashboards within Logger

Using Kiwi sysloggen utility (Windows utility)

From any Windows system, the Kiwi syslog gen utility can send a raw syslog event to any IP.

• Can download Kiwi syslog gen from: o https://www.kiwisyslog.com/downloads o Scroll down on page to “Kiwi SyslogGen” and download

• Run Kiwi Syslog Message Generator • Set the Target IP address to the DNS server (on which Collector installed)

o For example: 172.16.100.53 • Set the Source IP address from which you want events to show as coming from

o For example: 172.16.100.98 o Click on the Send options drop-down and select # of events to send

o Click Send

By default, the following message is used (where ### is automatically and randomly replaced with IP addresses)

• Can click “Message text to send” dropdown and either pick another default, or add your own text

Test user connected to website http://###.###.###.###/index.html

https://www.kiwisyslog.com/downloads

29

o Click Stop to stop sending test messages Bottom of Kiwi Syslog Gen utility will show # of messages sent

• Can verify event flow multiple ways: o From ArcMC:

Configuration Management > Manage Collectors / Connectors In Collector tab, view the “Syslog Lines Received” column

o From Logger: Click Analyze > Search, and search for the following string

• Note: May take a couple minutes for events to be pulled into Logger once sent from Kiwi Syslog Gen (have to route through EB topics, and pulled from Logger)

deviceCustomString1 contains SyslogGen

Review events that return • The name column will show random IPs for websites • deviceVendor and deviceProduct will both show Unix

• With the “All Fields” field set, deviceCustomString1 field will show the test message

30


Even though the DNS server not touched necessarily, snapshot taken of all 4 servers to have a consistent starting point for the next sections (Collector install).



shutdown -h now

• Shutdown dns.example.com (then take snapshot) • Shutdown arcmc.example.com (then take snapshot) • Shutdown logger.example.com (then take snapshot)

31

Appendix A: Troubleshooting and Status Checking Information Eb-Diag Script (automated log collection for EB)

1. On the master server, find the docker container id of web service. (In this example 278e86760803)

docker ps | grep atlas_web-service

278e86760803 localhost:5000/arcsightsecurity/atlas_web-service@sha256:c25b023afa7b7054de6aa188ed2802d24312f3c5de87b6537aa3e937476376d8 "/bin/bash -c 'source"

2. Copy the script archive from web service container (In this example 278e86760803)

docker cp 278e86760803:/eb/ws/eb_diag/eb_diag.tgz .

tar -tzf eb_diag.tgz

Following details return:

vertica-diag.sh

eb-diag.sh

eb-sys-diag.sh

3. Extract the diagnostic script and run it

tar -xvf eb_diag.tgz eb-diag.sh

sh eb-diag.sh

Collecting Logs from ArcMC with Snapshot Feature

NOTE: For the snapshot feature to work correctly (collect and generate a zip), there are OS-level pre-reqs, including zip and other packages. If snapshot appears to run, but the zip file created is only zero bytes, there may be missing OS-level packages. (For example, a “minimal” RedHat / CentOS install does not include all needed packages.)

• Logon to ArcMC UI

32

• Browse to Administration > Snapshot

o ArcMC logs will be collected from the system, and saved in a .zip file on the ArcMC server

• When status marked as Complete, click Download

o Download gets saved by default to the browser’s default download location

Log files from /etc, /opt, and /var are included, as well as exports from the ArcMC database

33

Remove and Re-add EB Host in ArcMC

For the current version of the guide (and with these VM examples), the following step is one way to re-synch the CEB deployment with Event Broker.

• For example, if initial CEB details display in the UI, after logging off/on ArcMC, they may no longer display (because they were in memory in ArcMC, but may not have been stored in the ArcMC DB tables, even though they have been deployed to EB.)

The following steps have been shown to synch CEB deployments into ArcMC correctly, and appear to only need to be done once.

• Logon to ArcMC UI • Click Node Management > View All Nodes • Click Hosts tab • Click on the eb1.example.com row

• Click Delete • Click Yes when prompted to confirm delete • Click Done at successful confirmation message • Click on Default from the left navigation panel • Under the Hosts tab, click Add Host

34

• In Hostname field: eb1.example.com o Replace with name of your EB server if different

• In Type drop-down, select: Event Broker 2.02 or later

• For Port and Cluster Port, leave defaults:

o Port: 38080 o Cluster Port: 5443

• For Cluster Username: o admin

• For Cluster Password: o <Use the value set on EB install>

• For the Cluster Certificate:

o As root on the EB server:

Type:

/opt/arcsight/kubernetes/scripts/arcsight-cert-util.sh

o Copy all of the text from the “BEGIN CERTIFICATE” line through the “END CERTIFICATE” line (including the leading and trailing -----‘s)

o Click Add

o Click Continue on the “Import Host Certificates” window

Certificates are imported, and the EB host is re-added

35

Verify the EB host (and any previously added CEBs {containers}) are displayed in the list at the bottom of the page:

o Click Node Management > View All Nodes

o Click Hosts tab

Verify the eb1.example.com host is listed

o Click Connectors tab

Verify the previously installed CEB is listed in ArcMC

NOTE: When EB is rebooted, the Container with the CEB will show as red with a status of “Unknown Issue” (and the Connectors tab will be blank) until all processes start, and ArcMC synchs fully with EB / CEB.

36

o Click Dashboard > Topology View Verify Event Broker displays the vmCEB1 entry Also shows default routing of eb-con-syslog to eb-cef topic

If Re-Add of EB Node Required, Shutdown All Servers and Take Snapshots

Even though the DNS and Logger servers not touched in this stage, snapshot taken of all 4 servers to have a consistent starting point for the next sections (Logger receiver configuration and Collector install).



shutdown -h now

• Once EB shut down completely, shutdown dns.example.com (then take snapshot) • Shutdown arcmc.example.com (then take snapshot) • Shutdown logger.example.com (then take snapshot)

Restart Lab Servers


DNS ArcMC

• Logon to ArcMC UI when bootup complete EB (wait for all EB pods to get to “running” status)

• As root, from ssh window, track with following command once EB processes boot up enough to allow connection:




37

Checking Ansible Log Files for Deployment Details (Collector install) • If the deployment fails, check the ansible log file on the ArcMC server to try and track down errors:

cd /opt/arcsight/userdata/logs/arcmc/ cat ansible.log.0

Can also tail the ansible log when doing a deployment to watch the status of the log as the installation is done:

tail –f ansible.log.0

Kubectl Command Examples kubectl get pods --all-namespaces -o wide [root@eb1 ~]# kubectl get pods --all-namespaces -o wide NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE core apiserver-eb1.example.com 1/1 Running 6 13d 172.16.100.191 eb1.example.com core arcsight-installer-7bdcb5886d-xjbvc 2/2 Running 0 4h 172.16.51.18 eb1.example.com core cdf-apiserver-84f87dc785-qhrtr 2/2 Running 0 4h 172.16.51.17 eb1.example.com core controller-eb1.example.com 1/1 Running 6 13d 172.16.100.191 eb1.example.com core idm-8465dc9b76-5pzbf 2/2 Running 0 4h 172.16.51.23 eb1.example.com core idm-8465dc9b76-j5878 2/2 Running 0 4h 172.16.51.22 eb1.example.com core itom-cdf-ingress-frontend-689xt 1/1 Running 3 5d 172.16.51.2 eb1.example.com core itom-postgresql-default-f8f7d5dff-6p25w 2/2 Running 0 4h 172.16.51.8 eb1.example.com core kube-dns-jgm9h 3/3 Running 9 5d 172.16.51.3 eb1.example.com core kube-registry-proxy-kghm7 2/2 Running 12 13d 172.16.51.7 eb1.example.com core kube-registry-vgldv 1/1 Running 3 5d 172.16.51.6 eb1.example.com core kubernetes-vault-p6km9 1/1 Running 9 5d 172.16.51.5 eb1.example.com core mng-portal-d7bdfd9bd-k6kmm 2/2 Running 0 4h 172.16.51.21 eb1.example.com core nginx-ingress-controller-vrcpj 1/1 Running 8 13d 172.16.51.4 eb1.example.com core scheduler-eb1.example.com 1/1 Running 6 13d 172.16.100.191 eb1.example.com core suite-db-594b9d9d97-8dhcl 2/2 Running 0 4h 172.16.51.16 eb1.example.com core suite-installer-frontend-6d74cdbd5b-lsqz8 2/2 Running 0 4h 172.16.51.13 eb1.example.com eventbroker1 eb-ceb-0 1/1 Running 0 4h 172.16.51.9 eb1.example.com eventbroker1 eb-kafka-0 1/1 Running 0 4h 172.16.51.19 eb1.example.com eventbroker1 eb-kafka-manager-7f7689c764-tpz9d 1/1 Running 0 4h 172.16.51.20 eb1.example.com eventbroker1 eb-routing-processor-0 1/1 Running 0 4h 172.16.51.12 eb1.example.com eventbroker1 eb-schemaregistry-5c66787cf5-q2hn4 1/1 Running 1 4h 172.16.51.14 eb1.example.com eventbroker1 eb-web-service-75fb7b7756-kstwp 2/2 Running 0 4h 172.16.51.24 eb1.example.com eventbroker1 eb-zookeeper-0 1/1 Running 0 4h 172.16.51.11 eb1.example.com eventbroker1 suite-reconf-pod-eventbroker-h272h 2/2 Running 0 4h 172.16.51.15 eb1.example.com kube-system heapster-apiserver-88f47fcbc-dn2jb 1/1 Running 0 4h 172.16.51.10 eb1.example.com

kubectl get services –n eventbroker1 –o wide [root@eb1 ~]# kubectl get services -n eventbroker1 -o wide NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR eb-arcmcweb-svc NodePort 172.30.78.251 <none> 8080:38080/TCP 13d svclink=webservice eb-kafka-svc ClusterIP 172.30.78.160 <none> 9092/TCP,9093/TCP,9999/TCP,10000/TCP 13d svclink=kafka eb-kafka-svc-nodeport NodePort 172.30.78.218 <none> 9092:39092/TCP,9093:39093/TCP 13d svclink=kafka eb-kafkamgr-svc ClusterIP 172.30.78.118 <none> 9000/TCP 13d svclink=kafkamanager eb-routing-streamp-svc ClusterIP None <none> <none> 13d svclink=routing-processor eb-schema-svc ClusterIP 172.30.78.126 <none> 8081/TCP 13d svclink=schema eb-syslog-svc01 NodePort 172.30.78.96 <none> 9001:39001/TCP 13d svclink=ceb9001 eb-syslog-svc02 NodePort 172.30.78.111 <none> 9002:39002/TCP 13d svclink=ceb9002 eb-syslog-svc03 NodePort 172.30.78.79 <none> 9003:39003/TCP 13d svclink=ceb9003

38

eb-syslog-svc04 NodePort 172.30.78.59 <none> 9004:39004/TCP 13d svclink=ceb9004 eb-syslog-svc05 NodePort 172.30.78.49 <none> 9005:39005/TCP 13d svclink=ceb9005 eb-syslog-svc06 NodePort 172.30.78.22 <none> 9006:39006/TCP 13d svclink=ceb9006 eb-syslog-svc07 NodePort 172.30.78.228 <none> 9007:39007/TCP 13d svclink=ceb9007 eb-syslog-svc08 NodePort 172.30.78.38 <none> 9008:39008/TCP 13d svclink=ceb9008 eb-syslog-svc09 NodePort 172.30.78.77 <none> 9009:39009/TCP 13d svclink=ceb9009 eb-syslog-svc10 NodePort 172.30.78.230 <none> 9010:39010/TCP 13d svclink=ceb9010 eb-syslog-svc11 NodePort 172.30.78.209 <none> 9011:39011/TCP 13d svclink=ceb9011 eb-syslog-svc12 NodePort 172.30.78.23 <none> 9012:39012/TCP 13d svclink=ceb9012 eb-syslog-svc13 NodePort 172.30.78.148 <none> 9013:39013/TCP 13d svclink=ceb9013 eb-syslog-svc14 NodePort 172.30.78.181 <none> 9014:39014/TCP 13d svclink=ceb9014 eb-syslog-svc15 NodePort 172.30.78.70 <none> 9015:39015/TCP 13d svclink=ceb9015 eb-syslog-svc16 NodePort 172.30.78.123 <none> 9016:39016/TCP 13d svclink=ceb9016 eb-syslog-svc17 NodePort 172.30.78.60 <none> 9017:39017/TCP 13d svclink=ceb9017 eb-syslog-svc18 NodePort 172.30.78.221 <none> 9018:39018/TCP 13d svclink=ceb9018 eb-syslog-svc19 NodePort 172.30.78.41 <none> 9019:39019/TCP 13d svclink=ceb9019 eb-syslog-svc20 NodePort 172.30.78.17 <none> 9020:39020/TCP 13d svclink=ceb9020 eb-syslog-svc21 NodePort 172.30.78.162 <none> 9021:39021/TCP 13d svclink=ceb9021 eb-syslog-svc22 NodePort 172.30.78.144 <none> 9022:39022/TCP 13d svclink=ceb9022 eb-syslog-svc23 NodePort 172.30.78.64 <none> 9023:39023/TCP 13d svclink=ceb9023 eb-syslog-svc24 NodePort 172.30.78.124 <none> 9024:39024/TCP 13d svclink=ceb9024 eb-syslog-svc25 NodePort 172.30.78.73 <none> 9025:39025/TCP 13d svclink=ceb9025 eb-syslog-svc26 NodePort 172.30.78.165 <none> 9026:39026/TCP 13d svclink=ceb9026 eb-syslog-svc27 NodePort 172.30.78.3 <none> 9027:39027/TCP 13d svclink=ceb9027 eb-syslog-svc28 NodePort 172.30.78.72 <none> 9028:39028/TCP 13d svclink=ceb9028 eb-syslog-svc29 NodePort 172.30.78.215 <none> 9029:39029/TCP 13d svclink=ceb9029 eb-syslog-svc30 NodePort 172.30.78.219 <none> 9030:39030/TCP 13d svclink=ceb9030 eb-syslog-svc31 NodePort 172.30.78.21 <none> 9031:39031/TCP 13d svclink=ceb9031 eb-syslog-svc32 NodePort 172.30.78.66 <none> 9032:39032/TCP 13d svclink=ceb9032 eb-syslog-svc33 NodePort 172.30.78.210 <none> 9033:39033/TCP 13d svclink=ceb9033 eb-syslog-svc34 NodePort 172.30.78.24 <none> 9034:39034/TCP 13d svclink=ceb9034 eb-syslog-svc35 NodePort 172.30.78.62 <none> 9035:39035/TCP 13d svclink=ceb9035 eb-syslog-svc36 NodePort 172.30.78.56 <none> 9036:39036/TCP 13d svclink=ceb9036 eb-syslog-svc37 NodePort 172.30.78.93 <none> 9037:39037/TCP 13d svclink=ceb9037 eb-syslog-svc38 NodePort 172.30.78.32 <none> 9038:39038/TCP 13d svclink=ceb9038 eb-syslog-svc39 NodePort 172.30.78.229 <none> 9039:39039/TCP 13d svclink=ceb9039 eb-syslog-svc40 NodePort 172.30.78.95 <none> 9040:39040/TCP 13d svclink=ceb9040 eb-syslog-svc41 NodePort 172.30.78.231 <none> 9041:39041/TCP 13d svclink=ceb9041 eb-syslog-svc42 NodePort 172.30.78.188 <none> 9042:39042/TCP 13d svclink=ceb9042 eb-syslog-svc43 NodePort 172.30.78.154 <none> 9043:39043/TCP 13d svclink=ceb9043

39

eb-syslog-svc44 NodePort 172.30.78.183 <none> 9044:39044/TCP 13d svclink=ceb9044 eb-syslog-svc45 NodePort 172.30.78.92 <none> 9045:39045/TCP 13d svclink=ceb9045 eb-syslog-svc46 NodePort 172.30.78.175 <none> 9046:39046/TCP 13d svclink=ceb9046 eb-syslog-svc47 NodePort 172.30.78.46 <none> 9047:39047/TCP 13d svclink=ceb9047 eb-syslog-svc48 NodePort 172.30.78.94 <none> 9048:39048/TCP 13d svclink=ceb9048 eb-syslog-svc49 NodePort 172.30.78.145 <none> 9049:39049/TCP 13d svclink=ceb9049 eb-syslog-svc50 NodePort 172.30.78.80 <none> 9050:39050/TCP 13d svclink=ceb9050 eb-zk-sts ClusterIP None <none> 2181/TCP,2888/TCP,3888/TCP 13d svclink=zookeeper eb-zook-svc ClusterIP 172.30.78.193 <none> 2181/TCP,2888/TCP,3888/TCP 13d svclink=zookeeper eb-zook-svc-nodeport NodePort 172.30.78.55 <none> 2181:32181/TCP 13d svclink=zookeeper suite-reconf-svc-eventbroker ClusterIP 172.30.78.214 <none> 8080/TCP,8081/TCP 13d name=suite-reconf-sel-eventbroker

40

Appendix B: Logger 2-way Authentication Update Process To set up two-way authentication, follow the steps in these sections:

• Step 1: Generate a CSR on the Logger Side • Step 2: Sign the Logger CSR on the Event Broker • Step 3: Import the Signed Certificate and Private Key to the Logger Keystore

• Step 1: Generate a CSR on the Logger side

o Logon to Logger as root In this example, using updated Logger 6.61 demo VM

o Cd to following dir

cd /opt/arcsight/logger/current/arcsight/logger/bin/scripts

o Run the eb_cert_tool script to generate a CSR

./eb_cert_tool.sh --generate-csr --eb-host eb1.example.com --key-length 2048

>> Script will provide output, including destination location for the csr file generated:

2018-09-08 06:02:41,953 INFO Key pair generated successfully in /opt/arcsight/logger/current/arcsight/logger/user/logger/eb_certs/eb1.example.com/key.pem 2018-09-08 06:02:41,974 DEBUG Gen CSR: Return code: 0, Output: 2018-09-08 06:02:41,974 INFO CSR generated successfully in /opt/arcsight/logger/current/arcsight/logger/user/logger/eb_certs/eb1.example.com/csr.csr ************************************************************************************************** PLEASE USE THE FOLLOWING CSR TO SIGN THIS LOGGER'S CERTIFICATE ON THE EVENT BROKER eb1.example.com **************************************************************************************************

o Copy the CSR text file generated in step 3 to the Event Broker host.

For example: scp /opt/arcsight/logger/current/arcsight/logger/user/logger/eb_certs/eb1.example.com/csr.csr [email protected]:/tmp/

When prompted, accept connection, and enter root password.

• Step 2: Sign the Logger CSR on the Event Broker

41

o Log in to the Event Broker host o Change to dir with openssl

cd /usr/bin

o Run the following command to sign the CSR:

./openssl x509 -req -CA /opt/arcsight/kubernetes/ssl/ca.crt -CAkey /opt/arcsight/kubernetes/ssl/ca.key -in /tmp/csr.csr -out /tmp/signedLoggerCert.pem -days 3650 -CAcreateserial -passin pass:qwerqawer -sha256

Results will be similar to following: [root@eb1 bin]# ./openssl x509 -req -CA /opt/arcsight/kubernetes/ssl/ca.crt -CAkey /opt/arcsight/kubernetes/ssl/ca.key -in /tmp/csr.csr -out /tmp/signedLoggerCert.pem -days 3650 -CAcreateserial -passin pass:qwerqawer -sha256 Signature ok subject=/C=US/ST=California/L=Sunnyvale/O=Arcsight/OU=Logger/CN=logger.example.com Getting CA Private Key

o Copy the signed certificate from EB back to the Logger host, for example: scp /tmp/signedLoggerCert.pem [email protected]:/tmp/

When prompted, confirm connection and root password for Logger

• Step 3: Import the Signed Certificate to the Logger Keystore

o Log in to the Logger host using operating system credentials.

Use the same credentials that were used to generate the CSR.

o Run the eb_cert_tool to import the certificate:

For example (demo VM Logger 6.61) /opt/arcsight/logger/current/arcsight/logger/bin/scripts/eb_cert_tool.sh --import-cert --eb-host eb1.example.com --cert-path /tmp/signedLoggerCert.pem

Default example:

/opt/arcsight/logger/current/arcsight/logger/bin/scripts/eb_cert_tool.sh --import-cert --eb-host <name or ip of EB host> --cert-path <location of cert signed by EB>

mailto:[email protected]:/tmp/

42

>> full path and filename of the pem file generated on the Logger

Example:

[root@logger scripts]# /opt/arcsight/logger/current/arcsight/logger/bin/scripts/eb_cert_tool.sh --import-cert --eb-host eb1.example.com --cert-path /tmp/signedLoggerCert.pem 2018-09-08 06:34:32,242 DEBUG Temp Keystore Import: Return code: 0, Output: 2018-09-08 06:34:32,242 INFO Key pair imported successfully into /opt/arcsight/logger/current/arcsight/logger/user/logger/eb_certs/eb1.example.com/eb_ks.pkcs12 2018-09-08 06:34:34,852 DEBUG Import Key Pair: Return code: 0, Output: Importing keystore /opt/arcsight/logger/current/arcsight/logger/user/logger/eb_certs/eb1.example.com/eb_ks.pkcs12 to /opt/arcsight/logger/current/arcsight/logger/user/logger/fips/receiver/bcfks_ks... 2018-09-08 06:34:34,853 INFO Key pair imported successfully into /opt/arcsight/logger/current/arcsight/logger/user/logger/fips/receiver/bcfks_ks *********************************************************************************************************************** CERTIFICATE IMPORT SUCCESSFUL PLEASE USE LOGGER UI TO CONFIGURE EVENT BROKER RECEIVERS FOR EVENT BROKER eb1.example.com ***********************************************************************************************************************

Repeat the steps in each section of this topic for all Event Brokers that do not have the same CA cert, from which Logger needs to receive events.

You can now configure Event Broker receivers on your Logger.

43

Micro Focus Trademark Information MICRO FOCUS and the Micro Focus logo, among others, are trademarks or registered trademarks of Micro Focus (IP) Limited or its subsidiaries in the United Kingdom, United States and other countries. All other marks are the property of their respective owners.

Company Details Company name: Micro Focus International plc

Place of registration: England and Wales

Registered number: 5134647

Registered address: The Lawn, 22-30 Old Bath Road, Berkshire, RG14 1Q

Date post:	17-Oct-2020
Category:	Documents
Upload:	others
View:	4 times
Download:	0 times

MacGyver Guide: Installing and Integrating EB CEB and Syslog …€¦ · o Vanilla install of ArcMC...

Documents