IBM Storage & SDI

1

Scale Security – File Audit Logging and Using Vagrant to setup Scale EnvironmentsChristopher D. MaestasSenior Architect – Spectrum Scale, IBM Systems

IBM Storage & SDI

Firewalls and SELinux

IBM Storage & SDI

gpfs 1191/tcp General Parallel File Systemgpfs 1191/udp General Parallel File System# Dave Craft gpfs@ibm.com November 2004Ports: https://www.ibm.com/support/knowledgecenter/STXKQY_5.0.0/com.ibm.spectrum.scale.v5r00.doc/bl1adv_firewall.htm

Spectrum Scale - firewall

© Copyright IBM Corporation 2017

IBM Storage & SDI

GPFS V3.5 and later run in'permissive' mode, and 'enforcing' mode with 'SELINUXTYPE=targeted'

GPFS commands have to run unconfined

No SELinux profiles supplied for GPFS daemons and utilitiesRunning GPFS command in a confined security context may failResult in a large volume of logged security exception events.

GPFS can hold files with per-inode security labels with limitationshttps://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20(GPFS)/page/SElinux

Spectrum Scale - SELinux

4

IBM Storage & SDI

EU GDPR

IBM Storage & SDI

http://www-03.ibm.com/support/techdocs/atsmastr.nsf/5cb5ed706d254a8186256c71006d2e0a/1d33b61a55b2787185258251004c0566/$FILE/GDPR%20Compliance-%20Spectrum%20Scale%20Technical%20Position.pdf

EU General Data Protection Regulation (GDPR)

© Copyright IBM Corporation 2017

IBM Storage & SDI

SUDO – don’t run as root

IBM Storage & SDI

https://www.ibm.com/support/knowledgecenter/en/STXKQY_5.0.0/com.ibm.spectrum.scale.v5r00.doc/bl1adm_sudowrapper.htm

Breaking news – installtoolkit mostly works! caveat with callhome and object configuration for CES

Configuring sudo – visudo/usr/lpp/mmfs/samples/sudoers.sample.

Configuring the cluster to use sudo wrapper scriptsmmchcluster command with the --use-sudo-wrapper option.

Configuring IBM Spectrum Scale GUI to use sudo wrapper

SUDO wrappers

© Copyright IBM Corporation 2017

IBM Storage & SDI

Immutability – WORM

IBM Storage & SDI

The immutability function in IBM Spectrum Scale Version 4.2 has been assessed for compliance in accordance to US SEC17a-4f rules, German and Swiss laws and regulations by a recognized auditor.

Assessment report: http://www.kpmg.de/bescheinigungen/RequestReport.aspx?41742

Certificate: https://www.kpmg.de/bescheinigungen/RequestReport.aspx?41743

Spectrum Scale immutability - certified for compliance

© Copyright IBM Corporation 2017

IBM Storage & SDI

Immutability means preventing changes and deletion of files during retention time

Spectrum Scale Immutability provides WORM storage in GPFS filesetImmutable files cannot be changed or deleted during retention period

Deletion is possible when retention time is expired

Managing immutability works similar to other productsRetention time can be set with last access dateWORM protection can be set by removing write permission

Spectrum Scale also supports append-only modeAn empty file can be set to append-only by removing and adding write permissionAppend-only file allows appends at the endAppend-only file can be made immutable by removing write permission once again

Immutability Overview

11

IBM Storage & SDI

none: Default setting for a normal fileset

advisory (ad): Allows setting retention times and WORM protectionBut files can be deleted with the proper permission

noncompliant (nc): Advisory mode plus Files cannot be deleted if retention time is not expired. But retention times can be reset and files can be deleted but not changed

compliant (co): noncompliant mode plusRetention time cannot be reset. When retention time has expired files can be deleted but not changed

Modes can be upgraded, but not downgraded

To set IAM use command: mmchfileset–iam-mode

Fileset Immutability Archive Manager Mode

© Copyright IBM Corporation 2017

IBM Storage & SDILook a man page! mmchfileset

© Copyright IBM Corporation 2017

IBM Storage & SDI

Setting retention time for filetouch –at MMddhhmmss filenamemmchattr –E yyyy-mm-dd[@hh:mm:ss] filename

Setting file immutable chmod –w filenamemmchattr –i yes filename

Setting file to append-onlyCreate Empty file

chmod –w filename; chmod +w filenamemmchattr –a yes

Set commands

© Copyright IBM Corporation 2017

IBM Storage & SDI

View fileset immutability mode

mmlsfilesetfsfset –iam-mode

Show file immutability setting

mmlsattr –L filename

Showing commands

© Copyright IBM Corporation 2017

IBM Storage & SDI

Deletion of file systems with compliant filesets (mmdelfs)Cluster-wide configuration parameter “indefiniteRetentionProtection” prevents this

Once set to yes deletion of file system is no longer possibleCannot be set back to no once set to yes

Deletion of compliant filesets (mmdelfileset)Not possible at GPFS 4.2 and higher

Backup and restore using mmbackupWorks with Spectrum Protect B/A client 7.1.3 and aboveIn-place restore cannot overwrite and existing immutable fileOut-of-place restore does not set the immutability attribute and retention time

Last access data will reflect retention time

Spectrum Protect for Space Management 7.1.4 and above supports this

Additional functions and options

IBM Storage & SDI

Spectrum Scale Immutability Whitepaper:http://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP102620

Recommended reading

© Copyright IBM Corporation 2017

IBM Storage & SDI

File Audit Logging

IBM Storage & SDI

19

New File Audit Logging capability (Data Management Edition only)

Track user accesses to filesystem and events

Supported across all nodes and all protocols

Parseable data stored in secure retention-protected fileset

Events that can be captured are: Open, Close, Destroy (Delete), Rename, Unlink, Remove Directory, Extended Attributed Change, Access Control List (ACL) change

Improved security and compliance

|

IBM Storage & SDI

20

Integration with audit tools like Varonis and IBM Guardium

http://www.redbooks.ibm.com/redpapers/pdfs/redp5426.pdfhttps://www.ibm.com/support/knowledgecenter/en/STXKQY_4.2.2/com.ibm.spectrum.scale.v4r22.doc/bl1adv_dpauditlogging.htm

Uses Light Weight Events (LWE) – What uses this today? Transparent Cloud Tiering - TCT

FAL - history

© Copyright IBM Corporation 2017

IBM Storage & SDI

21

Demonstrate monitoring of file activity including user name, timestamp, and file location regardless of client type

Demonstrate monitoring of file activity without endpoint (IBM Guardium or Varonis) agent on clients

Create CSV-formatted reports of file activity and directory activity

Create report containing variable days of activity and deliver via file system, email, and api

Sample Audit POC Tasks

IBM Storage & SDIAudit logging with Varonis DatAdvantage

© Copyright IBM Corporation 2017

IBM Storage & SDI

23

9 node cluster

Traffic FVT I/O Stress tests (autotest, mkfiles)Command Regression (as root)

STAPs installed on each node

Audit only policy right now

Audit removable media for NFS

Spectrum Scale Testing with IBM Guardium

IBM Storage & SDI

CommandsDELETEREADWRITE

Create file thru vi shows as a writeWe catch data in inodeCREATE system call shows up as a WRITE

EXEC (Execution)FILEOP (MKDIR, CHMOD, CHOWN)

Source Program

Db_user

OS-User

Object

What do we not catch

Spectrum Scale Testing with IBM Guardium

What do we catchGPFS administration commands like:

mmchattr –P sp1 /testfs/subdir/*#This changes the extended attributes of a file (root only)

mmapplypolicy /testfs/subdir -P mig.pol

# migrates data between storage pools (root only)To monitor root

§In guard_tap.ini file add : fam_protect_privileged=1

IBM Storage & SDI

2525

Spectrum Scale File Audit Logging - High Level Flow

IBM Storage & SDI

26

Now an API for 3rd party software IBM Guardium and Varonis

Light Weight Events (LWE) with Apache Kafka

Producer to publish stream of records: 1 million msg/s

Live inside mmfsd (gpfs) daemon

Consumer subscribe to one or more topics and process stream: 3 million msg/s

node classes – minimum of 3

Monitor via CLI, mmhealth ,logfile, msgqueue or GUI (Events panel)!

File Audit Logging (FAL)

26

IBM Storage & SDI

27

FAL - Architecture

Protocol NodesQuorum

IBM Storage & SDI

28

FAL – event flow

IBM Storage & SDI

Only Linux nodes (RHEL and Ubuntu)

Linux Kernel version above > 3.10

Minimum of 3 Linux quorum nodes

Minimum of 3 nodes must be designated as Broker nodes

Supported hardware platforms (x86 and PPCLE)RHEL supported on x86 and PPC LEUbuntu is only supported on x86

Advanced License edition or the Data Management edition

During Installation, most configuration is automatically done and stored in /opt/kafka folder

Free space requirements>1 GB local disk space per file

system being audited> 2 GB local disk space per file

system being audited on all broker nodes

Install and configuration

IBM Storage & SDI

30

Installation

IBM Storage & SDIInstallation and verification

© Copyright IBM Corporation 2017

IBM Storage & SDIWhat is logged

© Copyright IBM Corporation 2017

IBM Storage & SDI

33

Acquire most common types of file activity:open, close, delete, rename, POSIX permission changes, ACL changes, etc.Don’t capture internal operations (e.g., restripe)

Events captured within GPFS daemon – represent attributes of filesystem action at that pointExample audit log entry:

33

{"LWE_JSON": "0.0.1", "path": "/newfs/1Kfile2.restore", "oldPath": null, "clusterName": "pardie.cluster", "nodeName": "c6f2bc3n10", "nfsClientIp": "", "fsName": "newfs", "event": "OPEN", "inode": "26626", "openFlags": "32962", "poolName": "sp1", "fileSize": "0", "ownerUserId": "0", "ownerGroupId": "0", "atime": "2017-10-25_12:36:22-0400", "ctime": "2017-10-25_12:36:22-0400", "eventTime": "2017-10-25_12:36:22-0400", "clientUserId": "0", "clientGroupId": "0", "processId": "10437", "permissions": "200100644", "acls": "u::rwc, g::r, o::r, ", "xattrs": null }

What gets Monitored

IBM Storage & SDI

34

Each file system enabled has a dedicated fileset where the audit logs will go. • Default option is .audit_log at the root of the file

system.

.audit_log fileset is created as IAM mode noncompliant.• Advisory mode plus

Files cannot be deleted if retention time is not expired.

But retention times can be reset and files can be deleted but not changed

AuditLog files are nested within /FSNAME/.audit_log/topic/year/month/date/*

Log file is written in append only mode

Rotation to a new log file upon reaching a threshold (500,000 events), then compressed and marked immutable for the retention period.

Default retention period is 365 days

Live events can be monitored by tailing the current auditLogFile<…>

Easy to search and consume

Log Files for Auditing

IBM Storage & SDI

35

FAL in the GUI

© Copyright IBM Corporation 2017

IBM Storage & SDI

36

mmaudit all consumerStatus –N …

mmmsgqueue status

CLI Monitoring

IBM Storage & SDI

37

Periodic polling and event callback registration mechanism is used.Possible lag in determining the health due to polling constraints.

mmhealth cluster monitoring

IBM Storage & SDI

3838

mmhealth node monitoring

IBM Storage & SDI

3939

/var/adm/ras/mmmsgqueue.logContains information regarding the set up and configuration operations that take place that affect the message queue

Valid on any node containing a broker and/or zookeeper

/var/adm/ras/mmaudit.logContains information regarding the set up and configuration operations that take place that affect the File Audit Logging

Valid on any node running the File Audit Logging command or location where the subcommand may be run (such as a consumer)

/var/adm/ras/mmfs.log.latestDaemon log, and contains entries when major message queue or File Audit Logging activity occurs.

/var/log/messages (Redhat) or /var/log/syslog (Ubuntu)Contains messages from Kafka components as well as the producer and consumers that are running on a node.

Logs collected via gpfs.snap

Troubleshooting

IBM Storage & SDI

40

Antivirus

Take an action if something happens in a directory

TCT enhancements?!

Where could this go in the future?

© Copyright IBM Corporation 2017

IBM Storage & SDI

Running Spectrum Scale in a Vagrant Environment

© Copyright IBM Corporation 2017

IBM Storage & SDI

• Yes, we have a VM

• Stemmed from work to do an IBM Scale GUI Lab• Spin a VM with an RedHat based OS and kickstart file

• Use install toolkit and latest version of Scale!

• Tied to VMWare workstation

sudo genisoimage -U -r -v -T -J -joliet-long -V "CentOS 7 x86_64" -volset "CentOS-7.4" -A "CentOS-7.4" -b isolinux/isolinux.bin -c isolinux/boot.cat -no-emul-boot -boot-load-size 4 -boot-info-table -eltorito-alt-boot -e images/efiboot.img -no-emul-boot -o ISONAME .

Replicate a repeatable Scale environment

IBM Storage & SDI

Build and manage virtual machines on the fly

Plugins to configuration management utilities like:

ansible, chef , puppet, salt …

Scale runs anywhere but you need:

1. an OS installed2. time and name resolution working3. working network

What is vagrant and why??

IBM Storage & SDI

• Windows 7• needs a new powershell > 2• Use cmder.net for console§ built in git and ssh

• Linux and OS X environments seem to be fine

Can run on Windows, Linux and OS X

IBM Storage & SDI

• Virtualbox• Runs the published Scale and Archive VMs today• Scale Vagrant files tested on Linux and Windows

• KVM/libvirt• No problems with RHEL7, can work with RHEL6

Tested Hypervisors

IBM Storage & SDI

46

• Everything starts with vagrant

• To ssh: vagrant ssh VMNAME

• To start: vagrant up

• To halt: vagrant halt

• To reprovision: vagrant destroy

• The main definition is in a file called

• Vagrantfile – ruby syntax

• To cry or start from scratch: rm –fr $HOME/.vagrant.d

Vagrant Mini-HowTo

IBM Storage & SDI

• Certain plugins help with• Hosts file update§ vagrant plugin install vagrant-hosts

• if using Virtualbox, run• vagrant plugin install vagrant-vbguest

Setup plugins and add default OS to use§ else if using libvirt, run

§ vagrant plugin install \vagrant-libvirt

§ Sometimes trouble starting libvirt vms, so restart it

§ systemctl restart libvirtd

IBM Storage & SDI

48

• Select your hypervisor (recommend virtualbox or libvirt)• Add centos/7 vagrant box§ vagrant box add centos/7§ vagrant box list

• You should see centos/7 listed

Setup a local box to work from

IBM Storage & SDIVagrant file - Clients and Protocol nodes

Vagrantfile is Ruby code

IBM Storage & SDIVagrant file – libvirt SNC vs Shared

IBM Storage & SDI

51

Shared libvirt vs Virtualbox

KVM VS Virtualbox

IBM Storage & SDI

52

Virtualbox SNC

IBM Storage & SDI

53

Install a base box so you don’t have to pull updates

IBM Storage & SDI

54

Can call out ansible here

Currently calling a shell script

Points to a SCALESOURCE tree and extracts data

Provision Scripts

IBM Storage & SDI

55

Can call out ansible here

Currently calling a shell script

Points to a SCALESOURCE tree and extracts data

Provision Scripts

IBM Storage & SDI

56

Coming soon GIT tree public

vagrantbuild – sample Vagrant files for Scale

cssdeployenv – install toolkit and runbooks

Integrate with Ansible form others

Let’s demo

ibm.com/storage

Thank You.IBM Storage & SDI