Post on 17-Mar-2018
transcript
IRMA DocumentationRelease 2.0.4
Quarkslab
Mar 13, 2018
Contents
1 Introduction 31.1 Purpose . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.2 File Analysis Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.3 Infrastructure Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41.4 Hardware requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51.5 Supported Analyzers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2 Automated Install 92.1 Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92.2 Ansible scripts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92.3 Predefined Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92.4 Using Debian repos . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
3 Manual Installation 133.1 Brain . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133.2 Frontend . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223.3 Probe . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
4 Database migration 414.1 Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 414.2 Content . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 414.3 Usage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 424.4 Tips and tricks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
5 To evolve IRMA 475.1 Adding a new probe . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
6 References 516.1 Disclaimer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 516.2 License . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 516.3 Apache License, version 2.0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 516.4 Authors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
7 Frequently Asked Questions 537.1 Playing with tags . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 537.2 SSL settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 567.3 How to debug . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
i
7.4 How to migrate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 607.5 API documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 617.6 Connect to a vagrant box through ssh . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 637.7 Enable SSL using OpenSSL in ansible scripts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 637.8 Speed up your Vagrant VMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
8 Resources 65
9 Screenshots 679.1 Command Line Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 679.2 Web Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
ii
IRMA Documentation, Release 2.0.4
IRMA is an asynchronous & customizable analysis system for suspicious files.
This guide will explain how to set up IRMA, use it and customize it at will.
Contents 1
IRMA Documentation, Release 2.0.4
2 Contents
CHAPTER 1
Introduction
This is an introductory chapter to IRMA. It recalls IRMA’s overall architecture, hardware required to run it and therecommended order for installing the IRMA’s components.
1.1 Purpose
IRMA intends to be an open-source platform designed to help identifying and analyzing malicious files.
However, today’s defense is not only about learning about a file, but it is also getting a fine overview of the incidentyou dealt with: where / when a malicious file has been seen, who submitted a hash, where a hash has been noticed,which anti-virus detects it, . . .
An important value with IRMA comes from you keep control over where goes and who gets your data. Once youinstall IRMA on your network, your data stays on your network.
Each submitted files is analyzed in various ways. For now, we focus our efforts on multiple anti-virus engines, but weare working on other “probes” (feel free to submit your own).
1.2 File Analysis Process
1. An analysis begins when a user uploads files to the Frontend.
2. Frontend checks for existing files and results in SQL. If needed, it stores the new files and calls asynchronouslyscan jobs on Brain.
3. Brain worker sends as much subtasks to Probe(s) as needed.
4. Probe workers process their jobs and send back results to Brain.
5. Brain sends results to Frontend.
3
IRMA Documentation, Release 2.0.4
1.3 Infrastructure Overview
A drawing is better than a lot of explanations (sometimes ;)
4 Chapter 1. Introduction
IRMA Documentation, Release 2.0.4
1.4 Hardware requirements
IRMA platform is divided in three major components: the Frontend, the Brain and one or multiple Probes.
These three components can be installed on a unique host or on multiple hosts, according to the kind of probes youare using.
The Frontend and the Brain must be installed on a GNU/Linux system1. We recommend to use a Debian Stabledistribution which is supported and known to work.
According to the kind of probes and their dependencies, each analyzers can be installed on a separate hosts or share
1 Theorically, it should be possible, with some efforts, to make IRMA work on Microsoft Windows systems as most of the components used forthe platform are known to work or to have equivalents on these systems.
1.4. Hardware requirements 5
IRMA Documentation, Release 2.0.4
the same host as far as they do not interfere with each other2. So forth, only Debian Stable and Microsoft Windows 8and 10 hosts have been tested.
We can not give you any specific numbers. On one hand we managed to run the whole IRMA platform on a singlemachine by hosting it with multiple systems inside virtual machines: this setup gives fairly high throughput as longas it has reasonable IO (ideally, SSDs), and a good amount of memory (our setup was an i7 cpu with 16 GB ramon regular drives (at least 200 GB required), on the other hand, a lighter version of the system with the three partstogether3 was successfully installed on a single virtual machine (1 GB of Ram and 4 virtual processors).
For a large company, in theory, given a single high-memory machine, with 16+ cores, and SSDs, you could run IRMAplatform and bear the workload load with reasonable response time.
1.5 Supported Analyzers
We have mainly focused our efforts on multiple anti-virus engines but we are working on other kind of “probes”. Weenumerate the analyzers that are bundled with IRMA probe application. Feel free to submit your own probes.
1.5.1 Antiviruses
So far, we have instrumented the following antiviruses from their CLI:
Probe Name Anti-Virus Name PlatformASquaredCmd Emsisoft Command Line Microsoft Windows CLIAvira Avira Microsoft Windows CLIAvastCoreSecurity Avast GNU/Linux CLIAVGAntiVirusFree AVG GNU/Linux CLIBitdefenderForUnices Bitdefender GNU/Linux CLIClamAV ClamAV GNU/Linux CLIComodoCAVL Comodo Antivirus for Linux GNU/Linux CLIDrWeb Dr.Web GNU/Linux CLIEsetNod32 Eset Nod32 Business Edition GNU/Linux CLIEScan eScan GNU/Linux CLIFProt F-Prot GNU/Linux CLIFSecure F-Secure GNU/Linux CLIGData G Data Antivirus Microsoft Windows CLIKaspersky Kaspersky Internet Security Microsoft Windows CLIMcAfeeVSCL McAfee VirusScan Command Line GNU/Linux - Microsoft Windows CLISophos Sophos GNU/Linux - Microsoft Windows CLISymantec Symantec Endpoint Protection Microsoft Windows CLIVirusBlokAda VirusBlokAda GNU/Linux CLIZoner Zoner Antivirus GNU/Linux CLI
1.5.2 External analysis platforms
So far, we query the following external analysis platforms:
2 For instance, we managed to host several GNU/Linux anti-viruses on an unique probe by preventing it to launch daemons at startup. This isdifficult for Microsoft systems on which it is not recommended to install multiple anti-viruses on a single host.
3 with a limited set of probes
6 Chapter 1. Introduction
IRMA Documentation, Release 2.0.4
Probe Name Analysis Platform DescriptionVirusTotal VirusTotal Report is searched using the sha256 of the file which is not sent
1.5.3 File database
So far, we query the following file databases:
ProbeName
Database Description
NSRL National Software Reference Li-brary
collection of digital signatures of known, traceable software ap-plications
1.5.4 Metadata
So far, we implemented the following analyzers:
Probe Name DescriptionStaticAnalyzer PE File analyzer adapted from Cuckoo SandboxPEiD PE File packer analyzerYara Checks if a file match yara rules
1.5. Supported Analyzers 7
IRMA Documentation, Release 2.0.4
8 Chapter 1. Introduction
CHAPTER 2
Automated Install
IRMA platform can be easily installed with a set of ansible roles and playbooks. It will help you to build, install ormaintain different setups.
2.1 Requirements
• Ansible 2.2.1.0;
2.2 Ansible scripts
Get IRMA ansible scripts on github:
$ git clone https://github.com/quarkslab/irma$ cd irma/ansible
2.3 Predefined Environments
There are 2 different IRMA setups available. Dev/Testing will be installed in one or multiple virtual machines whileproduction could be used to install IRMA on physical machines or virtual machines already setup:
2.3.1 Development environment
This environment has been designed to help you to modify IRMA’s components and redeploy and test them. In thissetup, everything is installed in a single virtual machine with sources rsync-ed between the host and the guest.
9
IRMA Documentation, Release 2.0.4
Requirements
• Vagrant 1.8 or higher has to be installed
• As the installation work only for Virtualbox, you will need to install it
• Rsync to synchronize directories from host to VMs
• Read the Ansible introduction
Run Vagrant and create your VMs
To initialize and provision the Virtualbox VM.
$ cd <IRMA_SRC_DIR>/ansible$ VM_ENV=your_environment_name vagrant up
The template will be downloaded automatically and configured using environments/dev.yml file.
Note: Optionally, if you want to use your own environment, create it in environments directory and run:
$ VM_ENV=your_environment_name vagrant up
Configure your .ini files
Note: You can bypass this step, as this provisioning is sync with default username and password used in (fron-tend|brain|probe) config files.
As your config/*.ini file are transferred from host to VMs, you will need to modify them individually in thefrontend, probe and brain directories to match the user and password defined in playbooks/group_vars/*.
Modify your host and open IRMA frontend
Then, for proper use, update your /etc/hosts file and add:
172.16.1.30 www.frontend.irma
Then, with your web browser, IRMA allinone is available at www.frontend.irma.
Sync files between host and guest
Once rsync is installed inside your virtual machine and your environment is correctly set. You could easily sync yourcode with:
$ vagrant rsync # or vagrant rsync-auto to automatically initiates an rsync# transfer when changes are detected
Then, reload the modified application.
10 Chapter 2. Automated Install
IRMA Documentation, Release 2.0.4
2.3.2 Production environment
This environment is used to install IRMA on production-ready Debian servers.
Requirements
• One or multiple 64-bit Debian 7 servers.
Preparing servers
Create an account that is going to be used to provision IRMA on the server via Ansible, or use one which has alreadybeen created. To speed up provisioning, you can:
• Authorize your SSH key for password-less authentication (optional):
# On your local machine$ ssh-copy-id user@hostname # -i if you want to select your identity file
• If you do not want to have to type your password for sudo command, consider adding your user to sudoers,using visudo command (optional):
user ALL=(ALL) NOPASSWD: ALL
Configure for your installation
Modify settings in playbooks/group_vars/all especially the default_ssh_keys: section. You will needto add your public keys for SSH password-less connection to the default irma server user.
Configuration file used by the brain, the frontend and the probes applications are generated with default values that arespecified in playbooks/group_vars/brain, playbooks/group_vars/frontend and playbooks/group_vars/probe respectively. Make sure to adapt xxx_deployment_configs variables accordingly toyour installation. It is recommended to change all the default passwords defined in group_vars/* configurationfiles (password variables for most of them).
Finally, you will need to customize the hosts/example file and adapt it to describe your own server infrastruc-ture. There is three sections, one for each server role (frontend, brain, probe). Please refer to Ansible Inventorydocumentation for the expected syntax.
Run the Ansible Playbook
To run the main playbook with the hosts/example file you have defined, use the following command. Ansiblewill ask you the sudo password (-K option).
$ cd <IRMA_SRC_DIR>/ansible$ ansible-playbook -i ./hosts/example playbooks/playbook.yml -u <your_sudo_username> -→˓K
To run one or more specific actions and avoid running all the playbook, you can use tags. For example, if you want tore-provision Nginx, run the same command, but append --tags=nginx. You can combine multiple tags separatedwith commas.
2.3. Predefined Environments 11
IRMA Documentation, Release 2.0.4
Deploy a new version of IRMA
Assuming that you have already provisioned and deployed a version of IRMA, which you want to upgrade, you willneed to run the deployment script:
$ ansible-playbook -i ./hosts/example ./playbooks/deployment.yml -u irma
Make sure to replace irma with the default user if you have changed it in the group_vars/all file.
Access your IRMA installation
Once the provisioning and the deployment have finished, you should be able to perform scans and get their results viathe frontend hostname you specified.
2.4 Using Debian repos
If you planned to use only Debian official repository, you’ll need to change in playbooks/group_vars/all:
default_use_debian_repo: yes # no is the default value
12 Chapter 2. Automated Install
CHAPTER 3
Manual Installation
Warning: Manual installation needs an update since 1.5.3
as all repositories layout have changed. Try Automated install or ask for help on IRC.
In some specific cases where automatic install is not possible, you should go with manual installation.
Each major component of the IRMA platform comes with their own python-based application. As the Brain is thenerve center of the whole platform, it is recommended to install it first before installing other components. One canthen install either the Frontend or the Probes he wants.
Please refer to specific documentation for the manual installation process for each component.
3.1 Brain
The Brain is a python-based application that only dispatches analysis requests from different frontends1 to the avail-able Probes. Analyses are scheduled by the Brain on Probes through Celery, an open source task framework forPython.
3.1.1 Architecture
Let us recall first the inner architecture of the Brain. It uses multiple technologies with a specific purpose each:
• a Celery worker that handles scan requests from Frontends and results returned by the Probes.
• a RabbitMQ server used by Celery as a backend and as a broker for task queues and job queues used to scheduletasks for Probes (for scan jobs) and the Frontend (for scan results).
• an SFTP server where files to be scanned are uploaded by Frontends and downloaded by Probes,
1 This feature is not ready yet, we are currently working on its implementation.
13
IRMA Documentation, Release 2.0.4
3.1.2 Installing and Configuring RabbitMQ
The following explains how to install and to setup RabbitMQ for your setup.
Install RabbitMQ
RabbitMQ server can be installed with this command on Debian:
$ sudo apt-get install rabbitmq-server
Configuring RabbitMQ
RabbitMQ serves all components of IRMA platform. Each component has their own virtual host (i.e., message queuewhere to get data), a username and a password.
To configure RabbitMQ for IRMA platform, you have to create users, vhosts and add permissions for each user tocorresponding vhost. To easily create virtual hosts and users, we provide scripts in extras/ directory:
$ sh ./extras/scripts/rabbitmq/rmq_adduser.shUsage: sudo rmq_adduser.sh <user> <password> <vhost>
For instance, to create 3 users with the following parameters, one can do:
User-name
Password VirtualHost
Command
brain brain-rmq-password
mqbrain sudo ./extras/scripts/rabbitmq/rmq_adduser.sh brainbrain-rmq-password mqbrain
probe probe-rmq-password
mqprobe sudo ./extras/scripts/rabbitmq/rmq_adduser.sh probeprobe-rmq-password mqprobe
fron-tend
frontend-rmq-password
mqfron-tend
sudo ./extras/scripts/rabbitmq/rmq_adduser.shfrontend frontend-rmq-password mqfrontend
The script simply execute the following three commands:
$ sudo rabbitmqctl add_user <username> <password>$ sudo rabbitmqctl add_vhost <vhostname>$ sudo rabbitmqctl set_permissions -p <vhostname> <username> ".*" ".*" ".*"
Warning: Important
Make sure to note down the username, the password and the virtual host you just defined. You will be asked toretype them on each application configuration file (brain, frontend and probe)
Note: Disclaimer
Please ensure that only trusted sources can communicate with your RabbitMQ server, by setting up firewall rules forinstance, as your RabbitMQ may be exposed to Internet.
14 Chapter 3. Manual Installation
IRMA Documentation, Release 2.0.4
Verifying RabbitMQ configuration
We can verify that the RabbitMQ server has taken into account our modifications with some commands:
Checking for vhosts
$ sudo rabbitmqctl list_vhostsListing vhosts ...mqbrain/mqfrontendmqprobemqadmin...done.
If the defined virtual host are not listed by the above command, please execute once more the script.
Checking for users
$ sudo rabbitmqctl list_usersListing users ...probe []brain []frontend []...done.
If the defined users are not listed by the above command, please execute once more the script.
Changing password
If you do not remember the password you just typed, you can change it with rabbitmqctl command:
$ sudo rabbitmqctl change_password brain brain-rmq-passwordChanging password for user "brain" ......done.
Restarting the service
You may want to restart the service. Thus, the following command can be done:
$ sudo invoke-rc.d rabbitmq-server restart
3.1.3 Installing and configuring sftp
A SFTP server is used to transfer files Frontends and meant to be analyzed by Probes. We describe in the followinghow to set it up.
3.1. Brain 15
IRMA Documentation, Release 2.0.4
Installing sshd
sshd should already been installed but if not install it with:
$ sudo apt-get install openssh-sftp-server
Creating FTP specific users and groups
$ sudo groupadd sftpusers$ sudo useradd -g sftpusers -M -s /etc <frontend username>$ sudo useradd -g sftpusers -Match -s /etc <probe username>$ sudo passwd frontend
<enter frontend password>$ sudo passwd probe
<enter probe password>
Configure sshd
Edit /etc/sshd_config
Subsystem sftp internal-sftpMatch User <frontend username>
AllowTcpForwarding noChrootDirectory /sftp/<frontend username>ForceCommand internal-sftp -u 2PermitTunnel noX11Forwarding no
Match User <probe username>AllowTcpForwarding noChrootDirectory /sftpForceCommand internal-sftp -u 2PermitTunnel noX11Forwarding no
Create SFTP directories
The frontends need an account with /sftp/<frontend-name>/uploads as home directory and a single accountis shared between probes ( uploads directory is needed as in chrooted environment home is not writeable). The laterneeds to access to all frontends, thus the associated home directory is simply /sftp/. For instance, a frontend namedfrontend, execute the following to create the directories:
$ sudo mkdir -pv /sftp/frontend/uploadsmkdir: created directory `/sftp'mkdir: created directory `/sftp/frontend'mkdir: created directory `/sftp/frontend/uploads'$ sudo chown -R root:sftpusers /sftp$ sudo chmod 0750 /sftp$ sudo chmod 0750 /sftp/frontend$ sudo chown -R frontend:sftpusers /sftp/frontend/uploads$ sudo chmod -R 0775 /sftp/frontend/uploads
16 Chapter 3. Manual Installation
IRMA Documentation, Release 2.0.4
Restart the service
You may want to restart the service:
$ sudo invoke-rc.d sshd restart
3.1.4 Installation
The Brain must be installed on a GNU/Linux distribution. With some efforts, it should be possible to run it on aMicrosoft Windows system, but this has not been tested yet.
This section describes how to get the source code of the application for the Brain and to install it.
Pre-requisites
We assume that you have a command line interface on your system with the following tools installed:
• python 3.4.x and newer
• python3-pip (see Install pip for the recommended way to install pip package manager)
Furthermore, we assume that you have created an user irma that will be used to run the python application.
Note: On GNU/Linux, one can create the user (and the group) irma with:
$ sudo adduser --system --no-create-home --group irma
Installation on GNU/Linux
On GNU/Linux system, we will assume that the code for the Brain will be installed in /opt/irma/irma-brain/current directory, which is the configured default installation directory.
The source code is hosted on github.com. One can fetch an up-to-date version with the following commands. Let usnote that there is a common submodule named common that is a soft-link (lib), do not forget the dereferenceoption of cp (-L):
# If src repository not already cloned$ git clone https://github.com/quarkslab/irma /opt/irma/src$ cp -rL /opt/irma/src/brain /opt/irma/irma-brain/current
We need few dependencies for future steps:
$ sudo apt-get install python3-virtualenv python3-dev
then, we need to install python dependencies manually in a virtualenv :
$ virtualenv --system-site-packages /opt/irma/irma-brain/current/venv$ /opt/irma/irma-brain/current/venv/bin/pip install -r /opt/irma/irma-brain/current/→˓requirements.txt[...]
If everything has gone well, you should have installed the python application on your system. The next step is to installthe other components it relies on and finally to configure it for your platform.
3.1. Brain 17
IRMA Documentation, Release 2.0.4
3.1.5 Configuration
The configuration file is located at config/brain.ini in the installation directory. Update it with your specificinfo.
Note: Detailed meaning of each field in config/brain.ini:
Section Key Type Default Descriptionlog syslog integer 0 enable rsyslog (experimental)
prefix string irma-brain: prefix to append to rsyslog entriesdebug boolean False | enable Debug logsql_debug boolean False | enable SQL debug log
celery_options concurrency integer 0 number of concurrent workers (0means nb of cores)
soft_time_limit integer 300 (sec) time limit before task soft interrupttime_limit integer 1500 (sec) time limit before task is killed
broker_brain host string hostname for the RabbitMQ serverport integer 5672 port for the RabbitMQ servervhost string virtual host configured for brainusername string username used for brain on the Rab-
bitMQ serverpassword string password used for brain on the Rab-
bitMQ serverqueue string queue to poll new tasks on the Rab-
bitMQ serverbroker_probe host string hostname for the RabbitMQ server
port integer 5672 port for the RabbitMQ servervhost string virtual host configured for probesusername string username used for probes on the
RabbitMQ serverpassword string password used for probes on the
RabbitMQ serverqueue string queue to poll new tasks on the Rab-
bitMQ serverbroker_frontend host string hostname for the RabbitMQ server
port integer 5672 port for the RabbitMQ servervhost string virtual host configured for frontendusername string username used for frontend on the
RabbitMQ serverpassword string password used for frontend on the
RabbitMQ serverqueue string queue to poll new tasks on the Rab-
bitMQ serversqldb dbms string sqlite dbapi engine
dialect string sqlalchemy dialectusername string database usernamepassword string database passwordhost string database hostdbname string /var/irma/
db/brain.dbdatabase name
tables_prefix string database tables prefixContinued on next page
18 Chapter 3. Manual Installation
IRMA Documentation, Release 2.0.4
Table 3.1 – continued from previous pageSection Key Type Default Descriptionftp protocol string “sftp” choose File Transfer Protocol
(“sftp” or “ftps”)ftp_brain host string hostname for the FTP server
port integer 21 port for the FTP serverauth string “password”| SFTP authentication method (“password”
or “key”)key_path string sftp private key absolute path
username string username used by probe on the FTP server
password string password used by the probe on theFTP server
interprocess_lock
path string /var/run/lock/irma-brain.lock
Concurrency file lock
ssl_config activate_ssl boolean False Enable RabbitMQ sslca_certs string RabbitMQ SSL certskeyfile string RabbitMQ SSL keyfilecertfile string RabbitMQ SSL certfile
Generate a SQLite database for scan tracking
You could easily generate the user database by running the following command. The path of the database is takenfrom the configuration file and the folder where the database is going to be stored must be created beforehand.
Note: The default path for the database is /var/irma/db/ make sure it exists before creating user database.
$ cd /opt/irma/irma-brain/current/$ ./venv/bin/python -m scripts.create_userusage: create_user <username> <rmqvhost> <ftpuser>
with <username> a string<rmqvhost> the rmqvhost used for the frontend<ftpuser> the ftpuser used by the frontend
example: create_user test1 mqfrontend frontend
To create an entry in the database for the frontend named frontend and which uses the mqfrontend virtual hoston the RabbitMQ server, simply run the following commands:
$ ./venv/bin/python -m scripts.create_user frontend mqfrontend frontend
Note: There is a limitation due to SQLite. The folder where the database is stored, plus the database file must bewritable by the user running the worker:
$ sudo chown irma:irma /var/irma/db/brain.db$ sudo chmod a+w /opt/irma/irma-brain
3.1. Brain 19
IRMA Documentation, Release 2.0.4
3.1.6 Installation Checks
Celery Workers
Before going further, you should check that the python applications manages to communicate with the RabbitMQserver through Celery. To ensure that, from the installation directory, execute both Celery workers:
On GNU/Linux:
$ cd /opt/irma/irma-brain/current$ ./venv/bin/python -m brain.scan_tasks
-------------- celery@brain v3.1.23 (Cipater)---- **** -------- * *** * -- Linux-3.16.0-4-amd64-x86_64-with-debian-8.2-- * - **** ---- ** ---------- [config]- ** ---------- .> app: scantasks:0x7fbd7ee4c350- ** ---------- .> transport: amqp://brain:**@127.0.0.1:5672/mqbrain- ** ---------- .> results: amqp://- *** --- * --- .> concurrency: 2 (prefork)-- ******* ------- ***** ----- [queues]-------------- .> brain exchange=celery(direct) key=brain
[2016-07-15 15:00:36,155: WARNING/MainProcess] celery@brain ready.
This worker is responsible for splitting the whole scan job in multiples job per probe per file.
$ cd /opt/irma/irma-brain/current$ ./venv/bin/python -m brain.results_tasks
-------------- celery@brain v3.1.23 (Cipater)---- **** -------- * *** * -- Linux-3.16.0-4-amd64-x86_64-with-debian-8.2-- * - **** ---- ** ---------- [config]- ** ---------- .> app: resultstasks:0x7fa68f9aa590- ** ---------- .> transport: amqp://probe:**@127.0.0.1:5672/mqprobe- ** ---------- .> results: disabled://- *** --- * --- .> concurrency: 2 (prefork)-- ******* ------- ***** ----- [queues]-------------- .> results exchange=celery(direct) key=results
[2016-07-15 14:59:01,799: WARNING/MainProcess] celery@brain ready.
And this worker is responsible for collecting and tracking results.
If your Celery worker does not output something similar to the above output, you should check twice the parametersin the application configuration file you are using.
SFTP accounts
Try to login as frontend and upload a sample file in home dir (should raise an error as it is non writeable) then inuploads dir.
20 Chapter 3. Manual Installation
IRMA Documentation, Release 2.0.4
$ sftp frontend@localhostfrontend@localhost's password:Connected to localhost.sftp> put testUploading test to /testremote open("/test"): Permission deniedsftp> lsuploadssftp> cd uploads/sftp> put testUploading test to /uploads/testtest→˓100% 10 0.0KB/s 00:00
3.1.7 Running Brain applications at startup
We have ensured that the freshly installed Brain is ready to be integrated to your IRMA platform. Now, we can go astep further and make it launch automatically all daemons when the system starts up so you will not need to relaunchthem manually every time.
We are using supervisor to manage our celery daemons.
Installing Supervisor
Install it with pip (python2):
$ sudo pip install supervisor
We will create two new applications called scan_app and result_app.
Configure Scan APP
Create a file called scan_app located at /etc/supervisor/conf.d with the following content:
[program:scan_app]
numprocs = 1stopwaitsecs = 600killasgroup = Truestderr_logfile = /var/log/supervisor/scan_app.logcommand = /opt/irma/irma-brain/current/venv/bin/python -m brain.scan_tasksuser = irmaautostart = Truedirectory = /opt/irma/irma-brain/currentstdout_logfile = /var/log/supervisor/scan_app.log
Configure Result APP
Create a file called result_app located at /etc/supervisor/conf.d with the following content:
[program:result_app]
numprocs = 1
3.1. Brain 21
IRMA Documentation, Release 2.0.4
stopwaitsecs = 600killasgroup = Truestderr_logfile = /var/log/supervisor/result_app.logcommand = /opt/irma/irma-brain/current/venv/bin/python -m brain.results_tasksuser = irmaautostart = Truedirectory = /opt/irma/irma-brain/currentstdout_logfile = /var/log/supervisor/result_app.log
Ensure supervisor will read our files by checking /etc/supervisor/supervisord.conf last lines should be:
[...][include]files = /etc/supervisor/conf.d/*
Restart supervisor:
$ sudo service supervisor restart
Restart applications if needed (should be done automatically):
$ sudo supervisorctl restart allscan_app: stoppedresult_app: stoppedscan_app: startedresult_app: started
3.2 Frontend
The Frontend handles scan submission to the Brain, stores the results of the scanned files. These results can bedisplayed through a web graphical user interface or via the command line interface.
3.2.1 Architecture
Let us recall first the inner architecture of the Frontend. It uses multiple technologies with each a specific purpose:
• A client through which an user submits a file and get the analysis results. There are two clients bundled in therepository: a web user interface and a command-line client.
• A python-based restful API, served by a nginx web server and a uWSGI application server. It gets the results ofa file scan by querying a database.
• A worker that will handle scan submission to the Brain and store the results of analyzes scheduled by the Brain.The worker relies on Celery, a python-based distributed task queue.
• An database server (PostgreSQL) is used to store results of analyzes made on each file submitted either by theweb graphical interface or the CLI client.
3.2.2 Installation
The Frontend must be installed on a GNU/Linux system. With some efforts, it should be possible to run it on aMicrosoft Windows system, but this has not been tested yet.
This section describes how to get the source code of the application and to install it.
22 Chapter 3. Manual Installation
IRMA Documentation, Release 2.0.4
Pre-requisites
We assume that you have a command line interface on your system with the following tools installed:
• python 3.4.x and newer
• python3-pip (see Install pip for the recommended way to install pip package manager)
Furthermore, we assume that you have created an user irma that will be used to run the python application.
Note: On GNU/Linux, one can create the user (and the group) irma with:
$ sudo adduser --system --no-create-home --group irma
Installation on GNU/Linux
On GNU/Linux system, we will assume that the code for the Frontend will be installed in /opt/irma/irma-frontend/current directory, which is the configured default installation directory.
The source code is hosted on github.com. One can fetch an up-to-date version with the following commands. Let usnote that there is a common submodule named common that is a soft-link (lib), do not forget the dereferenceoption of cp (-L):
$ git clone https://github.com/quarkslab/irma /opt/irma/src$ cp -rL /opt/irma/src/frontend /opt/irma/irma-frontend/current
We need few dependencies for future steps:
$ sudo apt-get install python3-virtualenv python3-dev
then, we need to install python dependencies manually in a virtualenv :
$ virtualenv --system-site-packages /opt/irma/irma-frontend/current/venv$ /opt/irma/irma-frontend/current/venv/bin/pip install -r /opt/irma/irma-frontend/→˓current/requirements.txt[...]
Building the web client
The first step that should be performed when installing the frontend from the code source is to build the web clientwhich is composed of static HTML files mixed with javascript web framework (in particular, AngularJS).
One must install first some tools that are necessary to build the whole web client:
$ curl -sL https://deb.nodesource.com/setup | sudo bash -[...]$ sudo apt-get install -y nodejs[...]$ curl -sL https://www.npmjs.org/install.sh | sudo bash -[...]
Once the tools installed, you can build the static files for the web user interface with the following commands, executedfrom the installation directory:
3.2. Frontend 23
IRMA Documentation, Release 2.0.4
$ cd irma-frontend/current/web$ sudo npm install$ sudo node_modules/.bin/bower install --allow-root$ sudo node_modules/.bin/gulp dist
A new directory or an updated directory web/dist should appear now with HTML and javascript files that havebeen minified and “obfuscated” by gulp. For more details on the web interface, please refer to the dedicated chapter.
If everything has gone well, you should have installed the python application on your system. The next step is toconfigure it for your platform and to install the other components it relies on.
3.2.3 Configuration
The configuration file is located at config/frontend.ini in the installation directory.
Note: Detailed meaning of each field in config/frontend.ini:
Section Key Type Default Descriptionlog syslog integer 0 enable rsyslog (ex-
perimental)prefix string irma-frontend: prefix to append to
rsyslog entriesdebug boolean False enable Debug logsql_debug boolean False enable SQL debug
logsqldb username string database username
password string database passwordhost string database hostdbname string database nametables_prefix string database tables pre-
fixsamples_storage path string Samples storage
pathcelery_brain timeout integer 60 (sec) time before consid-
ering that the brainhas timed-out
celery_frontend timeout integer 30 (sec) time before consid-ering that the fron-tend has timed-out
celery_options concurrency integer 0 number of con-current workers (0means nb of cores)
soft_time_limit integer 300 (sec) time limit beforetask soft interrupt
time_limit integer 1500 (sec) time limit beforetask is killed
beat_schedule string /var/irma/frontend_beat_schedule
celery beat schedulefile
broker_brain host string hostname for theRabbitMQ server
port integer 5672 port for the Rab-bitMQ server
Continued on next page
24 Chapter 3. Manual Installation
IRMA Documentation, Release 2.0.4
Table 3.2 – continued from previous pageSection Key Type Default Description
vhost string virtual host config-ured for brain
username string username used forbrain on the Rab-bitMQ server
password string password used forbrain on the Rab-bitMQ server
queue string queue to poll newtasks on the Rab-bitMQ server
broker_frontend host string hostname for theRabbitMQ server
port integer 5672 port for the Rab-bitMQ server
vhost string virtual host config-ured for this fron-tend
username string username used forthis frontend on theRabbitMQ server
password string password used forthis frontend on theRabbitMQ server
queue string queue to poll newtasks on the Rab-bitMQ server
ftp protocol string “sftp” choose File TransferProtocol (“sftp” or“ftps”)
ftp_brain host string hostname for theFTP server
port integer 22 port for the FTPserver
auth string “password” SFTP authenti-cation method(“password” or“key”)
key_path string sftp private key ab-solute path
username string username used bythis frontend on theFTP server
password string password used bythis frontend on theFTP server
cron_clean_file_age clean_db_file
_max_age
integer 0 remove file after Xdays (0 means dis-abled)
Continued on next page
3.2. Frontend 25
IRMA Documentation, Release 2.0.4
Table 3.2 – continued from previous pageSection Key Type Default Description
clean_db_cron_hour
string 0 cron hour settings
clean_db_cron_minute
string 0 cron minute settings
clean_db_scan_day_of_week
string * cron day of weeksettings
cron_clean_file_size clean_fs_max
_size
string “0” space’smaxi-mumsize(bytes)dedi-cated tothe filesystemcronhoursettings
clean_fs_size_cron_hour
string *
clean_fs_size_cron_minute
string 0 cron minute settings
clean_fs_size_cron_day_of_week
string * cron day of weeksettings
interprocess_ lock path string /var/run/lock/ir ma-frontend.lock
Concurrency filelock
ssl_config activate_ssl boolean False Enable RabbitMQssl
ca_certs string RabbitMQ SSLcerts
keyfile string RabbitMQ SSL key-file
certfile string RabbitMQ SSLcertfile
Note: The default path for samples is /var/irma/samples/ make sure it exists with correct rights for irma user beforelaunching your first scan.
3.2.4 Installing and configuring uWSGI
The restful API is served by an uWSGI application server. This section will explain how to install an uWSGI serverand configure it for the Frontend.
26 Chapter 3. Manual Installation
IRMA Documentation, Release 2.0.4
Installation
On Debian, installing uWSGI to serve a python application is pretty straightforward:
$ cd /opt/irma/irma-frontend/current$ ./venv/bin/pip install https://projects.unbit.it/downloads/uwsgi-lts.tar.gz[...]
Please refer to the documentation of your preferred distribution to install an uWSGI server on it.
3.2.5 Installing SQL server
Note: Since version 1.5.0, IRMA required a postgreSQL server as we are using JSONB column.
The Frontend relies on a PostgreSQL database to keep track of all scans info. On Debian, one can install a PostgreSQLserver in few commands:
Install it with the following dependencies:
$ sudo apt-get install postgresql-9.4 python3-psycopg2[...]
You need to manually create the configured database and db user, the tables are automatically created on uwsgi startup.
3.2.6 Running Frontend applications at startup
We have ensured that the freshly installed Frontend is ready to be integrated to your IRMA platform. Now, we can goa step further and make it launch automatically all daemons when the system starts up so you will not need to relaunchthem manually every time.
We are using supervisor to manage our python daemons (uswgi for IRMA API and celery for IRMA frontend workers).
Installing Supervisor
Install it with apt:
$ sudo apt-get install python3-virtualenv python3-dev$ sudo pip install supervisor (python2 only)
We will create two new applications called frontend_api and frontend_app. Frontend_api is the python uwsgi serverand frontend_app the frontend celery daemon.
Configure Frontend API
Create a file called frontend_api located at /etc/supervisor/conf.d with the following content:
[program:frontend_api]
numprocs = 1redirect_stderr = Truestopwaitsecs = 600killasgroup = True
3.2. Frontend 27
IRMA Documentation, Release 2.0.4
stderr_logfile = /var/log/supervisor/frontend_api.logstopsignal = INTcommand = /opt/irma/irma-frontend/current/venv/bin/uwsgi -s 127.0.0.1:8081 --disable-→˓logging --master --workers 4 --need-app --chdir /opt/irma/irma-frontend/current --→˓home /opt/irma/irma-frontend/current/venv --python-path /opt/irma/irma-frontend/→˓current/venv --wsgi-file api/base.py --callable __hug_wsgi__ --lazy --offload-→˓threads 4user = irmaautostart = Truestdout_logfile = /var/log/supervisor/frontend_api.log
Configure Frontend APP
Create a file called frontend_app located at /etc/supervisor/conf.d with the following content:
[program:frontend_app]
numprocs = 1stopwaitsecs = 600killasgroup = Truestderr_logfile = /var/log/supervisor/frontend_app.logcommand = /opt/irma/irma-frontend/current/venv/bin/python -m api.tasks.frontend_appuser = irmaautostart = Truedirectory = /opt/irma/irma-frontend/currentstdout_logfile = /var/log/supervisor/frontend_app.log
Ensure supervisor will read our files by checking /etc/supervisor/supervisord.conf last lines should be:
[...][include]files = /etc/supervisor/conf.d/*
Restart supervisor:
$ sudo service supervisor restart
Restart applications if needed (should be done automatically):
$ sudo supervisorctl restart allfrontend_api: stoppedfrontend_app: stoppedfrontend_app: startedfrontend_api: started
3.2.7 Installing and configuring nginx
In the Frontend, we use a nginx web server to serve the uWSGI application and the static web site that query the APIin order to get results of scanned files and to present them to the user.
Installation
On Debian, installing nginx is done with few commands:
28 Chapter 3. Manual Installation
IRMA Documentation, Release 2.0.4
$ sudo apt-get install nginx[...]
Please refer to the documentation of your preferred distribution to install an uWSGI server on it.
Configuration
We provide several template scripts in the extras/ repository located at the root of the installation directory. Tem-plates for nginx are located in extras/nginx/. Copy the file to sites-available in nginx configurationfolder:
$ sudo cp extras/nginx/frontend /etc/nginx/sites-available/
The template is configured for a default installation of the frontend in /opt/irma/irma-frontend/current.You may need to edit it for you setup. In particular, please ensure that the root directive points to the folder web/dist (for a production ready version of the web site) or web/app (for a development version of the web site) inyour installation directory.
Activate the web site
By default, all websites in apps-available are not activated. One can enable the website described in frontendconfiguration file by creating a soft-link into the sites-enabled folder:
$ sudo ln -s /etc/nginx/sites-available/frontend /etc/nginx/sites-enabled/frontend
Note: The case of HTTPs
A template to set up a HTTPs server with nginx is also provided in the extras/nginx folder. Here is the way tosetup it:
$ sudo cp extras/nginx/frontend-https /etc/nginx/sites-available/$ sudo ln -s /etc/nginx/sites-available/frontend-https /etc/nginx/sites-enabled/→˓frontend-https$ sudo mkdir /etc/nginx/certificates/$ cd /etc/nginx/certificates/
Generate the required Diffie Hellman Ephemeral Parameters:
$ sudo openssl dhparam -out dhparam.pem 4096
Finally, generate a self signed certificate:
$ sudo openssl req -x509 -nodes -days 365 -newkey rsa:4096 -subj "/CN=$(hostname --→˓fqdn)/" -keyout frontend.key -out frontend.crt
Relaunch the service
The final step is to relaunch the service:
$ sudo invoke-rc.d nginx restart
3.2. Frontend 29
IRMA Documentation, Release 2.0.4
3.2.8 Installation Checks
Celery Workers
Before going further, you should check that the python application manages to communicate with the RabbitMQ serverand the Redis server through Celery. To ensure that, from the installation directory, execute the Celery worker:
On GNU/Linux:
$ cd /opt/irma/irma-frontend/current$ ./venv/bin/python -m api.tasks.frontend_app
-------------- celery@frontend v3.1.23 (Cipater)---- **** -------- * *** * -- Linux-3.2.0-4-amd64-x86_64-with-debian-7.6-- * - **** ---- ** ---------- [config]- ** ---------- .> app: frontend.tasks:0x1e18750- ** ---------- .> transport: amqp://probe:**@brain:5672/mqfrontend- ** ---------- .> results: disabled- *** --- * --- .> concurrency: 2 (prefork)-- ******* ------- ***** ----- [queues]-------------- .> frontend exchange=celery(direct) key=frontend
[2014-08-20 15:28:58,745: WARNING/MainProcess] celery@frontend ready.
If your Celery worker does not output something similar to the above output, you should check twice the parametersin the application configuration file you are using. Let us note that the Celery worker gives us useful information onthe analyzer that are enabled.
SFTP accounts
Defaut File Transport Protocol since v1.4.0 is now SFTP, you can check whether the configured account is valid.
$ sftp <user>@<hostname of the brain>
FTP-TLS accounts
Additionnally, if you have configured IRMA to use FTP-TLS, you can check whether the configured account is valid.On Debian, this can be done with the ftp-ssl package:
$ sudo apt-get install ftp-ssl[...]$ ftp-ssl <hostname of the brain>Connected to brain.220---------- Welcome to Pure-FTPd [privsep] [TLS] ----------220-You are user number 1 of 50 allowed.220-Local time is now 18:55. Server port: 21.220-This is a private system - No anonymous login220-IPv6 connections are also welcome on this server.220 You will be disconnected after 15 minutes of inactivity.Name (brain:root): frontend-ftp500 This security scheme is not implemented234 AUTH TLS OK.
30 Chapter 3. Manual Installation
IRMA Documentation, Release 2.0.4
[SSL Cipher DHE-RSA-AES256-GCM-SHA384]200 PBSZ=0200 Data protection level set to "private"331 User probe OK. Password requiredPassword: frontend-ftp-password230 OK. Current directory is /Remote system type is UNIX.Using binary mode to transfer files.ftp>
Restful API
One can verify that the restful API is up and running by querying a specific route on the web server or by checking thesystem logs:
$ curl http://localhost/api/v1.1/probes{"total": 9, "data": ["ClamAV", "ComodoCAVL", "EsetNod32", "FProt", "Kaspersky",→˓"McAfeeVSCL", "NSRL", "StaticAnalyzer", "VirusTotal"]}
$ sudo cat /var/log/supervisor/frontend_api.log[...]added /opt/irma/irma-frontend/current/venv/ to pythonpath.
*** uWSGI is running in multiple interpreter mode ***spawned uWSGI master process (pid: 3943)spawned uWSGI worker 1 (pid: 3944, cores: 1)spawned uWSGI worker 2 (pid: 3945, cores: 1)spawned uWSGI worker 3 (pid: 3946, cores: 1)spawned uWSGI worker 4 (pid: 3947, cores: 1)mounting frontend/api/base.py on /apimounting frontend/api/base.py on /apimounting frontend/api/base.py on /apimounting frontend/api/base.py on /apiWSGI app 0 (mountpoint='/api') ready in 0 seconds on interpreter 0x99a3e0 pid: 3945→˓(default app)WSGI app 0 (mountpoint='/api') ready in 0 seconds on interpreter 0x99a3e0 pid: 3946→˓(default app)WSGI app 0 (mountpoint='/api') ready in 0 seconds on interpreter 0x99a3e0 pid: 3944→˓(default app)WSGI app 0 (mountpoint='/api') ready in 0 seconds on interpreter 0x99a3e0 pid: 3947→˓(default app)
3.3 Probe
The Probes are python-based application that host a single or multiple analyzers. Each analyzer listens on a specificwork queue and waits for an analysis to be scheduled by the Brain through Celery, an open source task framework forPython. Python version should be at least 3.4 on linux, 3.5 on windows.
3.3.1 Architecture
Probes are mainly Celery workers that handle scan requests from Brain
3.3. Probe 31
IRMA Documentation, Release 2.0.4
3.3.2 Installation
Probes can be installed either on GNU/Linux and on Microsoft Windows systems. The installation procedure maydiffers between the two systems.
Downloading the source code from github.com
The source code is hosted on github.com. One can fetch an up-to-date version with the following commands. Let usnote that there is a common submodule named common that is a soft-link (lib), do not forget the dereferenceoption of cp1 (-L):
# If src repository not already cloned$ git clone https://github.com/quarkslab/irma /opt/irma/src$ cp -rL /opt/irma/src/probe /opt/irma/irma-probe/current
We assume that you have a command line interface on your system with the following tools installed:
• python 3.4.x and newer on Linux
• python 3.5.x and newer on Windows (see Python for Windows for prebuild MSI installer)
• python3-pip (see Install pip for the recommended way to install pip package manager)
Furthermore, we assume that you have created an user irma that will be used to run the python application.
Note: On GNU/Linux, one can create the user (and the group) irma with:
$ sudo adduser --system --no-create-home --group irma
Installation on Microsoft Windows
On windows system, we will assume that the code for the Probe will be installed at the root of the C:\ drive, namelyin C:\irma\irma-probe\current.
$ C:\Python27\Scripts\pip.exe install virtualenv$ C:\Python27\Scripts\virtualenv C:\irma\irma-probe\current\venv
$ cd C:\irma\irma-probe\current\$ .\venv\Scripts\pip.exe install -r requirements.txt[...]
Installation on GNU/Linux
On GNU/Linux system, we will assume that the code for the Probe will be installed in /opt/irma/irma-probe/current directory.
The source code is hosted on github.com. One can fetch an up-to-date version with the following commands. Let usnote that there is a common submodule named common that is a soft-link (lib), do not forget the dereferenceoption of cp (-L):
1 On Microsoft Windows, a Linux-like lightweight command line interface can be installed by installing MSYS or Git for windows.
32 Chapter 3. Manual Installation
IRMA Documentation, Release 2.0.4
# If src repository not already cloned$ git clone https://github.com/quarkslab/irma /opt/irma/src$ cp -rL /opt/irma/src/probe /opt/irma/irma-probe/current
We need few dependencies for future steps:
$ sudo apt-get install python3-virtualenv python3-dev
then, we need to install python dependencies manually in a virtualenv :
$ virtualenv --system-site-packages --python=/usr/bin/python3 /opt/irma/irma-probe/→˓current/venv$ /opt/irma/irma-probe/current/venv/bin/pip install -r /opt/irma/irma-probe/current/→˓requirements.txt[...]
If everything has gone well, the python application is now installed on your system. The next step is to configure it foryour platform and to enable the analyzers you need.
3.3.3 Configuration
The configuration file is config/probe.ini located in the installation directory.
Note: We recall in the following the meaning of each field in config/probe.ini:
Section Key Type Default Descriptionlog syslog integer 0 enable rsyslog (experimental)
prefix string irma-probe: prefix to append to rsyslog entriescelery_options concurrency integer 0 number of concurrent workers (0
means nb of cores)soft_time_limit integer 300 (sec) time limit before task soft interrupttime_limit integer 1500 (sec) time limit before task is killed
broker probe host string hostname for the RabbitMQ serverport integer 5672 port for the RabbitMQ servervhost string virtual host configured for probesusername string username used for probes on the
RabbitMQ serverpassword string password used for probes on the
RabbitMQ serverqueue string queue to poll new tasks on the Rab-
bitMQ serverftp_brain host string hostname for the FTP server
port integer 21 port for the FTP serverauth string “password”| SFTP authentication method (“password”
or “key”)key_path string sftp private key absolute path
username string username used by probe on the FTPserver
password string password used by the probe on theFTP server
3.3. Probe 33
IRMA Documentation, Release 2.0.4
3.3.4 Enabling Analyzers
The python application auto-discovers available analyzers. As long as the requirements are fulfilled for an analyzer,it is automatically discovered and enabled. By default, no analyzers are available. The following enumerates therequirements to enable each analyzer bundled with the application.
ClamAV - GNU/Linux
To enable ClamAV on Debian Stable distribution, one should install several packages:
$ sudo apt-get install clamav-daemon[...]$ sudo freshclam[...]$ sudo service clamav-daemon restart[...]
ComodoCAVL - GNU/Linux
Comodo Antivirus for Linux can be downloaded from the Comodo’s download page. The following instruction enableto install the Debian package. By default, the binaries are installed in /opt/COMODO/ directory. As ComodoCAVLis not packaged for the current Debian Stable distribution, we must install it manually:
$ sudo apt-get install binutils$ ar x cav-linux_1.1.268025-1_iXXX.deb$ sudo tar xvf ~/data.tar.gz -C /[...]
Updates for the database can be performed with the following command:
$ XAUTHORITY="$HOME/.Xauthority" sudo /opt/COMODO/menu/comodo-updater[...]
Note: Dependencies to update the database
To be able to update the database using the updater provided with Comodo Antivirus for Linux, some dependeciesneeded for the graphical interface may be missing from the distribution. On Debian Stable system, one can installthem with:
$ sudo apt-get install libqt4-sql libqt4-network libqtgui4
EsetNod32 - GNU/Linux
One can request a trial version of Eset Nod32 Business Edition for Linux on the Eset download page. Once down-loaded, the anti-virus can be installed with the following commands on Debian. Just follow the typical installation onthe GUI:
$ sudo chmod u+x eset_nod32av_64bit_en.linux$ sudo apt-get install libgtk2.0-0 libc6-i386$ sudo ./eset_nod32av_64bit_en.linux[...]
34 Chapter 3. Manual Installation
IRMA Documentation, Release 2.0.4
Binaries should be installed in /opt/eset/esets directory. Updates are performed from the GUI:
$ XAUTHORITY="$HOME/.Xauthority" sudo /opt/eset/esets/bin/esets_gui
Note: Disabling the antivirus daemon
To avoid the anti-virus to protect your system at startup, we deliberately disabled the script used to launch the anti-virusearly at boot:
$ sudo service esets stop$ sudo mv /etc/init.d/esets /etc/init.d/esets.disable
F-Prot - GNU/Linux
A copy of F-PROT anti-virus for Linux workstations is available on the F-PROT download page.
The binaries should be installed in /usr/local/f-prot to make the python application detect it automatically.
$ sudo tar xvf fp-Linux.x86.32-ws.tar.gz -C /usr/local/
To launch an update, a configuration step is mandatory:
$ sudo cp /usr/local/f-prot/f-prot.conf.default /etc/f-prot.conf
An update is launched with:
$ sudo ./fpupdateERROR: ld.so: object 'libesets_pac.so' from /etc/ld.so.preload cannot be preloaded:→˓ignored.[...]
Note: Error
If you see an error message like:
DownloadingWarning: Network - Connection failed (18), trying again...Downloading updateError: Update - Bad mergefile
Just relaunch the script.
Note: Dependencies to update the database
To be able to update the database using the updater provided with Comodo install them with:
$ sudo apt-get install libc6-i386
McAfeeVSCL - GNU/Linux or Microsoft Windows
A free evaluation of McAfee VirusScan Command Line can be downloaded from the editor download page.
3.3. Probe 35
IRMA Documentation, Release 2.0.4
The binaries should be installed in /usr/local/uvscan/ on GNU/Linux system and must be installed inC:\VSCL on Windows Systems. Let us note that updates must be performed manually. Anti-virus databases andengines can be downloaded here.
After downloading McAfee Virus Scan archive, create /usr/local/uvscan and extract the archive in it:
$ sudo mkdir /usr/local/uvscan$ sudo tar xvf vscl-XXX.tar.gz -C /usr/local/uvscan # replace using your values$ sudo chmod +x /usr/local/uvscan/uvscan
Extract also, using unzip program, the database:
$ sudo unzip avvepo7536dat.zip -d /usr/local/uvscan$ cd /usr/local/uvscan$ sudo unzip avvdat-XXXX.zip
Sophos - GNU/Linux or Microsoft Windows
A free evaluation of Sophos Endpoint Antivirus can be downloaded from the editor download page. You should receiveby email a username and a password to authenticate for updates.
It should be detected automatically by IRMA if the anti-virus is installed in its default installation directory.
On GNU/Linux:
• Download the archive for Linux, then execute:
$ tar zxf sav-linux-9-i386.tgz$ ./sophos-av/install.sh /opt/sophos-av --acceptlicence --autostart=False --→˓enableOnBoot=False --automatic --ignore-existing-installation$ /opt/sophos-av/bin/savconfig set EnableOnStart false$ /opt/sophos-av/bin/savconfig set PrimaryUpdateSourcePath "sophos:"$ /opt/sophos-av/bin/savconfig set PrimaryUpdateUsername "<your_username_from_email>"$ /opt/sophos-av/bin/savconfig set PrimaryUpdatePassword "<your_password_from_email>"$ /opt/sophos-av/bin/savupdate
Kaspersky - Microsoft Windows
A free evaluation of Kaspersky Internet Security can be requested on the editor download page. It should be detectedautomatically if the anti-virus is installed in its default installation directory.
Symantec - Microsoft Windows
The procedure to install a trial version of Symantec Endpoint Protection is particularly tedious. We will not documentits installation.
G Data - Microsoft Windows
A trial version of G Data is available on the editor download page<https://www.gdata.de/kundenservice/downloads.html>. It should be detected automatically if the anti-virus isinstalled in its default installation directory.
36 Chapter 3. Manual Installation
IRMA Documentation, Release 2.0.4
VirusTotal - GNU/Linux or Microsoft Windows
The VirusTotal analyzer can be installed easily by downloading the python packages it depends on and by modifyingits configuration file. From the installation directory, one can execute:
On GNU/Linux:
$ pip install -r modules/external/virustotal/requirements.txt[...]
then update configuration file located at modules/external/virustotal/config.ini.
Note: Meaning of the fields in the configuration file
Section Option Type Default DescriptionVirusTotal apikey string api_key used to query VirusTotal APIVirusTotal private boolean use private api (need a private api key)
NSRL - GNU/Linux
The National Software Reference Library can be downloaded on the NIST’s web page. The provided files are storedin the RDS (Reference Data Set) format. To use this analyzer, one must build first the database. We use LevelDB asfast key-value storage library.
To build the dabatase, one must install first the dependencies:
$ pip install -r modules/database/nsrl/requirements.txt
A (not optimized and very slow) helper script is provided to build the database:
$ mkdir /home/irma/leveldb$ python -m modules.database.nsrl.nsrl create -t os NSRLOS.txt /home/irma/leveldb/os_→˓db$ python -m modules.database.nsrl.nsrl create -t manufacturer NSRLMfg.txt /home/irma/→˓leveldb/mfg_db$ python -m modules.database.nsrl.nsrl create -t product NSRLProd.txt /home/irma/→˓leveldb/prod_db$ python -m modules.database.nsrl.nsrl create -t file NSRLFile.txt /home/irma/leveldb/→˓file_db
Finally, one must indicate to the analyzer where to find the files for the database by filling the configuration file locatedat modules/database/nsrl/config.ini.
Note: Meaning of the fields in the configuration file
Section Option Type Default DescriptionNSRL nsrl_os_db string location of the OS database
nsrl_mfg_db string location of the Manufacturer databasensrl_file_db string location of the File databasensrl_prod_db string location of the Product database
3.3. Probe 37
IRMA Documentation, Release 2.0.4
Note: Error
If you see an error message like:
fatal error: Python.h: No such file or directory
Then you’ll need to install python3-dev package (for Debian like systems).
Note: leveldb.LevelDBError: IO error: /home/irma/leveldb/file_db/LOCK: Permission denied
If you encounter this problem, you likely have a problem with unix permissions. Please ensure that the folder is ownedby the user running the probes. On supervisord-based installation (default for vagrant/ansible scripts), the folderowner should be set to nobody. For init.d-based installation, it should be irma instead.
StaticAnalyzer - GNU/Linux or Microsoft Windows
The PE File analyzer adapted from Cuckoo Sandbox can be installed easily. One need to install the python packagesit depends on. From the installation directory, one can execute:
On GNU/Linux:
$ pip install -r modules/metadata/pe_analyzer/requirements.txt[...]
On Microsoft Windows, you need cygwin to successfully install the requirements (see python3-magic documentationfor installation details):
$ C:\Python27\Scripts\pip.exe install -r modules/metadata/pe_analyzer/requirements.txt[...]
Yara - GNU/Linux or Microsoft Windows
Please refer to modules/metadata/yara/README.md file for the documentation.
Guide on Debian (credits to Carbonn)
1. Installing dependencies
$ sudo apt-get install libtool automake bison
2. Installing Yara python modules
$ git clone https://github.com/plusvic/yara.git$ autoreconf -i --force$ ./configure$ make$ sudo make install$ python setup.py build$ sudo python setup.py install$ sudo ldconfig
38 Chapter 3. Manual Installation
IRMA Documentation, Release 2.0.4
3. Configuring for IRMA
$ mkdir /opt/irma/yara_rules/$ cat /opt/irma/yara_rules/yara_rules.yar << EOF# Insert rule below inside the file
rule silent_banker : banker{
meta:description = "This is just an example"thread_level = 3in_the_wild = true
strings:$a = {6A 40 68 00 30 00 00 6A 14 8D 91}$b = {8D 4D B0 2B C1 83 C0 27 99 6A 4E 59 F7 F9}$c = "UVODFRYSIHLNWPEJXQZAKCBGMT"
condition:$a or $b or $c
}EOF
3.3.5 Installation Checks
Celery Workers
Before going further, you should check that the python application manages to communicate with the RabbitMQ serverand the Redis server through Celery. To ensure that, from the installation directory, execute the Celery worker:
On GNU/Linux:
$ cd /opt/irma/irma-probe/current$ ./venv/bin/python -m probe.tasksWARNING:root: *** [plugin] Plugin failed to load: Kaspersky miss dependencies: win32→˓(PlatformDependency).WARNING:root: *** [plugin] Plugin failed to load: Sophos miss dependencies: win32→˓(PlatformDependency).WARNING:root: *** [plugin] Plugin failed to load: Symantec miss dependencies: win32→˓(PlatformDependency).WARNING:root: *** [plugin] Plugin failed to load: NSRL miss dependencies: leveldict→˓(ModuleDependency). See requirements.txt for needed dependenciesWARNING:root: *** [plugin] Plugin failed to load: VirusTotal miss dependencies: virus_→˓total_apis (ModuleDependency). See requirements.txt for needed dependenciesWARNING:root: *** [plugin] Plugin failed to load: StaticAnalyzer miss dependencies:→˓pefile (ModuleDependency). See requirements.txt for needed dependencies
-------------- celery@irma-probe v3.1.13 (Cipater)---- **** -------- * *** * -- Linux-3.2.0-4-amd64-x86_64-with-debian-7.6-- * - **** ---- ** ---------- [config]- ** ---------- .> app: probe.tasks:0x1e18750- ** ---------- .> transport: amqp://probe:**@brain:5672/mqprobe- ** ---------- .> results: redis://brain:6379/1- *** --- * --- .> concurrency: 1 (prefork)-- ******* ----
3.3. Probe 39
IRMA Documentation, Release 2.0.4
--- ***** ----- [queues]-------------- .> ClamAV exchange=celery(direct) key=ClamAV
.> ComodoCAVL exchange=celery(direct) key=ComodoCAVL
.> EsetNod32 exchange=celery(direct) key=EsetNod32
.> FProt exchange=celery(direct) key=FProt
.> McAfeeVSCL exchange=celery(direct) key=McAfeeVSCL
[2014-08-19 18:25:06,469: WARNING/MainProcess] celery@irma-probe ready.
The equivalent command on Microsoft Windows is:
C:\> cd "C:\IRMA\irma-probe\current"C:\> .\venv\Scripts\python.exe -m probe.tasks[...]
If your Celery worker does not output something similar to the above output, you should check twice the parametersin the application configuration file you are using. Let us note that the Celery worker gives us useful information onthe analyzer that are enabled. As each analyzer connects to a dedicated queue, we can deduce, in this example, thatthe analyzers ClamAV, ComodoCAVL, EsetNod32, FProt and McAfeeVSCL are enabled and ready to serve.
SFTP accounts
Defaut File Transport Protocol since v1.4.0 is now SFTP, you can check whether the configured account is valid.
$ sftp <user>@<hostname of the brain>
FTP-SSL accounts
Additionnally, if you have configured IRMA to use FTP-ssl, you can check whether the configured account is valid.On Debian, this can be done with the ftp-ssl package:
$ sudo apt-get install ftp-ssl[...]$ ftp-ssl <hostname of the brain>Connected to brain.220---------- Welcome to Pure-FTPd [privsep] [TLS] ----------220-You are user number 1 of 50 allowed.220-Local time is now 18:55. Server port: 21.220-This is a private system - No anonymous login220-IPv6 connections are also welcome on this server.220 You will be disconnected after 15 minutes of inactivity.Name (brain:root): probe-ftp500 This security scheme is not implemented234 AUTH TLS OK.[SSL Cipher DHE-RSA-AES256-GCM-SHA384]200 PBSZ=0200 Data protection level set to "private"331 User probe OK. Password requiredPassword: probe-ftp-password230 OK. Current directory is /Remote system type is UNIX.Using binary mode to transfer files.ftp>
40 Chapter 3. Manual Installation
CHAPTER 4
Database migration
IRMA uses Alembic to manage and perform databases migration.
Note: Alembic is a useful tool to manage migration, but can’t surpass local engine implementation of SQL. AsSQLite doesn’t manage schema modifications such as ALTER_COLUMN, the whole migration system of IRMAwon’t support it. The preferred database engine is PostgreSQL.
You can still use SQLite, but you will be on your own for migrations.
Warning: Please note that most of the manipulations on this can and sometimes will alter your data. If you arenot sure about what you are doing, and even if you are sure, make backup.
4.1 Requirements
• Alembic
4.2 Content
Database migrations are managed in the frontend and brain IRMA components.
The files/directories used are:
alembic.iniextras/migration/
+- env.py+- script.py.mako+- versions/
+- <revision_1>.py
41
IRMA Documentation, Release 2.0.4
+- <revision_2>.py+- ...
Note: All the commands below will assert to be executed on top of this file system, as Alembic needs the alembic.ini configuration file.
You could also use the -c <path_to_conf_file>.
4.3 Usage
Alembic manage a ‘revision’ for each database evolution. These revisions are used to upgrade or downgrade thedatabase schema.
The command:
$ alembic current
. . . shows the current revision of the database.
The command to get the history of the latest alembic migrations is:
$ alembic history --verbose
4.3.1 Create database from scratch with Alembic
Configuration and creating database
Alembic will use the information in the [sqldb] section of the configuration files (respectively config/frontend.ini or conf/brain.ini for the repositories of the frontend or the brain components). Make surethey are accurate.
The database must already exist. This step is quite simple, the SQL command usually being:
sql$ CREATE DATABASE <db_name>;
Update your schema with Alembic
If you use a virtualenv, activate it. Then enter:
$ alembic upgrade head
Alembic applies each revision one after the other. At the end of the process, if no error occurs, your database shouldbe updated.
Note: You can update the database one revision at a time, or up to a specific revision. See the revisions section forfurther information.
42 Chapter 4. Database migration
IRMA Documentation, Release 2.0.4
4.3.2 If you already have a database WITHOUT Alembic
Alembic stores its current revision number in database. If your database doesn’t have this information, you are verylikely to encounter errors when using Alembic, as it will try to create already existing tables.
The easiest solution is to destroy your database and go for a fresh install.
Although, if you don’t want to lose your data, you could update the Alembic information manually.
You will need to:
1. Get the exact current Alembic revision of your database. Each migration file has a Revision ID in its header.Investigate the successive revisions to know which one matches your current database state.
2. Once you known your Alembic revision, run:
$ alembic stamp <your_alembic_revision_number>
3. Your database is now synchronized with Alembic! You should be able to use Alembic to upgrade/downgradeyour database now. Be aware that if the revision number you provided is false, you could encounter massiveerrors while attempting to upgrade/downgrade your database.
4.3.3 Generating a new revision
Creating a new revision can be done with the command:
$ alembic revision -m <revision_message>
This command produces a new <hash>_<revision_message>.py file in the extras/migration/versions/ directory. This file contains two functions upgrade and downgrade, respectively used to upgradethe database to the revision, or downgrade from it. These two functions are empty and must be completed with thedesired modifications (see the alembic documentation).
A revision could be produced automatically, from database metadata defined in the IRMA SQL objects descriptionthrough sqlalchemy, with the command:
$ alembic revision --autogenerate -m <revision_message>
These SQL objects are defined in:
• frontend/models/sqlobjects.py for the frontend,
• brain/models/sqlobjects.py for the brain.
Alembic scripts in IRMA repositories are already configured to use metadata defined in these files. You should be ableto use the --autogenerate option without further modifications.
Note: IRMA configuration allows to prefix table names through configuration. Our revision files use the function<frontend_or_brain>/config/parser.py:prefix_table_name to generate table names rather thankeeping alembic-generated plain string names. A good practice would be to keep using this function in revision files.
Warning: Alembic easily detects changes such as adding/removing columns, but could be blind on thin, innermodifications. Re-reading the auto-generated script is a strongly recommended step before actually performingthe migration.
See the alembic documentation for more information.
4.3. Usage 43
IRMA Documentation, Release 2.0.4
Warning: Database modifications using ALTER_COLUMN (such as changing the type of a column) can’t beperformed on SQLite databases. Be aware of this limitation if you absolutely want
to use migration scripts with this SQL engine.
4.3.4 Migrating between revisions
Once the revision is properly described, the migration is performed with:
$ alembic upgrade head
Alembic allows to migrate the database to any revision, relatively to the current revision or absolutely. Several exam-ples:
$ alembic upgrade +4$ alembic downgrade base$ alembic upgrade <revision_number>+3
4.4 Tips and tricks
Note: Don’t trust Alembic too much. It is nothing more than a tool, without any comprehension on the code.Cautiously read the revision scripts it generates.
Note: Database migration is hardly ever a painless step. Be sure to:
1. save your data before performing a migration,
2. test your application after the migration to ensure its compatibility with the new data schemes.
Note: With a PostgreSQL database, the Float type is tolerated but the real type name used by the databaseis Real. It means that SQL objects described in sqlalchemy with Float columns will be properly applied indatabase, but at each autogenerate revision, alembic will see Real type in database, against Float type in thecode metadata, and so will perform each time a useless alter_column from Real to Float. This problem couldbe avoided (with PostgreSQL) by declaring Real instead of Float.
See this page for more information on PostgreSQL numeric types.
Note:
Alembic can’t directly deal with many somehow complex operations, such as type migration with notrivial cast. In these cases, the operation must be manually described with a raw SQL command (whichcould be database-dependent).
For instance, alembic can’t perform the migration from real to datetime:
> alembic.alter_column('table', 'column',existing_type=sqlalchemy.REAL(),
44 Chapter 4. Database migration
IRMA Documentation, Release 2.0.4
type_=sqlalchemy.DateTime(),existing_nullable=False)
. . . because of an error a column "column" cannot be cast automatically to typetimestamp with time zone.
A proper migration for PostgreSQL would be (in Python):
> alembic.execute('ALTER TABLE "table" ALTER COLUMN "column" TYPE TIMESTAMP WITHOUT→˓TIME ZONE USING to_timestamp(column)')
And the reverse code to downgrade the migration could be:
> alembic.execute('ALTER TABLE "table" ALTER COLUMN "column" TYPE REAL USING→˓extract(epoch from column)')
Note: Rather than managing migrations directly with Alembic, we could generate SQL migration revision to be useddirectly on database with the command:
$ alembic upgrade <revision> --sql > migration.sql
Note: Deleting a revision R is simple:
• downgrade the database to the revision before R-1 the revision you want to delete;
• if any, edit the script of the following revision R+1 and update the down_revision variable to match therevision number of revision R-1;
• delete the script of the revision R you want to delete;
• upgrade your database.
The deleted revision want be applied any more.
4.4. Tips and tricks 45
IRMA Documentation, Release 2.0.4
46 Chapter 4. Database migration
CHAPTER 5
To evolve IRMA
5.1 Adding a new probe
5.1.1 Writing a Plugin for the probe
Note: To be a valid probe module, IRMA expects it to have a predefined structure. To save time, one can get aminimal working structure from the skeleton plugin. The new plugin is stored in the appropriate sub-directory of thedirectory probe/modules according to the type of the new probe (antivirus, metadata, external. . . ).
For a probe that is not a antivirus
1. Copy the directory skeleton to the new module (appropriate localisation). Example with a module my_module withmetadata type :
$cp -r probe/modules/custom/skeleton/ probe/modules/metadata/my_module
2. If there are packages to install, specify them in the file requirements.txt. Otherwise remove the file
3. Adjust the file plugin.py according to the module :
• Adjust the class’s name with the name of your probe
• Fill in the fields of the class :- _plugin_name_ = [the plugin name] - _plugin_display_name_ = [the field_name of the class of the probe] - _plugin_version_ = [the version number] - _plugin_category = [the typeof the probe: IrmaProbeType.] - _plugin_description = [uick description] - _plugin_dependencies = [listof dependencies: platform, binary or/and file]
=> if used import from lib.plugins PlatformDependency, BinaryDependency or/and FileDepen-dency
– _mimetype_regexp = [mimetype corresponding]
4. Implement the functions corresponding to the type of the plugin
47
IRMA Documentation, Release 2.0.4
For an antivirus
In the case of an antivirus, it is a little different because an Antivirus class was created to avoid code’s duplication.You can use the skeleton below:
plugin.py:
## Copyright (c) 2013-2018 Quarkslab.# This file is part of IRMA project.## Licensed under the Apache License, Version 2.0 (the "License");# you may not use this file except in compliance with the License.# You may obtain a copy of the License in the top-level directory# of this distribution and at:## http://www.apache.org/licenses/LICENSE-2.0## No part of the project, including this file, may be copied,# modified, propagated, or distributed except according to the# terms contained in the LICENSE file.
from .skeleton import Skeleton
from lib.plugins import PluginBase, PluginLoadErrorfrom lib.irma.common.utils import IrmaProbeType
class SkeletonPlugin(PluginBase, Skeleton):
# =================# plugin metadata# =================_plugin_name_ = "Skeleton"_plugin_display_name_ = Skeleton._name_plugin_author_ = "IRMA (c) Quarkslab"_plugin_version_ = "1.0.0"_plugin_category_ = "custom"_plugin_description_ = "Plugin skeleton"_plugin_dependencies_ = []_mimetype_regexp = None
# =============# constructor# =============
def __init__(self):self.module = Skeleton()
skeleton.py:
## Copyright (c) 2013-2018 Quarkslab.# This file is part of IRMA project.## Licensed under the Apache License, Version 2.0 (the "License");# you may not use this file except in compliance with the License.# You may obtain a copy of the License in the top-level directory# of this distribution and at:#
48 Chapter 5. To evolve IRMA
IRMA Documentation, Release 2.0.4
# http://www.apache.org/licenses/LICENSE-2.0## No part of the project, including this file, may be copied,# modified, propagated, or distributed except according to the# terms contained in the LICENSE file.
import logging
# Choose the class you need to inherit fromfrom modules.antivirus.base import AntivirusUnix, AntivirusWindows
log = logging.getLogger(__name__)
# Inhererit from AntivirusUnix or AntivirusWindows according to your plateformclass Skeleton(Antivirus):
name = "Skeleton for Antivirus"
# ==================================# Constructor and destructor stuff# ==================================
def __init__(self, *args, **kwargs):# class super class constructorsuper().__init__(*args, **kwargs)
# do your initialization stuff
# Finally initialize all the attributesself._init_attributes()
The recipe is the same, the files with the corresponding module name and differents fields need to be updated. Theattributes in Antivirus._attributes are meant to be defined by the instanciation. One can either:
• leave it blank, in this case the super class will assign it a default value (eg. "unavailable" for self.version);
• define it directly (eg. self.scan_path = Path("/opt/skeleton/skeleton"));
• define a function to be called to assign it (eg. def get_scan_path(self): ...), the super class willtake care of calling it and handling exceptions.
5.1.2 Testing the new plugin
Before testing, module’s necessary stuff (binaries, files, etc) must be provisioned to the VM.
$cd ansible$vagrant rsync$vagrant ssh$sudo su deploy$cd /opt/irma/irma-probe/current$venv/bin/python -m extras.tools.run_module
This last command lists available modules.
Now, if the new module is available, its launching can be done:
$venv/bin/python -m extras.tools.run_module my_module file
5.1. Adding a new probe 49
IRMA Documentation, Release 2.0.4
5.1.3 Automatic provisioning
Creating a new role
Create a new directory with this structure:
cd ansibletree roles/quarsklab.my_moduleroles/quarkslab.my_module/+-- defaults| +-- main.yml+-- tasks
+-- main.yml
tasks/main.yml is the default entry point for a role containing Ansible tasks. In this file, write the instruction to installthe module. Add the file tasks/update.yml to write the informations for the update if necessary. In defaults/main.yml itis usual to store default variables for this role. If there are particular instructions, for example how to obtain a licencefor a antivirus, add a README file.
Invoking the module role
Modify playbooks/provisioning.yml : add the module
-name : my_modulehosts: my_moduleroles:- { role: quarkslab.my_module, tags: 'my_module'}
If a task update was defined, add the module in playbooks/updating.yml :
-name : my_modulehosts: my_moduleroles:- { role: quarkslab.module, tags: 'my_module', task_from : update}
Defining hosts
Modify the environment to add the new probe.
For example for the allinone_dev :
$ cat environments/allinone_dev.yml[ ... snip ... ]
virustotal:- brain.irma
my_module:- brain.irma
"probe:children":- clamav- comodo- mcafee- static-analyzer- virustotal- my_module
50 Chapter 5. To evolve IRMA
CHAPTER 6
References
6.1 Disclaimer
IRMA is distributed as it is, in the hope that it will be useful, but without any warranty neither the implied mer-chantability or fitness for a particular purpose.
Whatever you do with this tool is uniquely your own responsibility.
6.2 License
IRMA source code is licensed under Apache License, version 2.0.
The full license text can be found below (Apache License, version 2.0).
6.3 Apache License, version 2.0
6.4 Authors
51
IRMA Documentation, Release 2.0.4
52 Chapter 6. References
CHAPTER 7
Frequently Asked Questions
7.1 Playing with tags
Tags are available in IRMA from version 1.3.0
7.1.1 Creating a tag
You could create tags by using the command line tools
>>> from irmacl.helpers import *>>> tag_list()[]
>>> tag_new("archive"){u'text': u'archive', u'id': 1}
>>> tag_list()[Tag archive [1]]
or directly from your terminal by using curl and posting a json with ‘text’ key:
$ curl -H "Content-Type: application/json; charset=UTF-8" -X POST -d '{"text":"<your→˓tag>"}' http://172.16.1.30/api/v1.1/tags
Note: There is currently no way to create a tag directly from the web IHM.
7.1.2 Tagging a File
Directly in web IHM, once you are on a file details page:
53
IRMA Documentation, Release 2.0.4
Just click the tag bar and you will see all available tags. You could add multiple tags.
It is also possible to add a tag through command line tools:
54 Chapter 7. Frequently Asked Questions
IRMA Documentation, Release 2.0.4
>>> from irmacl.helpers import *>>> file_tag_add?Signature: file_tag_add(sha256, tagid, verbose=False)Docstring:Add a tag to a File
:param sha256: file sha256 hash:type sha256: str of (64 chars):param tagid: tag id:type tagid: int:return: No return
>>> file_tag_add("346ae869f7c7ac7394196de44ab4cfcde0d1345048457d03106c1a0481fba853",1)
7.1.3 Searching by tag
You could specify one or more tags while searching for files too:
choose your tag list then hit the search button:
or by command line:
>>> from irmacl.helpers import *>>> file_search(tags=[1])(1, [<irma.apiclient.IrmaResults at 0x7f079ca23890>])
7.1. Playing with tags 55
IRMA Documentation, Release 2.0.4
7.2 SSL settings
7.2.1 Making RabbitMQ running SSL
Certificates generation
See rabbitmq detailled guide on how to generate server and clients certificates here
Update server settings on brain
Copy the RabbitMQ configuration template provided in “./extras/rabbitmq” to “/etc/rabbitmq”
Restart RabbitMQ:
$ sudo service rabbitmq-server restart
Allow irma to use SSL with RabbitMQ
Note: Frontend configuration file is <irma dir>/config/frontend.ini
Brain configuration file is <irma dir>/config/brain.ini
Probe configuration file is <irma dir>/config/probe.ini
Update PORT in configuration file from 5672 to 5671. Except if you change the default port defined in the theRabbitMQ configuration template provided in the precedent paragraph.
[broker_xxxx]host = 127.0.0.1port = 5671
Update activate_ssl switch in configuration file:
[ssl_config]activate_ssl = yesca_certs =keyfile =certfile =
Put the SSL certificates (ca_cert, key_file, cert_file) in <irma dir>/ssl Update ca_certs, keyfile andcertfile in configuration file according to the filenames in “./ssl”
[ssl_config]activate_ssl = yesca_certs = <ca_cert_filename>keyfile = <key_filename>certfile = <cert_filename>
Note: If you are switching to ssl from an already running no_ssl version, please do the following on irma-brainRabbitMQ server:
56 Chapter 7. Frequently Asked Questions
IRMA Documentation, Release 2.0.4
$ sudo rabbitmqctl stop_app$ sudo rabbitmqctl reset$ sudo rabbitmqctl start_app# create again the RabbitMQ vhosts, usernames and passwords:$ sudo ./extras/scripts/rabbitmq/rmq_adduser.sh probe probe mqprobe$ sudo ./extras/scripts/rabbitmq/rmq_adduser.sh brain brain mqbrain$ sudo ./extras/scripts/rabbitmq/rmq_adduser.sh frontend frontend mqfrontend
7.3 How to debug
7.3.1 Switch debug log on
Configuration file for frontend, brain and probe is located by default in the config folder and is named respectivelyfrontend.ini, brain.ini and probe.ini.
To turn on debug log just add the following line:
[log]syslog = 0debug = 1
and restart all related applications.
To turn on SQL debug log (warning: its verbose) just add the following line:
[log]syslog = 0debug = 1sql_debug = 1
and restart all related applications.
7.3.2 Debug a probe
Open a session on the probe machine and change directory to the irma-probe location. Try the run_module tool on afile to see what analyzer is detected and what is its output on a file.
$ sudo su deploy$ cd /opt/irma/irma-probe/current$ ./venv/bin/python -m extras.tools.run_module
[...]usage: run_module.py [-h] [-v]
{Unarchive,StaticAnalyzer,ClamAV,VirusTotal} filename[filename ...]
run_module.py: error: too few arguments
Here 4 probes are automatically detected. Now try one on a file:
$ ./venv/bin/python -m extras.tools.run_module ClamAV requirements.txt{'database': {'/var/lib/clamav/bytecode.cvd': {'ctime': 1458640823.285298,
'mtime': 1458640823.069295,'sha256':
→˓'82972e6cc5f1204829dba913cb1a0b5f8152eb73d3407f6b86cf388626cff1a1'},
7.3. How to debug 57
IRMA Documentation, Release 2.0.4
'/var/lib/clamav/daily.cvd': {'ctime': 1458640822.8932924,'mtime': 1458640822.6692889,'sha256':
→˓'9804c9b9aaf983f85b4f13a7053f98eb7cca5a5a88d3897d49b22182b228885f'},'/var/lib/clamav/main.cvd': {'ctime': 1458640821.6972747,
'mtime': 1458640813.9771628,'sha256':
→˓'4a8dfbc4c44704186ad29b5a3f8bdb6674b679cecdf83b156dd1c650129b56f2'}},'duration': 0.0045299530029296875,'error': None,'name': 'Clam AntiVirus Scanner','platform': 'linux2','results': None,'status': 0,'type': 'antivirus','version': '0.99'}
And check the output.
7.3.3 Debug Ansible Provisioning
To debug errors while provisioning (same goes with deployment) with following typical command:
$ ansible-playbook --private-key=~/.vagrant.d/insecure_private_key --inventory-file=.→˓vagrant/provisioners/ansible/inventory/vagrant_ansible_inventory -u vagrant→˓playbooks/provisioning.yml
Example output:
TASK [Mayeu.RabbitMQ : add rabbitmq user and set privileges] *******************[DEPRECATION WARNING]: Using bare variables is deprecated. Update your playbooks so→˓that the environment value uses thefull variable syntax ('{{rabbitmq_users_definitions}}').This feature will be removed in a future release. Deprecationwarnings can be disabled by setting deprecation_warnings=False in ansible.cfg.failed: [brain.irma] (item={u'vhost': u'mqbrain', u'password': u'brain', u'user': u→˓'brain'}) => {"failed": true, "item": {"password": "brain", "user": "brain", "vhost→˓": "mqbrain"}, "module_stderr": "", "module_stdout": "Traceback (most recent call→˓last):\r\n File \"/tmp/ansible_wKXoO5/ansible_module_rabbitmq_user.py\", line 302,→˓in <module>\r\n main()\r\n File \"/tmp/ansible_wKXoO5/ansible_module_rabbitmq_→˓user.py\", line 274, in main\r\n if rabbitmq_user.get():\r\n File \"/tmp/→˓ansible_wKXoO5/ansible_module_rabbitmq_user.py\", line 155, in get\r\n users =→˓self._exec(['list_users'], True)\r\n File \"/tmp/ansible_wKXoO5/ansible_module_→˓rabbitmq_user.py\", line 150, in _exec\r\n rc, out, err = self.module.run_→˓command(cmd + args, check_rc=True)\r\n File \"/tmp/ansible_wKXoO5/ansible_modlib.→˓zip/ansible/module_utils/basic.py\", line 1993, in run_command\r\n File \"/usr/lib/→˓python2.7/posixpath.py\", line 261, in expanduser\r\n if not path.startswith('~→˓'):\r\nAttributeError: 'list' object has no attribute 'startswith'\r\n", "msg":→˓"MODULE FAILURE", "parsed": false}
You could first increase ansible verbosity by adding -vvv option (-vvvv on windows for winrm debug), it will helpis the problem is linked to arguments.
$ ansible-playbook -vvv --private-key=~/.vagrant.d/insecure_private_key --→˓inventory-file=.vagrant/provisioners/ansible/inventory/vagrant_ansible_inventory -u→˓vagrant playbooks/provisioning.yml
TASK [Mayeu.RabbitMQ : add rabbitmq user and set privileges] *******************
58 Chapter 7. Frequently Asked Questions
IRMA Documentation, Release 2.0.4
task path: /home/alex/repo/irma-ansible/roles/Mayeu.RabbitMQ/tasks/vhost.yml:13[DEPRECATION WARNING]: Using bare variables is deprecated. Update your playbooks
→˓so that the environment value uses the fullvariable syntax ('{{rabbitmq_users_definitions}}').This feature will be removed in a future release. Deprecation warnings can bedisabled by setting deprecation_warnings=False in ansible.cfg.<127.0.0.1> ESTABLISH SSH CONNECTION FOR USER: vagrant<127.0.0.1> SSH: EXEC ssh -C -q -o ForwardAgent=yes -o Port=2222 -o
→˓'IdentityFile="/home/alex/.vagrant.d/insecure_private_key"' -o→˓KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-→˓keyex,hostbased,publickey -o PasswordAuthentication=no -o User=vagrant -o→˓ConnectTimeout=10 127.0.0.1 '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo $HOME/.→˓ansible/tmp/ansible-tmp-1468570550.09-211613386938202 `" && echo ansible-tmp-→˓1468570550.09-211613386938202="` echo $HOME/.ansible/tmp/ansible-tmp-1468570550.09-→˓211613386938202 `" ) && sleep 0'"'"''
<127.0.0.1> PUT /tmp/tmpiysJ6l TO /home/vagrant/.ansible/tmp/ansible-tmp-→˓1468570550.09-211613386938202/rabbitmq_user
<127.0.0.1> SSH: EXEC sftp -b - -C -o ForwardAgent=yes -o Port=2222 -o→˓'IdentityFile="/home/alex/.vagrant.d/insecure_private_key"' -o→˓KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-→˓keyex,hostbased,publickey -o PasswordAuthentication=no -o User=vagrant -o→˓ConnectTimeout=10 '[127.0.0.1]'
<127.0.0.1> ESTABLISH SSH CONNECTION FOR USER: vagrant<127.0.0.1> SSH: EXEC ssh -C -q -o ForwardAgent=yes -o Port=2222 -o
→˓'IdentityFile="/home/alex/.vagrant.d/insecure_private_key"' -o→˓KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-→˓keyex,hostbased,publickey -o PasswordAuthentication=no -o User=vagrant -o→˓ConnectTimeout=10 -tt 127.0.0.1 '/bin/sh -c '"'"'sudo -H -S -n -u root /bin/sh -c '"→˓'"'"'"'"'"'"'"'echo BECOME-SUCCESS-rbeeckncuxenewcwkayivqiwvarchlrd; LANG=fr_FR.UTF-→˓8 LC_ALL=fr_FR.UTF-8 LC_MESSAGES=fr_FR.UTF-8 /usr/bin/python /home/vagrant/.ansible/→˓tmp/ansible-tmp-1468570550.09-211613386938202/rabbitmq_user; rm -rf "/home/vagrant/.→˓ansible/tmp/ansible-tmp-1468570550.09-211613386938202/" > /dev/null 2>&1'"'"'"'"'"'"→˓'"'"' && sleep 0'"'"''
failed: [brain.irma] (item={u'vhost': u'mqbrain', u'password': u'brain', u'user→˓': u'brain'}) => {"failed": true, "invocation": {"module_name": "rabbitmq_user"},→˓"item": {"password": "brain", "user": "brain", "vhost": "mqbrain"}, "module_stderr→˓": "", "module_stdout": "Traceback (most recent call last):\r\n File \"/tmp/→˓ansible_Qo3lZl/ansible_module_rabbitmq_user.py\", line 302, in <module>\r\n→˓main()\r\n File \"/tmp/ansible_Qo3lZl/ansible_module_rabbitmq_user.py\", line 274,→˓in main\r\n if rabbitmq_user.get():\r\n File \"/tmp/ansible_Qo3lZl/ansible_→˓module_rabbitmq_user.py\", line 155, in get\r\n users = self._exec(['list_users→˓'], True)\r\n File \"/tmp/ansible_Qo3lZl/ansible_module_rabbitmq_user.py\", line→˓150, in _exec\r\n rc, out, err = self.module.run_command(cmd + args, check_→˓rc=True)\r\n File \"/tmp/ansible_Qo3lZl/ansible_modlib.zip/ansible/module_utils/→˓basic.py\", line 1993, in run_command\r\n File \"/usr/lib/python2.7/posixpath.py\",→˓ line 261, in expanduser\r\n if not path.startswith('~'):\r\nAttributeError:→˓'list' object has no attribute 'startswith'\r\n", "msg": "MODULE FAILURE", "parsed→˓": false}
In this particular case, verbose doesnt add much information as the problem is linked to ansible scripts. Lets go onelevel deeper so. Ansible output the temporary script executed on guest (highlighted in previous code block), but deleteit just after execution. To further debug it we will set ansible to keep remote files and the debug session will now takesplace inside the guest.
$ ANSIBLE_KEEP_REMOTE_FILES=1 ansible-playbook -vvv --private-key=~/.vagrant.d/→˓insecure_private_key --inventory-file=.vagrant/provisioners/ansible/inventory/→˓vagrant_ansible_inventory -u vagrant playbooks/provisioning.yml
7.3. How to debug 59
IRMA Documentation, Release 2.0.4
in debug log get the temporary ansible path to remote script:
/usr/bin/python /home/vagrant/.ansible/tmp/ansible-tmp-1468571039.87-134696488633275/→˓rabbitmq_user
Log in to remote machine and go to the temporary ansible dir. Explode the compressed script and run it locallly:
$ vagrant@brain:~/.ansible/tmp/ansible-tmp-1468571039.87-134696488633275$ lsrabbitmq_user
$ vagrant@brain:~/.ansible/tmp/ansible-tmp-1468571039.87-134696488633275$ python→˓rabbitmq_user explodeModule expanded into:/home/vagrant/.ansible/tmp/ansible-tmp-1468571039.87-134696488633275/debug_dir
$ vagrant@brain:~/.ansible/tmp/ansible-tmp-1468571039.87-134696488633275$ ls debug_→˓dir/ansibleansible_module_rabbitmq_user.pyargs
$ vagrant@brain:~/.ansible/tmp/ansible-tmp-1468571039.87-134696488633275$ python→˓rabbitmq_user executeTraceback (most recent call last):
File "/home/vagrant/.ansible/tmp/ansible-tmp-1468571039.87-134696488633275/debug_→˓dir/ansible_module_rabbitmq_user.py", line 302, in <module>
main()File "/home/vagrant/.ansible/tmp/ansible-tmp-1468571039.87-134696488633275/debug_
→˓dir/ansible_module_rabbitmq_user.py", line 274, in mainif rabbitmq_user.get():
File "/home/vagrant/.ansible/tmp/ansible-tmp-1468571039.87-134696488633275/debug_→˓dir/ansible_module_rabbitmq_user.py", line 155, in get
users = self._exec(['list_users'], True)File "/home/vagrant/.ansible/tmp/ansible-tmp-1468571039.87-134696488633275/debug_
→˓dir/ansible_module_rabbitmq_user.py", line 150, in _execrc, out, err = self.module.run_command(cmd + args, check_rc=True)
File "/home/vagrant/.ansible/tmp/ansible-tmp-1468571039.87-134696488633275/debug_→˓dir/ansible/module_utils/basic.py", line 1993, in run_command
args = [ os.path.expandvars(os.path.expanduser(x)) for x in args if x is not None→˓]File "/usr/lib/python2.7/posixpath.py", line 261, in expanduserif not path.startswith('~'):
AttributeError: 'list' object has no attribute 'startswith'
You could now add debug to source files and properly understand where the problem is. In our example case, it is anansible problem related to module_rabbitmq_user present in 2.1.0.0 see github PR
7.4 How to migrate
Note: If you need help to connect to your box through ssh, see vagrant FAQ
This part is only useful to someone willing to manually upgrade from an older version of IRMA.
60 Chapter 7. Frequently Asked Questions
IRMA Documentation, Release 2.0.4
7.4.1 Install alembic
$ sudo su deploy$ cd /opt/irma/irma-frontend/current$ ./venv/bin/pip install alembic$ export PYTHONPATH=.:$PYTHONPATH$ alembic history
430a70c8aa21 -> eb7141efd75a (head), version 1.3.02cc69d5c53eb -> 430a70c8aa21, version 1.2.1<base> -> 2cc69d5c53eb, DB revision creation
7.4.2 from 1.2.1 to 1.3.0
Fix nginx configuration
Introducing multiversion API means python code should receive the api version parameter. in file /etc/nginx/sites-available/irma-frontend.conf replace:
rewrite ^/api/v1/(.+) /$1 break;
by:
rewrite ^/api/(.+) /$1 break;
and restart nginx
Migrate Database
First you should tell alembic you are at version 1.2.1:
$ ./venv/bin/alembic stamp 430a70c8aa21
then upgrade model and data:
$ ./venv/bin/alembic upgrade head
Regenerate IHM
to regenerate IHM do the following:
$ sudo su deploy$ cd /opt/irma/irma-frontend/current/web$ ./node_modules/.bin/bower update$ ./node_modules/.bin/gulp dist
Its done.
7.5 API documentation
There is a dynamic documentation for IRMA API available on your instance
7.5. API documentation 61
IRMA Documentation, Release 2.0.4
It allows you to read documentation but also try request and see server response. Give it a try.
You could see detailed information about one specific API route:
and by clicking on the Try it button, see the server response:
62 Chapter 7. Frequently Asked Questions
IRMA Documentation, Release 2.0.4
7.6 Connect to a vagrant box through ssh
If you don’t already have it download vagrant insecure_private_key
$ wget https://raw.githubusercontent.com/mitchellh/vagrant/master/keys/vagrant -O→˓insecure_private_key
Then change rights on the key otherwise ssh will complains and connect to your vagrant box
$ chmod 700 insecure_private_key$ ssh vagrant@172.16.1.30 -i insecure_private_key
7.7 Enable SSL using OpenSSL in ansible scripts
If you want to activate SSL on the frontend server, you’ll need:
• modify frontend_openssl variables in group_vars/frontend:
frontend_openssl: True # Default is falsefrontend_openssl_dh_param: # put the DH file locationsfrontend_openssl_certificates: [] # an array of files {source, destination}
# to copy to the server
• Uncomment (and customize) the nginx_sites variable in the group_vars/frontend, a commented example is avail-able.
Then, provision or re-provision your infrastructure. Ansible will only change file related to OpenSSL and Nginxconfigurations.
7.6. Connect to a vagrant box through ssh 63
IRMA Documentation, Release 2.0.4
7.8 Speed up your Vagrant VMs
Install this softwares:
• vagrant-cachier (more info on vagrant-cachier)
$ vagrant plugin install vagrant-cachier
• vagrant-vbguest (more info on vagrant-vbguest)
$ vagrant plugin install vagrant-vbguest
64 Chapter 7. Frequently Asked Questions
CHAPTER 8
Resources
• Project website
• IRC (irc.freenode.net, #qb_irma)
• Twitter (@qb_irma)
65
IRMA Documentation, Release 2.0.4
66 Chapter 8. Resources
CHAPTER 9
Screenshots
9.1 Command Line Interface
A sample script can be found in frontend repository. Add your own frontend address before testing it.
67
IRMA Documentation, Release 2.0.4
9.2 Web Interface
Some screenshots of the irma user interface shipped with frontend package.
68 Chapter 9. Screenshots
IRMA Documentation, Release 2.0.4
9.2. Web Interface 69
IRMA Documentation, Release 2.0.4
70 Chapter 9. Screenshots
IRMA Documentation, Release 2.0.4
9.2. Web Interface 71
IRMA Documentation, Release 2.0.4
72 Chapter 9. Screenshots