Date post: | 02-Mar-2018 |
Category: |
Documents |
Upload: | truongtuong |
View: | 227 times |
Download: | 2 times |
www.elixir-europe.org
Provisioning Docker securely for HPC
AbdulrahmanAzab&BjoernGrueningELIXIR-NO,EXCELERATE-NO
GCC2016
Commonissues/requirementsinaproductionsystem
• Ihaveanoldkernel• OldversionofDocker
• Iwanttoallowalluserstoruncontainers
• Andtodosoasthemselves,notasroot
• ThisispossibleJ
Docker inside HPC SLURM
Socker
Docker Engine
-rwxr--r-- 1 root root -rwsr-sr-x 1 root root
socker socker_core
Socker Wrapper
Use
r Docker binary
Sys
Adm
in
root
SLURM(socker)
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --partition=docker
srun -n 32 socker bowtie1 -p 8 -S -v 2 -m 1 /work/hg19/hg19
/work/bowtie-input/$(TASK_ID).normal.fastq
SLURM(socker)
[azab@compute-1-27~]$ cat /proc/4822/cgroup
11:blkio:/docker/d034ac18549fed91deff35e13a63767684ff43f3a961...
10:net_cls:/
8:devices:/docker/d034ac18549fed91deff35e13a63767684ff43f3a961...
6:cpuacct:/docker/d034ac18549fed91deff35e13a63767684ff43f3a961...
5:cpu:/docker/d034ac18549fed91deff35e13a63767684ff43f3a961...
3:memory:/slurm/uid_238869/job_14576313
2:cpuset:/slurm/uid_238869/job_14576313
1:freezer:/slurm/uid_238869/job_14576313
Docker on the cluster - SLURM
Docker repositories
Compute Node
Docker Engine
Compute Node
Docker Engine
Compute Node
Docker Engine
Compute Node
Docker Engine
Local Registry
Compute Node
Docker Engine
Compute Node
Docker Engine
Compute Node
Docker Engine
Compute Node
Docker Engine
Pull
Abe
l C
olos
sus
Docker inside HPC HTCondor
HTCondor
universe = docker executable = /usr/local/bin/bowtie arguments = -p 8 -S -v 2 -m 1 /work/hg19/hg19
/work/bowtie-input/$(process).normal.fastq docker_image = genomicpariscentre/bowtie1 output = $(process).normal.sam error = $(process).normal.err log = normal.log queue 46
Galaxy Docker with Docker enabled HPC
HTCondor
GalaxyDocker
Docker on the cluster – HTCondor (Infrastructure)
Docker repositories
Invivo 64 cores
Docker Engine
Invitro 64 cores
Docker Engine
insilico 32 cores
Docker Engine
Pull
HTC
ondo
r Poo
l Condor CM
Docker Container
Tools
Docker on the cluster – HTCondor on the Cloud (cPouta)
Docker repositories
Compute Node
Docker Engine
Compute Node
Docker Engine
Compute Node
Docker Engine
Compute Node
Docker Engine
Local ePouta Registry
Pull
HTC
ondo
r Poo
l Condor CM
Docker Container
VM
Tools