Date post: | 05-Jan-2016 |
Category: |
Documents |
Upload: | daniella-may |
View: | 218 times |
Download: | 0 times |
GAAIN Virtual Appliances: Virtual Machine Technology for Scientific Data Analysis
Arihant PatawariUSC Stevens Neuroimaging and Informatics Institute
July 9th 2015
Organization
1) GAAIN Virtual Appliances
-Expanding the GAAIN application with Docker as well as Virtual Machines
- Objectives: Support production data analysis in GAAIN
2) Medical Datasets Element Name Matching- Integration into larger GEM system
- Scalability issues
- Mine features from data
- Neural Network classifiers9/8/14
The Virtual Machine
• A (computing) machine purely “made of software”• A machine within a machine• WHY ? : Sharable, transportable over a network
GAAIN Virtual Machines
Investigator
Data Partner
Virtual Appliances
9/8/14
6
• How do we provide a scientific investigator a dedicated analysis development resource
• How do we ensure that an analysis resource is sharable• How do we run applications that require graphical display (such
as a UI)• How can we connect client and server applications• How do we ensure automated cloud backups• How do we send over analysis machine to data partners• How do we access data partner data • How do we get beck analysis results into GAAIN network • …..
Objectives ….
7
•Designed to provide framework for (specific) application encapsulation
•Provide minimal support for application•Not intended as a general purpose computing machine•Other aspects
- Dockerfile management- Docker Hub- Security
• Relatively new and evolving framework
Recap: Docker Framework
8
•Intended as full computing machine•Command line control, not scripts•Interoperability with Open Virtualization Format (OVF)•Also VM environments like VMSphere, XenServer and others
Recap: Virtual Machines
PC
VBox VM
(Pipeline)Server
PC
VBox VM
(Pipeline)Server
PC
Docker
(Pipeline)Server
PC
VBox VM
Docker
(Pipeline)Server
Many Possibilities !
client on PC, server in VM client and server in VM
client and server in Docker client in VM, server in Docker, Docker in VM
√
10
Docker LifeCycle
Docker File
Hub
Data Partner’s Machine
Result in Shared Folder
Docker vs Virtual Machines
AspectsVirtualBox Docker
Virtual Image Type (Formats)
The open-virtual-format as ‘.OVF’ and ‘’.OVA’ files
Proprietary Docker image format
Requirements Any virtualization hypervisor that can run the open virtualization format images
Docker Engine
Architecture Typically the core virtual image contains a complete operating system of choice
Minimal system layer is provided and components are added only as required
‘Typical’ Image Sizes Encapsulating a simple application (for instance a single workflow) results in a machine of size ~ 1.5GB. However options are recently becoming available for including only a liminal operating system layer.
Typically only a few hundred MB for the same applications
Management and Sharing
No specific capabilities provided Docker Hub for centralized Docker image storage, tagging, and sharing
Access Control No specific capabilities provided Docker Hub provides account management and access control
Network Access Can provide network access between Virtual Box VM and external machines/networks.
External network access to Docker image can be provided but with limitations
Host Folder Mounting
Possible but with some additional software installation
Host folder mounting can be done more easily with a single command
12
Virtual Machines and Docker
• Virtual Machines provide
- More robust platform
- Interoperability
- Network access
• Docker provides
- Small application packages
- Hub management
- Security and access control
- “On-demand”
13
Docker File- Docker can build images automatically by reading the instructions
from a Docker file.
- Only requirement is to have docker installed
- Docker file can be created automatically by recording the actions performed just by a command (Using Auto-commit module), which makes it flexible for any user unfamiliar with docker commands
- Just by executing simple text file, the whole system can built from the scratch.
- The idea behind using docker file, it helps to manage size with requirements.
14
• Best supported for Linux but some challenges with Windows and Mac
• Graphical Displays- Could be achieved with X Window and other software on Windows- Challenges with Mac OS
• Network access- Restricted due to security issues- Port forwarding
• Frequent updates to framework
Some Challenges with Docker
15
Working Prototype
Virtual Machine Manager
Auto Pipeline pop-up
Investigator
Data Partner
Image push to Hub
Docker HUB
Web ServiceGAAIN Server
Docker Auto-Invocation Results
16
Features• Flexible and no interoperability issues.
• Better control and management of workflows images through docker hub.
• Better accessibility and ease of use directly.
• Automated invocation of workflows at data partner’s end using java based web-service.
• Dedicated application just for creating and testing workflows, with automated script for pushing it to the hub.
• Minimal size of overall system (1.5-2.0 GB)
17
Some technical issues, we faced- Virtual appliance creation with various minimal install OS’s
- Scripts for automatic invocation of pipeline
- Installation of docker on different Operating system version.
- Compatibility of pipeline with different OS’s
- Memory bubble and deleting existing images from the system.
- Memory Overrun can be solved by deletion of images which are not required.
- Web services
18
• Choice of VM OS
• Choice of Docker OS
• How to get a minimal VM working
• How to work with a minimal Docker image
• How to enable network access in VMs
• Limitations of network access
• How to mount folders from host
• Differences between Linux, Windows and Mac hosts
• How to get GUI displays to work in different Docker images• and VMs
• How to enable external (client/server access) to VMs and• Docker images
• How to autostart applications
• How to manage scripts
Issues Addressed
19
• All files and code for system is provided on the Google Drive shared folder
• Comprehensive “How-To” Manual
Miscellaneous