© Hitachi Data Systems Corporation 2015. All rights reserved.
Efficient Image Management
using Cinder Volumes
for Virtual and Baremetal Machines
Tomoki Sekiyama
Mitsuhiro Tanino
1
© Hitachi Data Systems Corporation 2015. All rights reserved.
Background
New features for Efficient Image Handling
Use case of Volume-backed Images
Usage of Volume-backed Images
Current Limitations
Future Work
2
Contents
© Hitachi Data Systems Corporation 2015. All rights reserved.
(3) Storage
virtualization
and automatic data
optimization using
thin provision and
tiering
Volume
Requirement for block storage solution
4
VM
(4) Taking backup and
snapshot instantly via
storage feature
(2) Provide stable
IOPS and low
latency using boot
from volume
Volume
(1) Create a boot volume
using COW snapshot
SATA
SAS
Flash
Stable Performance
Virtualization/ Optimization
Business Continuity
Agility of booting an instance
© Hitachi Data Systems Corporation 2015. All rights reserved.
HDD
Problems (1/3)
Agility is an important factor for OpenStack clouds
• Required to rapidly boot instances
When the image size is large, “nova boot” takes a long time
to download the image from Glance
• Nova caches images (per host), but not effective to first boot
• Cinder-volume-boot always downloads the image by default
5
Download Image Cached
Image
Guest
Image
VM
Volume VM
Image
Compute
Node
Storage
Array
© Hitachi Data Systems Corporation 2015. All rights reserved.
Booting baremetal nodes requires to copy the image to local disk
• Download & copy takes a long time
• Causes high network traffic
• Ironic caches the downloaded images
same as Nova but image copy to
attached disk is always required.
Problems (2/3)
6
HDD
1. Boot with
deploy image
2. Export the disk
as iSCSI target
Image
3. Download & Copy Image
4. Reboot
Attach
Baremetal
Node
© Hitachi Data Systems Corporation 2015. All rights reserved.
Image copy may affect disk I/O performance of instances
• Example: sysbench OLTP benchmark during volume creation
– Measured on a KVM instance with a volume (LVM-iSCSI backend)
– During the image copy to a volume, I/O performance degrades
– The interference can be mitigated by image copy I/O bandwidth limit, but
image copy takes longer time
Volume creation from a snapshot is rapid, but snapshots cannot be
shared among tenants
• Not suitable for public images (e.g. operating systems)
Problems (3/3)
7
© Hitachi Data Systems Corporation 2015. All rights reserved.
Target Environment
Nova: VM, Baremetal (Ironic)
Cinder + Storage array
Cinder
Glance
Nova
Ironic
VM
* Currently baremetal does not
support Cinder volumes
8
Baremetal nodes
VM
VM VM
VM VM
VM VM
Storage Array
Compute Nodes
Control Node
SAN (FC/iSCSI)
etc.
© Hitachi Data Systems Corporation 2015. All rights reserved.
New features for Efficient Image Handling
9
© Hitachi Data Systems Corporation 2015. All rights reserved.
Cinder New features for Image Handling In Liberty release, some features are added to Cinder for efficient
image management
• Image-Volume Cache
– Cache recently used images as “image volumes”
• Volume-backed Image
– Store an Glance image data in a Cinder volume
In both features, a new bootable volume can be created rapidly by
cloning the image volume
Both features are disabled by default
10
© Hitachi Data Systems Corporation 2015. All rights reserved.
Internal tenant
Overview of Image-Volume Cache
Images recently used by Cinder are cached as Cinder volumes
• Each image is stored in a volume (Image-Volume)
• Image volumes are placed in the internal tenant
Automatic management
• If cache volumes exceed the specific amount size, recently
unused cache volumes are deleted
Support various disk formats (converted to raw before cached)
Can coexist with the volume-backed Image feature
11
Download
(First time only)
Image Vol.
(Cache)
Image
Volume VM
Clone Boot Storage
Array
© Hitachi Data Systems Corporation 2015. All rights reserved.
Overview of Volume-backed Image
Register an image volume as a Glance image
• Utilize Glance’s Cinder store
– The image data is stored in a Cinder volume (Image-Volume)
• No image data transfer between Cinder and Glance
New volume can be created rapidly by cloning the image volume
12
Image
Volume
Volume-
backed
Image
Volume VM
Clone Boot
location = cinder://1234-abc..
Storage
Array
© Hitachi Data Systems Corporation 2015. All rights reserved.
Rapid Boot for Virtual Machine Instances
Rapidly boot an instance with a new volume from an image
• Reduce time to launch instances
• Reduce I/O workload
Significant on booting multiple instances
Example:
• Booting an instance from a new Cinder volume
created from 20 GB operating system image
– Measured with thin-provisioning LVM backend
14
0 50 100 150 200 250
default
Volume-backed
Setup instance Volume creation Volume attach Spawning
[s]
246s
11s
© Hitachi Data Systems Corporation 2015. All rights reserved.
Boot Volume Boot Volume
Rapid Boot for Baremetal Instances (TBD) Currently baremetal boot requires image copy to local HDD drive
Ongoing work to support volume boot for Ironic
Combined with volume-backed images, deploy baremetal nodes
without copying image data to local HDD
15
Image Volume
Boot Volume
iSCSI / FC
1. Clone
2. Attach & Boot
Baremetal Nodes
© Hitachi Data Systems Corporation 2015. All rights reserved.
Copy-offload of Image Data
Leverage the storage array’s copy-offloading features
• Some storages support copy-on-write based cloning
– No data transfer
– No interference to instances’ performance
Example: sysbench OLTP benchmark during volume creation
• When thin-provisioning is supported, this feature also improve
storage capacity
16
© Hitachi Data Systems Corporation 2015. All rights reserved.
Admin
tenant
Sharing volume data among tenants
Sharing volume data among tenants
• The visibility of the Volume-backed image is managed by
Glance’s ACL feature
– Public image volume
– Sharing among specific members
• Useful to share base images such as operating system images,
master datasets, etc.
17
Base OS
image
Tenant A Tenant B
Guest 2 Guest 4
Guest 1 Guest 3
Public
Volume-
backed
Image
*The image must be
registered by volume owner
© Hitachi Data Systems Corporation 2015. All rights reserved.
Registration of Volume-backed Image (1/2) Using Glance CLI (with Image API v2)
• glance image-create --disk-format raw \ --container-format bare --name <image-name>
• glance location-add <image-uuid> \ --url cinder://<volume-uuid>
• NOTE: the registered volume data shouldn’t be modified
+------------------+-------------------------------------------------------------+ | Property | Value | +------------------+-------------------------------------------------------------+ | checksum | None | | container_format | bare | | created_at | 2015-09-22T21:31:34Z | | disk_format | raw | | id | f698173d-b96f-43be-aae9-fa0d13751c09 | | locations | [{"url": "cinder://95e571f9-5ccd-45eb-b282-441c3ce9a5db", | | | "metadata": {}}] | … | size | 1073741824 | | visibility | private | +------------------+-------------------------------------------------------------+
19
© Hitachi Data Systems Corporation 2015. All rights reserved.
Registration of Volume-backed Image (2/2) “cinder upload-to-image” command can be optionally configured
to create a volume-backed image
• cinder upload-to-image <volume> <new-image-name>
• The specified volume is cloned to create a new image volume
• The cloned volume is set to read-only
• The cloned volume’s URL is registered to a new Glance image
• To enable this behavior, the following options is required:
– image_upload_use_cinder_backend = True
• Only raw format is supported
20
Image
Volume
New
Image
Volume
Clone
Register
location
Storage
Array
© Hitachi Data Systems Corporation 2015. All rights reserved.
New Volume from Volume-backed Image
To create a new volume from a volume-backed Image:
• cinder create --image <image-uuid> <size>
• The new volume is cloned from the image volume
• The volume is extended when the volume size is larger than the
image volume
In Horizon, new instance
can be launched from a new
volume cloned from the
volume-backed image
21
© Hitachi Data Systems Corporation 2015. All rights reserved.
Enable Volume-backed Image Features
Glance settings (/etc/glance/glance-api.conf)
• Enable Cinder store
– [glance_store] :: stores = file,http,swift,cinder
• Expose image locations (URL)
– [DEFAULT] :: show_multiple_locations = True
Cinder settings (/etc/cinder/cinder.conf)
• Enable Glance API version 2
– [DEFAULT] :: glance_api_version = 2
• Enable volume creation by cloning image volumes
– [DEFAULT] :: allowed_direct_url_schemes = cinder
• (optional) To use “cinder upload-to-image” to create volume-backed image
– Backend section :: image_upload_use_cinder_backend = True
22
© Hitachi Data Systems Corporation 2015. All rights reserved.
Enable Image-Volume Cache
Image-Volume Cache and Volume-backed image feature can
coexist
Cinder settings (/etc/cinder/cinder.conf)
• Enable internal tenant
– [DEFAULT] :: cinder_internal_tenant_project_id = ...
– [DEFAULT] :: cinder_internal_tenant_user_id = ...
• Enable Image-Volume Cache
– Backend section :: image_volume_cache_enabled = True
• (optional) Limit max capacity / number of cache volumes
– Backend section :: image_volume_cache_max_size_gb = ...
and/or
– Backend section :: image_volume_cache_max_count = ...
23
© Hitachi Data Systems Corporation 2015. All rights reserved.
Current Limitation (1/2)
Cannot clone volumes between multiple hosts / backends
• Need to create an image volume on each host / backend
• and register their locations to the volume-backed image
Volume-backed images must be in raw format
• Image-Volume Cache supports various disk formats
– Automatically converts to raw type
25
© Hitachi Data Systems Corporation 2015. All rights reserved.
Current Limitation (2/2) Volume-backed images can only be used to create new volumes
• Nova and other components cannot access volume-backed image
contents
– Currently Glance’s Cinder store functionality is quite limited
– We are proposing a patch to enable Glance download from /
upload to volume-backed images
Image volumes created by “upload-to-image” are visible
• The volume owner may destroy the image volumes
• The image volumes can be hidden from users by storing them in
the internal tenant
– “upload_image_use_internal_tenant = True” in cinder.conf
– Require the Glance patch
26
© Hitachi Data Systems Corporation 2015. All rights reserved.
Current Situation of Glance Cinder-store Lacking important features (considered almost “broken”)
• Only provides a pointer to an existing volume
• Only can be used to create a new volume
• Glance cannot access to Image Volume contents
→ Clients cannot upload & download images
• Doesn’t delete the image volume when the image is deleted
• Cannot specify user/tenant to access Cinder
– Always use current context
– The image volume is visible to users
27
Image
Volume
Volume
backed
Image
Volume
© Hitachi Data Systems Corporation 2015. All rights reserved.
Proposed Cinder-store feature for Glance We are proposing patches for cinder store:
Glance patch: https://review.openstack.org/#/c/186201/
Glance_store patch: https://review.openstack.org/#/c/166414/
• Implement download/upload
– Attach Cinder volumes to Glance node using os-brick library
• Support image volume deletion
• Add settings for user/tenant to store images
– Useful to store images in the internal tenant
This enables cinder store to be used as default store
Nova will also be able to download the image for image-boot
28
Image
Volume
Volume
backed
Image Attach
Volume
© Hitachi Data Systems Corporation 2015. All rights reserved.
Boot Volume Boot Volume
Ironic: Baremetal Volume-Boot
Specs for supporting volume-boot of baremetal nodes is proposed:
• Ironic-spec: https://review.openstack.org/#/c/200496/
• Nova-spec: https://review.openstack.org/#/c/211101/
Combining with volume-boot and cinder-backed images, the image
can be rapidly deployed to the baremetal node.
30
Image Volume
Boot Volume
iSCSI / FC
1. Clone
2. Attach & Boot
Baremetal Nodes
© Hitachi Data Systems Corporation 2015. All rights reserved.
Nova: Copyless Image Boot
Attaching volume to Nova nodes instead of download the base
image
• Bypass the first image download
• Improve boot time of image-boot instances
31
Guest
Image
VM
Volume-
backed
Image
Compute
Node
Storage
Array
Image
Volume
Attach
Image
Volume
© Hitachi Data Systems Corporation 2015. All rights reserved.
Wrap-up
Cinder volume-backed images are useful to:
• Rapid boot of volume-boot VM instances
• Rapid boot of baremetal instances in the future
– Ironic work for volume-boot is ongoing
• Share volume data (e.g. base OS image) among tenants
• Leverage the storage features for image management
32
© Hitachi Data Systems Corporation 2015. All rights reserved.
Before Liberty & after: Image and Volume boot
Wrap-up
33
# Boot method Boot image type Kilo Liberty Mitaka
1 Boot from volume Image file
2 Boot from snapshot
(creates a new volume) Cinder snapshot
3 Boot from image
(creates a new volume)
Cinder volume *1
*2
4 Boot from image Image file
*1 Limitation: Clients cannot upload & download images at this version.
*2 We are proposing Glance to improve cinder store to enable Cinder volumes to
be used as default images store
• Reviews are welcome!
© Hitachi Data Systems Corporation 2015. All rights reserved.
Disclaimer
The OpenStack® Word Mark and OpenStack Logo are either registered
trademarks/service marks or trademarks/service marks of the OpenStack
Foundation in the United States and other countries and are used with the
OpenStack Foundation's permission. We are not affiliated with, endorsed or
sponsored by the OpenStack Foundation, or the OpenStack community.
Other company, product or service names may be trademarks or service mark
of others.
34