Creating block volumes

OpenStack provides two classes of block storage, ephemeral storage and persistent volumes. Ephemeral storage exists only for the life of an instance, it will persist across reboots of the guest operating system but when the instance is deleted so is the associated storage. The size of the Ephemeral storage is defined in the flavor of the virtual machine and is fixed for all virtual machines of that particular flavor. The service level for ephemeral storage is dependent on the underlying hardware.

Volumes are persistent virtualized block devices independent of any particular instance. Volumes may be attached to a single instance at a time (i.e. not like a distributed filesystem such as AFS or DFS), but they may be detached or reattached to a different instance while retaining all data, much like a USB drive. Volumes are stored on redundant storage and are therefore more resilient against hardware failures.

You can define additional volumes to be attached to a virtual machine in addition to the system disk. This is useful if you need more space than is provided on the initial flavor and can also allow some interesting flexiblity for snapshotting or moving volumes around between virtual machines.

The steps are

Define a disk volume of the size required (10GB in example below) with the volume name defined (testvol in the example below)
Check that the volume is available for mounting using openstack volume list
Attach that volume to the instance that you would like it to be available on
Log in to the instance and format/mount as required

For example, to define a volume of 10GB named timvol143, attach it to the VM test143 and fornat/mount it as ext4, you do the following on a machine which has the unified openstack command line tool installed.

The volume type describes the type of storage that is being requested.

The following storage types are currently available. Note: the standard type is allocated for projects by default. To request volumes of the other types, please fill out the project quota request form.

Storage Type	Description	Datacentre	Location	Maximum Performance
standard	The 'standard' volume type provides a reliable store which is tolerant to at least one disk failure without user impact or data loss. This is the default.	Meyrin	Main room / Vault *	80MB/s and 100 IO operations (both, read and write)
io1	The 'io1' volume type provides the same reliability level as 'standard' volumes, but is meant for applications with higher IO demands.	Meyrin	Main room / Vault *	120MB/s and 500 IO operations (both, read and write)
io2	The 'io2' volume type provides the same reliability level as 'standard' volumes, but is meant for applications with higher IO demands.	Meyrin	Main room	300MB/S and 1000 IO operations (both, read and write)
io3	The 'io3' volume type provides the same reliability level as 'standard' volumes, but is meant for applications with higher IO demands.	Meyrin	Main room	300MB/S and a rate of 5 IO operation per gigabyte with a guaranteed minimum of 500 IO operations and a maximum of 2000 IO operations (both, read and write)
cp1	The 'cp1' volume type provides the same performance level as 'standard' volumes, but is meant for applications with higher reliability demands: 'cp1' volumes are hosted on critical power which guarantees availability even in the event of a power cut. 'cp1' volumes should hence be used for projects mapped to the critical area.	Meyrin	Critical Area	80MB/s and 100 IO operations (both, read and write)
cpio1	The 'cpio1' volume type combines the reliability of 'cp1' with the performance of 'io1' volumes.	Meyrin	Critical Area	120MB/s and 500 IO operations (both, read and write)
cpio2	The 'cpio2' volume type combines the reliability of 'cp1' with the performance of 'io2' volumes.	Meyrin	Critical Area	300MB/s and 1000 IO operations (both, read and write)
cpio3	The 'cpio3' volume type combines the reliability of 'cp1' with the performance of 'io3' volumes.	Meyrin	Critical Area	300MB/s and 500 IO operations minimum + 5 IO/GB up to 2000 IO (both, read and write)

Note: The location of the volumes for standard/io1 is defined by the storage availability zone where the volume is allocated.

Volume creation

$ openstack volume create --description "Volume for documentation" --size 10 timvol143
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| attachments         | []                                   |
| availability_zone   | ceph-geneva-1                        |
| bootable            | false                                |
| created_at          | 2014-12-18T13:27:27.754856           |
| display_description | Volume for Documentation             |
| display_name        | timvol143                            |
| encrypted           | False                                |
| id                  | d8d8be88-f1a4-4120-b1c2-50621a981c74 |
| properties          |                                      |
| size                | 10                                   |
| snapshot_id         | None                                 |
| source_volid        | None                                 |
| status              | creating                             |
| type                | standard                             |
+---------------------+--------------------------------------+

The size is measured in GB.

The status reported will become available when the volume has been created. This status can be checked using openstack volume show.

The volume just created was allocated automatically to the ceph-geneva-1 storage availability zone. This occurred as it was not specified at creation time.

If you want to create a volume in an specific storage availability zone you can use --availability-zone during the volume creation such as:

$ openstack volume create --description "Volume for documentation" --availability-zone ceph-geneva-2 --size 10 timvol144
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| attachments         | []                                   |
| availability_zone   | ceph-geneva-1                        |
| bootable            | false                                |
| created_at          | 2014-12-18T13:27:27.754856           |
| display_description | Volume for Documentation             |
| display_name        | timvol143                            |
| encrypted           | False                                |
| id                  | d8d8be88-f1a4-4120-b1c2-50621a981c74 |
| properties          |                                      |
| size                | 10                                   |
| snapshot_id         | None                                 |
| source_volid        | None                                 |
| status              | creating                             |
| type                | standard                             |
+---------------------+--------------------------------------+

At creation time, you can also specify the availability zone of the volume ()

Note: storage availability zones are only supported for standard and io1 volume types.

Attaching a volume

Once the volume has been created, it can be attached to the virtual machine.

$ openstack server add volume tim-centos7-143 timvol143

The arguments are

tim-centos7-143 is the virtual machine where you would like to attach the image. This is as previously created using openstack server create.
timvol143 is the volume id from openstack volume list as created above

Note: The optional device argument to specify the device name should not be used. It does not work properly, and will be removed in the near future.

The operation can be checked by seeing if this is now shown in the openstack server list command. Log into the VM to check and format the disk:

$ ssh root@tim-centos7-143
# grep vdb /proc/partitions
 252       48   12582912 vdb

Create a file system on it and mount it. Using a volume label is useful in the future to avoid needing to find the exact device name as you attach different volumes. Using a label, such as testvol in the example below, which is the same as the name in OpenStack also makes tracking easier when volumes are not mounted. The filesystem type can be one supported by the Linux distribution but xfs and ext4 are the most widely used.

# mkfs -t ext4 -L testvol /dev/vdb
# mount -L testvol /mnt
# df

If you'd like non-root users to be able to access the mounted volume similar to the way /tmp is accessible , you'll need to set the sticky bit on the mount point.

# chmod +t /mnt

The filesystem should be added to /etc/fstab if the desired behaviour is to mount automatically on restart. Using the LABEL parameter here will ensure the correct volume is chosen if multiple volumes are attached.

LABEL=testvol /mnt ext4 noatime,nodiratime,user_xattr,nofail    0       0

Note: The use of nofail is recommended to skip (and not block on) mounting the volume if it is unavailable, e.g. in case of network issues. Remove this option from the fstab line if you want your VM to block the boot process if the volume is unavailable.

Note: The use of nobarrier for the mount options is not recommended as volumes are accessed via a cache, so ignoring the correct ordering of journal commits may leave you with a corrupted file system in case of a hardware problem.

Removing disk volumes

The steps to properly remove disk volumes are:

Umount the volume on the virtual machine
Detach it from the virtual machine
Delete the volume
Logged into the virtual machine that has the volume mounted, unmount it first

# umount /mnt

Note: detaching a volume while it is being mounted is possible, but the next access on the VM to that volume will return I/O errors.

Then use the openstack command line interface to detach the volume from the virtual machine:

$ openstack server remove volume tim-centos7-143 timvol143

where tim-centos7-143 is the virtual machine and the second parameter, timvol143, is the volume as shown in openstack volume list.

Check that the volume is in state 'available' again.

If that's the case, the volume is now ready to either be attached to another virtual machine or, if it is not needed any longer, to be completely removed (note that this step cannot be reverted!):

$ openstack volume delete timvol143

Your volume will now go into state 'deleting' and completely disappear from the openstack volume list output

Volume snapshots

A user can also create snapshots of a volume, that then later can be used to create other volumes or to rollback to a precedent point in time. Volume snapshots are pointers in the RW history of a volume. The creation of a snapshot takes a few seconds and it can be done while the volume is in-use.

Once you have the snapshot, you can use it to create other volumes based on this snapshot. Creation time for these volumes may depend on the type of the volume you are creating as it may entitle some data transfer.

$ openstack volume snapshot create --volume timvol143 timvolsnap143
+-------------+--------------------------------------+
| Field       | Value                                |
+-------------+--------------------------------------+
| created_at  | 2020-12-10T09:16:45.749407           |
| description | None                                 |
| id          | 7c63c503-8cb5-40bd-b434-9b638ecb4fc3 |
| name        | timvolsnap143                        |
| properties  |                                      |
| size        | 1                                    |
| status      | creating                             |
| updated_at  | None                                 |
| volume_id   | f3709518-b67b-49a8-ad03-0ba5f115ab67 |
+-------------+--------------------------------------+

Note: if the volume is in-use, you may need to specify --force

You can list the volume snapshots with the following command.

$ openstack volume snapshot list
+--------------------------------------+---------------+-------------+-----------+------+
| ID                                   | Name          | Description | Status    | Size |
+--------------------------------------+---------------+-------------+-----------+------+
| 7c63c503-8cb5-40bd-b434-9b638ecb4fc3 | timvolsnap143 | None        | available |    1 |
+--------------------------------------+---------------+-------------+-----------+------+

Once the volume snapshot is in available state, then you can create other volumes based on that snapshot. You don't need to specify the size of the volume, it will use the size of the snapshot.

$ openstack volume create --description --source timvolsnap143 "Volume from an snapshot" timvol143
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| attachments         | []                                   |
| availability_zone   | ceph-geneva-1                        |
| bootable            | false                                |
| consistencygroup_id | None                                 |
| created_at          | 2020-12-10T09:19:45.000000           |
| description         | None                                 |
| encrypted           | False                                |
| id                  | 68fb4e47-39e6-40b5-a12d-371b34989b83 |
| multiattach         | False                                |
| name                | timvol144                            |
| properties          |                                      |
| replication_status  | None                                 |
| size                | 1                                    |
| snapshot_id         | 7c63c503-8cb5-40bd-b434-9b638ecb4fc3 |
| source_volid        | None                                 |
| status              | creating                             |
| type                | standard                             |
| updated_at          | None                                 |
+---------------------+--------------------------------------+

You can delete the snapshots just by issuing the following command

$ openstack volume snapshot delete timvolsnap143

Common Error Messages

From the openstack volume command, the following errors can be received

Error	Cause
VolumeSizeExceedsAvailableQuota: Requested volume or snapshot exceeds allowed Gigabytes quota ERROR: openstack Request Entity Too Large (HTTP 413)	Your total space for the volume service exceeds the allowance for the project. If you require further quota, please contact the helpdesk. Please note that additional quota is only granted for shared projects. Additional personal project quota requests will not be accepted.
VolumeLimitExceeded: Maximum number of volumes allowed (N) exceeded	You have exceeded the number of volumes. Please contact the helpdesk to request to raise this limit for a project. Additional personal project quota requests will not be accepted.
BadRequest: Failed to delete volume with name or ID: Invalid volume: Volume status must be available or error or error_restoring or error_extending or error_managing and must not be migrating, attached, belong to a group, have snapshots or be disassociated from snapshots after volume transfer. (HTTP 400)	If you are deleting a volume and has
associated snapshots, you need to delete them first, before deleting the volume.

Software RAID (md) with volumes

Since they provide a block device interface, volumes can be used to create software RAID (md) devices. While this is usually sensible when using real disk devices to either increase data reliability, to create large devices or to improve the overall IO performance, it usually does not make sense for volumes. Volumes can be created with arbitrary sizes, are replicated already on the backend, and there are special volume types for higher IO demands. Depending on the RAID level, RAIDing volumes can even have a negative impact as all IO goes over the network and some transactions, such as small writes, require preceeding read operations. If in doubt, please open a ticket to the cloud team via opening a service desk ticket.