Creating block volumes
OpenStack provides two classes of block storage, ephemeral storage and persistent volumes. Ephemeral storage exists only for the life of an instance, it will persist across reboots of the guest operating system but when the instance is deleted so is the associated storage. The size of the Ephemeral storage is defined in the flavor of the virtual machine and is fixed for all virtual machines of that particular flavor. The service level for ephemeral storage is dependent on the underlying hardware.
Volumes are persistent virtualized block devices independent of any particular instance. Volumes may be attached to a single instance at a time (i.e. not like a distributed filesystem such as AFS or DFS), but they may be detached or reattached to a different instance while retaining all data, much like a USB drive. Volumes are stored on redundant storage and are therefore more resilient against hardware failures.
You can define additional volumes to be attached to a virtual machine in addition to the system disk. This is useful if you need more space than is provided on the initial flavor and can also allow some interesting flexiblity for snapshotting or moving volumes around between virtual machines.
The steps are
- Define a disk volume of the size required (10GB in example below) with the volume name defined (testvol in the example below)
- Check that the volume is available for mounting using
openstack volume list - Attach that volume to the instance that you would like it to be available on
- Log in to the instance and format/mount as required
For example, to define a volume of 10GB named timvol143, attach it to the VM test143 and fornat/mount it as ext4, you do the following on a machine which has the unified openstack command line tool installed.
The volume type describes the type of storage that is being requested.
The following storage types are currently available. Note: the standard type is allocated for projects by default. To request volumes of the other types, please fill out the project quota request form.
| Storage Type | Description | Datacentre | Location | Maximum Performance |
|---|---|---|---|---|
| standard | The 'standard' volume type provides a reliable store which is tolerant to at least one disk failure without user impact or data loss. This is the default. | Meyrin | Main room / Vault * | 80MB/s and 100 IO operations (both, read and write) |
| io1 | The 'io1' volume type provides the same reliability level as 'standard' volumes, but is meant for applications with higher IO demands. | Meyrin | Main room / Vault * | 120MB/s and 500 IO operations (both, read and write) |
| io2 | The 'io2' volume type provides the same reliability level as 'standard' volumes, but is meant for applications with higher IO demands. | Meyrin | Main room | 300MB/S and 1000 IO operations (both, read and write) |
| io3 | The 'io3' volume type provides the same reliability level as 'standard' volumes, but is meant for applications with higher IO demands. | Meyrin | Main room | 300MB/S and a rate of 5 IO operation per gigabyte with a guaranteed minimum of 500 IO operations and a maximum of 2000 IO operations (both, read and write) |
| cp1 | The 'cp1' volume type provides the same performance level as 'standard' volumes, but is meant for applications with higher reliability demands: 'cp1' volumes are hosted on critical power which guarantees availability even in the event of a power cut. 'cp1' volumes should hence be used for projects mapped to the critical area. | Meyrin | Critical Area | 80MB/s and 100 IO operations (both, read and write) |
| cpio1 | The 'cpio1' volume type combines the characteristics of 'cp1' and 'io1' volumes. | Meyrin | Critical Area | 120MB/s and 500 IO operations (both, read and write) |
Note: The location of the volumes for standard/io1 is defined by the storage availability zone where the volume is allocated.
Volume creation
$ openstack volume create --description "Volume for documentation" --size 10 timvol143
+---------------------+--------------------------------------+
| Field | Value |
+---------------------+--------------------------------------+
| attachments | [] |
| availability_zone | ceph-geneva-1 |
| bootable | false |
| created_at | 2014-12-18T13:27:27.754856 |
| display_description | Volume for Documentation |
| display_name | timvol143 |
| encrypted | False |
| id | d8d8be88-f1a4-4120-b1c2-50621a981c74 |
| properties | |
| size | 10 |
| snapshot_id | None |
| source_volid | None |
| status | creating |
| type | standard |
+---------------------+--------------------------------------+
The size is measured in GB.
The status reported will become available when the volume has been created. This status can be checked using openstack volume show.
The volume just created was allocated automatically to the ceph-geneva-1 storage availability zone. This occurred as it was not specified at creation time.
If you want to create a volume in an specific storage availability zone you can use --availability-zone during the volume creation such as:
$ openstack volume create --description "Volume for documentation" --availability-zone ceph-geneva-2 --size 10 timvol144
+---------------------+--------------------------------------+
| Field | Value |
+---------------------+--------------------------------------+
| attachments | [] |
| availability_zone | ceph-geneva-1 |
| bootable | false |
| created_at | 2014-12-18T13:27:27.754856 |
| display_description | Volume for Documentation |
| display_name | timvol143 |
| encrypted | False |
| id | d8d8be88-f1a4-4120-b1c2-50621a981c74 |
| properties | |
| size | 10 |
| snapshot_id | None |
| source_volid | None |
| status | creating |
| type | standard |
+---------------------+--------------------------------------+
At creation time, you can also specify the availability zone of the volume ()
Note: storage availability zones are only supported for standard and io1 volume types.
Attaching a volume
Once the volume has been created, it can be attached to the virtual machine.
$ openstack server add volume tim-centos7-143 timvol143
The arguments are
- tim-centos7-143 is the virtual machine where you would like to attach the image. This is as previously created using
openstack server create. - timvol143 is the volume id from
openstack volume listas created above
Note: The optional device argument to specify the device name should not be used. It does not work properly, and will be removed in the near future.
The operation can be checked by seeing if this is now shown in the openstack server list command.
Log into the VM to check and format the disk:
$ ssh root@tim-centos7-143
# grep vdb /proc/partitions
252 48 12582912 vdb
Create a file system on it and mount it. Using a volume label is useful in the future to avoid needing to find the exact device name as you attach different volumes. Using a label, such as testvol in the example below, which is the same as the name in OpenStack also makes tracking easier when volumes are not mounted. The filesystem type can be one supported by the Linux distribution but xfs and ext4 are the most widely used.
# mkfs -t ext4 -L testvol /dev/vdb
# mount -L testvol /mnt
# df
If you'd like non-root users to be able to access the mounted volume similar to the way /tmp is accessible , you'll need to set the sticky bit on the mount point.
# chmod +t /mnt
The filesystem should be added to /etc/fstab if the desired behaviour is to mount automatically on restart. Using the LABEL parameter here will ensure the correct volume is chosen if multiple volumes are attached.
LABEL=testvol /mnt ext4 noatime,nodiratime,user_xattr,nofail 0 0
Note: The use of nofail is recommended to skip (and not block on) mounting the volume if it is unavailable, e.g. in case of network issues. Remove this option from the fstab line if you want your VM to block the boot process if the volume is unavailable.
Note: The use of nobarrier for the mount options is not recommended as volumes are accessed via a cache, so ignoring the correct ordering of journal commits may leave you with a corrupted file system in case of a hardware problem.
Removing disk volumes
The steps to properly remove disk volumes are:
- Umount the volume on the virtual machine
- Detach it from the virtual machine
- Delete the volume
- Logged into the virtual machine that has the volume mounted, unmount it first
# umount /mnt
Note: detaching a volume while it is being mounted is possible, but the next access on the VM to that volume will return I/O errors.
Then use the openstack command line interface to detach the volume from the virtual machine:
$ openstack server remove volume tim-centos7-143 timvol143
where tim-centos7-143 is the virtual machine and the second parameter, timvol143, is the volume as shown in openstack volume list.
Check that the volume is in state 'available' again.
If that's the case, the volume is now ready to either be attached to another virtual machine or, if it is not needed any longer, to be completely removed (note that this step cannot be reverted!):
$ openstack volume delete timvol143
Your volume will now go into state 'deleting' and completely disappear from the openstack volume list output
Volume snapshots
A user can also create snapshots of a volume, that then later can be used to create other volumes or to rollback to a precedent point in time. Volume snapshots are pointers in the RW history of a volume. The creation of a snapshot takes a few seconds and it can be done while the volume is in-use.
Once you have the snapshot, you can use it to create other volumes based on this snapshot. Creation time for these volumes may depend on the type of the volume you are creating as it may entitle some data transfer.
$ openstack volume snapshot create --volume timvol143 timvolsnap143
+-------------+--------------------------------------+
| Field | Value |
+-------------+--------------------------------------+
| created_at | 2020-12-10T09:16:45.749407 |
| description | None |
| id | 7c63c503-8cb5-40bd-b434-9b638ecb4fc3 |
| name | timvolsnap143 |
| properties | |
| size | 1 |
| status | creating |
| updated_at | None |
| volume_id | f3709518-b67b-49a8-ad03-0ba5f115ab67 |
+-------------+--------------------------------------+
Note: if the volume is in-use, you may need to specify --force
You can list the volume snapshots with the following command.
$ openstack volume snapshot list
+--------------------------------------+---------------+-------------+-----------+------+
| ID | Name | Description | Status | Size |
+--------------------------------------+---------------+-------------+-----------+------+
| 7c63c503-8cb5-40bd-b434-9b638ecb4fc3 | timvolsnap143 | None | available | 1 |
+--------------------------------------+---------------+-------------+-----------+------+
Once the volume snapshot is in available state, then you can create other volumes based on that snapshot. You don't need to specify the size of the volume, it will use the size of the snapshot.
$ openstack volume create --description --source timvolsnap143 "Volume from an snapshot" timvol143
+---------------------+--------------------------------------+
| Field | Value |
+---------------------+--------------------------------------+
| attachments | [] |
| availability_zone | ceph-geneva-1 |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2020-12-10T09:19:45.000000 |
| description | None |
| encrypted | False |
| id | 68fb4e47-39e6-40b5-a12d-371b34989b83 |
| multiattach | False |
| name | timvol144 |
| properties | |
| replication_status | None |
| size | 1 |
| snapshot_id | 7c63c503-8cb5-40bd-b434-9b638ecb4fc3 |
| source_volid | None |
| status | creating |
| type | standard |
| updated_at | None |
+---------------------+--------------------------------------+
You can delete the snapshots just by issuing the following command
$ openstack volume snapshot delete timvolsnap143
Common Error Messages
From the openstack volume command, the following errors can be received
| Error | Cause |
|---|---|
| VolumeSizeExceedsAvailableQuota: Requested volume or snapshot exceeds allowed Gigabytes quota ERROR: openstack Request Entity Too Large (HTTP 413) |
Your total space for the volume service exceeds the allowance for the project. If you require further quota, please contact the helpdesk. Please note that additional quota is only granted for shared projects. Additional personal project quota requests will not be accepted. |
| VolumeLimitExceeded: Maximum number of volumes allowed (N) exceeded | You have exceeded the number of volumes. Please contact the helpdesk to request to raise this limit for a project. Additional personal project quota requests will not be accepted. |
| BadRequest: Failed to delete volume with name or ID: Invalid volume: Volume status must be available or error or error_restoring or error_extending or error_managing and must not be migrating, attached, belong to a group, have snapshots or be disassociated from snapshots after volume transfer. (HTTP 400) | If you are deleting a volume and has |
| associated snapshots, you need to delete them first, before deleting the volume. |
Software RAID (md) with volumes
Since they provide a block device interface, volumes can be used to create software RAID (md) devices. While this is usually sensible when using real disk devices to either increase data reliability, to create large devices or to improve the overall IO performance, it usually does not make sense for volumes. Volumes can be created with arbitrary sizes, are replicated already on the backend, and there are special volume types for higher IO demands. Depending on the RAID level, RAIDing volumes can even have a negative impact as all IO goes over the network and some transactions, such as small writes, require preceeding read operations. If in doubt, please open a ticket to the cloud team via opening a service desk ticket.