Backup and snapshots
The virtual machine systems disks backups are covered in Using OpenStack. This section covers backup and recovery of volumes allocated external to the virtual machine system disks.
As mentioned in the system disk backup section, there is no default backup performed for the cloud. Data can be stored on one of the IT reliable data stores such as
- Database-on-Demand
- Oracle
- AFS
- DFS
However, if you wish to use an external volume attached to a virtual machine and have that data backed up, there are a number of possible approaches.
If creating a snapshot of a Windows host, beware that a Local Administrator account needs to be present in order to boot later off the snapshot. Without the Administrator account locally available, the backup may be unusable.
Do not use the openstack backup create command. This has not been configured for the CERN OpenStack cloud and will produce the error "Service cinder-backup could not be found"
It's worth to mention that there are 2 types of machines and their procedures to snapshot the instance data is different on each of them
- Boot from Image. In this case the machines boots from a local file in the server
- Boot from Volume. In this case the machine boots from a network attached storage device
You can identify the type of your machine by looking at the Image attribute of your machine.
- If the value has a value, then it is a boot from image instance
- If the value is None (or:empty) and there are volumes attached then it's a boot from volume instance
$ openstack server show jcastdev01 -c id -c name -c image -c volumes_attached
+------------------+-------------------------------------------+
| Field | Value |
+------------------+-------------------------------------------+
| id | 696b79ab-8669-436a-88ea-4f3aef5952de |
| image | |
| name | jcastdev01 |
| volumes_attached | id='e5c51107-eb5f-4a4d-bd23-774bc6f0bea8' |
+------------------+-------------------------------------------+
$ openstack server show jcastdev02 -c id -c name -c image -c volumes_attached
+------------------+-------------------------------------------+
| Field | Value |
+------------------+-------------------------------------------+
| id | 747001ff-d23e-49cf-8287-1bcc82d0f51e |
| image | 430fd6c0-1abc-4e53-8e64-03e2edf5b8b1 |
| name | jcastdev02 |
| volumes_attached | id='7dc23ace-1abb-48c8-8ec5-fd3df5be7def' |
+------------------+-------------------------------------------+
In this case, the first machine is a boot from volume one and the second is a boot from instance
To do an snapshot, you need to follow a different procedures depending if it's a boot from image instance or a boot from volume one.
Boot from image instance => image snapshot boot from volume instance => volume snapshot
Image snapshot
On boot from image instances, the snapshots are done directly on the instance panel (create snapshot action) in the website or using the CLI
$ openstack server image create jcastdev02 --name testimage
If you are running it on a live machine, this operation may slow down the disk access of your instance.
Volume snapshot
On boot from volume instances, the instances do not have any data in the server as they are using a volume as boot. In this case you need to do an snapshot of the volume in the volumes panel in the website or using the CLI as follows. First you need to identify the volume in which the machine is booting from (/dev/vda)
$ openstack volume list
+--------------------------------------+-------------------+--------+------+-------------------------------------+
| ID | Name | Status | Size | Attached to |
+--------------------------------------+-------------------+--------+------+-------------------------------------+
| e5c51107-eb5f-4a4d-bd23-774bc6f0bea8 | jcastdev01_system | in-use | 80 | Attached to jcastdev01 on /dev/vda |
| 7dc23ace-1abb-48c8-8ec5-fd3df5be7def | jcastdata | in-use | 54 | Attached to jcastdev02 on /dev/vdb |
+--------------------------------------+-------------------+--------+------+-------------------------------------+
$ openstack volume snapshot create --volume jcastdev01_system testsnapshot --force
+-------------+--------------------------------------+
| Field | Value |
+-------------+--------------------------------------+
| created_at | 2020-09-11T08:42:31.040913 |
| description | None |
| id | eb1953f6-a74b-4308-8d4a-b276be56a501 |
| name | testsnapshot |
| properties | |
| size | 80 |
| status | creating |
| updated_at | None |
| volume_id | e5c51107-eb5f-4a4d-bd23-774bc6f0bea8 |
+-------------+--------------------------------------+
In this case, we do a list of volumes to check where the volume was mounted and then create an snapshot on it. In order to do it on a volume 'in-use' we must use the force parameter. This operation is safe to do on a live system.
Note: If you incidentally perform an image snapshot on a boot from volume instance, as the instance does not have local data in the server in which it's running, the procedure will succeed but with an image with zero size. If you notice this case, please change to the volume snapshot procedure.
Locally installed backup client
To use a local backup client (as described in Backups and snapshots), the volume would need to be mounted on the virtual machine where the client is installed.
The include/exclude lists for filesystems and files can then be adjusted to include the newly mounted file system. This is especially useful where you want to be able to restore individual files.
Use cases
Those are some examples of different use cases in which snapshots can be used.
Rollback & rebuild
Typical cases for rollback/rebuild are test environments. A reference volume is created at the start of a loop of tests, each test resets to its initial state and then rolls back when the test is completed for the next test to run.
This can be performed as follows:
- A master volume is created as described in Create volumes
- A clone volume is created using the master contents as a source
- Work is done using the volume, such as running a test
- The clone volume is deleted and re-created from the master contents
The state of the VM before the operation is as follows
- A volume timmaster143 has been created
- It has been mounted on /mnt/master on the VM and its contents initialised
- It has been unmounted and is available in
openstack volume list
On the client: detach the volume, clone it and attach it
$ openstack server remove volume timhost143 timmaster143
$ openstack volume create --source timmaster143 timclone143
$ openstack server add volume timhost143 timclone143
The tests can then be run by re-mounting the clone volume and working with it. Once the tests are completed, the
volume can be detached from the host using openstack server remove, deleted and re-created from the master.
Errors
Invalid volume: must be available
Snapshots for testing and cloning
The typical use case for testing and cloning is a production service where there is a significant change, such as new application software to be validated. This assumes that the application executables and data are stored on an external volume. The steps that would be needed would be
- Create a new VM for the test environment (the approach outlined in Backups and snapshots could be used or a new virtual machine created with configuration management tools such as Puppet)
- Create a snapshot of the existing application volume
- Create a volume from the snapshot and attach it to the test VM
- Run the tests
- Delete the test VM and volume on completion
The snapshotting steps are as follows assuming a master volume puppet-train-vol.
$ openstack volume snapshot create --volume puppet-train-vol puppet-train-vol-snapshot
+-------------+--------------------------------------+
| Field | Value |
+-------------+--------------------------------------+
| created_at | 2020-07-27T12:15:56.447368 |
| description | None |
| id | 1c4051e0-ae74-487b-ab94-95ddb6ffd8b3 |
| name | puppet-train-vol-snapshot |
| properties | |
| size | 1 |
| status | creating |
| updated_at | None |
| volume_id | cc3ec55f-dfa1-4398-b067-3c44e2e37a10 |
+-------------+--------------------------------------+
$ openstack volume snapshot list
+--------------------------------------+---------------------------+-------------+-----------+------+
| ID | Name | Description | Status | Size |
+--------------------------------------+---------------------------+-------------+-----------+------+
| 1c4051e0-ae74-487b-ab94-95ddb6ffd8b3 | puppet-train-vol-snapshot | None | available | 1 |
+--------------------------------------+---------------------------+-------------+-----------+------+
$ openstack volume create --snapshot 1c4051e0-ae74-487b-ab94-95ddb6ffd8b3 --size 1 volume-from-snapshot
+---------------------+--------------------------------------+
| Field | Value |
+---------------------+--------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2020-07-27T12:23:23.000000 |
| description | None |
| encrypted | False |
| id | ac0c557b-9cfd-4ce0-88b4-1ce44b3563f4 |
| multiattach | False |
| name | volume-from-snapshot |
| properties | |
| replication_status | None |
| size | 1 |
| snapshot_id | 1c4051e0-ae74-487b-ab94-95ddb6ffd8b3 |
| source_volid | None |
| status | creating |
| type | standard |
| updated_at | None |
| user_id | zmatonis |
+---------------------+--------------------------------------+
$ openstack volume list
+--------------------------------------+-------------------------------+-----------+------+---------------------------------------+
| ID | Name | Status | Size | Attached to |
+--------------------------------------+-------------------------------+-----------+------+---------------------------------------+
| ac0c557b-9cfd-4ce0-88b4-1ce44b3563f4 | volume-from-snapshot | available | 1 | |
| 3919bfec-2c08-431d-a89f-70368108fff6 | C8 TEST - x86_64 [2020-07-08] | available | 100 | |
| b065e674-6315-4d23-8471-d5960baee423 | devstack01-vol | in-use | 40 | Attached to devstack01 on /dev/vdb |
| cc3ec55f-dfa1-4398-b067-3c44e2e37a10 | puppet-train-vol | available | 1 | |
+--------------------------------------+-------------------------------+-----------+------+---------------------------------------+
The newly created volume puppet-train-vol-snapshot can now be attached to the test VM.
While OpenStack can be forced to snapshot an attached volume, it is recommended to detach the master volume from the master VM before creating the snapshot in order to guarantee consistency.
In order to delete a clone volume the corresponding snapshot needs to be deleted first.