VM management
OpenStack provides many facilities to manipulate virtual machines. The full details are available in the OpenStack site user guide (although some features may not yet be implemented at CERN).
Note: Care should be taken for Puppet managed VMs since some of the operations below should be performed using the appropriate ai-* commands from the configuration user guide. The specific cases will be noted in each section below but not all OpenStack command options have been tried.
Stopping and Starting
VMs can be stopped and started in different ways. These operations are performed by the openstack command with appropriate actions.
- Reboot is equivalent to powering down the machine and then restarting it. A complete boot sequence takes place and thus the machine returns to use in a few minutes. A soft reboot attempts a graceful shut down and restart of the instance. A hard reboot power cycles the instance.
- Pause/Unpause allows for temporary suspension of the VM. The VM is kept in memory but it is not allocated any CPU time. This is useful if you are performing an intervention on a set of servers and want to avoid that a VM continues to process while the intervention is underway. Unpause takes less than a second.
- Suspend/Resume stores the VM onto disk and recovers it later (in less than a minute). This is faster than stop/start and the VM returns to the place is was when the suspend was performed rather than a new boot cycle.
Note: Shelving and unshelving are not yet supported in the CERN OpenStack cloud
These various approaches can be very useful in environments such as software builds or scale-out services to pre-create VMs and then rapidly bring them online.
The actions can all be performed using the openstack unified CLI with
$ openstack server <operation> <hostname>
such as
$ openstack server resume timdoc148
The status of a VM can be shown using openstack server show:
$ openstack server show timdoc148
+--------------------------------------+-------------------------------------------------------------------------+
| Field | Value |
+--------------------------------------+-------------------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | cern-geneva-b |
| OS-EXT-STS:power_state | 1 |
| OS-EXT-STS:task_state | None |
| OS-EXT-STS:vm_state | active |
| OS-SRV-USG:launched_at | 2014-09-26T11:53:37.000000 |
| OS-SRV-USG:terminated_at | None |
| accessIPv4 | |
| accessIPv6 | |
| addresses | CERN_NETWORK=188.184.140.199 |
| config_drive | |
| created | 2014-09-26T11:44:11Z |
| flavor | m2.small (2) |
| hostId | 56ac70c93d69a6266318fe631af50856d64fa3e6419f2857dddbe49f |
| id | 45a84b1e-38fc-4dc0-a0c2-88d08f1a4350 |
| image | CC7 Server - x86_64 [2014-08-12] (60366d54-63d8-4e58-97cf-9923bb21c538) |
| key_name | mykey |
| name | timdoc148 |
| os-extended-volumes:volumes_attached | [] |
| progress | 0 |
| security_groups | [{u'name': u'default'}] |
| status | ACTIVE |
| project_id | 841615a3-ece9-4622-9fa0-fdc178ed34f8 |
| updated | 2014-09-26T11:53:37Z |
| user_id | my-user-name |
+--------------------------------------+-------------------------------------------------------------------------+
| Error | Description |
|---|---|
| ERROR: openstack Cannot 'reboot' instance UUID while it is in task_state powering-off (HTTP 409) | The reboot operation is not possible as the machine has already been requested to reboot. If this request was done and the machine has not returned to running state, please raise a ticket with the service desk to resolve the problem. |
Renaming VMs
Once a VM is created, its name is fixed when the openstack server create command is run. Renaming a virtual machine is not supported.
An IP alias can be defined so that the new name maps to the same IP address as the old one. Alternatively, a new VM can be created using a snapshot of the old VM as follows:
- Snapshotting the source VM to create a copy of the virtual machine.
- Creating the destination VM with the new name from the source snapshot.
- Check that the new service is working as desired.
- Delete the source VM
Note: For Puppet managed machines, a new VM should be created using the Puppet configuration as defined in the Configuration documentation.
Rescue a VM
In some cases, a virtual machine may fail to boot. Reasons such as incorrect configuration or a problem with the system disk can block the boot process. The virtual machine console can provide useful diagnostic information on the root cause and may help to resolve the problem.
The other approach is to use the rescue functions of OpenStack where a virtual machine is booted using the original image and the system disk is provided as a second disk to the virtual machine. This disk can then be manipulated such as fsck to clear a filesystem problem or mounting and editing the configuration.
Note: this procedure has not been tested for Windows virtual machines on the CERN cloud.
$ openstack server rescue <my VM name>
This will reboot the virtual machine and you can then log in using the key pair previously defined. You will see two disks, /dev/vda which is the new system disk and /dev/vdb which is the old one to be repaired.
The old one can be mounted and configuration files edited or fsck'd.
# cat /proc/diskstats
# mkdir /tmp/rescue
# mount /dev/vdb1 /tmp/rescue
On completion, the VM can be returned to active state with openstack server unrescue, and rebooted.
$ openstack server unrescue <my VM name>