Skip to content

Performance Tuning

Virtual machines generally show similar performance to physical hardware and the overheads are reducing as the virtualisation technology improves. However, in some cases, performance can be affected such as high disk I/O rates or additional network latency. A few hints and tips to investigate are described below. For questions or issues concerning the performance of a VM please contact the cloud team via the service desk at https://cern.service-now.com/service-portal/.

Define a workload

In order to test different options, it is necessary to define a representative workload such as the time to perform a compilation or transfer a file. With memory caching, these tests can vary the number of times you run them as the disk contents could be already in memory and therefore served more quickly. Tuned-Adm

Tuned-adm is a Red Hat tool for setting kernel parameters, buffer sizes and disk I/O schedulers according to the workload. Red Hat identify some common profiles and find the appropriate setting for these. Thus, there is no need to go into the details of each kernel parameter. By default, a VM is set up with virtual-guest as the setting when a VM is created from the standard images. However, this may need to be set if user defined images are used. The virtual-guest is a good initial starting point for benchmarking the workload. However, for applications such as web servers, it may be that alternative profiles are better suited. tuned-adm list provides a set of profiles to investigate.

tuned-adm profile default

RAM disks

Some I/O patterns are very intense on a small volume of data which does not need to be permanently stored. These areas are often stored in /tmp or /var/tmp but applications generally allow you to configure the TMPDIR variable or a configuration parameter for other areas. Depending on the size of the temporary data, a RAM disk can be used to mount a section of RAM as a file system. This gives excellent performance at the expense of eating into the RAM available for the application. Selecting a larger flavor of VM may allow a RAM disk to be allocated for this purpose. To mount a RAM file system, use the mount command.

mkdir /tmp/ram
mount -t tmpfs -o size=4G tmpfs /tmp/ram/

Care should be taken not to swap memory. This is an expensive operation and would lead to a reduction in overall performance.

AFS client configuration

AFS, by default, stores blocks from the remote server onto a local partition. This creates additional I/O load which can be avoided by using a smaller memory cache.

In the /etc/sysconfig/afsd file, memory cacheing can be enabled using

# OpenAFS 1.6.x
AFSD_ARGS="-afsdb -daemons 10 -fakestat -memcache -nosettime -volumes 1024"

Network offloading

Linux kernels, by default, offload some processing to the network card. In the case of a VM, this does not improve the performance. The effects are most visible on high performance networks.

To disable this, run

ethtool -K eth0 gro off tso off gso off

The current settings can be checked with ethtool -k.

This can be made persistent by setting the ETHTOOL_OPTS value in the network scripts such as /etc/sysconfig/network-scripts/ifcfg-eth0

IO caching with ZFS

The combination of an ephemeral drive on SSDs and a volume can be used to leverage ZFS caching features for read (l2arc) and write (ZIL). The setup would be like this:

Prepare the system

  • Resize the LVM partition to 20GB
  • Disable Puppet

Install ZFS

  • Add zfs-kmod repo 
    [zfs-kmod]
name=ZFS on Linux for EL $releasever (KMOD)
baseurl=http://linuxsoft.cern.ch/mirror/archive.zfsonlinux.org/epel/6/kmod/$basearch/
enabled=1
gpgcheck=0
priority=9
  • Install ZFS packages and load the module 
    yum install zfs
modprobe zfs
  • Create the data pool from the attached Cinder volume 
    zpool create -o ashift=12 -f zfspool01 /dev/disk/by-id/virtio-106b905d-b1aa-4bf8-9
  • Set sensible defaults 
    zfs set mountpoint=none zfspool01
zfs set compression=lz4 zfspool01
  • Create partition for ZIL (4GB)
  • Create partition for l2arc (20GB) 
    Number  Start   End     Size    Type     File system  Flags
 1      1049kB  420MB   419MB   primary  ext4         boot
 2      420MB   21.9GB  21.5GB  primary               lvm
 3      21.9GB  26.2GB  4295MB  primary
 4      26.2GB  47.7GB  21.5GB  primary
  • Attach the 3rd partition as ZIL 
    zpool add zfspool01 log /dev/vda3
  • Attach the 4th partition as L2ARC 
    zpool add zfspool01 cache /dev/vda4
  • Check the layout of the ZFS pool with 
    zpool status
  • Create the file system 
    zfs create -o refquota=500G zfspool01/data1
  • Potential additional settings 
    zfs set recordsize=4k zfspool01/data1
zfs set sync=always zfspool01/data1 # this may cost performance!
  • Set the mountpoint 
    zfs set mountpoint=/data zfspool01/data1

Save the setting

Once a good combination of settings is found, the configuration of the machine (or the associated configuration management system) should be permanently set so it persists over reboots. A tool such as Puppet can ensure this is done consistently over a number of virtual machines