Skip to content

Multiple Interfaces

Certain use cases may require Virtual Machines with more network interfaces than the default ones.

The guide below is aimed at a single active->multiple standby setup, meaning a single instance serving requests at a time. Goal is to have a floating network interface moved around different VMs providing failover.

The network interface can be moved around different instances, even if the VM where it is currently attached is no longer available.

Requirements

Due to networking architecture at CERN, VMs sharing a port need to be hosted in the same LanDB Primary network service - all IPs on a subset of related subnets. If this is not the case, then an existing port will not attach to the targeted VM. The below steps will help you ensure this condition is fulfilled by creating VMs only within the same IPv4 and IPv6 subnet.

These steps can be summarized as:

  1. Launch a VM for your application.
  2. Create one additional network interface, and attach it to the VM.
  3. Create new VMs as needed, making sure they are in the same subnet as the first VM.

Setup

Remember to configure the interface

After following these instructions to create and attach the port, remember to also configure the interface before testing it. It won't work before configuration.

Below is an example of such a deployment, step by step:

  1. Create a new VM as normal. If you already have a running instance for an existing service to which you would like to add a standby instance, you may skip this step. 
    $ openstack server create --image MYIMAGE --keypair MYKEYPAIR VM1
  2. Create a new network port, to be used to move the IP from one VM to another when necessary. 
    $ openstack port create --network CERN_NETWORK MYPORT
  3. The IPs of the above port are not determined until it is attached to a VM for the first time. To attach it, run: 
    $ openstack server add port VM1 MYPORT

  4. At this point, you can inspect the port to obtain its new IP(s): 

    $ openstack port show MYPORT
+-------------------------+------------------------------------------------------------------------------------------+
| Field                   | Value                                                                                    |
+-------------------------+------------------------------------------------------------------------------------------+
...
| fixed_ips               | ip_address='188.184.89.10', subnet_id='082e96fb-d6d9-49a8-a054-a0549f3f8f2f'             |
|                         | ip_address='2001:1458:201:e4::100:631', subnet_id='fac90ad0-2894-49cb-8c68-e32f61911ef9' |
...

  5. Login to the VM and check the new Ethernet interface is available 

    $ ssh root@VM1
# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether fa:16:3e:31:ce:ba brd ff:ff:ff:ff:ff:ff
    inet 137.138.xx.yy/24 brd 137.138.xx.yy scope global noprefixroute dynamic eth0
       valid_lft 603899sec preferred_lft 603899sec
    inet6 fe80::f816:3eff:fe31:ceba/64 scope link noprefixroute
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether fa:16:3e:ac:a6:ef brd ff:ff:ff:ff:ff:ff
    You should see both eth0 and eth1, and can go ahead with the interface configuration as desired.

  6. Confirm that the interface corresponding to this port shows up in LanDB.

  7. Now, for any standby VMs to which you may want to attach the port with the set IPs in the future, it is important that it lands within the same subnet(s) as the first one and its ports. Take note of the subnet_id from the port show output above, and use the ID or the corresponding names (MYSUBNET) for subsequent VM creations: 

    openstack subnet show <subnet_id>
+----------------------+--------------------------------------+
| Field                | Value                                |
+----------------------+--------------------------------------+
...
| id                   | 082e96fb-d6d9-49a8-a054-a0549f3f8f2f |
| ip_version           | 4                                    |
...
| name                 | S513-V-VM50                          |
...

  8. Ensure that new VMs land in the same subnet by pre-allocating a port for each using these subnets (both IPv4 and IPv6 if desired):

    openstack port create --network CERN_NETWORK --fixed-ip subnet=MYSUBNET --fixed-ip subnet=MYSUBNET-V6 VM2PORT
openstack server create --image MYIMAGE --keypair MYKEYPAIR --port VM2PORT VM2
    You probably want to split the machines across different physical hosts, check anti-affinity to achieve it.

Configuring the interface

On the VM with the attached port you need to change the following sysctl setting: 

# ssh root@VM1
# sysctl net.ipv4.conf.all.rp_filter=2

Make sure it's persisted (if you use puppet there's a module for that): 

# vim /etc/sysctl.conf
...
net.ipv4.conf.all.rp_filter=2

You can then trigger a DHCP request on the second interface: 

# dhclient eth1
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether fa:16:3e:6a:49:96 brd ff:ff:ff:ff:ff:ff
    inet 137.138.31.21/24 brd 137.138.31.255 scope global noprefixroute dynamic eth0
       valid_lft 603150sec preferred_lft 603150sec
    inet6 fe80::f816:3eff:fe6a:4996/64 scope link noprefixroute
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether fa:16:3e:ac:a6:ef brd ff:ff:ff:ff:ff:ff
    inet 137.138.31.44/24 brd 137.138.31.255 scope global dynamic eth1
       valid_lft 604796sec preferred_lft 604796sec

One option is to add the configuration for eth1 on all VM instances, even if it's only attached/active in one at a time. This simplifies the switch by only bringing eth1 up after moving the port.

Here's an example: 

# vim /etc/sysconfig/network-scripts/ifcfg-eth1

# Generated by dracut initrd
NAME="eth1"

ONBOOT="yes"
NETBOOT="yes"
IPV6INIT="no"
BOOTPROTO="dhcp"
TYPE="Ethernet"
PROXY_METHOD="none"
BROWSER_ONLY="no"
DEFROUTE="yes"
IPV4_FAILURE_FATAL="no"
IPV6_AUTOCONF="no"
IPV6_DEFROUTE="no"
IPV6_FAILURE_FATAL="no"
DHCPV6C=no
PERSISTENT_DHCLIENT=1
NOZEROCONF=1

Moving the interface

A manual example of moving the network interface between VMs: 

$ openstack server remove port VM1 MYPORT
$ openstack server add port VM2 MYPORT

Common Questions

How to automate failover between active and standby?

The required commands are explained above, but a system triggering them is out of scope of these instructions. Corosync and Pacemaker are popular options for such a setup - there's a puppet module to help with such a setup.