Skip to content

Manual CephFS Config

CephFS Manual Client Configuration

For PCs which are not managed by the CERN-IT Puppet Infrastructure, you may manually configure a CephFS client to mount an OpenStack Manila share.

Requirements:

To follow this tutorial you will need the following information.

  • Which CephFS cluster are you mounting?
    • The examples below use Meyrin CephFS, aka cephflax.cern.ch
  • What is the remote path you are mounting?
    • This will look like /volumes/_nogroup/<uuid>
    • $CEPHFS_PREFIX in the recipes below.
  • What are the cephx id and secret to access the mount?
    • $CEPHX_ID and $CEPHX_SECRET in the recipes below.

Installing Required Packages:

We first add the ceph.repo and install ceph-fuse and ceph-common RPMs.

Begin by adding the ceph.repo file in the /etc/yum.repos.d directory. In the example below, replace {ceph-release} with the current stable major release of Ceph (e.g., octopus or pacific as of Jan 2022) and {distro} with your Linux distribution (e.g., el7 or el8). You may browse the rpm-{ceph-release} dirs at https://download.ceph.com/ to see which distributions Ceph supports. Some Ceph packages (e.g., EPEL) must take priority over standard packages, so you must ensure that you set priority=1. (If running CentOS 7, also ensure that yum-priorities is installed -- yum install yum-priorities).

> sudo -i

# rpm --import http://linuxsoft.cern.ch/mirror/download.ceph.com/release.asc

# cat << EOF > /etc/yum.repos.d/ceph.repo
[ceph]
name=Ceph {ceph-release} packages for $basearch
baseurl=http://linuxsoft.cern.ch/mirror/download.ceph.com/rpm-{ceph-release}/{distro}/$basearch
enabled=1
gpgcheck=1
priority=1
type=rpm-md
gpgkey=http://linuxsoft.cern.ch/mirror/download.ceph.com/release.asc

[ceph-noarch]
name=Ceph {ceph-release} noarch packages
baseurl=http://linuxsoft.cern.ch/mirror/download.ceph.com/rpm-{ceph-release}/{distro}/noarch
enabled=1
gpgcheck=1
priority=1
type=rpm-md
gpgkey=http://linuxsoft.cern.ch/mirror/download.ceph.com/release.asc
EOF

# yum install ceph-fuse ceph-common

Manual Client Configuration:

You can choose to use ceph-fuse or the native Linux ceph kernel driver to mount your CephFS share. As of Jan 2022, the latter is recommended.

Ceph Kernel Client

Ensure that ceph-common is already installed and that you are already running the latest kernel for your distro. (yum update && reboot).

Then, create a cephx secret file:

# CEPHX_ID=<create and get from openstack>
# CEPHX_SECRET=<create and get from openstack>
# cat << EOF > /etc/ceph/ceph.${CEPHX_ID}.secret
${CEPHX_SECRET}
EOF

Next, create an entry in /etc/fstab to mount the share at boot. The example given is for Meyrin CephFS, aka cephflax.cern.ch. The hostname to connect to should be changed if using a different cluster (see Share types).

# CEPHFS_PREFIX=<get from openstack>
# cat << EOF >> /etc/fstab
cephflax.cern.ch:6789:{$CEPHFS_PREFIX}  /cephfs ceph    name=${CEPHX_ID},secretfile=/etc/ceph/ceph.${CEPHX_ID}.secret,x-systemd.device-timeout=30,x-systemd.mount-timeout=30,noatime,_netdev,rw 0   2
EOF

Tip: If you would like to enable automounting for /cephfs, you can append ,noauto,x-systemd.automount,x-systemd.idle-timeout=30 to the fstab mount options in the example above, then run as root systemctl daemon-reload and systemctl restart remote-fs.target for this to take effect.

Now you can mkdir -p /cephfs; mount /cephfs to access your share.

Ceph FUSE Client

Ensure that both ceph-fuse and ceph-common are installed as above.

If you want to mount CephFS using ceph-fuse, you must first create the needed config files. The example given is for Meyrin CephFS, aka cephflax.cern.ch. The parameter mon host should be changed if using a different cluster (see Share types).

First, the ceph.conf file:

# cat << EOF > /etc/ceph/ceph.conf
[global]
admin socket = /var/run/ceph/\$cluster-\$name-\$pid.asok
client reconnect stale = true
debug client = 0/2
fuse big writes = true
mon host = cephflax.cern.ch:6789
EOF

Next, create a cephx keyring file:

# CEPHX_ID=<create and get from openstack>
# CEPHX_SECRET=<create and get from openstack>
# cat << EOF > /etc/ceph/ceph.client.${CEPHX_ID}.keyring
[client.${CEPHX_ID}]
    key = ${CEPHX_SECRET}
EOF

Then, create an entry in /etc/fstab to mount the share at boot, for example:

# CEPHFS_PREFIX=<get from openstack>
# cat << EOF >> /etc/fstab
none    /cephfs fuse.ceph   ceph.id=${CEPHX_ID},ceph.conf=/etc/ceph/ceph.conf,ceph.client_mountpoint=${CEPHFS_PREFIX},x-systemd.device-timeout=30,x-systemd.mount-timeout=30,noatime,_netdev,rw 0   2
EOF

Tip: If you would like to enable automounting for /cephfs, you can append ,noauto,x-systemd.automount,x-systemd.idle-timeout=30 to the fstab mount options in the example above, then run as root systemctl daemon-reload and systemctl restart remote-fs.target for this to take effect.

Now you can mkdir -p /cephfs; mount /cephfs to access your share.

Last update: January 12, 2022