Skip to content

Manual CephFS Config

CephFS Manual Client Configuration

For PCs which are not managed by the CERN-IT Puppet Infrastructure, you may manually configure a CephFS client to mount an OpenStack Manila share.

Requirements:

To follow this tutorial you will need the following information.

  • Which CephFS cluster are you mounting?
    • The examples below use Meyrin CephFS, aka cephflax.cern.ch
  • What is the remote path you are mounting?
    • This will look like /volumes/_nogroup/<uuid>
    • $CEPHFS_PREFIX in the recipes below.
  • What are the cephx id and secret to access the mount?
    • $CEPHX_ID and $CEPHX_SECRET in the recipes below.

Installing Required Packages:

We first add the ceph.repo and install ceph-fuse and ceph-common RPMs.

Begin by adding the ceph.repo file in the /etc/yum.repos.d directory. In the example below, replace {ceph-release} with the current stable major release of Ceph (e.g., pacific or quincy as of Feb. 2023) and {distro} with your Linux distribution (e.g., el9). You may browse the rpm-{ceph-release} dirs at https://download.ceph.com/ to see which distributions Ceph supports. Some Ceph packages (e.g., EPEL) must take priority over standard packages, so you must ensure that you set priority=1.

> sudo -i

# rpm --import http://linuxsoft.cern.ch/mirror/download.ceph.com/release.asc

# cat << EOF > /etc/yum.repos.d/ceph.repo
[ceph]
name=Ceph {ceph-release} packages for $basearch
baseurl=http://linuxsoft.cern.ch/mirror/download.ceph.com/rpm-{ceph-release}/{distro}/$basearch
enabled=1
gpgcheck=1
priority=1
type=rpm-md
gpgkey=http://linuxsoft.cern.ch/mirror/download.ceph.com/release.asc

[ceph-noarch]
name=Ceph {ceph-release} noarch packages
baseurl=http://linuxsoft.cern.ch/mirror/download.ceph.com/rpm-{ceph-release}/{distro}/noarch
enabled=1
gpgcheck=1
priority=1
type=rpm-md
gpgkey=http://linuxsoft.cern.ch/mirror/download.ceph.com/release.asc
EOF

# yum install ceph-fuse ceph-common

Manual Client Configuration:

You can choose to use ceph-fuse or the native Linux ceph kernel driver to mount your CephFS share. As of Jan 2022, the latter is recommended. These config files are needed both for FUSE and native kernel module.

The example given is for Meyrin CephFS, aka cephflax.cern.ch. The hostname to connect to should be changed if using a different cluster (see Share types).

Create the the ceph.conf file:

# cat << EOF > /etc/ceph/ceph.conf
[global]
admin socket = /var/run/ceph/\$cluster-\$name-\$pid.asok
client reconnect stale = true
debug client = 0/2
fuse big writes = true
mon host = cephflax.cern.ch:6789
EOF

Next, create a cephx keyring file:

# CEPHX_ID=<create and get from openstack>
# CEPHX_SECRET=<create and get from openstack>
# cat << EOF > /etc/ceph/ceph.client.${CEPHX_ID}.keyring
[client.${CEPHX_ID}]
    key = ${CEPHX_SECRET}
EOF

Ceph Kernel Client

Ensure that ceph-common is already installed and that you are already running the latest kernel for your distro. (yum update && reboot).

Then, create a cephx secret file:

# CEPHX_ID=<create and get from openstack>
# CEPHX_SECRET=<create and get from openstack>
# cat << EOF > /etc/ceph/ceph.${CEPHX_ID}.secret
${CEPHX_SECRET}
EOF

Next, create an entry in /etc/fstab to mount the share at boot. The example given is for Meyrin CephFS, aka cephflax.cern.ch. The hostname to connect to should be changed if using a different cluster (see Share types). 

# CEPHFS_PREFIX=<get from openstack>
# cat << EOF >> /etc/fstab
cephflax.cern.ch:6789:{$CEPHFS_PREFIX}  /cephfs ceph    name=${CEPHX_ID},secretfile=/etc/ceph/ceph.${CEPHX_ID}.secret,x-systemd.device-timeout=30,x-systemd.mount-timeout=30,noatime,_netdev,rw 0   2
EOF

Tip: If you would like to enable automounting for /cephfs, you can append ,noauto,x-systemd.automount,x-systemd.idle-timeout=30 to the fstab mount options in the example above, then run as root systemctl daemon-reload and systemctl restart remote-fs.target for this to take effect.

Now you can mkdir -p /cephfs; mount /cephfs to access your share.

Ceph FUSE Client

Ensure that both ceph-fuse and ceph-common are installed as above.

If you want to mount CephFS using ceph-fuse, you must first create the needed config files. The example given is for Meyrin CephFS, aka cephflax.cern.ch. The parameter mon host should be changed if using a different cluster (see Share types).

Then, create an entry in /etc/fstab to mount the share at boot, for example:

# CEPHFS_PREFIX=<get from openstack>
# cat << EOF >> /etc/fstab
none    /cephfs fuse.ceph   ceph.id=${CEPHX_ID},ceph.conf=/etc/ceph/ceph.conf,ceph.client_mountpoint=${CEPHFS_PREFIX},x-systemd.device-timeout=30,x-systemd.mount-timeout=30,noatime,_netdev,rw 0   2
EOF

Tip: If you would like to enable automounting for /cephfs, you can append ,noauto,x-systemd.automount,x-systemd.idle-timeout=30 to the fstab mount options in the example above, then run as root systemctl daemon-reload and systemctl restart remote-fs.target for this to take effect.

Now you can mkdir -p /cephfs; mount /cephfs to access your share.