Skip to content

Manual CephFS Config

CephFS Manual Client Configuration

For machines which are not managed by the CERN-IT Puppet Infrastructure, you may manually configure a CephFS client to mount an OpenStack Manila share. Whenever possible, we recommend using the Puppet module to mount CephFS shares.

Requirements:

To follow this tutorial you will need the following information.

  • Which CephFS cluster are you mounting?
    • The examples below use Meyrin CephFS, aka cephflax.cern.ch
  • What is the remote path you are mounting?
    • This will look like /volumes/_nogroup/<uuid>
    • $CEPHFS_PREFIX in the recipes below.
  • What are the cephx id and secret to access the mount?
    • $CEPHX_ID and $CEPHX_SECRET in the recipes below.

Installing Required Packages:

We first add the ceph.repo and install ceph-fuse and ceph-common RPMs.

Begin by adding the ceph.repo file in the /etc/yum.repos.d directory.

Upstream mirror

In the example below, replace {ceph-release} with the current stable major release of Ceph (e.g., pacific or quincy as of Feb. 2023) and {distro} with your Linux distribution (e.g., el9). You may browse the rpm-{ceph-release} dirs at https://download.ceph.com/ to see which distributions Ceph supports. Some Ceph packages (e.g., EPEL) must take priority over standard packages, so you must ensure that you set priority=1.

> sudo -i

# rpm --import http://linuxsoft.cern.ch/mirror/download.ceph.com/release.asc

# cat << EOF > /etc/yum.repos.d/ceph.repo
[ceph]
name=Ceph {ceph-release} packages for $basearch
baseurl=http://linuxsoft.cern.ch/mirror/download.ceph.com/rpm-{ceph-release}/{distro}/$basearch
enabled=1
gpgcheck=1
priority=1
type=rpm-md
gpgkey=http://linuxsoft.cern.ch/mirror/download.ceph.com/release.asc

[ceph-noarch]
name=Ceph {ceph-release} noarch packages
baseurl=http://linuxsoft.cern.ch/mirror/download.ceph.com/rpm-{ceph-release}/{distro}/noarch
enabled=1
gpgcheck=1
priority=1
type=rpm-md
gpgkey=http://linuxsoft.cern.ch/mirror/download.ceph.com/release.asc
EOF

# yum install ceph-fuse ceph-common

First, install the key (unless you have it already). The examples here are for RHEL9. For Alma9, repositories and tags are a bit different.

The gpg key for koji can be found in the repository http://linuxsoft.cern.ch/cern/rhel/9-testing/CERN/x86_64/Packages/c/ and is named like cern-gpg-keys-*. Download and install the newest version, e.g.:

For RHEL9 do 

yum install http://linuxsoft.cern.ch/cern/rhel/9/CERN/x86_64/Packages/c/cern-gpg-keys-1.0-1.rh9.cern.noarch.rpm
Next, configure the CEPH repository like in the example below. As an example, the distro_tag should be 9el for EL9. If in doubt, check the URL to see what is available.

# cat << EOF > /etc/yum.repos.d/ceph.repo
[ceph-quincy9el-stable]
name=Ceph quincy binaries from CERN Koji [stable]
baseurl=http://linuxsoft.cern.ch/internal/repos/ceph-quincy9el-stable/x86_64/os
enabled=1
gpgcheck=True
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-kojiv2
priority=9
EOF

Install the software

To install the package, run 

# yum update
# yum install ceph-fuse ceph-common

Manual Client Configuration:

You can choose to use ceph-fuse or the native Linux ceph kernel driver to mount your CephFS share. If uncertain which of the two to use, please reach-out to the Ceph Support Line to identify the best choice for your use case.

These config files are needed both for FUSE and native kernel module.

The example given is for Meyrin CephFS, aka cephflax.cern.ch. The hostname to connect to should be changed if using a different cluster (see Share types).

Create the the ceph.conf file:

# cat << EOF > /etc/ceph/ceph.conf
[global]
admin socket = /var/run/ceph/\$cluster-\$name-\$pid.asok
client reconnect stale = true
debug client = 0/2
fuse big writes = true
mon host = cephflax.cern.ch:6789
EOF

Next, create a cephx keyring file:

# CEPHX_ID=<create and get from openstack>
# CEPHX_SECRET=<create and get from openstack>
# cat << EOF > /etc/ceph/ceph.client.${CEPHX_ID}.keyring
[client.${CEPHX_ID}]
    key = ${CEPHX_SECRET}
EOF

where: - CEPHX_ID is the name of the access identity (e.g., myauthid01 in the Quickstart example, reported in the "Access To" column of openstack share access list <myshare>) - CEPHX_SECRET is the secret used to authenticate (e.g., AQAdFCNYDC... in the Quickstart example, reported in the "Access Key" column)

Ceph Kernel Client

Ensure that ceph-common is already installed and that you are already running the latest kernel for your distro. (yum update && reboot).

Then, create a cephx secret file:

# CEPHX_ID=<create and get from openstack>
# CEPHX_SECRET=<create and get from openstack>
# cat << EOF > /etc/ceph/ceph.${CEPHX_ID}.secret
${CEPHX_SECRET}
EOF

Next, create an entry in /etc/fstab to mount the share at boot. The example given is for Meyrin CephFS, aka cephflax.cern.ch. The hostname to connect to should be changed if using a different cluster (see Share types). 

# CEPHFS_PREFIX=<get from openstack>
# cat << EOF >> /etc/fstab
cephflax.cern.ch:6789:${CEPHFS_PREFIX}  /cephfs ceph    name=${CEPHX_ID},secretfile=/etc/ceph/ceph.${CEPHX_ID}.secret,x-systemd.device-timeout=30,x-systemd.mount-timeout=30,noatime,_netdev,rw 0   2
EOF

Tip: If you would like to enable automounting for /cephfs, you can append ,noauto,x-systemd.automount,x-systemd.idle-timeout=30 to the fstab mount options in the example above, then run as root systemctl daemon-reload and systemctl restart remote-fs.target for this to take effect.

Now you can mkdir -p /cephfs; mount /cephfs to access your share.

Ceph FUSE Client

Ensure that both ceph-fuse and ceph-common are installed as above.

If you want to mount CephFS using ceph-fuse, you must first create the needed config files. The example given is for Meyrin CephFS, aka cephflax.cern.ch. The parameter mon host should be changed if using a different cluster (see Share types).

Then, create an entry in /etc/fstab to mount the share at boot, for example:

# CEPHFS_PREFIX=<get from openstack>
# cat << EOF >> /etc/fstab
none    /cephfs fuse.ceph   ceph.id=${CEPHX_ID},ceph.conf=/etc/ceph/ceph.conf,ceph.client_mountpoint=${CEPHFS_PREFIX},x-systemd.device-timeout=30,x-systemd.mount-timeout=30,noatime,_netdev,rw 0   2
EOF

Tip: If you would like to enable automounting for /cephfs, you can append ,noauto,x-systemd.automount,x-systemd.idle-timeout=30 to the fstab mount options in the example above, then run as root systemctl daemon-reload and systemctl restart remote-fs.target for this to take effect.

Now you can mkdir -p /cephfs; mount /cephfs to access your share.