Certificates

Warning

This documentation is deprecated, please check here for its new home

There are different types of certificates described in this section:

Let's Encrypt certificates for Kubernetes Ingress
Internal cluster certificates, issued by an internal cluster CA
CERN credentials, including X509 host certificates and kerberos keytab

Let's Encrypt and Ingress

Let's Encrypt supports the ACME protocol to manage X509 certificates, you can find more details here.

From the different available ACME challenges, at CERN we can only rely on the TLS challenge today - ongoing work to get DNS also working. This means you'll need to first request a firewall opening, stating that you will immediately after request the certificate to do proper TLS.

Setup

The example below shows the required change to get Let's Encrypt certificates for an Ingress resource in a Kubernetes clusters with Traefik enabled. Only the acme sections should be required to add, the remaining configuration should already be present.

It uses myservice.cern.ch as an example, with an additional alias myadditionalalias.cern.ch - the latter is optional, drop it if not needed.

Edit the ingress-traefik ConfigMap adding an acme section:

apiVersion: v1
data:
  traefik.toml: |-
    logLevel = "INFO"
    defaultEntryPoints = ["http", "https"]
    [api]
    [kubernetes]
    [acme]
      email = "YOUREMAIL"
      storage = "acme.json"
      entryPoint = "https"
      ACMELogging = true
    [acme.tlsChallenge]
    [[acme.domains]]
      main = "myservice.cern.ch"
      sans = ["myadditionalalias.cern.ch"]
    [entryPoints]
...

Kill the ingress-traefik pod to force a restart which will trigger the certificate request:

kubectl -n kube-system delete $(kubectl -n kube-system get pod -o name | grep traefik)

NOTE: For the moment this setup requires that only one node is set as an ingress controller - check here for more details.

Internal

This functionality is only available for Kubernetes.

To enable it, pass the cert_manager_api label as described here.

$ openstack coe cluster create <name> --cluster-template <cluster-template> \
    --merge-labels \
    --labels cert_manager_api=true

You can then use the kubernetes certificate manager api, as described here.

CERN

Cluster nodes can be configured with both a keytab and X509 host certificate.

The label 'cern_enabled' controls this behavior, check the cluster template to see if your cluster has it enabled or not. You can enable it on cluster creation by passing the label:

$ openstack coe cluster create <name> --cluster-template <cluster-template> \
    --merge-labels \
    --labels cern_enabled=true

When enabled, the following additional files will be set on each node:

/etc/krb5.keytab: the host's kerberos keytab
/etc/grid-security/hostcert.pem: the host's X509 certificate
/etc/grid-security/hostkey.pem: the host's X509 key

All tied to the hostname.

Usage

Below we give an example of checking the keytab and/or certificate by bind mounting the host's /etc directory. A similar procedure can be done to use the credentials on a real application.

Kubernetes

Note we're setting hostNetwork to true, this might be needed if you need dns resolution to match the certificate.

$ cat sample-cert.yaml 
apiVersion: v1
kind: Pod
metadata:
  name: sample-cert
  namespace: default
spec:
  hostNetwork: true
  containers:
  - image: centos:7
    command:
      - sleep
      - "3600"
    imagePullPolicy: IfNotPresent
    name: sample-cert
    volumeMounts:
    - mountPath: /etc/grid-security
      name: etc-grid-security
    - mountPath: /etc/krb5.keytab
      name: etc-keytab
  volumes:
  - name: etc-grid-security
    hostPath:
      path: /etc/grid-security
  - name: etc-keytab
    hostPath:
      path: /etc/krb5.keytab

$ kubectl exec -it sample-cert bash
[root@rocha-hostcert-kub-001-rm6c5woy62ta-minion-0 /]# yum install -q -y openssl krb5-workstation
[root@rocha-hostcert-kub-001-rm6c5woy62ta-minion-0 /]# openssl x509 -in /etc/grid-security/hostcert.pem -subject -noout
subject= /CN=rocha-hostcert-kub-001-rm6c5woy62ta-minion-0.cern.ch

[root@rocha-hostcert-kub-001-rm6c5woy62ta-minion-0 /]# klist -kt /etc/krb5.keytab 
Keytab name: FILE:/etc/krb5.keytab
KVNO Timestamp         Principal
---- ----------------- --------------------------------------------------------
...
   4 04/12/18 08:57:03 host/rocha-hostcert-kub-001-rm6c5woy62ta-minion-0.cern.ch@CERN.CH
   4 04/12/18 08:57:03 host/rocha-hostcert-kub-001-rm6c5woy62ta-minion-0.cern.ch@CERN.CH
   4 04/12/18 08:57:03 host/rocha-hostcert-kub-001-rm6c5woy62ta-minion-0.cern.ch@CERN.CH

Swarm

As above, we're running this example with host networking which might be (or not) required.

$ docker run -it --net host -v /etc:/etc-host centos:7 bash
[root@rocha-hostcert-swarm001-qc2arp6o7m3s-primary-master-0 /]# yum install -q -y openssl krb5-workstation^C
[root@rocha-hostcert-swarm001-qc2arp6o7m3s-primary-master-0 /]# openssl x509 -in /etc-host/grid-security/hostcert.pem -noout -subject
subject= /CN=rocha-hostcert-swarm001-qc2arp6o7m3s-primary-master-0.cern.ch

[root@rocha-hostcert-swarm001-qc2arp6o7m3s-primary-master-0 /]# klist -kt /etc-host/krb5.keytab 
Keytab name: FILE:/etc-host/krb5.keytab
KVNO Timestamp         Principal
---- ----------------- --------------------------------------------------------
...
   4 04/12/18 09:56:04 host/rocha-hostcert-swarm001-qc2arp6o7m3s-primary-master-0.cern.ch@CERN.CH
   4 04/12/18 09:56:04 host/rocha-hostcert-swarm001-qc2arp6o7m3s-primary-master-0.cern.ch@CERN.CH
   4 04/12/18 09:56:04 host/rocha-hostcert-swarm001-qc2arp6o7m3s-primary-master-0.cern.ch@CERN.CH

Last update: June 1, 2022