Skip to content

Kube Monkey

Kube Monkey

kube-monkey is an implementation of Netflix's Chaos Monkey for Kubernetes clusters, encouraging and validating the development of failure-resilient services.

Below we give examples of how to use kube-monkey with Kubernetes, check here for additional information on how to use it with Kubernetes and Helm.

We plan to integrate kube-monkey by default in all clusters soon.

Installation

Helm is used for installation, check here for how to use it.

Full instructions for deployment of kube-monkey are available here.

Here's an example deployment, where we: - set to 8am the time to schedule terminations for the day's time window - set a custom time window for kube-monkey to run (in this case between 10h and 16h) - blacklist the namespace kube-system, as we don't want it to be touched - set whitelisted namespaces to "" so that any namespace can trigger terminations

helm install --name kube-monkey --namespace kube-system \
  cern/kube-monkey \
  --version 1.2-cern-0.1.0-6+d6fc355 \
  --set config.debug.enabled=false \
  --set config.dryRun=false \
  --set config.runHour=8 \
  --set config.startHour=10 \
  --set config.endHour=16 \
  --set config.blacklistedNamespaces="kube-system" \
  --set config.whitelistedNamespaces=""

A useful set of additional options make debugging easier (but likely not ideal for production setups):

  --set config.debug.enabled=true \
  --set config.debug.schedule_immediate_kill=true

Alternatively, one can use a values.yaml file with all the configs set. Here is an example of such a file:

config:
  dryRun: false
  runHour: 8
  startHour: 10
  endHour: 16
  blacklistedNamespaces: "kube-system"
  whitelistedNamespaces: ""
  timeZone: Europe/Paris
  debug:
    enabled: false
  # schedule_immediate_kill: true

The complete set of available configurations can be found here.

Usage

kube-monkey is deployed but will only act against resources tagged with certain labels, so by default it will not schedule any terminations.

Here's an example of how to set a Deployment as a kube-monkey target using labels:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
    kube-monkey/enabled: enabled
    kube-monkey/identifier: "frontend"
    kube-monkey/kill-mode: "fixed"
    kube-monkey/kill-value: "1"
    kube-monkey/mtbf: "1"
spec:
  template:
    metadata:
      labels:
        kube-monkey/enabled: enabled
        kube-monkey/identifier: frontend
        kube-monkey/kill-mode: "fixed"
        kube-monkey/kill-value: "1"
        kube-monkey/mtbf: "1"
...

This adds the Deployment to the list of scheduled terminations for kube-monkey. You can find here the full set of available configurations when you opt-in to chaos.

You can check scheduled terminations and executions in the kube-monkey logs:

kubectl -n kube-system logs --tail 50 -f deployment/kube-monkey

Happy testing!