Kube Monkey
Kube Monkey
kube-monkey is an implementation of Netflix's Chaos Monkey for Kubernetes clusters, encouraging and validating the development of failure-resilient services.
Below we give examples of how to use kube-monkey with Kubernetes, check here for additional information on how to use it with Kubernetes and Helm.
We plan to integrate kube-monkey by default in all clusters soon.
Installation
Helm is used for installation, check here for how to use it.
Full instructions for deployment of kube-monkey are available here.
Here's an example deployment, where we:
- set to 8am the time to schedule terminations for the day's time window
- set a custom time window for kube-monkey to run (in this case between 10h and 16h)
- blacklist the namespace kube-system
, as we don't want it to be touched
- set whitelisted namespaces to "" so that any namespace can trigger terminations
helm install --name kube-monkey --namespace kube-system \
cern/kube-monkey \
--version 1.2-cern-0.1.0-6+d6fc355 \
--set config.debug.enabled=false \
--set config.dryRun=false \
--set config.runHour=8 \
--set config.startHour=10 \
--set config.endHour=16 \
--set config.blacklistedNamespaces="kube-system" \
--set config.whitelistedNamespaces=""
A useful set of additional options make debugging easier (but likely not ideal for production setups):
--set config.debug.enabled=true \
--set config.debug.schedule_immediate_kill=true
Alternatively, one can use a values.yaml
file with all the configs set. Here is an example of such a file:
config:
dryRun: false
runHour: 8
startHour: 10
endHour: 16
blacklistedNamespaces: "kube-system"
whitelistedNamespaces: ""
timeZone: Europe/Paris
debug:
enabled: false
# schedule_immediate_kill: true
The complete set of available configurations can be found here.
Usage
kube-monkey is deployed but will only act against resources tagged with certain labels, so by default it will not schedule any terminations.
Here's an example of how to set a Deployment as a kube-monkey target using labels:
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
labels:
app: nginx
kube-monkey/enabled: enabled
kube-monkey/identifier: "frontend"
kube-monkey/kill-mode: "fixed"
kube-monkey/kill-value: "1"
kube-monkey/mtbf: "1"
spec:
template:
metadata:
labels:
kube-monkey/enabled: enabled
kube-monkey/identifier: frontend
kube-monkey/kill-mode: "fixed"
kube-monkey/kill-value: "1"
kube-monkey/mtbf: "1"
...
This adds the Deployment to the list of scheduled terminations for kube-monkey. You can find here the full set of available configurations when you opt-in to chaos.
You can check scheduled terminations and executions in the kube-monkey logs:
kubectl -n kube-system logs --tail 50 -f deployment/kube-monkey
Happy testing!