Deploying Grafana into a Kubernetes Cluster
I recently needed to set up a new Grafana instance. Previously I've run Grafana using Docker, but this time, I decided to run it in my K8S cluster instead.
There was no particular reason for choosing to run it in Kubernetes other than that I had the cluster sat there, with spare resources and it avoided needing to make other arrangements.
Grafana have some documentation on deploying into Kubernetes, but it skips over things like provisioning storage and customising configuration (to do things such as enabling SMTP so that alert emails can be sent).
This documentation details how to deploy and configure Grafana in a Kubernetes cluster, with some guidance on how to provision some persistent storage so that changes made in Grafana survive pod restart and replacement.
Configuring Grafana
Although it's the main aim, we're not just going to stand up a Grafana instance, we also want to be able to configure it.
In this doc, we'll configure SMTP, but it's possible to take any configuration item from grafana.ini and define it via environment variable - the variable name is of the form $GF_[section]_[name].
For example
[paths]
logs = "foo"
Becomes $GF_PATHS_LOGS = 'foo'
To Namespace or Not
Creating separate namespaces for separate things is best practice and the Grafana docs observe this, running
kubectl create namespace my-grafana
However, it's worth being aware that, if you're not used to doing this, you can make quite a mess if you later accidentally apply the manifests without specifying the relevant namespace. Every command you run will need to include it (or you can use something like kubectx to switch between)
kubectl -n my-grafana get all
To avoid confusion, example calls in this doc won't specify a namespace
Storage
Grafana's Doc doesn't really go into persistent storage, so if you follow the guide to the letter you'll likely be left with an unbound or pending Persistent Volume Claim (PVC).
You need some kind of persistent storage so that you don't lose dashboards/users/alerts etc whenever a pod is restarted.
The manifest creation stage later in this doc includes an example of using NFS, but when I first set this up, I opted to create a volume using local-storage (I was experimenting on a single node Kubernetes instance).
Note: If you're running a multi-node cluster you almost certainly don't want to do this (because it requires affinity to a single node).
Create a directory to house files
mkdir -p /home/ben/volumes/grafana
Create a manifest (I called mine grafana-storage.yml)
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: grafana-local-pvc
spec:
storageClassName: local-storage
volumeName: "grafana-local-storage"
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: grafana-local-storage
spec:
claimRef:
name: grafana-local-pvc
# If you're not using the default
# namespace you'll need to specify
# the name here
#namespace: my-grafana
capacity:
storage: 2Gi
volumeMode: Filesystem
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Delete
storageClassName: local-storage
local:
path: /home/ben/volumes/grafana
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
# Hostname of the node that holds the storage
- bumblebee
Provision the storage by applying the manifest
kubectl apply grafana-storage.yml
Create a Manifest
With storage provisioned, we now need to define the Grafana instance.
First, we create a secret so that we don't have SMTP credentials sat in a random manifest:
kubectl create secret generic notifications-smtp \
--from-literal=user=<smtp username> \
--from-literal=password=<smtp password> \
--from-literal=host=<smtp server:port>
Then we start to create a manifest (as grafana.yml):
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: grafana
name: grafana
spec:
selector:
matchLabels:
app: grafana
template:
metadata:
labels:
app: grafana
spec:
securityContext:
fsGroup: 472
supplementalGroups:
- 0
containers:
- name: grafana
image: grafana/grafana:latest
imagePullPolicy: IfNotPresent
env:
# Enable SMTP
- name: GF_SMTP_ENABLED
value: "true"
- name: GF_SMTP_FROM_ADDRESS
value: "notifications@example.com"
- name: GF_SMTP_FROM_NAME
value: "Bens Grafana"
# Use the secret to define these values
- name: GF_SMTP_HOST
valueFrom:
secretKeyRef:
name: notifications-smtp
key: host
- name: GF_SMTP_PASSWORD
valueFrom:
secretKeyRef:
name: notifications-smtp
key: password
- name: GF_SMTP_USER
valueFrom:
secretKeyRef:
name: notifications-smtp
key: user
ports:
- containerPort: 3000
name: http-grafana
protocol: TCP
# Have k8s monitor container health
readinessProbe:
failureThreshold: 3
httpGet:
path: /robots.txt
port: 3000
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 30
successThreshold: 1
timeoutSeconds: 2
livenessProbe:
failureThreshold: 3
initialDelaySeconds: 30
periodSeconds: 10
successThreshold: 1
tcpSocket:
port: 3000
timeoutSeconds: 1
# Limit the resources that
# can be consumed
resources:
requests:
cpu: 250m
memory: 750Mi
# Mount our PV
volumeMounts:
- mountPath: /var/lib/grafana
name: grafana-local-pv
# Link the volume to the claim
volumes:
- name: grafana-local-pv
persistentVolumeClaim:
claimName: grafana-local-pvc
If you're intending to use a NFS share you'll want to change that last section to something like
volumes:
# If you change the name, remember to update
# the name in volumeMounts too
- name: grafana-local-pv
nfs:
server: 192.168.3.233
path: /volume1/kubernetes_grafana_data
readOnly: false
If we applied the manifest now, a Grafana pod would spin up but would be inaccessible: it needs a service definition to expose port 3000.
Add the following to the end of grafana.yml
---
apiVersion: v1
kind: Service
metadata:
name: grafana
spec:
ports:
- port: 3000
protocol: TCP
targetPort: http-grafana
selector:
app: grafana
sessionAffinity: None
type: LoadBalancer
Apply the manifest to stand Grafana up:
kubectl apply -f grafana.yml
Wait a minute or two and then verify that there's a Grafana pod running
kubectl get pods
Retrieve the service details
kubectl get service grafana
This should show you details similar to the following:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
grafana LoadBalancer 10.98.119.76 <pending> 3000:32574/TCP 128h
If you're not using a Cloud Kubernetes cluster, it's quite likely that external IP will always be in a pending state (because your network doesn't have the gubbins needed to assign one).
However, hopefully you've set up routes to your cluster range and so can now hit Grafana via the Cluster IP:
curl -v http://10.98.119.76:3000
If that works, you should be able to visit Grafana in your browser. Use admin/admin to log in for the first time and set a new admin password.
If you go to http://<your cluster ip>:3000/alerting/notifications you should be able to create a Contact Point and successfully send a test email.
Although not covered here, you might also want to look at accessing Grafana via a HTTPS reverse proxy, whether thats a Nginx instance, or by using a Nginx ingress
