I recently needed to set up a new Grafana instance. Previously I've run Grafana using Docker, but this time, I decided to run it in my K8S cluster instead.
There was no particular reason for choosing to run it in Kubernetes other than that I had the cluster sat there, with spare resources and it avoided needing to make other arrangements.
Grafana have some documentation on deploying into Kubernetes, but it skips over things like provisioning storage and customising configuration (to do things such as enabling SMTP so that alert emails can be sent).
This documentation details how to deploy and configure Grafana in a Kubernetes cluster, with some guidance on how to provision some persistent storage so that changes made in Grafana survive pod restart and replacement.
Although it's the main aim, we're not just going to stand up a Grafana instance, we also want to be able to configure it.
In this doc, we'll configure SMTP, but it's possible to take any configuration item from
grafana.ini and define it via environment variable - the variable name is of the form
[paths] logs = "foo"
$GF_PATHS_LOGS = 'foo'
To Namespace or Not
Creating separate namespaces for separate things is best practice and the Grafana docs observe this, running
kubectl create namespace my-grafana
However, it's worth being aware that, if you're not used to doing this, you can make quite a mess if you later accidentally apply the manifests without specifying the relevant namespace. Every command you run will need to include it (or you can use something like
kubectx to switch between)
kubectl -n my-grafana get all
To avoid confusion, example calls in this doc won't specify a namespace
Grafana's Doc doesn't really go into persistent storage, so if you follow the guide to the letter you'll likely be left with an unbound or pending Persistent Volume Claim (PVC).
You need some kind of persistent storage so that you don't lose dashboards/users/alerts etc whenever a pod is restarted.
The manifest creation stage later in this doc includes an example of using NFS, but when I first set this up, I opted to create a volume using
local-storage (I was experimenting on a single node Kubernetes instance).
Note: If you're running a multi-node cluster you almost certainly don't want to do this (because it requires affinity to a single node).
Create a directory to house files
mkdir -p /home/ben/volumes/grafana
Create a manifest (I called mine
--- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: grafana-local-pvc spec: storageClassName: local-storage volumeName: "grafana-local-storage" accessModes: - ReadWriteOnce resources: requests: storage: 1Gi --- apiVersion: v1 kind: PersistentVolume metadata: name: grafana-local-storage spec: claimRef: name: grafana-local-pvc # If you're not using the default # namespace you'll need to specify # the name here #namespace: my-grafana capacity: storage: 2Gi volumeMode: Filesystem accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Delete storageClassName: local-storage local: path: /home/ben/volumes/grafana nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: # Hostname of the node that holds the storage - bumblebee
Provision the storage by applying the manifest
kubectl apply grafana-storage.yml
Create a Manifest
With storage provisioned, we now need to define the Grafana instance.
First, we create a secret so that we don't have SMTP credentials sat in a random manifest:
kubectl create secret generic notifications-smtp \ --from-literal=user=<smtp username> \ --from-literal=password=<smtp password> \ --from-literal=host=<smtp server:port>
Then we start to create a manifest (as
--- apiVersion: apps/v1 kind: Deployment metadata: labels: app: grafana name: grafana spec: selector: matchLabels: app: grafana template: metadata: labels: app: grafana spec: securityContext: fsGroup: 472 supplementalGroups: - 0 containers: - name: grafana image: grafana/grafana:latest imagePullPolicy: IfNotPresent env: # Enable SMTP - name: GF_SMTP_ENABLED value: "true" - name: GF_SMTP_FROM_ADDRESS value: "email@example.com" - name: GF_SMTP_FROM_NAME value: "Bens Grafana" # Use the secret to define these values - name: GF_SMTP_HOST valueFrom: secretKeyRef: name: notifications-smtp key: host - name: GF_SMTP_PASSWORD valueFrom: secretKeyRef: name: notifications-smtp key: password - name: GF_SMTP_USER valueFrom: secretKeyRef: name: notifications-smtp key: user ports: - containerPort: 3000 name: http-grafana protocol: TCP # Have k8s monitor container health readinessProbe: failureThreshold: 3 httpGet: path: /robots.txt port: 3000 scheme: HTTP initialDelaySeconds: 10 periodSeconds: 30 successThreshold: 1 timeoutSeconds: 2 livenessProbe: failureThreshold: 3 initialDelaySeconds: 30 periodSeconds: 10 successThreshold: 1 tcpSocket: port: 3000 timeoutSeconds: 1 # Limit the resources that # can be consumed resources: requests: cpu: 250m memory: 750Mi # Mount our PV volumeMounts: - mountPath: /var/lib/grafana name: grafana-local-pv # Link the volume to the claim volumes: - name: grafana-local-pv persistentVolumeClaim: claimName: grafana-local-pvc
If you're intending to use a NFS share you'll want to change that last section to something like
volumes: # If you change the name, remember to update # the name in volumeMounts too - name: grafana-local-pv nfs: server: 192.168.3.233 path: /volume1/kubernetes_grafana_data readOnly: false
If we applied the manifest now, a Grafana pod would spin up but would be inaccessible: it needs a service definition to expose port 3000.
Add the following to the end of
--- apiVersion: v1 kind: Service metadata: name: grafana spec: ports: - port: 3000 protocol: TCP targetPort: http-grafana selector: app: grafana sessionAffinity: None type: LoadBalancer
Apply the manifest to stand Grafana up:
kubectl apply -f grafana.yml
Wait a minute or two and then verify that there's a Grafana pod running
kubectl get pods
Retrieve the service details
kubectl get service grafana
This should show you details similar to the following:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE grafana LoadBalancer 10.98.119.76 <pending> 3000:32574/TCP 128h
If you're not using a Cloud Kubernetes cluster, it's quite likely that external IP will always be in a pending state (because your network doesn't have the gubbins needed to assign one).
However, hopefully you've set up routes to your cluster range and so can now hit Grafana via the Cluster IP:
curl -v http://10.98.119.76:3000
If that works, you should be able to visit Grafana in your browser. Use
admin to log in for the first time and set a new admin password.
If you go to
http://<your cluster ip>:3000/alerting/notifications you should be able to create a Contact Point and successfully send a test email.