Monitoring & Alerting Setup for cluster

Prometheus is an open-source systems monitoring and alerting toolkit originally built at SoundCloud

prometheus-operator chart includes multiple components and is suitable for a variety of use-cases.

The default installation is intended to suit monitoring a kubernetes cluster the chart is deployed onto. It closely matches the kube-prometheus project.

• prometheus-operator

• prometheus

• alertmanager

• node-exporter

• kube-state-metrics

• grafana

• service monitors to scrape internal kubernetes components

  • kube-apiserver

  • kube-scheduler

  • kube-controller-manager

  • etcd

  • kube-dns/coredns

  • kube-proxy

With the installation, the chart also includes dashboards and alerts.

Deployment steps:

1. Add environment variable to the respective env config file

Update the configs branch (like for qa.yaml added qa branch)

2. Add monitoring-dashboards folder to respective configs branch.

3. Enable the nginx-ingress monitoring and redeploy the nginx-ingress.

4. Add alertmanager secret in respective <env>.secrets.yaml

If you want you can change the slack channel and other details like group_wait , group_interval and repeat_interval according to your values.

5. Deploy the prometheus-operator using go cmd or deploy using Jenkins.

            go run main.go deploy -e <environment_name> -c 'prometheus-operator,grafana,prometheus-kafka-exporter'

To create a new panel in the existing dashboard:-  

1.  Login to dashboard and click on add panel  

2. Set all required queries and apply the changes. Export the JSON file by clicking on the save dashboard

3. Update the existing *-dashboard.json file from configs monitoring-dashboards folder with a newly exported JSON file.