Prometheus is an open-source systems monitoring and alerting toolkit originally built at SoundCloud

Open

prometheus-operator chart includes multiple components and is suitable for a variety of use-cases.

The default installation is intended to suit monitoring a kubernetes cluster the chart is deployed onto. It closely matches the kube-prometheus project.

• prometheus-operator

• prometheus

• alertmanager

• node-exporter

• kube-state-metrics

• grafana

• service monitors to scrape internal kubernetes components

  • kube-apiserver

  • kube-scheduler

  • kube-controller-manager

  • etcd

  • kube-dns/coredns

  • kube-proxy

With the installation, the chart also includes dashboards and alerts.

Deployment steps:

1. Add environment variable to the respective env config file

   1. Chose your respective env config file, if you are deploying monitoring and alerting into dev environment chose dev.yaml similarly for uat and other environments.

Open

Depending upon your selected environment config file update the configs repo branch (like for qa.yaml added qa branch)

2. Add monitoring-dashboards folder to the respective configs repo branch.

3. Enable the serviceMonitor in the nginx-ingress configs which are available in <env>.yaml and redeploy the nginx-ingress.

Open

4. Add alertmanager secret in respective <env>.secrets.yaml

If you want you can change the slack channel and other details like group_wait , group_interval and repeat_interval according to your values.

Open

5. Deploy the prometheus-operator using go cmd or deploy using Jenkins.

1. Deploy using go lang deployer

            go run main.go deploy -e <environment_name> -c 'prometheus-operator,grafana,prometheus-kafka-exporter'

2. Deploy using Jenkin’s respective deployment job.

Open

To create a new panel in the existing dashboard:-  

1.  Login to dashboard and click on add panel  

Open

2. Set all required queries and apply the changes. Export the JSON file by clicking on the save dashboard

Open

3. Update the existing *-dashboard.json file from configs monitoring-dashboards folder with a newly exported JSON file.