5.2. Metrics Developer Guide¶
5.2.1. Anible File Organization¶
5.2.1.1. Ansible-Server¶
Please follow the following file structure:
ansible-server
| ansible.cfg
| hosts
|
+---group_vars
| all.yml
|
+---playbooks
| clean.yaml
| setup.yaml
|
\---roles
+---clean-monitoring
| \---tasks
| main.yml
|
+---monitoring
+---files
| | monitoring-namespace.yaml
| |
| +---alertmanager
| | alertmanager-config.yaml
| | alertmanager-deployment.yaml
| | alertmanager-service.yaml
| | alertmanager1-deployment.yaml
| | alertmanager1-service.yaml
| |
| +---cadvisor
| | cadvisor-daemonset.yaml
| | cadvisor-service.yaml
| |
| +---collectd-exporter
| | collectd-exporter-deployment.yaml
| | collectd-exporter-service.yaml
| |
| +---grafana
| | grafana-datasource-config.yaml
| | grafana-deployment.yaml
| | grafana-pv.yaml
| | grafana-pvc.yaml
| | grafana-service.yaml
| |
| +---kube-state-metrics
| | kube-state-metrics-deployment.yaml
| | kube-state-metrics-service.yaml
| |
| +---node-exporter
| | nodeexporter-daemonset.yaml
| | nodeexporter-service.yaml
| |
| \---prometheus
| main-prometheus-service.yaml
| prometheus-config.yaml
| prometheus-deployment.yaml
| prometheus-pv.yaml
| prometheus-pvc.yaml
| prometheus-service.yaml
| prometheus1-deployment.yaml
| prometheus1-service.yaml
|
\---tasks
main.yml
5.2.1.2. Ansible - Client¶
Please follow the following file structure:
ansible-server
| ansible.cfg
| hosts
|
+---group_vars
| all.yml
|
+---playbooks
| clean.yaml
| setup.yaml
|
\---roles
+---clean-collectd
| \---tasks
| main.yml
|
+---collectd
+---files
| collectd.conf.j2
|
\---tasks
main.yml
5.2.2. Summary of Roles¶
A brief description of the Ansible playbook roles, which are used to deploy the monitoring cluster
5.2.2.1. Ansible Server Roles¶
Ansible Server, this part consists of the roles used to deploy Prometheus Alertmanager Grafana stack on the server-side
5.2.2.1.1. Role: Monitoring¶
Deployment and configuration of PAG stack along with collectd-exporter, cadvisor and node-exporter.
5.2.2.1.2. Role: Clean-Monitoring¶
Removes all the components deployed by the Monitoring role.
5.2.3. File-Task Mapping and Configurable Parameters¶
5.2.3.1. Ansible Server¶
5.2.3.1.1. Role: Monitoring¶
5.2.3.1.1.1. Alert Manager¶
5.2.3.1.1.1.1. File: alertmanager-config.yaml¶
Path : monitoring/files/alertmanager/alertmanager-config.yaml
Task: Configures Receivers for alertmanager
Summary: A configmap, currently configures webhook for alertmanager, can be used to configure any kind of receiver
- Configurable Parameters:
- receiver.url: change to the webhook receiver’s URL route: Can be used to add receivers
5.2.3.1.1.1.2. File: alertmanager-deployment.yaml¶
Path : monitoring/files/alertmanager/alertmanager-deployment.yaml
Task: Deploys alertmanager instance
Summary: A Deployment, deploys 1 replica of alertmanager
5.2.3.1.1.1.3. File: alertmanager-service.yaml¶
Path : monitoring/files/alertmanager/alertmanager-service.yaml
Task: Creates a K8s service for alertmanager
Summary: A Nodeport type of service, so that user can create “silences”, view the status of alerts from the native alertmanager dashboard / UI.
- Configurable Parameters:
- spec.type: Options : NodePort, ClusterIP, LoadBalancer spec.ports: Edit / add ports to be handled by the service
Note: alertmanager1-deployment, alertmanager1-service are the same as alertmanager-deployment and alertmanager-service respectively.
5.2.3.1.1.2. CAdvisor¶
5.2.3.1.1.2.1. File: cadvisor-daemonset.yaml¶
Path : monitoring/files/cadvisor/cadvisor-daemonset.yaml
Task: To create a cadvisor daemonset
Summary: A daemonset, used to scrape data of the kubernetes cluster itself, its a daemonset so an instance is run on every node.
- Configurable Parameters:
- spec.template.spec.ports: Port of the container
5.2.3.1.1.2.2. File: cadvisor-service.yaml¶
Path : monitoring/files/cadvisor/cadvisor-service.yaml
Task: To create a cadvisor service
Summary: A ClusterIP service for cadvisor to communicate with prometheus
- Configurable Parameters:
- spec.ports: Add / Edit ports
5.2.3.1.1.3. Collectd Exporter¶
5.2.3.1.1.3.1. File: collectd-exporter-deployment.yaml¶
Path : monitoring/files/collectd-exporter/collectd-exporter-deployment.yaml
Task: To create a collectd replica
Summary: A deployment, acts as receiver for collectd data sent by client machines, prometheus pulls data from this exporter
- Configurable Parameters:
- spec.template.spec.ports: Port of the container
5.2.3.1.1.3.2. File: collectd-exporter.yaml¶
Path : monitoring/files/collectd-exporter/collectd-exporter.yaml
Task: To create a collectd service
Summary: A NodePort service for collectd-exporter to hold data for prometheus to scrape
- Configurable Parameters:
- spec.ports: Add / Edit ports
5.2.3.1.1.4. Grafana¶
5.2.3.1.1.4.1. File: grafana-datasource-config.yaml¶
Path : monitoring/files/grafana/grafana-datasource-config.yaml
Task: To create config file for grafana
Summary: A configmap, adds prometheus datasource in grafana
5.2.3.1.1.4.2. File: grafana-deployment.yaml¶
Path : monitoring/files/grafana/grafana-deployment.yaml
Task: To create a grafana deployment
Summary: The grafana deployment creates a single replica of grafana, with preconfigured prometheus datasource.
- Configurable Parameters:
- spec.template.spec.ports: Edit ports spec.template.spec.env: Add / Edit environment variables
5.2.3.1.1.4.3. File: grafana-pv.yaml¶
Path : monitoring/files/grafana/grafana-pv.yaml
Task: To create a persistent volume for grafana
Summary: A persistent volume for grafana.
- Configurable Parameters:
- spec.capacity.storage: Increase / decrease size spec.accessModes: To change the way PV is accessed. spec.nfs.server: To change the ip address of NFS server spec.nfs.path: To change the path of the server
5.2.3.1.1.4.4. File: grafana-pvc.yaml¶
Path : monitoring/files/grafana/grafana-pvc.yaml
Task: To create a persistent volume claim for grafana
Summary: A persistent volume claim for grafana.
- Configurable Parameters:
- spec.resources.requests.storage: Increase / decrease size
5.2.3.1.1.4.5. File: grafana-service.yaml¶
Path : monitoring/files/grafana/grafana-service.yaml
Task: To create a service for grafana
Summary: A Nodeport type of service, so that users actually connect to, view the dashboard / UI.
- Configurable Parameters:
- spec.type: Options : NodePort, ClusterIP, LoadBalancer spec.ports: Edit / add ports to be handled by the service
5.2.3.1.1.5. Kube State Metrics¶
5.2.3.1.1.5.1. File: kube-state-metrics-deployment.yaml¶
Path : monitoring/files/kube-state-metrics/kube-state-metrics-deployment.yaml
Task: To create a kube-state-metrics instance
Summary: A deployment, used to collect metrics of the kubernetes cluster iteself
- Configurable Parameters:
- spec.template.spec.containers.ports: Port of the container
5.2.3.1.1.5.2. File: kube-state-metrics-service.yaml¶
Path : monitoring/files/kube-state-metrics/kube-state-metrics-service.yaml
Task: To create a collectd service
Summary: A NodePort service for collectd-exporter to hold data for prometheus to scrape
- Configurable Parameters:
- spec.ports: Add / Edit ports
5.2.3.1.1.6. Node Exporter¶
5.2.3.1.1.6.1. File: node-exporter-daemonset.yaml¶
Path : monitoring/files/node-exporter/node-exporter-daemonset.yaml
Task: To create a node exporter daemonset
Summary: A daemonset, used to scrape data of the host machines / node, its a daemonset so an instance is run on every node.
- Configurable Parameters:
- spec.template.spec.ports: Port of the container
5.2.3.1.1.6.2. File: node-exporter-service.yaml¶
Path : monitoring/files/node-exporter/node-exporter-service.yaml
Task: To create a node exporter service
Summary: A ClusterIP service for node exporter to communicate with Prometheus
- Configurable Parameters:
- spec.ports: Add / Edit ports
5.2.3.1.1.7. Prometheus¶
5.2.3.1.1.7.1. File: prometheus-config.yaml¶
Path : monitoring/files/prometheus/prometheus-config.yaml
Task: To create a config file for Prometheus
Summary: A configmap, adds alert rules.
- Configurable Parameters:
- data.alert.rules: Add / Edit alert rules
5.2.3.1.1.7.2. File: prometheus-deployment.yaml¶
Path : monitoring/files/prometheus/prometheus-deployment.yaml
Task: To create a Prometheus deployment
Summary: The Prometheus deployment creates a single replica of Prometheus, with preconfigured Prometheus datasource.
- Configurable Parameters:
- spec.template.spec.affinity: To change the node affinity,
- make sure only 1 instance of prometheus is running on 1 node.
spec.template.spec.ports: Add / Edit container port
5.2.3.1.1.7.3. File: prometheus-pv.yaml¶
Path : monitoring/files/prometheus/prometheus-pv.yaml
Task: To create a persistent volume for Prometheus
Summary: A persistent volume for Prometheus.
- Configurable Parameters:
- spec.capacity.storage: Increase / decrease size spec.accessModes: To change the way PV is accessed. spec.hostpath.path: To change the path of the volume
5.2.3.1.1.7.4. File: prometheus-pvc.yaml¶
Path : monitoring/files/prometheus/prometheus-pvc.yaml
Task: To create a persistent volume claim for Prometheus
Summary: A persistent volume claim for Prometheus.
- Configurable Parameters:
- spec.resources.requests.storage: Increase / decrease size
5.2.3.1.1.7.5. File: prometheus-service.yaml¶
Path : monitoring/files/prometheus/prometheus-service.yaml
Task: To create a service for prometheus
Summary: A Nodeport type of service, prometheus native dashboard available here.
- Configurable Parameters:
- spec.type: Options : NodePort, ClusterIP, LoadBalancer spec.ports: Edit / add ports to be handled by the service
5.2.3.1.1.7.6. File: main-prometheus-server.yaml¶
Path: monitoring/files/prometheus/main-prometheus-service.yaml
Task: A service that connects both prometheus instances.
Summary: A Nodeport service for other services to connect to the Prometheus cluster. As HA Prometheus needs to independent instances of Prometheus scraping the same inputs having the same configuration
Note: prometheus-deployment, prometheus1-service are the same as prometheus-deployment and prometheus-service respectively.
5.2.3.2. Ansible Client Roles¶
5.2.3.2.1. Role: Collectd¶
5.2.3.2.1.1. File: main.yml¶
Path: collectd/tasks/main.yaml
Task: Install collectd along with prerequisites
Associated template file:
collectd.conf.j2 Path: collectd/files/collectd.conf.j2
Summary: Edit this file to change the default configuration to be installed on the client’s machine