======================= Metrics Developer Guide ======================= Anible File Organization ======================== Ansible-Server -------------- Please follow the following file structure: .. code-block:: bash ansible-server | ansible.cfg | hosts | +---group_vars | all.yml | +---playbooks | clean.yaml | setup.yaml | \---roles +---clean-monitoring | \---tasks | main.yml | +---monitoring +---files | | monitoring-namespace.yaml | | | +---alertmanager | | alertmanager-config.yaml | | alertmanager-deployment.yaml | | alertmanager-service.yaml | | alertmanager1-deployment.yaml | | alertmanager1-service.yaml | | | +---cadvisor | | cadvisor-daemonset.yaml | | cadvisor-service.yaml | | | +---collectd-exporter | | collectd-exporter-deployment.yaml | | collectd-exporter-service.yaml | | | +---grafana | | grafana-datasource-config.yaml | | grafana-deployment.yaml | | grafana-pv.yaml | | grafana-pvc.yaml | | grafana-service.yaml | | | +---kube-state-metrics | | kube-state-metrics-deployment.yaml | | kube-state-metrics-service.yaml | | | +---node-exporter | | nodeexporter-daemonset.yaml | | nodeexporter-service.yaml | | | \---prometheus | main-prometheus-service.yaml | prometheus-config.yaml | prometheus-deployment.yaml | prometheus-pv.yaml | prometheus-pvc.yaml | prometheus-service.yaml | prometheus1-deployment.yaml | prometheus1-service.yaml | \---tasks main.yml Ansible - Client ---------------- Please follow the following file structure: .. code-block:: bash ansible-server | ansible.cfg | hosts | +---group_vars | all.yml | +---playbooks | clean.yaml | setup.yaml | \---roles +---clean-collectd | \---tasks | main.yml | +---collectd +---files | collectd.conf.j2 | \---tasks main.yml Summary of Roles ================ A brief description of the Ansible playbook roles, which are used to deploy the monitoring cluster Ansible Server Roles -------------------- Ansible Server, this part consists of the roles used to deploy Prometheus Alertmanager Grafana stack on the server-side Role: Monitoring ~~~~~~~~~~~~~~~~ Deployment and configuration of PAG stack along with collectd-exporter, cadvisor and node-exporter. Role: Clean-Monitoring ~~~~~~~~~~~~~~~~~~~~~~ Removes all the components deployed by the Monitoring role. File-Task Mapping and Configurable Parameters ================================================ Ansible Server ---------------- Role: Monitoring ~~~~~~~~~~~~~~~~~~~ Alert Manager ^^^^^^^^^^^^^^^ File: alertmanager-config.yaml ''''''''''''''''''''''''''''''''' Path : monitoring/files/alertmanager/alertmanager-config.yaml Task: Configures Receivers for alertmanager Summary: A configmap, currently configures webhook for alertmanager, can be used to configure any kind of receiver Configurable Parameters: receiver.url: change to the webhook receiver's URL route: Can be used to add receivers File: alertmanager-deployment.yaml '''''''''''''''''''''''''''''''''' Path : monitoring/files/alertmanager/alertmanager-deployment.yaml Task: Deploys alertmanager instance Summary: A Deployment, deploys 1 replica of alertmanager File: alertmanager-service.yaml ''''''''''''''''''''''''''''''''' Path : monitoring/files/alertmanager/alertmanager-service.yaml Task: Creates a K8s service for alertmanager Summary: A Nodeport type of service, so that user can create "silences", view the status of alerts from the native alertmanager dashboard / UI. Configurable Parameters: spec.type: Options : NodePort, ClusterIP, LoadBalancer spec.ports: Edit / add ports to be handled by the service **Note: alertmanager1-deployment, alertmanager1-service are the same as alertmanager-deployment and alertmanager-service respectively.** CAdvisor ^^^^^^^^^^^ File: cadvisor-daemonset.yaml ''''''''''''''''''''''''''''''''' Path : monitoring/files/cadvisor/cadvisor-daemonset.yaml Task: To create a cadvisor daemonset Summary: A daemonset, used to scrape data of the kubernetes cluster itself, its a daemonset so an instance is run on every node. Configurable Parameters: spec.template.spec.ports: Port of the container File: cadvisor-service.yaml ''''''''''''''''''''''''''''''''' Path : monitoring/files/cadvisor/cadvisor-service.yaml Task: To create a cadvisor service Summary: A ClusterIP service for cadvisor to communicate with prometheus Configurable Parameters: spec.ports: Add / Edit ports Collectd Exporter ^^^^^^^^^^^^^^^^^^^^ File: collectd-exporter-deployment.yaml '''''''''''''''''''''''''''''''''''''''''' Path : monitoring/files/collectd-exporter/collectd-exporter-deployment.yaml Task: To create a collectd replica Summary: A deployment, acts as receiver for collectd data sent by client machines, prometheus pulls data from this exporter Configurable Parameters: spec.template.spec.ports: Port of the container File: collectd-exporter.yaml ''''''''''''''''''''''''''''''''' Path : monitoring/files/collectd-exporter/collectd-exporter.yaml Task: To create a collectd service Summary: A NodePort service for collectd-exporter to hold data for prometheus to scrape Configurable Parameters: spec.ports: Add / Edit ports Grafana ^^^^^^^^^ File: grafana-datasource-config.yaml '''''''''''''''''''''''''''''''''''''''''' Path : monitoring/files/grafana/grafana-datasource-config.yaml Task: To create config file for grafana Summary: A configmap, adds prometheus datasource in grafana File: grafana-deployment.yaml ''''''''''''''''''''''''''''''''' Path : monitoring/files/grafana/grafana-deployment.yaml Task: To create a grafana deployment Summary: The grafana deployment creates a single replica of grafana, with preconfigured prometheus datasource. Configurable Parameters: spec.template.spec.ports: Edit ports spec.template.spec.env: Add / Edit environment variables File: grafana-pv.yaml ''''''''''''''''''''''''''''''''' Path : monitoring/files/grafana/grafana-pv.yaml Task: To create a persistent volume for grafana Summary: A persistent volume for grafana. Configurable Parameters: spec.capacity.storage: Increase / decrease size spec.accessModes: To change the way PV is accessed. spec.nfs.server: To change the ip address of NFS server spec.nfs.path: To change the path of the server File: grafana-pvc.yaml ''''''''''''''''''''''''''''''''' Path : monitoring/files/grafana/grafana-pvc.yaml Task: To create a persistent volume claim for grafana Summary: A persistent volume claim for grafana. Configurable Parameters: spec.resources.requests.storage: Increase / decrease size File: grafana-service.yaml ''''''''''''''''''''''''''''''''' Path : monitoring/files/grafana/grafana-service.yaml Task: To create a service for grafana Summary: A Nodeport type of service, so that users actually connect to, view the dashboard / UI. Configurable Parameters: spec.type: Options : NodePort, ClusterIP, LoadBalancer spec.ports: Edit / add ports to be handled by the service Kube State Metrics ^^^^^^^^^^^^^^^^^^^^ File: kube-state-metrics-deployment.yaml '''''''''''''''''''''''''''''''''''''''' Path : monitoring/files/kube-state-metrics/kube-state-metrics-deployment.yaml Task: To create a kube-state-metrics instance Summary: A deployment, used to collect metrics of the kubernetes cluster iteself Configurable Parameters: spec.template.spec.containers.ports: Port of the container File: kube-state-metrics-service.yaml ''''''''''''''''''''''''''''''''''''' Path : monitoring/files/kube-state-metrics/kube-state-metrics-service.yaml Task: To create a collectd service Summary: A NodePort service for collectd-exporter to hold data for prometheus to scrape Configurable Parameters: spec.ports: Add / Edit ports Node Exporter ^^^^^^^^^^^^^^^ File: node-exporter-daemonset.yaml '''''''''''''''''''''''''''''''''' Path : monitoring/files/node-exporter/node-exporter-daemonset.yaml Task: To create a node exporter daemonset Summary: A daemonset, used to scrape data of the host machines / node, its a daemonset so an instance is run on every node. Configurable Parameters: spec.template.spec.ports: Port of the container File: node-exporter-service.yaml ''''''''''''''''''''''''''''''''' Path : monitoring/files/node-exporter/node-exporter-service.yaml Task: To create a node exporter service Summary: A ClusterIP service for node exporter to communicate with Prometheus Configurable Parameters: spec.ports: Add / Edit ports Prometheus ^^^^^^^^^^^^^ File: prometheus-config.yaml '''''''''''''''''''''''''''''''''''''''''' Path : monitoring/files/prometheus/prometheus-config.yaml Task: To create a config file for Prometheus Summary: A configmap, adds alert rules. Configurable Parameters: data.alert.rules: Add / Edit alert rules File: prometheus-deployment.yaml ''''''''''''''''''''''''''''''''' Path : monitoring/files/prometheus/prometheus-deployment.yaml Task: To create a Prometheus deployment Summary: The Prometheus deployment creates a single replica of Prometheus, with preconfigured Prometheus datasource. Configurable Parameters: spec.template.spec.affinity: To change the node affinity, make sure only 1 instance of prometheus is running on 1 node. spec.template.spec.ports: Add / Edit container port File: prometheus-pv.yaml ''''''''''''''''''''''''''''''''' Path : monitoring/files/prometheus/prometheus-pv.yaml Task: To create a persistent volume for Prometheus Summary: A persistent volume for Prometheus. Configurable Parameters: spec.capacity.storage: Increase / decrease size spec.accessModes: To change the way PV is accessed. spec.hostpath.path: To change the path of the volume File: prometheus-pvc.yaml ''''''''''''''''''''''''''''''''' Path : monitoring/files/prometheus/prometheus-pvc.yaml Task: To create a persistent volume claim for Prometheus Summary: A persistent volume claim for Prometheus. Configurable Parameters: spec.resources.requests.storage: Increase / decrease size File: prometheus-service.yaml ''''''''''''''''''''''''''''''''' Path : monitoring/files/prometheus/prometheus-service.yaml Task: To create a service for prometheus Summary: A Nodeport type of service, prometheus native dashboard available here. Configurable Parameters: spec.type: Options : NodePort, ClusterIP, LoadBalancer spec.ports: Edit / add ports to be handled by the service File: main-prometheus-server.yaml ''''''''''''''''''''''''''''''''''' Path: monitoring/files/prometheus/main-prometheus-service.yaml Task: A service that connects both prometheus instances. Summary: A Nodeport service for other services to connect to the Prometheus cluster. As HA Prometheus needs to independent instances of Prometheus scraping the same inputs having the same configuration **Note: prometheus-deployment, prometheus1-service are the same as prometheus-deployment and prometheus-service respectively.** Ansible Client Roles ---------------------- Role: Collectd ~~~~~~~~~~~~~~~~~~ File: main.yml ^^^^^^^^^^^^^^^^ Path: collectd/tasks/main.yaml Task: Install collectd along with prerequisites Associated template file: collectd.conf.j2 Path: collectd/files/collectd.conf.j2 Summary: Edit this file to change the default configuration to be installed on the client's machine