Environment Prerequisites

The following infrastructure services are needed in the customer environment: 

• Swarm 15 solution stack installed and configured, with nodeExporter enabled and nodeExporterFrequency set to 120 (Do not set it too fast; this number is in seconds). 

• Enabled Swarm metrics:  

  1. metrics.nodeExporterFrequency = 120 

  2. metrics.enableNodeExporter = True    

• For deploying a telemetry solution, use the following software versions:

  1. Node_exporter 1.6.0

  2. Prometheus 2.45.0

  3. Grafana server 9.3.2

  4. Elasticsearch_exporter 1.5.0

• DHCP server for first-time boot and configuration. 

• DNS server (recommended if you do not want to see IP addresses in your dashboards, but can also be solved by configuring static entries in /etc/hosts on this VM). This step is optional.

Configuration

VMware Network Configuration

1. Verify that the VM can reach the Swarm storage nodes directly via port 9100 before proceeding with configuring Grafana and Prometheus.

2. By default, the VM uses a single Nic configured with DHCP. 

3. For a deployed dual-network SCS/Swarm configuration, select the appropriate "storage vlan" for the second virtual network card. 

4. Boot the VM and configure the second virtual network card inside the OS.  

5. Edit /etc/sysconfig/network-scripts/ifcfg-ens160 and modify/add the following to it:

   Code Block
   ONBOOT=yes  NETMASK=255.255.255.0 (Match the same netmask as your storage vlan)  IPADDR=Storage VLAN IP (Picked from 3rd party range to avoid conflicts)   BOOTPROTO=none GATEWAY=SCS IP (Usually, this is the gateway in the Swarm vlan)

6. Type the following to enable it:

   Code Block
   ifdown ens160 ifup ens160

7. Verify the new IP is coming up correctly with "ip a".

 The second network device is currently hardcoded to 172.29.x.x, so change it to fit the Swarm storage network.

Time Synchronization

1. Prometheus requires correct time synchronization for it to work and present data to Grafana. 

2. The following is already applied on SwarmTelemetry VM, but mentioning it here in case you need to reapply it.

   Code Block
   timedatectl set-timezone UTC

3. Edit /etc/chrony.conf and add server 172.29.0.3 iburst (set to your SCS IP), if missing. 

   Code Block
   systemctl stop chronyd  hwclock --systohc  systemctl start chronyd

Prometheus Master Configuration

1. Tell Prometheus to collect metrics from Swarm storage nodes.

2. Inside the /etc/prometheus/prometheus.yml file, a list of swarm nodes is displayed to modify the following section: 

   Code Block
   - job_name: 'swarm'     scrape_interval: 30s     static_configs:  - targets: ['10.10.10.84:9100','10.10.10.85:9100','10.10.10.86:9100']

3. Verify that targets are changed to match your Swarm storage node IP's.

4. If a Content Gateway is available in the deployment, add that Content Gateway to prometheus.yml as follows: 

   Code Block
   - job_name: 'swarmcontentgateway' scrape_interval: 30s static_configs: - targets: ['10.10.10.20:9100','10.10.10.21:9100' ] relabel_configs: - source_labels: [__address__] regex: "([^:]+):\\d+" target_label: instance

5. Use a human-friendly job_name displayed in the gateway dashboard. 

6. Modify the swarmUI template in /etc/prometheus/alertmanager/template/basic-email.tmpl. This is used for the HTML email template, which shows a button to the chosen URL.

7. Change the part in bold.

8. Modify the gateway job name in /etc/prometheus/alertmanager/alertmanager.yml. It must match what you chose in prometheus.yml.

   Code Block
   routes: - match: job: swarmcontentgateway

9. Modify the gateway job name in /etc/prometheus/alert.rules.yml.

   Code Block
   - alert: gateway_down expr: up{job="swarmcontentgateway"} == 0

10. To restart the service, type:

    Code Block
    systemctl restart prometheus

11. To enable it for reboots, type:

    Code Block
    systemctl enable prometheus

12. Open a browser and go to http://YourVMIP:9090/targets to test if Prometheus is up. This page shows which targets it is currently collecting metrics for and if they are reachable. 
    Use the below command to test the same from a terminal:

Gateway Node Exporter Configuration

1. Starting from Swarm 15.3, the gateway dashboard requires the node_exporter service to be running on the gateways.

2. Configure the systemd service to listen on port 9095 because Gateway metrics component uses the default port 9100.

3. Put the node_exporter golang binary in the /usr/local/bin directory. (Example: systemd config file for the node exporter)

   Code Block

   [Unit] Description=Node Exporter Wants=network-online.target After=network-online.target [Service] User=root Group=root Type=simple ExecStart=/usr/local/bin/node_exporter --web.listen-address=:9095 --collector.diskstats.ignored-devices=^(ram|loop|fd|(h|s|v|xv)d[a-z])\\d+$ --collector.filesystem.ignored-mount-points=^/(dev|proc|sys|var/lib/docker)($|/) --collector.filesystem.ignored-fs-types=^/(autofs|binfmt_misc|cgroup|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|mqueue|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|sysfs|tracefs)($|/) --collector.meminfo_numa --collector.ntp --collector.processes --collector.tcpstat --no-collector.nfs --no-collector.nfsd --no-collector.xfs --no-collector.zfs --no-collector.infiniband --no-collector.vmstat --no-collector.textfile --collector.conntrack --collector.qdisc --collector.netclass [Install] WantedBy=multi-user.target

4. Enable and configure the service.

   Code Block
   systemctl enable node_exporter systemctl start node_exporter

5. Add a job definition for it in the Prometheus master configuration file.

Example:

- job_name: 'gateway-node-exporter' 
   scrape_interval: 30s 
   static_configs: 
    - targets: ['10.10.10.20:9095'] 
relabel_configs: 
  - source_labels: [__address__] 
  regex: "([^:]+):\\d+" 
  target_label: instance

SCS Node Exporter Configuration

1. Starting from Swarm 15.3, the SCS requires the node_exporter service to monitor partition capacity information which is exposed at the end of the Swarm Node View dashboard.

2. Use the same systemd script provided by the gateway and the default listen port of 9100. SCS 1.5.1 has been modified to add a firewall rule for port 9100 on the Swarm storage network.

3. Put the node_exporter golang binary in the /usr/local/bin directory.

4. Enable and configure the service.

   Code Block
   systemctl enable node_exporter systemctl start node_exporter

5. Add a job definition for it in the Prometheus master configuration file.

Example:

- job_name: 'scs-node-exporter' 
   scrape_interval: 30s 
   static_configs: 
    - targets: ['10.10.10.2:9100'] 
relabel_configs: 
  - source_labels: [__address__] 
  regex: "([^:]+):\\d+" 
  target_label: instance

Elasticsearch Exporter Configuration

1. The Swarm Search v7 Dashboard requires a new elasticsearch_exporter service that runs locally on the Telemetry VM.

2. Modify the systemd script to tell the IP address of your Elasticsearch nodes.

3. Modify /usr/lib/systemd/system/elasticsearch_exporter.service if the Elasticsearch node IP is different.

4. The --uri needs to be pointing at the IP address of one of your elasticsearch nodes. It will auto-discover other nodes from the metrics. 

5. The new elasticsearch_exporter needs its job and replaces the old way of scraping metrics from Elasticsearch nodes via plugins. 

6. Add the job if it is missing in /etc/prometheus/prometheus.yml using the following script:

   Code Block
   - job_name: 'elasticsearch' scrape_interval: 30s static_configs: - targets: ['127.0.0.1:9114'] relabel_configs: - source_labels: [__address__] regex: "([^:]+):\\d+" target_label: instance

7. Verify the Elasticsearch exporter is running and configured to start on a reboot. 
   

   Code Block
   systemctl enable elasticsearch_exporter  systemctl start elasticsearch_exporter

Prometheus Retention Time

1. By default, Prometheus keeps metrics for 15 days (It can be modified to store for 30 days). To change the duration, follow the below instructions: 
   

   1. Edit the /root/prometheus.service file and select your default retention time for the collected metrics. (30 days are sufficient for POC's and demo's. Modify the flag --storage.tsdb.retention.time=30d to get new desired retention time.)

   2. The rule of thumb is 600MB of disk space for 30 days per Swarm node. This VM comes with a 50 GB dedicated vmdk partition for Prometheus. (This means Swarm can handle up to 32 chassis for 30 days). 

   3. If the retention time is modified, then commit the change: 

      Code Block
      cp /root/prometheus.service /usr/lib/systemd/system systemctl daemon-reload promtool check config /etc/prometheus/prometheus.yml systemctl restart prometheus

Prometheus Security

It may be desirable to restrict the Prometheus server to only allow queries from the local host since the grafana-server is running on the same VM. To do so, edit the prometheus.service file and add the flag --web.listen-address=127.0.0.1:9090.

Grafana Configuration 

1. Modify the /etc/grafana/grafana.ini file to set up the IP address and the server should be listening too. By default, it binds to all local IPs on port 80.  

2. Review the admin_password parameter.

3. Grafana has several authentication options including google-auth / oAuth / ldap. The default option is basic http auth. See https://docs.grafana.org/ for more details. 

4. To start the service, type "service grafana-server start" or "systemctl start grafana-server".

5. To enable it for reboots, type "systemctl enable grafana-server".

Alertmanager Configuration

1. Four alerts are defined in /etc/prometheus/alert.rules.yml including”

   1. Service_down: Triggered if any Swarm storage node is down for more than 30 minutes.

   2. Gateway_down: Triggered if the cloudgateway service is down for more than 2 minutes.

   3. Elasticsearch_cluster_state: Triggered if the cluster state is changed to "red" after 5 minutes.

   4. Swarm_volume_missing: Triggered if the reported drive count is decreasing in 10 minutes.

2. /etc/prometheus/prometheus.yml file contains a section that points to the alertmanager service on port 9093 as well as which alert.rules.yml file to use. 

3. The configuration to send alerts to, is defined in /etc/prometheus/alertmanager/alertmanager.yml.

4. By default, the route is disabled as it requires manual input from the environment (smtp server, user, pass, etc.). An example of a working route to email alerts via Gmail: 

- name: 'swarmtelemetry' 
  email_configs: 
  - to: swarmtelemetry@gmail.com 
    from: swarmtelemetry@gmail.com 
    smarthost: smtp.gmail.com:587 
    auth_username: swarmtelemetry@gmail.com 
    auth_identity: swarmtelemetry@gmail.com 
    auth_password: YOUGMAILPASSWORD or APPPASSWORD 
    send_resolved: true

Example configuration for a local SMTP relay in the enterprise environment.

- name: 'emailchannel' 
  email_configs: 
  - to: admin@acme.com 
    from: swarmtelemetry@acme.com
    smarthost: smtp.acme.com:25 
    require_tls: false
    send_resolved: true

5. Once the configuration is completed, restart the alertmanager.

   Code Block
   systemctl restart alertmanager

6. Verify the alertmanager.yml has the correct syntax using the below command:

   Code Block
   amtool check-config /etc/prometheus/alertmanager/alertmanager.yml

   This returns the following output:

   Code Block
   Checking '/etc/prometheus/alertmanager/alertmanager.yml' SUCCESS Found: - global config - route - 1 inhibit rules - 2 receivers - 1 templates SUCCESS

7. Run the below command to show a list of active alerts.

   Code Block
   amtool alert

9. Run the below command to show which alert route is enabled:
   

   Code Block
   amtool config routes show Routing tree: └── default-route  receiver: disabled

   Example Email Alert: 

Image Removed

10. The easiest way to trigger an alert for testing purposes is to shut down one gateway.

Image Removed

Dashboards on Grafana