...

...

...

Environment Prerequisites

The following infrastructure services are needed in the customer environment: 

1. Swarm 15 solution stack installed and configured, with nodeExporter enabled and nodeExporterFrequency set to 120 ( do Do not set it too fast; this number is in seconds). 

2. Enabled Swarm metrics:  

   1. metrics.nodeExporterFrequency = 120 

   2. metrics.enableNodeExporter = True    

3. If your deploying you deploy your own telemetry solution, you will need to use the following software versions:

   1. nodeNode_exporter 1.6.0

   2. prometheus Prometheus 2.45.0

   3. grafana Grafana server 9.3.2

   4. elasticsearchElasticsearch_exporter 1.5.0

4. DHCP server, needed for first-time boot and configuration (optional) configuration. 

5. DNS server (recommended if you don't do not want to see IP addresses in your dashboards, but can also be solved by configuring static entries in /etc/hosts on this VM) .

Configuration

VMware Network

...

Configuration

1. Make sure the VM can reach the Swarm storage nodes directly via port 9100 before proceeding with configuring Grafana and Prometheus.

...

2. By default, the VM uses a single Nic configured with DHCP. 

3. If you have deployed a dual-network SCS/Swarm configuration then we must first select the appropriate "storage vlan" for the second virtual network card. 

4. Boot the VM and configure the second virtual network card inside the OS.  

5. Edit /etc/sysconfig/network-scripts/ifcfg-ens160 and modify/add the following to it:

...

6. Enable it by typing:

7. Verify the new

...

7. IP is coming up correctly with "ip a"

...

7. .

...

Time Synchronization

1. Prometheus requires correct time synchronization for it to work and present data to

...

1. Grafana. 

2. The following has already been done on SwarmTelemetry VM, but it is mentioned in case you need to re-apply it.

...

3. Edit /etc/chrony.conf and

...

3. add server 172.29.0.3

...

3. iburst (

...

3. Set to your SCS IP), if missing. 

Prometheus

...

Master Configuration

1. We need to tell Prometheus which Swarm storage nodes we wish to collect metrics from.  

2. Inside the /etc/prometheus/prometheus.yml file, you will

...

1. see a list of swarm nodes

...

1. to modify

...

1. the following section: 

   Code Block
   - job_name: 'swarm'     scrape_interval: 30s     static_configs:  - targets: ['10.10.10.84:9100','10.10.10.85:9100','10.10.10.86:9100']

2. Make sure to change the targets to match your Swarm storage node IP's.

...

...

...

...

...

...

4. If you have content gateway in your deployment, you can add them to prometheus.yml as follows: 

...

...

Note: if you have multiple gateways , just add them to the targets list. 

4. Code Block
   - job_name: 'swarmcontentgateway' scrape_interval: 30s static_configs: - targets: ['10.10.10.20:9100','10.10.10.21:9100' ] relabel_configs: - source_labels: [__address__] regex: "([^:]+):\\d+" target_label: instance

...

5. You will see the job_name

...

5. in the gateway dashboard, so make sure to make it human friendly. 

6. Modify the swarmUI template in /etc/prometheus/alertmanager/template/basic-email.tmpl

...

5. . This will be used for the email html template showing a button to the chosen URL.

...

6. Change the part in bold:

...

8. Modify the gateway job name in /etc/prometheus/alertmanager/alertmanager.yml

...

8. . It must match what you chose in prometheus.yml.

...

8. Code Block
   routes: - match: job: swarmcontentgateway

9. Modify the gateway job name in /etc/prometheus/alert.rules.

...

8. yml

   Code Block
   - alert: gateway_down expr: up{job="swarmcontentgateway"} == 0

9. To

...

8. restart the service, type:

11. To enable it for reboots, type:

12. You can test

...

12. Prometheus is up by opening a browser and going to http://YourVMIP:9090/targets

...

12. . This page will show which targets it is currently collecting metrics for and if they are reachable.

...

12.  You can also do this from a terminal by doing:

Gateway

...

Node Exporter Configuration

1. Starting Swarm 15.3, the gateway dashboard require you to run the node_exporter service on the gateways.

2. The systemd service must be configured to listen on port 9095, because the default port 9100 is used by the gateway metrics component.

3. Make sure to put the node_exporter golang binary in the /usr/local/bin directory.

...

1. (Example: systemd config file for the node exporter.)

   Code Block

   [Unit] Description=Node Exporter Wants=network-online.target After=network-online.target [Service] User=root Group=root Type=simple ExecStart=/usr/local/bin/node_exporter --web.listen-address=:9095 --collector.diskstats.ignored-devices=^(ram|loop|fd|(h|s|v|xv)d[a-z])\\d+$ --collector.filesystem.ignored-mount-points=^/(dev|proc|sys|var/lib/docker)($|/) --collector.filesystem.ignored-fs-types=^/(autofs|binfmt_misc|cgroup|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|mqueue|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|sysfs|tracefs)($|/) --collector.meminfo_numa --collector.ntp --collector.processes --collector.tcpstat --no-collector.nfs --no-collector.nfsd --no-collector.xfs --no-collector.zfs --no-collector.infiniband --no-collector.vmstat --no-collector.textfile --collector.conntrack --collector.qdisc --collector.netclass [Install] WantedBy=multi-user.target

...

1. Enable and configure the service.

...

5. Add a job definition for it in the

...

5. Prometheus master configuration file

...

5. .

exampleExample:

- job_name: 'gateway-node-exporter' 
   scrape_interval: 30s 
   static_configs: 
    - targets: ['10.10.10.20:9095'] 
relabel_configs: 
  - source_labels: [__address__] 
  regex: "([^:]+):\\d+" 
  target_label: instance

SCS

...

Node Exporter Configuration

1. Starting Swarm 15.3, the SCS requires the node_exporter service to monitor partition capacity information which is exposed at the end of the Swarm Node View dashboard.

2. Use the same systemd script provided by the gateway, but here you must use the default listen port of 9100

...

1. . SCS 1.5.1 has been modified to add a firewall rule for port 9100 on the swarm storage network

...

1. .

2. Make sure to put the node_exporter golang binary in the /usr/local/bin directory.

...

1. Enable and configure the service.

5. You will have to add a job definition for it in the

...

5. Prometheus master configuration file

...

5. .

exampleExample:

- job_name: 'scs-node-exporter' 
   scrape_interval: 30s 
   static_configs: 
    - targets: ['10.10.10.2:9100'] 
relabel_configs: 
  - source_labels: [__address__] 
  regex: "([^:]+):\\d+" 
  target_label: instance

Elasticsearch

...

Exporter Configuration

1. The Swarm Search v7 Dashboard requires a new elasticsearch_exporter service that runs locally on the Telemetry VM.

2. You will need to modify the systemd script to tell

...

1. what the

...

1. IP address is of one of your elasticsearch nodes

...

1. .

...

1. Modify /usr/lib/systemd/system/elasticsearch_exporter.service if the elasticsearch node IP is

...

1. different.

2. The --uri needs to be pointing at the

...

1. IP address of one of your elasticsearch nodes

...

1. . It will auto-discover the other nodes from the metrics. 

2. The new elasticsearch_exporter

...

1. need it's own job and replaces the old way of scraping metrics from elasticsearch nodes via plugins. 

2. The job you need to add if missing in /etc/prometheus/prometheus.yml  is as follows:

...

1. Code Block
   - job_name: 'elasticsearch' scrape_interval: 30s static_configs: - targets: ['127.0.0.1:9114'] relabel_configs: - source_labels: [__address__] regex: "([^:]+):\\d+" target_label: instance

2. Make sure the elasticsearch exporter is running and configured to start on a reboot. 

Prometheus

...

Retention Time

1. By default, Prometheus will keep metrics for 15 days

...

1. (It can be modified to store for 30 days

...

1. ). If you wish to change this, follow

...

1. the below instructions: 

• Edit the /root/prometheus.service file and choose select your default retention time for the collected metrics. Note:

...

• The rule of thumb is 600MB of disk space for 30 days per Swarm Node. This VM comes with a 50 GB dedicated vmdk partition for prometheusPrometheus. (This means we can handle up to 32 chassis for 30 days). 

• If you have modified the retention, then you need to commit the change: 

...

Prometheus Security

It may be desirable to restrict prometheus Prometheus server to only allow query's from the local host, since grafana-server is running on the same VM. This can be done by editing prometheus.service file and adding the flag --web.listen-address=127.0.0.1:9090  Note: if you decide to bind only to localhost , you 9090.

Grafana

...

Configuration 

1. The /etc/grafana/grafana.ini file should be modified to setup the IP address, the server should be listening too. By default,

...

1. it will bind to all local IP's on port 80.  

2. Review the admin_password parameter.

...

...

...

3. Grafana has several authentication options including google-auth / oAuth / ldap and by default basic http auth.

...

3.  See https://docs.grafana.org/ for more details. 

4. To start the service, type "service grafana-server start" or "systemctl start grafana-server"

...

3. .

4. To enable it for reboots, type "systemctl enable grafana-server"

...

3. .

Alertmanager

...

Configuration

1. We currently have 4 alerts defined in /etc/prometheus/alert.rules.

...

1. yml

   1. Service_down:

...

1. 1. Triggered if any swarm storage node is down for more than 30

...

1. 1. minutes.

   2. Gateway_down:

...

1. 1. Triggered if the cloudgateway service is down for more than 2

...

1. 1. minutes.

   2. Elasticsearch_cluster_state:

...

1. 1. Triggered if the cluster state changed to "red" after 5

...

1. 1. minutes.

   2. Swarm_volume_missing:

...

1. 1. Triggered if reported drive count is decreasing over a period of 10

...

1. 1. minutes.

...

2. /etc/prometheus/prometheus.yml now contains a section that points to the alertmanager service on port 9093 as well as which alert.rules.yml file to use. 

3. The configuration for where to send alerts is defined in /etc/prometheus/alertmanager/alertmanager.yml.

...

2. By default, the route is disabled as it requires manual input from your environment (smtp server , user, pass etc.)

...

1. . Here is an example of a working route to email alerts via

...

1. Gmail: 

- name: 'swarmtelemetry' 
  email_configs: 
  - to: swarmtelemetry@gmail.com 
    from: swarmtelemetry@gmail.com 
    smarthost: smtp.gmail.com:587 
    auth_username: swarmtelemetry@gmail.com 
    auth_identity: swarmtelemetry@gmail.com 
    auth_password: YOUGMAILPASSWORD or APPPASSWORD 
    send_resolved: true

...

...

...

...

- name: 'emailchannel' 
  email_configs: 
  - to: admin@acme.com 
    from: swarmtelemetry@acme.com
    smarthost: smtp.acme.com:25 
    require_tls: false
    send_resolved: true

5. Once configuration has completed, restart the alertmanager:

6. To verify the alertmanager.yml has the correct syntax, run: 

...

7. You will

...

7. get the following output: 

Checking '/etc/prometheus/alertmanager/alertmanager.yml'  SUCCESS 
Found: 
 - global config 
 - route 
 - 1 inhibit rules 
 - 2 receivers 
 - 1 templates 
  SUCCESS 

8. To show a list of active alerts, run: 

...

9. To show which alert route is enabled, run: 

Example Email Alert: 

...

10. The easiest way to trigger an alert for testing purposes is to shutdown 1 gateway.

...

Dashboards on

...

Grafana

...

General Advice

...

Around Defining New Alerts

• Pages should be urgent, important, actionable, and real. 

• They should represent either ongoing or imminent problems with your service. 

• Err on the side of removing noisy alerts – over-monitoring is a harder problem to solve than under-monitoring. 

• You should almost always be able to classify the problem into one of: availability &

  • Availability and basic functionality

  ; latency; correctness

  • Latency

  • Correctness (completeness, freshness and durability of data)

  ;

  • and

  feature

  • Feature-specific problems

  . 

• Symptoms are a better way to capture more problems more comprehensively and robustly with less effort. 

• Include cause-based information in symptom-based pages or on dashboards, but avoid alerting directly on causes. 

• The further up your serving stack you go, the more distinct problems you catch in a single rule.  But  But don't go so far you can't sufficiently distinguish what's going on. 

• If you want a quiet on call rotation, it's imperative to have a system for dealing with things that need timely response, but are not imminently critical. 

...