Introduction and Prerequisites

This document guides to effectively deploy a Swarm cluster using the OVF VM bundle package – Rocky Linux 8 edition. 

Both the OVF bundle package and the standalone software are available in the DataCore downloads website.

There are two main sections in this document:

• Swarm deployment using the OVF VM bundle package.

• Deploying Swarm from scratch.

The present document is based on a traditional deployment of Swarm, where the management and access layer run virtualized in one or more VMware ESXi hosts, while the storage nodes are physical x86-64 machines that will hold the data. See the below diagram.

Swarm Components

The Swarm stack utilizes several components grouped in two different layers:

• Storage Layer: Comprised by the Swarm storage nodes which hold the information and take care of data protection.

• Management and Access Layer: As the name implies, this layer provides both the administration of the Swarm cluster as well as access to the storage for users and client applications. No data storage or caching is happening in this layer.  

Below are the software components of the entire Swarm stack, their functions, and count recommendations for durability and availability purposes:

Swarm Storage Nodes

• Swarm is an in-purpose built on-premises object storage solution. It runs on standard physical x86-64 servers providing a single pool of resources, supporting billions of objects/files in the same cluster and extending its capabilities to multiple sites (data replication).

• Swarm will leverage all hardware resources the node (server where it runs) provides: CPU, RAM, network, and any direct-attached disk drives.

• Minimum recommended storage nodes count: Four (4).

Platform Server - Swarm Cluster Services (SCS)

• The SCS software provides Swarm cluster configuration and boot services as well as log aggregation and Swarm version management.

• The SCS is not in the data path, but it does require access to the same layer 2 network as the Swarm storage nodes.

• Minimum recommended SCS count is one.

Elasticsearch

• Provides listing and search capabilities based on object name and object metadata.

• Minimum recommended Elasticsearch VM count for production environments is three.

• For functional Proof-of-Concepts, one instance should suffice.

Content Gateway

• The Content Gateway provides S3 and HTTP access as well as a Content Portal (web interface) that users and administrators can leverage to create buckets, upload data, use collections to perform searches (based on metadata), and many more. Hence, the Content Gateway is in the data path.

• Content Gateway also enforces multitenancy features such as user authentication against LDAP, Active Directory or Single-Sign-on (SAML), permissions, quotas, and so on.

• Minimum recommended Content Gateway count for production environments is two.

• For functional Proof-of-Concepts, one instance should suffice.

Telemetry (Optional)

• Prometheus integration and Grafana dashboards.

• Minimum recommended Telemetry count is often one, but there could be as many as needed.

Load Balancers (Optional)

• To balance the client load across all the Content Gateway instances, an HTTP Load Balancer in front of the Content Gateways can be leveraged. This load balancer can be a software solution such as HAProxy, NGINX, or others. Also, it could be a hardware-based, appliance one.

Networking Requirements and Recommendations

Swarm utilizes a dual networking configuration, where there is a Storage (Backend) network and a Service (Frontend) one. As per the diagram above, the Swarm storage nodes are only connected to the Backend network, while the management and access layer components have presence in both (dual-homed). Hence, this Backend/storage network must be configured in VMware ESXi as well.

The Backend network could be just a VLAN in the existent switching environment. However, this VLAN/network has to be dedicated exclusively to Swarm and it is usually isolated from the rest of the network environment. At any rate, no other system outside the Swarm stack should be connected to it.

The switch ports used by the Swarm storage nodes must be in access mode, as the Swarm nodes cannot tag VLAN traffic. Also, ‘port fast’ should be enabled to facilitate the PXE boot process (see below).

Open Ports Overview 

Environment Prerequisites

The following table illustrates the requirements for a typical Swarm deployment.

Optionally, the end-user organization should generate a valid SSL certificate to enable HTTPS access.

Site Survey

To configure the Swarm cluster, the following information is required:

Swarm Deployment Using VMware Bundle

The VM bundled is comprised of OVF packages to be deployed in VMware ESXi 7.0U2 and above. The operating system and the Swarm software are both pre-installed. They are based on Rocky Linux 8.9. 

The pre-configured Backend network/VLAN range is 172.29.0.0/16, but it can be changed as desired. Make sure the selected range is not in use by another environment.

The default credentials are: 

• SSH and console access: root - datacore 

• Web UIs: admin - datacore  

These are the templates included in the VM bundle Swarm-16.1-ESX-8.0-RL8: 

• SCS - PXE-boot the Swarm storage nodes, support tools 

  • Template: SwarmClusterServices.ovf  

  • Associated disks: datacore-swarm-16.1.0-ESX-RL8-disk1.vmdk, datacore-swarm-16.1.0-ESX-RL8-disk2.vmdk 

• Swarmsearch (Elasticsearch) - Indexer and search engine 

  • Template: SwarmSearch1.ovf  

  • Associated disks: datacore-swarm-16.1.0-ESX-RL8-disk3.vmdk, datacore-swarm-16.1.0-ESX-RL8-disk4.vmdk 

• Content Gateway - S3 access, Content Portal 

  • Template: SwarmContentGateway.ovf  

  • Associated disks: datacore-swarm-16.1.0-ESX-RL8-disk5.vmdk 

• Telemetry (optional component) - Grafana dashboards 

  • Template: SwarmTelemetry.ovf 

  • Associated disks: datacore-swarm-16.1.0-ESX-RL8-disk6.vmdk, datacore-swarm-16.1.0-ESX-RL8-disk7.vmdk

The bundle also includes an OVF template that will deploy all VMs as a vAPP:  

datacore-swarm-16.1.0-ESX-RL8.ovf  

Platform Server – Swarm Cluster Services (SCS)

Preparation Steps

1. Deploy SCS VM (SwarmClusterServices.ovf) and its associated virtual disks (vmdk).

2. Change the IP configuration and verify the connection information for the frontend network.

   nmcli con mod ens192 ipv4.addresses <SCS_FRONTEND_IP>/<FRONTEND_NETWORK> 
   nmcli con mod ens192 ipv4.gateway <FRONTEND_GATEWAY> 
   nmcli con mod ens192 ipv4.dns <DNS_SERVER_1>,<DNS_SERVER_2> 
   nmcli con mod ens192 ipv4.dns-search <DNS_DOMAIN> 
    
   nmcli con mod ens192 ipv4.method manual 
   nmcli con mod ens192 connection.autoconnect yes 
   
   nmcli con reload 
   nmcli con down ens192 
   nmcli con up ens192 
   
   nmcli device show ens192

3. Change the IP configuration and verify the connection information for the backend network. 

   nmcli con mod ens224 ipv4.addresses <SCS_BACKEND_IP>/<BACKEND_NETWORK> 
   nmcli con mod ens224 ipv4.method manual 
   nmcli con mod ens224 connection.autoconnect yes 
   
   nmcli con reload 
   nmcli con down ens224 
   nmcli con up ens224 
    
   nmcli device show ens224 

4. The network configuration can be verified with the command: ip a or nmcli con show.

Offline Installation

For offline installation (i.e., when no Internet access is available).  

1. Edit /etc/hosts comment out the http://k8s.gcr.io entry and the docker-repo.tx.caringo.com one.
   The first line should read: 

   <SCS_FRONTEND_IP> www.datacore.com

2. Set the time zone according to your local clock. 
   timedatectl set-timezone <timezone> 
   hwclock --systohc 

3. Configure chrony (NTP daemon) to connect to a valid NTP server. 

4. Edit the file /etc/chrony.conf and add the proper IP addresses or names of those NTP servers. Remove the lines referred to as the default ones if they are not reachable. 

   server <NTP_SERVER_1> iburst 
   server <NTP_SERVER_2> iburst 
   
   allow <SCS_BACKEND_NETWORK/BACKEND_NETWORK>  

   The following is an example of the allow line: allow 192.168.90.0/24 

5. Restart chrony daemon: systemctl restart chronyd  

6. Verify the clock is in sync with: chronyc tracking

SCS Configuration

Once the auxiliary services of SCS are configured, the SCS setup can take place.

1. Run the configurator wizard:
   scsctl init wizard -a 

2. Running step [1/37]: Set site name.
   Type the <CLUSTER_NAME> 

3. Running step [2/37]: Set the administrative password.
   Type the admin password for the cluster
   Re-enter to confirm 

4. Running step [4/37]: Choose the Swarm-internal interface.
   Specify the network interface that will be used for internal Swarm operations:
      lo
      ens192
   > ens224

5. Select ens224, press Enter.

6. Running step [5/37]: Choose external (client-facing) interface.
   Specify the network interface that will be used for operations OUTSIDE of swarm:
      lo
    > ens192
   ens224

7. Select ens192, press Enter.

8. Running step [7/37]: Define Swarm-internal network.
   The internal interface requires a *static* IP address to be defined on it!
   It looks like your internal interface is already configured with an IP address: x.x.x.x/yz
   Do you wish to continue to use this address and netmask? [Y/n]:

9. Reply Y and hit Enter 

   The provisioning process will commence, and it takes a few minutes to complete. 

   Continue the configuration process running:
   scsctl diagnostics config scan_missing

10. Missing setting: network_boot/network.dnsServers
    Update this setting as a default at group level.

11. Press Enter and type the IP addresses of the DNS servers below separated with a blank space:
    network.dnsServers [type: array[str]] (Required: DNS servers to be used):
    <DNS_SERVER_1> <DNS_SERVER_2> 

12. Missing setting: platform/network.dnsDomain
    Update this setting as a default at group level

13. Hit Enter and type the DNS domain used
    network.dnsDomain [type: str] (Required: The DNS domain name that will be used.): 
    <DNS_DOMAIN> 

14. Fix the rsyslog template
    Run: cp -f /root/swarm-platform-fix.conf /etc/rsyslog.d/swarm-platform.conf

15. In the next step, the Swarm image will be added and configured. 

    Run: scsctl repo component add -f /root/swarm-scs-storage-16.1.0.tgz 

    During this process, the feature “encryption at rest” (EAR) can be configured.

Optional 

This is an optional functionality that will encrypt the data when it hits the plate of the disks. It comes at a cost of usually 15-20% performance penalty as the nodes need to use processing power to encrypt/decrypt data. 

This guide assumes EAR will be configured. If it is not a requirement, select False on the next step when the wizard asks about disk.encryptNewVolumes configuration. 

Also, the configuration steps will ask about multicast traffic, and whether it will be allowed. As it is a best practice to keep it enabled, this guide will follow that.

1. Missing setting: storage/disk.encryptNewVolumes
   Update this setting as a component-wide default

2. Hit Enter
   disk.encryptNewVolumes [type: bool] (Whether to encrypt new Swarm volumes. Enabling encryptNewVolumes means that any newly-formatted Swarm volume will be encrypted)
   True
   False 

3. Missing setting: storage/cip.multicastEnabled
   Update this setting as a component-wide default

4. Press Enter.
   cip.multicastEnabled [type: bool] (Whether multicast should be used for communication within Swarm.)
   > True 
   Press Enter
   Finally, the configuration wizard asks for what drives will be used to store data, this guide assumes “all” drives will be used as the server should be dedicated exclusively to Swarm. 

5. Missing setting: storage/disk.volumes
   Update this setting as a component-wide default

6. Press Enter
   disk.volumes [type: str] (Required: Specifies the volume storage devices for Swarm to use)
   all 

7. At this stage, the Swarm image is added. The configuration wizard will ask about the cluster name and a description.
   added: storage - 16.1.0 (16.1.0)
   Enter a name for the group (FQDN format encouraged):
   <CLUSTER_NAME>
   Enter a description for the group (purpose, etc.). [OPTIONAL]:

SCS needs to know what IP range can be used to PXE boot the Swarm storage nodes in the backend network. To avoid collisions with other Swarm services, we can reserve several IP addresses at the beginning and/or the end of the range, so SCS will not assign those IP addresses to the nodes. To do this,

1. Run the below command:

   scsctl init dhcp --dhcp-reserve-lower=50 --dhcp-reserve-upper=10

Adjusting the values to whatever makes sense in the backend network. For example, In a /24 network, the above will use from .51 to .244 to PXE boot and assign IP addresses to the Swarm storage nodes. 

2. If the physical servers have SSD/NVMe or smaller drives that are not required for Swarm, can be excluded by running:

   scsctl storage config set -d "disk.minGB=4096"

As an example, the above command will exclude any drive that is smaller than 4TB.

3. Unzip and add the license key. This key should be a plain text file:

   scsctl license add -f license.txt 

4. Override the swarm-platform.conf config file:

cp /root/swarm-platform-fix.conf /etc/rsyslog.d/swarm-platform.conf

5. It is recommended to enable Swarm node stats for the Telemetry VM (Prometheus/Grafana). To do this, run:

scsctl storage config set -d "metrics.enableNodeExporter=true" 
scsctl storage config set -d "metrics.nodeExporterFrequency=120" 

6. If the Swarm storage nodes use an Intel Skylake based CPU or similar, run the following:

scsctl network_boot config set kernel.extraArgs=clocksource.max_cswd_read_retries=50 -d  
systemctl restart swarm-platform 

For more information, see Intel Skylake/Cascade Lake CPU Performance Issue 

7. Finally, create a backup of the SCS configuration. Run:

scsctl backup -o backup-config-<date> 

At this point, SCS has been configured and it is ready to PXE boot Swarm storage nodes.

Elasticsearch

Preparation Steps

1. Deploy SwarmSearch VM (SwarmSearch.ovf) and its associated virtual disks (vmdk). Please note the virtual network interfaces are inverted. The first one corresponds to the backend network, while the second interface is connected to the frontend network. 
   The preconfigured IP address for the backend network is 172.29.1.20/16. 

   Below are the steps to change it, if required:

2. Change the IP configuration and verify the connection information for the backend network. 

   nmcli con mod ens192 ipv4.addresses <ES_BACKEND_IP>/<BACKEND_NETWORK> 
   nmcli con mod ens192 ipv4.gateway <SCS_BACKEND_IP>
   nmcli con reload 
   nmcli con down ens192 
   nmcli con up ens192 
   nmcli device show ens192

The network configuration can be verified with the command: ip a or nmcli con show.

3. Set the time zone according to your local clock:

   timedatectl set-timezone <timezone>
   hwclock --systohc 

4. disable swapping by executing the following command:

   echo "vm.swappiness=1" > /etc/sysctl.d/51-swappiness.conf
   sysctl -w vm.swappiness=1

5. Edit the file /etc/chrony.conf and add the proper IP addresses or names of those NTP servers. Remove the lines referred to the default ones if they are not reachable.

   server <NTP_SERVER_1> iburst 
   server <NTP_SERVER_2> iburst 

   Ensure that the NTP servers are reachable.

6. Restart chrony daemon.

   systemctl restart chronyd 

7. Verify the clock is in sync.

   chronyc tracking

8. Edit /etc/elasticsearch/elasticsearch.yml and replace 172.29.1.20 with the IP address configured in the previous step for this VM in the following sections of the file: 

   network.host: <ES_BACKEND_IP> 
   discovery.seed_hosts: ["<ES_BACKEND_IP>"] 
   cluster.initial_master_nodes: ["<ES_BACKEND_IP>"

9. Restart the service.

   systemctl restart elasticsearch 

10. Verify it is up and running.

    curl -XGET "http://<ES_BACKEND_IP>:9200/_cat/health?v"

    The response should be "green" or "yellow". 

nmcli con modify ens224 ipv4.addresses <ES_FRONTEND_IP>/<FRONTEND_NETWORK> 
nmcli con modify ens224 ipv4.gateway <FRONTEND_GATEWAY> 
nmcli con modify ens224 ipv4.dns <DNS_SERVER_1>,<DNS_SERVER_2> 
nmcli con modify ens224 ipv4.dns-search <DNS_DOMAIN> 
nmcli con reload 
nmcli con down ens224 
nmcli con up ens224 
nmcli device show ens224 

With the above steps, only one Elasticsearch VM will be provisioned. The status will appear in “Yellow” the moment there is any data in Elasticsearch as there is no redundancy.

This configuration is enough for Proof-of-Concept or Proof-of-Value scenarios. However, for production environments, the recommendation is to have at least three Elasticsearch VMs up and running forming a cluster by themselves. 

The steps to deploy a full Elasticsearch cluster are explained below:

1. Deploy the SwarmSearch1.ovf template two more times.

2. Update the static IP address for the backend adapter ens192.

3. Update /etc/hostname of the two new VMs, e.g., “swarmsearch2”, “swarmsearch3”.

4. Stop the elasticsearch service and delete the pre-generated data:

   systemctl stop elasticsearch 
   cd /var/lib/elasticsearch 
   rm -rf nodes 

5. Delete the predefined elasticsearch.yml config file and run the configurator wizard:  

   rm /etc/elasticsearch/elasticsearch.yml
   /usr/share/caringo-elasticsearch-search/bin/configure_elasticsearch_with_swarm_search.py

6. Enter Elasticsearch cluster name [A string]: swarmsearch

7. Enter List of all the Elasticsearch server names in the cluster [Comma-separated list of DNS-resolvable names or IP addresses]: <ES_NODE1_BACKEND_IP>,<ES_NODE2_BACKEND_IP>,<ES_NODE3_BACKEND_IP> 

8. Enter this Elasticsearch node's name [A string name from the list entered above]: <ES_NODE_NAME> 

Repeat these steps for every Elasticsearch VM, including the original (the first was deployed) one. For more information, see Configuring Elasticsearch  

Once all Elasticsearch VM’s is configured, restart the elasticsearch service in all of them:

systemctl restart elasticsearch 

Finally to check the health of the Elasticsearch cluster, run:

curl -XGET "http://<ES_BACKEND_IP>:9200/_cat/health?v" 

Three nodes should appear under the “node.total” column, and the status should be “green”.

Swarm Storage Nodes

1. Before starting the PXE boot process, enter the BIOS of each server that will be the Swarm storage node and check: 

   1. The HBA/Disk controller is configured in passthrough mode. Essentially, this is a non-RAID configuration where all the disk drives are presented to the operating system individually. It is also called IT mode, HBA mode, pass-thru, or non-RAID. 

   2. The network card port connected to the Backend VLAN/network must be enabled for PXE booting, no other port should be PXE-boot enabled. Moreover, there should not be any other port connected to any other network, with the exception of the dedicated port for out of band management (OOB, IPMI, BMC…) 

2. Once these are verified, the PXE boot process can begin.

3. Start with a single node, making sure it boots properly. 

4. Continue with the rest. A successful Swarm storage node boot looks like this on the screen / IPMI console of the server: 

   

5. Swarm version, IP address of the node, and “Storage Processes: RUNNING” should appear on the screen.

Content Gateway

The final step to have everything needed to have a functional Swarm cluster is to get Content Gateway up and running. 

1. Deploy SwarmContentGateway.ovf. The IP addresses must be configured next.

2. Change the IP configuration and verify the connection information for the frontend network. 

   nmcli con mod ens192 ipv4.addresses <GW_FRONTEND_IP>/<FRONTEND_NETWORK> 
   nmcli con mod ens192 ipv4.gateway <FRONTEND_GATEWAY> 
   nmcli con mod ens192 ipv4.dns <DNS_SERVER_1>,<DNS_SERVER_2> 
   nmcli con mod ens192 ipv4.dns-search <DNS_DOMAIN> 
   nmcli con mod ens192 ipv4.method manual 
   nmcli con mod ens192 connection.autoconnect yes 
   
   nmcli con reload 
   nmcli con down ens192 
   nmcli con up ens192
   
   nmcli device show ens192

3. Change the IP configuration and verify the connection information for the backend network. 

   nmcli con mod ens224 ipv4.addresses <GW_BACKEND_IP>/<BACKEND_NETWORK> 
   nmcli con mod ens224 ipv4.method manual 
   nmcli con mod ens224 connection.autoconnect yes 
   nmcli con reload 
   nmcli con down ens224 
   nmcli con up ens224 
    
   nmcli device show ens224

4. The network configuration can be verified with the command: ip a or nmcli con show 

5. Set the time zone according to your local clock. 

   timedatectl set-timezone <timezone> 
   hwclock --systohc

7. Configure chrony (NTP daemon) to connect to a valid NTP server. 

8. Edit the file /etc/chrony.conf and add the proper IP addresses or names of those NTP servers. Remove the lines referred to as the default ones. 

   server <NTP_SERVER_1> iburst 
   server <NTP_SERVER_2> iburst

9. Restart the chrony daemon.

   systemctl restart chronyd  

10. Verify the clock is in sync.

    chronyc tracking 

The Content Gateway configuration comes next. 

To proceed with the Content Gateway configuration:

1. Edit /etc/caringo/cloudgateway/gateway.cfg

   adminDomain = admin.<CLUSTER_NAME>
   hosts = <SWARM_NODE1_IP> < SWARM_NODE2_IP > <SWARM_NODE3_IP> < SWARM_NODE4_IP>
   indexerHosts = <ES_BACKEND_IP> 
   
   managementPassword = <CLUSTER_MGMT_PASSWORD>

2. Metering and quotas can be enabled. This is optional.
   For more information, see Content Metering and Setting Quotas

   [metering]
   enabled = true

   By default, metering is false.

   [quota]
   enabled = true

   By default, the quota is false.

3. Run the below command.

   /opt/caringo/cloudgateway/bin/initgateway 
   systemctl enable cloudgateway
   systemctl start cloudgateway
   systemctl status cloudgateway

Content Gateway should be up and running now.

As the final step, let’s configure the desired default protection scheme and connect Swarm to Elasticsearch.

1. Open a web browser and go to: http://<GW_FRONTEND_IP>:91/_admin/storage

2. Click Storage Management.

3. Click Cluster and then on Feeds. 

4. On the top right corner, click +Add and select Search Metadata feed. 

5. On server host(s) or IP(s), type all the IP addresses of all the Elasticsearch VMs that are up and running separated by a blank space. <ES_BACKEND_IP>

6. Click Save. Now the Swarm nodes are connected to Elasticsearch, every time a new object/file gets uploaded to the cluster, its metadata will be also copied to Elasticsearch for search and listing purposes. 

7. Due to a bug in Gateway 8.0.4 you will need to restart cloudgateway for it to see the new search feed.

   systemctl restart cloudgateway

8. To finalize the setup, the default protection scheme should be set. Also, features like lifecycle policies and versioning can be enabled, if desired.
   For more information about these features, see Object Versioning and Bucket Lifecycle Policy.

9. Versioning is required to enable “S3 object locking” (immutability).
   For more information, see SCSP Object Locking.

10. Click Settings and select Cluster.
    In the “Policy” section, change the protection scheme as desired, for example with 4 Swarm storage nodes:

    policy.eCEncoding 4:2 
    policy.eCMinStreamSize 1Mb 
    policy.lifecycle enabled 
    policy.replicas min:3 max:16 default:3 
    policy.versioning allowed 

11. Click Save at the top right corner.

12. Finally, test uploads and downloads using the provided Content Portal.

13. Open a web browser and go to http://<GW_FRONTEND_IP>/_admin/portal  

14. Click System Tenant at the upper right corner and click +Add.

To create a bucket:

1. Click the domain that you just created.

2. Click +Add this time selecting “Bucket”. Provide a name such as “bucket1” or “test1”.

3. Click the bucket you just created and click +Add or drop files.

4. Look for some files of various sizes on the client machine used, from KBs to MBs and upload them.

5. Click the bucket name at the top. 

Swarm utilizes FQDNs to identify which storage domain (endpoint) the client is connecting to. Hence, create DNS entries according to the Storage Domains used in the environment. 

At this point Swarm is up and running and its basic functionality has been verified.

Create an S3 Key Pair (Optional)

To access the storage layer using the S3 protocol, a S3 key pair must be created.  

It is comprised by the S3 access key and the S3 secret key. 

1. Open a web browser and go to http://<GW_FRONTEND_IP>/_admin/portal

2. Click the domain (endpoint) desired, but not on the admin one.

3. Click the cog/wheel in the top right corner and select Tokens. There will be an +Add button again in the top right corner.

4. Provide a description, an expiration date, and click the checkmark by “S3 secret key”.

5. Upon clicking on Add, a green message will appear with all the information needed.

6. With this information and the name of the domain used, it is possible to create a connection to the Swarm repository over the S3 protocol.

Configuring an SSL Certificate (Optional)

1. By default, the Content Gateway VM template comes with HAProxy unconfigured and no self-signed certificate.

2. If you wish to configure HAProxy as an SSL offloader, follow the steps outlined in Configuring haproxy SSL offloading with a Self Signed Certificate on CentOS7/8

Central Logging (Optional)

It is recommended that the Content Gateway logs all actions and their status to the central Syslog server. The SCS can act as the central repository for logs.

To configure this, edit /etc/caringo/cloudgateway/logging.yaml and modify the following lines:

Syslog: 
   - name: audit_syslog 
     host: <SCS_BACKEND_IP> 

It is localhost by default.

   - name: server_syslog 
     host: <SCS_BACKEND_IP> 

Loggers: 

# Global logging configuration 

Root: 
level: "${logLevel}" 
AppenderRef: 
  - ref: file 
  - ref: server_syslog    

Logger: 
   # Audit logger 
   - name: audit 
     level: info 
     additivity: false 
     AppenderRef: 
       - ref: audit 
       - ref: audit_syslog  

There is no need to restart the Content Gateway service; the new logging configuration will be applied automatically after a few seconds.

Telemetry (Optional)

The Telemetry VM provides an all-in-one reference implementation of Prometheus, Alertmanager, and Grafana.

Preparation Steps

1. Deploy SwarmTelemetry.ovf. IP addresses must be configured next.

2. Change the IP configuration and verify the connection information for the frontend network. 

   nmcli con mod ens192 ipv4.addresses <TM_FRONTEND_IP>/<FRONTEND_NETWORK> 
   nmcli con mod ens192 ipv4.gateway <FRONTEND_GATEWAY> 
   nmcli con mod ens192 ipv4.dns <DNS_SERVER_1>,<DNS_SERVER_2> 
   nmcli con mod ens192 ipv4.dns-search <DNS_DOMAIN> 
   nmcli con mod ens192 ipv4.method manual 
   nmcli con mod ens192 connection.autoconnect yes 
   nmcli con reload 
   nmcli con down ens192 
   nmcli con up ens192 
    
   nmcli device show ens192

3. Change the IP configuration and verify the connection information for the backend network. 

   nmcli con mod ens224 ipv4.addresses <TM_BACKEND_IP>/<BACKEND_NETWORK> 
   nmcli con mod ens192 ipv4.gateway <SCS_BACKEND_IP>
   nmcli con reload 
   nmcli con down ens224 
   nmcli con up ens224 
    
   nmcli device show ens224

4. The network configuration can be verified with the command: ip a or nmcli con show

5. Set the time zone according to the local clock.

   timedatectl set-timezone <timezone>
   hwclock --systohc

6. Configure chrony (NTP daemon) to connect to a valid NTP server. Edit the file /etc/chrony.conf and add the proper IP addresses or names of those NTP servers. Remove the lines referred to as the default ones. 

   server <NTP_SERVER_1> iburst 
   server <NTP_SERVER_2> iburst

7. Restart the chrony daemon.

   systemctl restart chronyd

8. Verify the clock is in sync.

   chronyc tracking

Prometheus Master Configuration

The next step is to configure Prometheus.

1. Edit /etc/prometheus/prometheus.yml to include all the IP addresses of all the Swarm components to be monitored, uncomment lines as needed:

   # THIS IS THE ELASTICSEARCH EXPORTER DEFINITION 
     # IP ADDRESS SHOULD BE Telemetry loopback 
   
     - job_name: 'elasticsearch' 
       scrape_interval: 30s 
       static_configs: 
       - targets: ['127.0.0.1:9114'] 
       relabel_configs: 
       - source_labels: [__address__] 
         regex: "([^:]+):\\d+" 
         target_label: instance 
   
     # THIS IS THE CLOUD CONTENT GATEWAY JOB DEFINITION 
     # IP ADDRESS SHOULD BE CLOUD GATEWAY STORAGE VLAN IP 
   
     - job_name: 'swarmcontentgateway' 
       static_configs: 
       - targets: ['<GW_BACKEND_IP>:9100'] 
       relabel_configs: 
       - source_labels: [__address__] 
         regex: "([^:]+):\\d+" 
         target_label: instance 
   
     # THIS IS THE CLOUD GATEWAY NODE_EXPORTER JOB DEFINITION 
     # IP ADDRESS SHOULD BE CLOUD GATEWAY STORAGE VLAN IP 
    
     - job_name: 'gateway-nodeexporter' 
       scrape_interval: 30s 
       static_configs: 
       - targets: ['<GW_BACKEND_IP>:9095'] 
       relabel_configs: 
       - source_labels: [__address__] 
         regex: "([^:]+):\\d+" 
         target_label: instance 
   
     # THIS IS THE SWARM JOB DEFINITON 
     # IP ADDRESS SHOULD BE STORAGE VLAN IP 
   
     - job_name: 'swarm' 
       scrape_interval: 30s 
       static_configs: 
       - targets: ['<SWARM_NODE1_IP>:9100','<SWARM_NODE2_IP>:9100','<SWARM_NODE3_IP>:9100','<SWARM_NODE4_IP>:9100'] 
       relabel_configs: 
       - source_labels: [__address__] 
         regex: "([^:]+):\\d+" 
         target_label: instance 

YAML (.yml) files are quite sensitive to spaces and indentation. The following command will check that there is no errors.  

promtool check config /etc/prometheus/prometheus.yml

Elasticsearch Node Exporter

1. To gather statistics and status about Elasticsearch edit: /usr/lib/systemd/system/elasticsearch_exporter.service updating the IP address of the (first) Elasticsearch VM (instead of the pre-configured 172.29.1.20). 

   ExecStart = /usr/local/bin/elasticsearch_exporter --es.all --
   es.cluster_settings --es.indices --es.indices_settings --
   es.indices_mappings --es.shards --es.snapshots --es.uri 
   http://<ES_BACKEND_IP>:9200 --es.timeout 20s --web.listen-address :9114

2. Run the below commands to enable and start the service.

   systemctl daemon-reload 
   systemctl enable elasticsearch_exporter 
   systemctl start elasticsearch_exporter 

3. Once the Prometheus master config changes are applied, the service can be enabled and started.

   systemctl enable prometheus 
   systemctl restart prometheus 

4. To verify that Prometheus is up and running, open a web browser and go to: http://<TM_FRONTEND_IP>:9090/targets  

This page shows which targets it is currently collecting metrics for and if they are reachable. Click Status and select “Targets”. It will take a few minutes to be updated. All states should appear as “UP”.

Alertmanager Configuration

There are four (4) alerts defined in /etc/prometheus/alert.rules.yml

• Service_down: Triggered if any swarm storage node is down for more than 30 minutes.

• Gateway_down: Triggered if the cloudgateway service is down for more than 2 minutes.

• Elasticsearch_cluster_state: Triggered if the cluster state changed to "red" after 5 minutes.

• Swarm_volume_missing: Triggered if the reported drive count is decreasing over 10 minutes. This is due to a failed disk drive that needs to be replaced.

The /etc/prometheus/prometheus.yml contains a section that points to the alertmanager service on port 9093, as well as which alert.rules.yml file to use.

To customize the alerts:

1. Modify the swarmUI template in /etc/prometheus/alertmanager/template/basic-email.tmpl. This will be used for the email HTML template showing a button to the chosen URL.

   {{ define "__swarmuiURL" }}http://<GW_FRONTEND_IP>:91/_admin/storage/{{ end }} 

2. The configuration for where to send alerts is defined in the file: /etc/prometheus/alertmanager/alertmanager.yml  

3. By default, the route is disabled as it requires manual input for every specific environment, values such as:
   SMTP server, username, password (if applicable), etc.

- name: 'emailchannel'  
  email_configs: 
  - to: admin@acme.com  
    from: swarmtelemetry@acme.com  
    smarthost: smtp.acme.com:25  
    require_tls: false  
    send_resolved: true  

6. Once the configuration is complete, restart the alertmanager.

   systemctl restart alertmanager  

7. To verify the alertmanager.yml has the correct syntax, run:  

   amtool check-config /etc/prometheus/alertmanager/alertmanager.yml  

It should give the following output: 

Checking '/etc/prometheus/alertmanager/alertmanager.yml' SUCCESS  

Found:  

• global config  

• route  

• 1 inhibit rules  

• 2 receivers  

• 1 templates SUCCESS 

Grafana Configuration

The password for the “admin” user can be changed on the configuration file /etc/grafana/grafana.ini, look for admin_password.

For more information, see Documentation | Grafana Labs.

To enable on boot and start the service type:

systemctl enable grafana-server 
systemctl restart grafana-server

Grafana has all the Swarm dashboards pre-installed. Open a web browser and go to http://<TM_FRONTEND_IP>

The default period is 7 days, modify it to 5 minutes to see some stats appearing on the charts.

The latest Swarm dashboards are available on the Grafana website.

Job Name (Optional)

In /etc/prometheus/prometheus.yml the job_name of the Content Gateway can be defined. This job_name will be displayed on the Content Gateway Grafana dashboard.

If the Content Gateway job_name is changed there are a couple of additional changes required:

Modify the gateway job name in /etc/prometheus/alertmanager/alertmanager.yml; it must match what appears in prometheus.yml.

routes:  
- match:  
      job: <new_job_name>

Modify the gateway job name in /etc/prometheus/alert.rules.yml.

alert: gateway_down  
expr: up{job="<new_job_name>"} == 0  

DNS names can be used. In the absence of a DNS server, first, modify /etc/hosts file with the desired names for each Swarm storage node and then use those in the configuration file. This is recommended in scenarios where the dashboards are publicly accessible.

Prometheus Retention Time (Optional)

By default, the Prometheus configuration in Telemetry keeps metrics for 30 days. If there is a need to increase or decrease this retention, follow the next steps: 

1. Edit the /root/prometheus.service file. 

2. Select the default retention time for the collected metrics. 

3. Modify the --storage.tsdb.retention.time=30d flag to the new desired retention time.

4. Finally, commit the change:

cp /root/prometheus.service /usr/lib/systemd/system  
systemctl daemon-reload  
promtool check config /etc/prometheus/prometheus.yml  
systemctl restart Prometheus

Prometheus Security (Optional)

It may be desirable to restrict Prometheus server to only allow queries from the local host, since Grafana server is running on the same VM. This can be done by editing /root/prometheus.service file and adding the flag --web.listen-address=127.0.0.1:9090  

If Prometheus is bound only to localhost, the built-in Prometheus UI on port 9090 will not be accessible remotely. 

Planning and Storage Nodes Prerequisites

Deployment Planning

Hardware Requirements for Storage

SCS

Swarm Cluster Services (SCS) Implementation

Elasticsearch

Hardware Requirements for Elasticsearch

Preparing the Search Cluster

Installing Elasticsearch

Configuring Elasticsearch

Setup Elasticsearch Cluster

Managing Feeds

Content Gateway

Gateway Requirements

Gateway Installation

Gateway Configuration

Gateway Verification

Telemetry (Prometheus and Grafana)

Prometheus Node Exporter and Grafana