Introduction and Prerequisites

This document guides to effectively deploy a Swarm cluster using the OVF VM bundle package – Rocky Linux 8 edition. 

Both the OVF bundle package and the standalone software are available in the DataCore downloads website.

There are two main sections in this document:

• Swarm deployment using the OVF VM bundle package.

• Deploying Swarm from scratch.

The present document is based on a traditional deployment of Swarm, where the management and access layer run virtualized in one or more VMware ESXi hosts, while the storage nodes are physical x86-64 machines that will hold the data. See the below diagram.

Swarm Components

The Swarm stack utilizes several components grouped in two different layers:

• Storage Layer: Comprised by the Swarm storage nodes which hold the information and take care of data protection.

• Management and Access Layer: As the name implies, this layer provides both the administration of the Swarm cluster as well as access to the storage for users and client applications. No data storage or caching is happening in this layer.  

Below are the software components of the entire Swarm stack, their functions, and count recommendations for durability and availability purposes:

Swarm Storage Nodes

• Swarm is an in-purpose built on-premises object storage solution. It runs on standard physical x86-64 servers providing a single pool of resources, supporting billions of objects/files in the same cluster and extending its capabilities to multiple sites (data replication).

• Swarm will leverage all hardware resources the node (server where it runs) provides: CPU, RAM, network, and any direct-attached disk drives.

• Minimum recommended storage nodes count: Four (4).

Platform Server - Swarm Cluster Services (SCS)

• The SCS software provides Swarm cluster configuration and boot services as well as log aggregation and Swarm version management.

• The SCS is not in the data path, but it does require access to the same layer 2 network as the Swarm storage nodes.

• Minimum recommended SCS count is one.

Elasticsearch

• Provides listing and search capabilities based on object name and object metadata.

• Minimum recommended Elasticsearch VM count for production environments is three.

• For functional Proof-of-Concepts, one instance should suffice.

Content Gateway

• The Content Gateway provides S3 and HTTP access as well as a Content Portal (web interface) that users and administrators can leverage to create buckets, upload data, use collections to perform searches (based on metadata), and many more. Hence, the Content Gateway is in the data path.

• Content Gateway also enforces multitenancy features such as user authentication against LDAP, Active Directory or Single-Sign-on (SAML), permissions, quotas, and so on.

• Minimum recommended Content Gateway count for production environments is two.

• For functional Proof-of-Concepts, one instance should suffice.

Telemetry (Optional)

• Prometheus integration and Grafana dashboards.

• Minimum recommended Telemetry count is often one, but there could be as many as needed.

Load Balancers (Optional)

• To balance the client load across all the Content Gateway instances, an HTTP Load Balancer in front of the Content Gateways can be leveraged. This load balancer can be a software solution such as HAProxy, NGINX, or others. Also, it could be a hardware-based, appliance one.

Networking Requirements and Recommendations

Swarm utilizes a dual networking configuration, where there is a Storage (Backend) network and a Service (Frontend) one. As per the diagram above, the Swarm storage nodes are only connected to the Backend network, while the management and access layer components have presence in both (dual-homed). Hence, this Backend/storage network must be configured in VMware ESXi as well.

The Backend network could be just a VLAN in the existent switching environment. However, this VLAN/network has to be dedicated exclusively to Swarm and it is usually isolated from the rest of the network environment. At any rate, no other system outside the Swarm stack should be connected to it.

The switch ports used by the Swarm storage nodes must be in access mode, as the Swarm nodes cannot tag VLAN traffic. Also, ‘port fast’ should be enabled to facilitate the PXE boot process (see below).

Open Ports Overview 

Environment Prerequisites

The following table illustrates the requirements for a typical Swarm deployment.

Optionally, the end-user organization should generate a valid SSL certificate to enable HTTPS access.

Site Survey

To configure the Swarm cluster, the following information is required:

Swarm Deployment Using VMware Bundle

The VM bundled is comprised of OVF packages to be deployed in VMware ESXi 7.0U2 and above. The operating system and the Swarm software are both pre-installed. They are based on Rocky Linux 8.9. 

The pre-configured Backend network/VLAN range is 172.29.0.0/16, but it can be changed as desired. Make sure the selected range is not in use by another environment.

The default credentials are: 

• SSH and console access: root - datacore 

• Web UIs: admin - datacore  

These are the templates included in the VM bundle Swarm-16.1-ESX-8.0-RL8: 

• SCS - PXE-boot the Swarm storage nodes, support tools 

  • Template: SwarmClusterServices.ovf  

  • Associated disks: datacore-swarm-16.1.0-ESX-RL8-disk1.vmdk, datacore-swarm-16.1.0-ESX-RL8-disk2.vmdk 

• Swarmsearch (Elasticsearch) - Indexer and search engine 

  • Template: SwarmSearch1.ovf  

  • Associated disks: datacore-swarm-16.1.0-ESX-RL8-disk3.vmdk, datacore-swarm-16.1.0-ESX-RL8-disk4.vmdk 

• Content Gateway - S3 access, Content Portal 

  • Template: SwarmContentGateway.ovf  

  • Associated disks: datacore-swarm-16.1.0-ESX-RL8-disk5.vmdk 

• Telemetry (optional component) - Grafana dashboards 

  • Template: SwarmTelemetry.ovf 

  • Associated disks: datacore-swarm-16.1.0-ESX-RL8-disk6.vmdk, datacore-swarm-16.1.0-ESX-RL8-disk7.vmdk

The bundle also includes an OVF template that will deploy all VMs as a vAPP:  

datacore-swarm-16.1.0-ESX-RL8.ovf  

Platform Server – Swarm Cluster Services (SCS)

Preparation Steps

1. Deploy SCS VM (SwarmClusterServices.ovf) and its associated virtual disks (vmdk).

2. Change the IP configuration and verify the connection information for the frontend network.

   Code Block

   nmcli con mod ens192 ipv4.addresses <SCS_FRONTEND_IP>/<FRONTEND_NETWORK> nmcli con mod ens192 ipv4.gateway <FRONTEND_GATEWAY> nmcli con mod ens192 ipv4.dns <DNS_SERVER_1>,<DNS_SERVER_2> nmcli con mod ens192 ipv4.dns-search <DNS_DOMAIN> nmcli con mod ens192 ipv4.method manual nmcli con mod ens192 connection.autoconnect yes nmcli con reload nmcli con down ens192 nmcli con up ens192 nmcli device show ens192

3. Change the IP configuration and verify the connection information for the backend network. 

   Code Block
   nmcli con mod ens224 ipv4.addresses <SCS_BACKEND_IP>/<BACKEND_NETWORK> nmcli con mod ens224 ipv4.method manual nmcli con mod ens224 connection.autoconnect yes nmcli con reload nmcli con down ens224 nmcli con up ens224 nmcli device show ens224

4. The network configuration can be verified with the command: ip a or nmcli con show.

Offline Installation

For offline installation (i.e., when no Internet access is available).  

1. Edit /etc/hosts comment out the http://k8s.gcr.io entry and the docker-repo.tx.caringo.comone.
   The first line should read: 

   Code Block
   <SCS_FRONTEND_IP> www.datacore.com

2. Set the time zone according to your local clock. 
   timedatectl set-timezone <timezone> 
   hwclock --systohc 

3. Configure chrony (NTP daemon) to connect to a valid NTP server. 

4. Edit the file /etc/chrony.conf and add the proper IP addresses or names of those NTP servers. Remove the lines referred to as the default ones if they are not reachable. 

   Code Block
   server <NTP_SERVER_1> iburst server <NTP_SERVER_2> iburst allow <SCS_BACKEND_NETWORK/BACKEND_NETWORK>

   The following is an example of the allow line: allow 192.168.90.0/24 

5. Restart chrony daemon: systemctl restart chronyd  

6. Verify the clock is in sync with: chronyc tracking

SCS Configuration

Once the auxiliary services of SCS are configured, the SCS setup can take place.

1. Run the configurator wizard:
   scsctl init wizard -a 

2. Running step [1/37]: Set site name.
   Type the <CLUSTER_NAME> 

3. Running step [2/37]: Set the administrative password.
   Type the admin password for the cluster
   Re-enter to confirm 

4. Running step [4/37]: Choose the Swarm-internal interface.
   Specify the network interface that will be used for internal Swarm operations:
      lo
      ens192
   > ens224

5. Select ens224, press Enter.

6. Running step [5/37]: Choose external (client-facing) interface.
   Specify the network interface that will be used for operations OUTSIDE of swarm:
      lo
    > ens192
   ens224

7. Select ens192, press Enter.

8. Running step [7/37]: Define Swarm-internal network.
   The internal interface requires a *static* IP address to be defined on it!
   It looks like your internal interface is already configured with an IP address: x.x.x.x/yz
   Do you wish to continue to use this address and netmask? [Y/n]:

9. Reply Y and hit Enter 

   The provisioning process will commence, and it takes a few minutes to complete. 

   Continue the configuration process running:
   scsctl diagnostics config scan_missing

10. Missing setting: network_boot/network.dnsServers
    Update this settingas a default at group level.

11. Press Enter and type the IP addresses of the DNS servers below separated with a blank space:
    network.dnsServers [type: array[str]] (Required: DNS servers to be used):
    <DNS_SERVER_1> <DNS_SERVER_2> 

12. Missing setting: platform/network.dnsDomain
    Update this setting as a default at group level

13. Hit Enter and type the DNS domain used
    network.dnsDomain [type: str] (Required: The DNS domain name that will be used.): 
    <DNS_DOMAIN> 

14. Fix the rsyslog template
    Run: cp -f /root/swarm-platform-fix.conf /etc/rsyslog.d/swarm-platform.conf

15. In the next step, the Swarm image will be added and configured. 

    Run: scsctl repo component add -f /root/swarm-scs-storage-16.1.0.tgz 

    During this process, the feature “encryption at rest” (EAR) can be configured.

Optional 

This is an optional functionality that will encrypt the data when it hits the plate of the disks. It comes at a cost of usually 15-20% performance penalty as the nodes need to use processing power to encrypt/decrypt data. 

This guide assumes EAR will be configured. If it is not a requirement, select False on the next step when the wizard asks about disk.encryptNewVolumes configuration. 

Also, the configuration steps will ask about multicast traffic, and whether it will be allowed. As it is a best practice to keep it enabled, this guide will follow that.

1. Missing setting: storage/disk.encryptNewVolumes
   Update this setting as a component-wide default

2. Hit Enter
   disk.encryptNewVolumes [type: bool] (Whether to encrypt new Swarm volumes. Enabling encryptNewVolumes means that any newly-formatted Swarm volume will be encrypted)
   True
   False 

3. Missing setting: storage/cip.multicastEnabled
   Update this setting as a component-wide default

4. Press Enter.
   cip.multicastEnabled [type: bool] (Whether multicast should be used for communication within Swarm.)
   > True 
   Press Enter
   Finally, the configuration wizard asks for what drives will be used to store data, this guide assumes “all” drives will be used as the server should be dedicated exclusively to Swarm. 

5. Missing setting: storage/disk.volumes
   Update this setting as a component-wide default

6. Press Enter
   disk.volumes [type: str] (Required: Specifies the volume storage devices for Swarm to use)
   all 

7. At this stage, the Swarm image is added. The configuration wizard will ask about the cluster name and a description.
   added: storage - 16.1.0 (16.1.0)
   Enter a name for the group (FQDN format encouraged):
   <CLUSTER_NAME>
   Enter a description for the group (purpose, etc.). [OPTIONAL]:

SCS needs to know what IP range can be used to PXE boot the Swarm storage nodes in the backend network. To avoid collisions with other Swarm services, we can reserve several IP addresses at the beginning and/or the end of the range, so SCS will not assign those IP addresses to the nodes. To do this,

1. Run the below command:

   Code Block
   scsctl init dhcp --dhcp-reserve-lower=50 --dhcp-reserve-upper=10

Adjusting the values to whatever makes sense in the backend network. For example, In a /24 network, the above will use from .51 to .244 to PXE boot and assign IP addresses to the Swarm storage nodes. 

2. If the physical servers have SSD/NVMe or smaller drives that are not required for Swarm, can be excluded by running:

   Code Block
   scsctl storage config set -d "disk.minGB=4096"

As an example, the above command will exclude any drive that is smaller than 4TB.

3. Unzip and add the license key. This key should be a plain text file:

   Code Block
   scsctl license add -f license.txt

4. Override the swarm-platform.conf config file:

cp /root/swarm-platform-fix.conf /etc/rsyslog.d/swarm-platform.conf

5. It is recommended to enable Swarm node stats for the Telemetry VM (Prometheus/Grafana). To do this, run:

scsctl storage config set -d "metrics.enableNodeExporter=true" 
scsctl storage config set -d "metrics.nodeExporterFrequency=120" 

6. If the Swarm storage nodes use an Intel Skylake based CPU or similar, run the following:

scsctl network_boot config set kernel.extraArgs=clocksource.max_cswd_read_retries=50 -d  
systemctl restart swarm-platform 

For more information, see Intel Skylake/Cascade Lake CPU Performance Issue 

7. Finally, create a backup of the SCS configuration. Run:

scsctl backup -o backup-config-<date> 

At this point, SCS has been configured and it is ready to PXE boot Swarm storage nodes.

Elasticsearch

Preparation Steps

1. Deploy SwarmSearch VM (SwarmSearch.ovf) and its associated virtual disks (vmdk). Please note the virtual network interfaces are inverted. The first one corresponds to the backend network, while the second interface is connected to the frontend network. 
   The preconfigured IP address for the backend network is 172.29.1.20/16. 

   Below are the steps to change it, if required:

2. Change the IP configuration and verify the connection information for the backend network. 

   Code Block
   nmcli con mod ens192 ipv4.addresses <ES_BACKEND_IP>/<BACKEND_NETWORK>  nmcli con mod ens192 ipv4.gateway <SCS_BACKEND_IP> nmcli con reload  nmcli con down ens192  nmcli con up ens192  nmcli device show ens192

The network configuration can be verified with the command: ip a or nmcli con show.

3. Set the time zone according to your local clock:

   Code Block
   timedatectl set-timezone <timezone> hwclock --systohc

4. disable swapping by executing the following command:

   Code Block
   echo "vm.swappiness=1" > /etc/sysctl.d/51-swappiness.conf sysctl -w vm.swappiness=1

5. Edit the file /etc/chrony.conf and add the proper IP addresses or names of those NTP servers. Remove the lines referred to the default ones if they are not reachable.

   Code Block
   server <NTP_SERVER_1> iburst  server <NTP_SERVER_2> iburst

   Ensure that the NTP servers are reachable.

6. Restart chrony daemon.

   Code Block
   systemctl restart chronyd

7. Verify the clock is in sync.

   Code Block
   chronyc tracking

8. Edit /etc/elasticsearch/elasticsearch.yml and replace 172.29.1.20 with the IP address configured in the previous step for this VM in the following sections of the file: 

   Code Block
   network.host: <ES_BACKEND_IP>  discovery.seed_hosts: ["<ES_BACKEND_IP>"]  cluster.initial_master_nodes: ["<ES_BACKEND_IP>"

9. Restart the service.

   Code Block
   systemctl restart elasticsearch

10. Verify it is up and running.

    Code Block
    curl -XGET "http://<ES_BACKEND_IP>:9200/_cat/health?v"

    The response should be "green" or "yellow". 

nmcli con modify ens224 ipv4.addresses <ES_FRONTEND_IP>/<FRONTEND_NETWORK> 
nmcli con modify ens224 ipv4.gateway <FRONTEND_GATEWAY> 
nmcli con modify ens224 ipv4.dns <DNS_SERVER_1>,<DNS_SERVER_2> 
nmcli con modify ens224 ipv4.dns-search <DNS_DOMAIN> 
nmcli con reload 
nmcli con down ens224 
nmcli con up ens224 
nmcli device show ens224 

With the above steps, only one Elasticsearch VM will be provisioned. The status will appear in “Yellow” the moment there is any data in Elasticsearch as there is no redundancy.

This configuration is enough for Proof-of-Concept or Proof-of-Value scenarios. However, for production environments, the recommendation is to have at least three Elasticsearch VMs up and running forming a cluster by themselves. 

The steps to deploy a full Elasticsearch cluster are explained below:

1. Deploy the SwarmSearch1.ovf template two more times.

2. Update the static IP address for the backend adapter ens192.

3. Update /etc/hostname of the two new VMs, e.g., “swarmsearch2”, “swarmsearch3”.

4. Stop the elasticsearch service and delete the pre-generated data:

   Code Block
   systemctl stop elasticsearch  cd /var/lib/elasticsearch  rm -rf nodes

5. Delete the predefined elasticsearch.yml config file and run the configurator wizard:  

   Code Block
   rm /etc/elasticsearch/elasticsearch.yml /usr/share/caringo-elasticsearch-search/bin/configure_elasticsearch_with_swarm_search.py

6. Enter Elasticsearch cluster name [A string]: swarmsearch

7. Enter List of all the Elasticsearch server names in the cluster [Comma-separated list of DNS-resolvable names or IP addresses]: <ES_NODE1_BACKEND_IP>,<ES_NODE2_BACKEND_IP>,<ES_NODE3_BACKEND_IP> 

8. Enter this Elasticsearch node's name [A string name from the list entered above]: <ES_NODE_NAME> 

Repeat these steps for every Elasticsearch VM, including the original (the first was deployed) one. For more information, see Configuring Elasticsearch  

Once all Elasticsearch VM’s is configured, restart the elasticsearch service in all of them:

systemctl restart elasticsearch 

Finally to check the health of the Elasticsearch cluster, run:

curl -XGET "http://<ES_BACKEND_IP>:9200/_cat/health?v" 

Three nodes should appear under the “node.total” column, and the status should be “green”.

Swarm Storage Nodes

1. Before starting the PXE boot process, enter the BIOS of each server that will be the Swarm storage node and check: 

   1. The HBA/Disk controller is configured in passthrough mode. Essentially, this is a non-RAID configuration where all the disk drives are presented to the operating system individually. It is also called IT mode, HBA mode, pass-thru, or non-RAID. 

   2. The network card port connected to the Backend VLAN/network must be enabled for PXE booting, no other port should be PXE-boot enabled. Moreover, there should not be any other port connected to any other network, with the exception of the dedicated port for out of band management (OOB, IPMI, BMC…) 

2. Once these are verified, the PXE boot process can begin.

3. Start with a single node, making sure it boots properly. 

4. Continue with the rest. A successful Swarm storage node boot looks like this on the screen / IPMI console of the server: 

   

5. Swarm version, IP address of the node, and “Storage Processes: RUNNING” should appear on the screen.

Content Gateway

The final step to have everything needed to have a functional Swarm cluster is to get Content Gateway up and running. 

1. Deploy SwarmContentGateway.ovf. The IP addresses must be configured next.

2. Change the IP configuration and verify the connection information for the frontend network. 

   Code Block

   nmcli con mod ens192 ipv4.addresses <GW_FRONTEND_IP>/<FRONTEND_NETWORK>  nmcli con mod ens192 ipv4.gateway <FRONTEND_GATEWAY>  nmcli con mod ens192 ipv4.dns <DNS_SERVER_1>,<DNS_SERVER_2>  nmcli con mod ens192 ipv4.dns-search <DNS_DOMAIN>  nmcli con mod ens192 ipv4.method manual  nmcli con mod ens192 connection.autoconnect yes  nmcli con reload  nmcli con down ens192  nmcli con up ens192 nmcli device show ens192

3. Change the IP configuration and verify the connection information for the backend network. 

   Code Block
   nmcli con mod ens224 ipv4.addresses <GW_BACKEND_IP>/<BACKEND_NETWORK>  nmcli con mod ens224 ipv4.method manual  nmcli con mod ens224 connection.autoconnect yes  nmcli con reload  nmcli con down ens224  nmcli con up ens224    nmcli device show ens224

4. The network configuration can be verified with the command: ip a or nmcli con show 

5. Set the time zone according to your local clock. 

   Code Block
   timedatectl set-timezone <timezone>  hwclock --systohc

7. Configure chrony (NTP daemon) to connect to a valid NTP server. 

8. Edit the file /etc/chrony.conf and add the proper IP addresses or names of those NTP servers. Remove the lines referred to as the default ones. 

   Code Block
   server <NTP_SERVER_1> iburst  server <NTP_SERVER_2> iburst

9. Restart the chrony daemon.

   Code Block
   systemctl restart chronyd

10. Verify the clock is in sync.

    Code Block
    chronyc tracking

The Content Gateway configuration comes next. 

To proceed with the Content Gateway configuration:

1. Edit /etc/caringo/cloudgateway/gateway.cfg

   Code Block
   adminDomain = admin.<CLUSTER_NAME> hosts = <SWARM_NODE1_IP> < SWARM_NODE2_IP > <SWARM_NODE3_IP> < SWARM_NODE4_IP> indexerHosts = <ES_BACKEND_IP> managementPassword = <CLUSTER_MGMT_PASSWORD>

2. Metering and quotas can be enabled. This is optional.
   For more information, see Content Metering and Setting Quotas

   Code Block
   [metering] enabled = true

   By default, metering is false.

   Code Block
   [quota] enabled = true

   By default, the quota is false.

3. Run the below command.

   Code Block
   /opt/caringo/cloudgateway/bin/initgateway systemctl enable cloudgateway systemctl start cloudgateway systemctl status cloudgateway

Content Gateway should be up and running now.

As the final step, let’s configure the desired default protection scheme and connect Swarm to Elasticsearch.

1. Open a web browser and go to: http://<GW_FRONTEND_IP>:91/_admin/storage

2. Click Storage Management.

3. Click Cluster and then on Feeds. 

4. On the top right corner, click +Add and select Search Metadata feed. 

5. On server host(s) or IP(s), type all the IP addresses of all the Elasticsearch VMs that are up and running separated by a blank space. <ES_BACKEND_IP>

6. Click Save. Now the Swarm nodes are connected to Elasticsearch, every time a new object/file gets uploaded to the cluster, its metadata will be also copied to Elasticsearch for search and listing purposes. 

7. Due to a bug in Gateway 8.0.4 you will need to restart cloudgateway for it to see the new search feed.

   Code Block
   systemctl restart cloudgateway

8. To finalize the setup, the default protection scheme should be set. Also, features like lifecycle policies and versioning can be enabled, if desired.
   For more information about these features, see Object Versioning and Bucket Lifecycle Policy.

9. Versioning is required to enable “S3 object locking” (immutability).
   For more information, see SCSP Object Locking.

10. Click Settings and select Cluster.
    In the “Policy” section, change the protection scheme as desired, for example with 4 Swarm storage nodes:

    Code Block
    policy.eCEncoding 4:2  policy.eCMinStreamSize 1Mb  policy.lifecycle enabled  policy.replicas min:3 max:16 default:3  policy.versioning allowed

11. Click Save at the top right corner.

12. Finally, test uploads and downloads using the provided Content Portal.

13. Open a web browser and go to http://<GW_FRONTEND_IP>/_admin/portal  

14. Click System Tenant at the upper right corner and click +Add.

To create a bucket:

1. Click the domain that you just created.

2. Click +Add this time selecting “Bucket”. Provide a name such as “bucket1” or “test1”.

3. Click the bucket you just created and click +Add or drop files.

4. Look for some files of various sizes on the client machine used, from KBs to MBs and upload them.

5. Click the bucket name at the top. 

Swarm utilizes FQDNs to identify which storage domain (endpoint) the client is connecting to. Hence, create DNS entries according to the Storage Domains used in the environment. 

At this point Swarm is up and running and its basic functionality has been verified.

Create an S3 Key Pair (Optional)

To access the storage layer using the S3 protocol, a S3 key pair must be created.  

It is comprised by the S3 access key and the S3 secret key. 

1. Open a web browser and go to http://<GW_FRONTEND_IP>/_admin/portal

2. Click the domain (endpoint) desired, but not on the admin one.

3. Click the cog/wheel in the top right corner and select Tokens. There will be an +Add button again in the top right corner.

4. Provide a description, an expiration date, and click the checkmark by “S3 secret key”.

5. Upon clicking on Add, a green message will appear with all the information needed.

6. With this information and the name of the domain used, it is possible to create a connection to the Swarm repository over the S3 protocol.

Configuring an SSL Certificate (Optional)

1. By default, the Content Gateway VM template comes with HAProxy unconfigured and no self-signed certificate.

2. If you wish to configure HAProxy as an SSL offloader, follow the steps outlined in Configuring haproxy SSL offloading with a Self Signed Certificate on CentOS7/8

Central Logging (Optional)

It is recommended that the Content Gateway logs all actions and their status to the central Syslog server. The SCS can act as the central repository for logs.

To configure this, edit /etc/caringo/cloudgateway/logging.yaml and modify the following lines:

Code Block
Syslog: - name: audit_syslog host: <SCS_BACKEND_IP>

It is localhost by default.

Code Block
- name: server_syslog host: <SCS_BACKEND_IP>

Code Block
Loggers: # Global logging configuration Root: level: "${logLevel}" AppenderRef: - ref: file - ref: server_syslog

Note

Commented by default, remove the # at the beginning.

Code Block
Logger: # Audit logger - name: audit level: info additivity: false AppenderRef: - ref: audit - ref: audit_syslog

There is no need to restart the Content Gateway service; the new logging configuration will be applied automatically after a few seconds.

Telemetry (Optional)

The Telemetry VM provides an all-in-one reference implementation of Prometheus, Alertmanager, and Grafana.

Preparation Steps

1. Deploy SwarmTelemetry.ovf. IP addresses must be configured next.

2. Change the IP configuration and verify the connection information for the frontend network. 

   Code Block

   nmcli con mod ens192 ipv4.addresses <TM_FRONTEND_IP>/<FRONTEND_NETWORK>  nmcli con mod ens192 ipv4.gateway <FRONTEND_GATEWAY>  nmcli con mod ens192 ipv4.dns <DNS_SERVER_1>,<DNS_SERVER_2>  nmcli con mod ens192 ipv4.dns-search <DNS_DOMAIN>  nmcli con mod ens192 ipv4.method manual  nmcli con mod ens192 connection.autoconnect yes  nmcli con reload  nmcli con down ens192  nmcli con up ens192    nmcli device show ens192

3. Change the IP configuration and verify the connection information for the backend network. 

   Code Block
   nmcli con mod ens224 ipv4.addresses <TM_BACKEND_IP>/<BACKEND_NETWORK>  nmcli con mod ens192 ipv4.gateway <SCS_BACKEND_IP> nmcli con reload  nmcli con down ens224  nmcli con up ens224    nmcli device show ens224

4. The network configuration can be verified with the command: ip a or nmcli con show

5. Set the time zone according to the local clock.

   Code Block
   timedatectl set-timezone <timezone> hwclock --systohc

Info
Note  All available time zones can be listed with the command: timedatectl list-timezones

6. Configure chrony (NTP daemon) to connect to a valid NTP server. Edit the file /etc/chrony.conf and add the proper IP addresses or names of those NTP servers. Remove the lines referred to as the default ones. 

   Code Block
   server <NTP_SERVER_1> iburst  server <NTP_SERVER_2> iburst

7. Restart the chrony daemon.

   Code Block
   systemctl restart chronyd

8. Verify the clock is in sync.

   Code Block
   chronyc tracking

Prometheus Master Configuration

The next step is to configure Prometheus.

1. Edit /etc/prometheus/prometheus.yml to include all the IP addresses of all the Swarm components to be monitored, uncomment lines as needed:

   Code Block

   # THIS IS THE ELASTICSEARCH EXPORTER DEFINITION # IP ADDRESS SHOULD BE Telemetry loopback - job_name: 'elasticsearch' scrape_interval: 30s static_configs: - targets: ['127.0.0.1:9114'] relabel_configs: - source_labels: [__address__] regex: "([^:]+):\\d+" target_label: instance # THIS IS THE CLOUD CONTENT GATEWAY JOB DEFINITION # IP ADDRESS SHOULD BE CLOUD GATEWAY STORAGE VLAN IP - job_name: 'swarmcontentgateway' static_configs: - targets: ['<GW_BACKEND_IP>:9100'] relabel_configs: - source_labels: [__address__] regex: "([^:]+):\\d+" target_label: instance # THIS IS THE CLOUD GATEWAY NODE_EXPORTER JOB DEFINITION # IP ADDRESS SHOULD BE CLOUD GATEWAY STORAGE VLAN IP - job_name: 'gateway-nodeexporter' scrape_interval: 30s static_configs: - targets: ['<GW_BACKEND_IP>:9095'] relabel_configs: - source_labels: [__address__] regex: "([^:]+):\\d+" target_label: instance # THIS IS THE SWARM JOB DEFINITON # IP ADDRESS SHOULD BE STORAGE VLAN IP - job_name: 'swarm' scrape_interval: 30s static_configs: - targets: ['<SWARM_NODE1_IP>:9100','<SWARM_NODE2_IP>:9100','<SWARM_NODE3_IP>:9100','<SWARM_NODE4_IP>:9100'] relabel_configs: - source_labels: [__address__] regex: "([^:]+):\\d+" target_label: instance

Note

If there are multiple gateways, add them to the targets list like: 

targets: ['<GW1_BACKEND_IP>:9100','<GW2_BACKEND_IP>:9100'] 

YAML (.yml) files are quite sensitive to spaces and indentation. The following command will check that there is no errors.  

Code Block
promtool check config /etc/prometheus/prometheus.yml

Elasticsearch Node Exporter

1. To gather statistics and status about Elasticsearch edit: /usr/lib/systemd/system/elasticsearch_exporter.service updating the IP address of the (first) Elasticsearch VM (instead of the pre-configured 172.29.1.20). 

   Code Block
   ExecStart = /usr/local/bin/elasticsearch_exporter --es.all -- es.cluster_settings --es.indices --es.indices_settings -- es.indices_mappings --es.shards --es.snapshots --es.uri http://<ES_BACKEND_IP>:9200 --es.timeout 20s --web.listen-address :9114

2. Run the below commands to enable and start the service.

   Code Block
   systemctl daemon-reload  systemctl enable elasticsearch_exporter  systemctl start elasticsearch_exporter

3. Once the Prometheus master config changes are applied, the service can be enabled and started.

   Code Block
   systemctl enable prometheus  systemctl restart prometheus

4. To verify that Prometheus is up and running, open a web browser and go to: http://<TM_FRONTEND_IP>:9090/targets  

This page shows which targets it is currently collecting metrics for and if they are reachable. Click Status and select “Targets”. It will take a few minutes to be updated. All states should appear as “UP”.

Alertmanager Configuration

There are four (4) alerts defined in /etc/prometheus/alert.rules.yml

• Service_down: Triggered if any swarm storage node is down for more than 30 minutes.

• Gateway_down: Triggered if the cloudgateway service is down for more than 2 minutes.

• Elasticsearch_cluster_state: Triggered if the cluster state changed to "red" after 5 minutes.

• Swarm_volume_missing: Triggered if the reported drive count is decreasing over 10 minutes. This is due to a failed disk drive that needs to be replaced.

The /etc/prometheus/prometheus.yml contains a section that points to the alertmanager service on port 9093, as well as which alert.rules.yml file to use.

To customize the alerts:

1. Modify the swarmUI template in /etc/prometheus/alertmanager/template/basic-email.tmpl. This will be used for the email HTML template showing a button to the chosen URL.

   Code Block
   {{ define "__swarmuiURL" }}http://<GW_FRONTEND_IP>:91/_admin/storage/{{ end }}

2. The configuration for where to send alerts is defined in the file: /etc/prometheus/alertmanager/alertmanager.yml  

3. By default, the route is disabled as it requires manual input for every specific environment, values such as:
   SMTP server, username, password (if applicable), etc.

Info

Important Prometheus alertmanager does not support SMTP NTLM authentication, as such it cannot be used to send authenticated emails directly to Microsoft Exchange. Instead, smarthost should be configured to connect to localhost:25 without authentication. This is where the default Rocky Linux Postfix server is running. It will know how to send the email to your corporate relay as it is auto-discovered via DNS. Add require_tls: false to the email definition config section in alertmanager.yml. Example configuration for a local SMTP relay: Prometheus alertmanager does not support SMTP NTLM authentication, as such it cannot be used to send authenticated emails directly to Microsoft Exchange. Instead, smarthost should be configured to connect to localhost:25 without authentication. This is where the default CentOS Postfix server is running. It will know how to send the email to your corporate relay as it is auto-discovered via DNS. Add require_tls: false to the email definition config section in alertmanager.yml.

Code Block
- name: 'emailchannel' email_configs: - to: admin@acme.com from: swarmtelemetry@acme.com smarthost: smtp.acme.com:25 require_tls: false send_resolved: true

6. Once the configuration is complete, restart the alertmanager.

   Code Block
   systemctl restart alertmanager

7. To verify the alertmanager.yml has the correct syntax, run:  

   Code Block
   amtool check-config /etc/prometheus/alertmanager/alertmanager.yml

It should give the following output: 

Checking '/etc/prometheus/alertmanager/alertmanager.yml' SUCCESS  

Found:  

• global config  

• route  

• 1 inhibit rules  

• 2 receivers  

• 1 templates SUCCESS 

Info
Tip The easiest way to trigger an alert for testing purposes is to shut down one Content Gateway.

Grafana Configuration

The password for the “admin” user can be changed on the configuration file /etc/grafana/grafana.ini, look for admin_password.

If you are using the SwarmTelemetry VM then the admin password can only be changed via the grafana GUI.

Note

Grafana has several authentication options including google-auth / oAuth / LDAP and by default basic http auth.

For more information, see Documentation | Grafana Labs.

To enable on boot and start the service type:

Code Block
systemctl enable grafana-server systemctl restart grafana-server

Grafana has all the Swarm dashboards pre-installed. Open a web browser and go to http://<TM_FRONTEND_IP>

The default period is 7 days, modify it to 5 minutes to see some stats appearing on the charts.

The latest Swarm dashboards are available on the Grafana website.

Job Name (Optional)

In /etc/prometheus/prometheus.yml the job_name of the Content Gateway can be defined. This job_name will be displayed on the Content Gateway Grafana dashboard.

If the Content Gateway job_name is changed there are a couple of additional changes required:

Modify the gateway job name in /etc/prometheus/alertmanager/alertmanager.yml; it must match what appears in prometheus.yml.

routes:  
- match:  
      job: <new_job_name>

Modify the gateway job name in /etc/prometheus/alert.rules.yml.

alert: gateway_down  
expr: up{job="<new_job_name>"} == 0  

DNS names can be used. In the absence of a DNS server, first, modify /etc/hosts file with the desired names for each Swarm storage node and then use those in the configuration file. This is recommended in scenarios where the dashboards are publicly accessible.

Prometheus Retention Time (Optional)

By default, the Prometheus configuration in Telemetry keeps metrics for 30 days. If there is a need to increase or decrease this retention, follow the next steps: 

1. Edit the /root/prometheus.service file. 

2. Select the default retention time for the collected metrics. 

3. Modify the --storage.tsdb.retention.time=30d flag to the new desired retention time.

4. Finally, commit the change:

cp /root/prometheus.service /usr/lib/systemd/system  
systemctl daemon-reload  
promtool check config /etc/prometheus/prometheus.yml  
systemctl restart Prometheus

Prometheus Security (Optional)

It may be desirable to restrict Prometheus server to only allow queries from the local host, since Grafana server is running on the same VM. This can be done by editing /root/prometheus.service file and adding the flag --web.listen-address=127.0.0.1:9090  

If Prometheus is bound only to localhost, the built-in Prometheus UI on port 9090 will not be accessible remotely. 

Planning and Storage Nodes Prerequisites

Deployment Planning

Hardware Requirements for Storage

SCS

Swarm Cluster Services (SCS) Implementation

Elasticsearch

Hardware Requirements for Elasticsearch

Preparing the Search Cluster

Installing Elasticsearch

Configuring Elasticsearch

Setup Elasticsearch Cluster

Managing Feeds

Content Gateway

Gateway Requirements

Gateway Installation

Gateway Configuration

Gateway Verification

Telemetry (Prometheus and Grafana)

Prometheus Node Exporter and Grafana