Import Prometheus metrics from another SwarmTelemetry VM

This KB explains what you need todo to import metrics from another SwarmTelemetry VM.

First you need to backup the metrics data in the customer environment, this means you will need the following data:

tar cvzf prometheus-state-ddmmyyyy.tgz /var/lib/prometheus/* /etc/prometheus/prometheus.yml /usr/lib/systemd/system/prometheus.service

The reason you need a copy of the prometheus systemd script is because it determines the database storage location as well as the data retention policy of the customer ( default its 15 days )

If you are just setting up another SwarmTelemetry VM to visualize the data, make sure to set the retention time to 365 days, otherwise Prometheus will delete it within 15 days by default.

--storage.tsdb.path=/var/lib/prometheus/data

--storage.tsdb.retention.time=15d

Note: Prometheus itself does not need to be scraping any targets as you just want to read historical data, so make sure /etc/prometheus/prometheus.yml has no job definitions. You can double check the syntax of the file is correct by running promtool check config /etc/prometheus/prometheus.yml

To restore the metrics on another SwarmTelemetry VM, you simply need to do the following:

systemctl stop prometheus
# You dont need the old metrics data
rm -rf /var/lib/prometheus 
cd /
tar xvzf prometheus-state-ddmmyyyy.tgz
chown -R prometheus. /var/lib/prometheus
systemctl daemon-reload
systemctl start prometheus

Verify that prometheus started without critical errors, you should see a message “Server is ready to receive web requests”

journalctl -xe -t prometheus

If you now access grafana you should see the old metrics on the dashboards, you will have to change the date range since it will be somewhere in the past. I would set the range to 30days first then use the mouse to select the start and end of the data in one of the historical charts so that your date range ends up specifying exactly where in time the data is. Example

Note: Keep in mind its important to have the right version of grafana as our latest dashboards are specific to grafana 9.x and above

That’s it.