Increasing Prometheus storage retention

The default retention of Prometheus on SwarmTelemetry is 15 days, so data will be removed automatically after 15 days. Below is how to change the data retention of Prometheus.

1. Edit the prometheus.service file

   vi /etc/systemd/system/prometheus.service

2. Change the value of “--storage.tsdb.retention.time=15d" below to "ExecStart=/usr/local/bin/prometheus \" line.
   The config will look like below for 45 days of data retention. (e.g. 1 year: 1y or 365d)

   # /etc/systemd/system/prometheus.service
   [Unit]
   Description=Prometheus Server
   Documentation=https://prometheus.io/docs/introduction/overview/
   After=network-online.target
   
   [Service]
   User=prometheus
   Restart=on-failure
   ExecStart=/usr/local/bin/prometheus \
                                   --config.file=/etc/prometheus/prometheus.yml \
                                   --storage.tsdb.path=/var/lib/prometheus/data \
                                   --storage.tsdb.retention.time=45d \
                                   --web.console.templates=/etc/prometheus/consoles \
                                   --web.console.libraries=/etc/prometheus/console_libraries
   
   [Install]
   WantedBy=multi-user.target

3. Reload systemd manager configuration

   systemctl daemon-reload

4. Restart Prometheus service

   systemctl restart prometheus

See the Prometheus documentation.