It is recommended that, at installation time, the SCS includes a dedicated /var/log partition so that excessive logging does not overrun the root partition. Having a dedicated /var partition is not sufficient from preventing problems arising from excessive logging as the podman containers live in /var/lib. Overrunning the /var partition will prevent bootup of storage nodes among other problems. The following instructions only cover how to add a new disk to a SCS VMWare-based installation as a /var/log partition to your environment and do not cover expanding or reconfiguring existing logical volumes.
To verify if you have a dedicated partition for /var/log, run this on your SCS:
df -h
If you see a line for /var/log under the “Mounted on” column, the SCS already has a dedicated /var/log partition. In the example following, I have a dedicated /var partition, but not /var/log partition:
[root@ace-scs2-scs1 ~]# df -h Filesystem Size Used Avail Use% Mounted on devtmpfs 1.9G 0 1.9G 0% /dev tmpfs 1.9G 0 1.9G 0% /dev/shm tmpfs 1.9G 8.9M 1.9G 1% /run tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup /dev/mapper/centos-root 15G 2.2G 13G 15% / /dev/sda1 1014M 146M 869M 15% /boot /dev/mapper/centos-var 30G 228M 30G 1% /var tmpfs 379M 0 379M 0% /run/user/0
Make a note of the output of the above command in your environment as it may be useful later.
Proceed with the Instructions to create a dedicated /var/log partition. The instructions presume:
You are an administrator of your VMWare environment (or can get someone to perform actions).
You have a maintenance window with approved SCS downtime - this should not impact cluster operations.
You have SSH access to the SCS, Content Gateways, and any other devices in the Swarm architecture that are sending logs to the SCS.
You have rsync installed on the SCS or can install it beforehand.
\uD83D\uDCD8 Instructions
Follow these steps during an approved maintenance window:
Before you continue, backup your SCS virtual machine in case anything goes wrong.
If the SCS is receiving logs from other sources- Content Gateways, Elasticsearch nodes, etc- stop these sources from sending logs to the SCS temporarily.
Power down the SCS and have the VMWare administrator add a new disk to the SCS VM. 20 GB is a good size depending on how much log history you want to keep.
Power on the SCS after the new disk has been added.
Stop rsyslog on the SCS:
systemctl stop rsyslogEnsure that you can see the new disk with:
fdisk -l- if you had a single disk previously, the new disk may be/dev/sdband the size reported should match your expectations.Create a new temporary directory where we will mount the disk:
mkdir /mnt/var_newAssuming the new disk is /dev/sdb for the following instructions. Type:
fdisk /dev/sdbtype n to create the new partition
type p for primary partition on this disk
type 1 for partition number
keep defaults for First sector and Last sector
type w to save
Next, we will format the disk with xfs:
mkfs.xfs -f -L varlog -b size=1024 /dev/sdbNow, we will mount the new disk into our temporary directory:
mount /dev/sdb /mnt/var_new/Next we will rsync the data from the existing /var/log directory onto the new mount point:
rsync -avHPSAX /var/log/ /mnt/var_new/Verify that the temporary mount point, /mnt/var_new, has the existing log files:
ls -al /mnt/var_new/- you can compare them with:diff -r /var/log/ /mnt/var_new/Now we will move the old /var/log/ directory out of the way so that we can move the new temporary mount point into its place.
mv /var/log /var/log_oldRecreate /var/log:
mkdir /var/logAdd this mount point in /etc/fstab for the new disk to mount to /var/log:
/dev/sdb /var/log xfs defaults 0 0
Unmount the temporary mountpoint /mnt/var_new since we will mount it to /var/log in the following step:
umount /mnt/var_newMount the disk as defined in /etc/fstab by simply typing:
mount /var/logAt this point, you should see all of your log files in /var/log:
ls -la /var/logdf -hshould show that your /var/log directory has the expected space.# df -h | egrep '^Filesystem|log' Filesystem Size Used Avail Use% Mounted on /dev/sdb 20G 52M 20G 1% /var/log
Restart rsyslog:
systemctl start rsyslogResume logging from external servers if applicable: Content Gateways, etc.
Remove the /var/log_old directory after verifying that your logging is working as expected on the new disk:
rm -rf /var/log_old