VCSA filesystem is out of disk space

This week while doing the normal check of the environment I came to run into a warning on the VCSA 6.5 stating: PostgresSQL Service Health Alarm

This was the first time I saw this kind of warning on the environment.

After logging in to the vSphere Appliance Managment I could see more detailed information about the issue.

Time for a deeper investigation

Nothing else is visible relate to this incident so I logged into VCSA using SSH to check the respective filesystem disk space using the command:

df -h

Here it is the problem, filesystem /dev/mapper/log_vg-log mounted on /storage/log has ZERO Available space

After identifying the low space filesystem there are two options that we can follow, increase the size of the disk on VCSA and for this one that is several blog post about it (Settlersoman) or try to find out the file and clean it.

I followed the second one and for that, I used the following command to search the filesystem for huge files/folders:

du -a /storage/log/ | sort -n -r | head -n 20

messages.1 with 20GB in size?????? This should not happen!!!

Now we know what is consuming the free space. For some reason, the file is not being compressed, I deleted the file using the command:

rm -Rf messages.1

Hoping I fixed it I checked again the filesystem usage and there was no change in the free space, /storage/log was still without free space.

Now what?

It looks like VCSA was not reclaiming the free space… After searching some blogs and VMware community posts the solution was to reboot VCSA but that was out of the question for now as it would impact production, and we don’t want that.

Why was the filesystem still full? I just deleted a 20GB file…

In Linux when a file is removed/deleted/unlinked if it’s still held open by any process only the entry of the file is removed, not the file’s data. To be able to free the data we have to close the process that is using the file.

Let’s see how many open deleted files we have on the system by running the following command:

lsof +L1

After identifying the file we have two options, or we kill the process or we restart the process. Because it was rsyslog I decided to restart it using the command:

service rsyslog restart

After I checked the filesystems and there it is, 19GB free on /storage/log

And this is how you claim free space on filesystem without restarting the VCSA.

As always,
Have fun and keep it simple!