Here below I would like to share with you the issue I had in the past. One day I found NSX Manager VM non-responding. Reboot didn’t help because of the filesystem error as shown below:
If you press Enter it just power off the VM. There is no way to login to the shell and fix the filesystem in a usual way. So I started looking for a way to fix the failed file system.
Warning: Before you start any action do not forget to create a fresh backup/snapshot of NSX Manager VM.
To fix the file system you need to download some Linux Live CD, for example Ubuntu Desktop. Once you have ISO prepared, write down the failed file system (in our case /dev/sda6), check if your NSX Manager appliance VM is powered off. Open NSX Manager VM settings, go to Manage -> Settings -> VM Hardware -> Edit -> VM Options -> Boot Options, choose “Force BIOS setup”.
Then power on the VM, open VM Console. So you are in BIOS now.
Go back to VM configuration. Connect downloaded Live CD ISO file as CD-ROM
Now in VM Console change boot order to have CD-ROM Drive as the first boot device in the list:
Choose ‘Exit saving changes’
After VM boot from ISO choose ‘Try Ubuntu’
Run application ‘Terminal’
In order to see all file systems, you need to type:
ls -l /dev/sd*
Then run the filesystem check for the device you noted at the beginning (/dev/sda6)
sudo fsck /dev/sda6
this command will find corrupted blocks and ask for confirmation to fix it, so press “y” to confirm
After all checks are completed and fixed you should get something like following view:
Run the file system check once again to be sure there is no errors any more.
sudo fsck /dev/sda6
Now you can shutdown the VM
shutdown
Dismount ISO image from CD-ROM and start your NSX Manager appliance VM. If you are lucky enough you will see login prompt again 🙂
Check your NSX Manager and remove unnecessary backups/snapshots.
If below steps did not help and you still see the filesystem errors, I hope you have build-in NSX Manager backups scheduled and performed to external FTP/SFTP server on daily/weekly basis. In such case you will need to deploy new fresh NSX Manager and restore NSX configuration from backup.
Yevgeniy Steblyanko
Latest posts by Yevgeniy Steblyanko (see all)
- VMware NSX VPN tunnels statistics collection with PowerShell - February 29, 2024
- vSphere & NSX: Active Directory over LDAPs authentication - February 3, 2022
- Export / Backup VMware NSX ESG/DLR configurations with PowerNSX (part 2, DLRs) - March 26, 2020
