Time to time we have to upgrade products to resolve bugs/vulnerabilities and get new features. During VMware NSX upgrade from 3.2.x, 4.0.x, 4.1.x to version 4.2.x you might get the error on Pre-Checks run step:
This happen because internal NSX services use certificates which already expired or close to expiration date. NSX Upgrade requires all certificates to be fixed before you continue. To simplify our lives, VMware by Broadcom provides special script for this purpose – Certificate Analyzer, Results and Recovery (CARR) Script.
Here is the link to original KB
At the bottom of the KB latest version of the script attached:
Download it and place on one of your NSX Manager node appliance to /root folder.
Unpack the script with following command (1.21 is the current version as of now but can be changed in the future, adjust file name accordingly if needed):
tar -xvf carr-1.21.tar.gz
Change folder:
cd carr-1.21
Start the script in dry run mode. On this step no changes will be made, you can safely run it any time. Prepare NSX Manager local admin and root passwords, it will ask during run.
./start.sh -d
Here is the example of output:
############################################################
# #
# NSX Certificates Analyzer, Results and Recovery #
# #
############################################################
Enter 10.xxx.xxx.197's 'Admin' user password (will not be displayed):
Enter 10.xxx.xxx.197's 'root' user password (will not be displayed):
Starting to validate certificate on 'LM' nodes
'LM' node IPs: 10.xxx.xxx.199, 10.xxx.xxx.197, 10.xxx.xxx.198
Validating 'VIP' certificate …
Validating 'STALE-CERTIFICATES' certificate …
Validating 'APH_AR' certificate …
Validating 'COMPUTE_MANAGER' certificate …
Validating 'API' certificate …
Validating 'SITE-TO-SITE' certificate …
Validating 'HOST' certificate …
Validating 'EDGE' certificate …
Validating 'CCP' certificate …
Validating 'APH_TN' certificate …
Validating 'LOCAL-MANAGER-PI' certificate …
Validating 'CORFU_CLIENTS' certificate …
Validating 'CORFU_SERVER' certificate …
Validating 'CBM-FILE-PERMISSIONS' certificate …
Validation completed
Time taken to complete validation: 4m:15s
All validations done, generating report
Detailed transport nodes report generated: dry_run_transport_nodes_validation_report.yaml
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| CARR Script Validation Report |
+-------------------------+-------------------------------------------------------------------------------+--------------------------------------------------------------+
| Certificate Checks | Validation Results | Probable Fix |
+-------------------------+-------------------------------------------------------------------------------+--------------------------------------------------------------+
| VIP | WARNING : Certificate is expiring in 187 days | Certificate of service type: 'VIP' will be replaced. |
| | | |
+-------------------------+-------------------------------------------------------------------------------+--------------------------------------------------------------+
| STALE-CERTIFICATES | SUCCESS: No stale certificates found. | |
| | | |
+-------------------------+-------------------------------------------------------------------------------+--------------------------------------------------------------+
| APH_AR | SUCCESS: 10.xxx.xxx.199 | |
| | SUCCESS: 10.xxx.xxx.197 | |
| | SUCCESS: 10.xxx.xxx.198 | |
| | | |
+-------------------------+------------------------------------------------------------------------------- [...skip...] +--------------------------------------------------------------+
| CBM_MP | ERROR : 10.xxx.xxx.199 : Certificate CBM_MP has expired | Certificate of service type: 'CBM_MP' will be replaced. |
| | ERROR : 10.xxx.xxx.197 : Certificate CBM_MP has expired | Certificate of service type: 'CBM_MP' will be replaced. |
| | ERROR : 10.xxx.xxx.198 : Certificate CBM_MP has expired | Certificate of service type: 'CBM_MP' will be replaced. |
| | | |
+-------------------------+-------------------------------------------------------------------------------+--------------------------------------------------------------+
| CBM_CORFU | SUCCESS: 10.xxx.xxx.199 | |
| | SUCCESS: 10.xxx.xxx.197 | |
| | SUCCESS: 10.xxx.xxx.198 | |
| | | |
+-------------------------+-------------------------------------------------------------------------------+--------------------------------------------------------------+
| CBM-FILE-PERMISSIONS | SUCCESS: 10.xxx.xxx.199 | |
| | SUCCESS: 10.xxx.xxx.197 | |
| | SUCCESS: 10.xxx.xxx.198 | |
| | | |
+-------------------------+-------------------------------------------------------------------------------+--------------------------------------------------------------+
All validations done
Script was run in 'Dry Run' mode. No recovery will be performed
The recovery yaml has been generated at: /root/carr-1.19/validation_config_recovery_mode.yaml
If you don't want to rotate all the certs having issues, you may modify the recovery yaml to validate only the certs that needs to be rotated.
To use this config in the fix mode, run the script with -r flag
After checking output you need to plan certificates replacement. This is online operation but if you prefer to be on the safe side, perform it after fresh NSX backup and during maintenance window.
./start.sh
Example of output during remediation:
############################################################
# #
# NSX Certificates Analyzer, Results and Recovery #
# #
############################################################
Enter 10.xxx.xxx.197's 'Admin' user password (will not be displayed):
Enter 10.xxx.xxx.197's 'root' user password (will not be displayed):
Starting to validate certificate on 'LM' nodes
'LM' node IPs: 10.xxx.xxx.199, 10.xxx.xxx.198, 10.xxx.xxx.197
Validating 'VIP' certificate ...
Validating 'STALE-CERTIFICATES' certificate ...
Validating 'APH_AR' certificate ...
Validating 'COMPUTE_MANAGER' certificate ...
Validating 'API' certificate ...
Validating 'SITE-TO-SITE' certificate ...
Validating 'HOST' certificate ...
Validating 'EDGE' certificate ...
Validating 'CCP' certificate ...
Validating 'APH_TN' certificate ...
Validating 'LOCAL-MANAGER-PI' certificate ...
Validating 'CORFU_CLIENTS' certificate ...
Validating 'CORFU_SERVER' certificate ...
Validating 'CBM-FILE-PERMISSIONS' certificate ...
Validation completed
Time taken to complete validation: 5m:29s
All validations done, generating report
Detailed transport nodes report generated: before_recovery_transport_nodes_validation_report.yaml
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| CARR Script Validation Report |
+-------------------------+---------------------------------------------------------------------------------------+--------------------------------------------------------------+
| Certificate Checks | Validation Results | Probable Fix |
+-------------------------+---------------------------------------------------------------------------------------+--------------------------------------------------------------+
| VIP | SUCCESS: 10.xxx.xxx.199 | |
| | SUCCESS: 10.xxx.xxx.198 | |
| | SUCCESS: 10.xxx.xxx.197 | |
| | | |
+-------------------------+---------------------------------------------------------------------------------------+--------------------------------------------------------------+
| STALE-CERTIFICATES | SUCCESS: No stale certificates found. | |
| | | |
+-------------------------+---------------------------------------------------------------------------------------+--------------------------------------------------------------+
| APH_AR | SUCCESS: 10.xxx.xxx.199 | |
| | SUCCESS: 10.xxx.xxx.198 | |
| | SUCCESS: 10.xxx.xxx.197 | |
| | | |
+-------------------------+---------------------------------------------------------------------------------------
[...skip...]
+-------------------------+--------------------------------------------------------------------------------------+--------------+
| CBM_CORFU | SUCCESS: 10.xxx.xxx.199 | |
| | SUCCESS: 10.xxx.xxx.198 | |
| | SUCCESS: 10.xxx.xxx.197 | |
| | | |
+-------------------------+--------------------------------------------------------------------------------------+--------------+
| CBM-FILE-PERMISSIONS | SUCCESS: 10.xxx.xxx.199 | |
| | SUCCESS: 10.xxx.xxx.198 | |
| | SUCCESS: 10.xxx.xxx.197 | |
| | | |
+-------------------------+--------------------------------------------------------------------------------------+--------------+
All validations done
Cluster status is : STABLE
All certificates have been fixed and you can continue upgrading NSX.
The following two tabs change content below.
Yevgeniy Steblyanko
Yevgeniy Steblyanko is an Infrastructure Architect/SME with experience in virtualization area for more than 15 years. His areas of interest are VMware vSphere, vSAN, NSX, automation on PowerCLI/PowerNSX. He has VMware certifications: VCIX-DCV, VCIX-NV.
Latest posts by Yevgeniy Steblyanko (see all)
- VMware NSX Manager certificate renew during upgrade - December 31, 2025
- How to stretch networks over VMware NSX Edge Bridge - December 30, 2025
- Automate bulk Windows and Linux VMs creation from template with Guest OS customization - July 18, 2024
