Updated: Building vSphere 6 with VSAN on IBM Softlayer: Part 4 – Configuring VSAN on Softlayer

UPDATE: I do not recommend using this guide for any future deployment. The information included is quite outdated and is not relevant anymore. The article should only be used as reference for some commands. My recommendation is to check  IBM’s official offering of VMware on IBM Cloud. A lot of flexibility was introduced in recent years. here is a link.

UPDATED(24.02.2016): Driver version changed, additional details added.

UPDATED(12.07.2016): Driver changed. VMware KB added

Updated(28.07.2016): Softlayer automated Firmware Update method added.

Update(03.02.2017): It looks like new servers ordered from Softlayer come with proper megaraid driver preinstalled. Please check if that’s the case for you, before making changes.

Update(26.04.2017): If you are an IBMer building a VSAN on IBM Cloud for your customer, you can find official GTS VSAN Cookbook using this IBM internal link. You can also reach me or Jack Cherkas over Sametime for more details.

Welcome back. In this part we will discuss specific steps to have VSAN configured on servers we’ve ordered in previous articles.

What we have is:

1. 3 ESXi hosts
2. 1 vCenter server
3. (optionally) a dedicated VLAN with portable subnet for VSAN

The approach I will be describing is the one which was tested by me and some of my colleges and proved to be working.

Here is the summary of what we will do:

1. Update driver of Storage controller to the latest version. (On all hosts one by one.),
2. Check and update Firmware of storage controller to the latest version.(On all hosts one by one.),
3. Reconfigure Disk subsystem to suit our needs.
4. Create networking on vSphere level for VSAN,
5. Configure VSAN.

Note: the RAID controller used in this example is MegaRAID SAS 9361-8i (aka LSI LSI00451 on Softlayer portal). As far as I was told, this is the default controller in Softlayer and the one which is supported by VMware.

Updating driver of MegaRAID SAS 9361-8i on ESXi 6

First we need to download the driver. I will include link to the latest supported(according to my info from VMware) driver, for today, but please make sure you check which one is latest and supported on the day you are performing configuration.

So the driver i was recommended to use can be downloaded from this link. Use this one unless you have more up-to-date info from VMware.

Just in case you need, here is how you can find the latest driver for vSphere 6 is to first go to the official website of the raid controller here. There, under Downloads, download what you find as a latest driver for vSphere 6, but it is actually text file which contains link to VMware website with latest supported driver.

Download the ZIP from here and unpack it. We will need the .vib file form inside. In my case its called scsi-megaraid-sas_6.610.15.00-1OEM.600.0.0.2494585.vib

Thing is that the driver bundled with ESXi is the so called Native driver Invader only one, and we need to install the so called Legacy driver. This will require an additional step after the installation of the .vib.

The process is pretty straight forward, like any other VIB:
1. Upload the file to the local datastore of ESXI host,
2. Put host to Maintenance mode,
3. Login to the ESXi host using SSH,
4. Execute the following command.

esxcli software vib install -v="/vmfs/volumes/DATASTORENAME/scsi-megaraid-sas_6.610.15.00-1OEM.600.0.0.2494585.vib"

5. After you have done this. you need to disable the old lsi_mr3 driver. To do that, run:

esxcfg-module -d lsi_mr3

6. Now reboot your host.

After reboot you can check if lsi_mr3 driver was replaced by megaraid driver by running:

esxcli storage core adapter list

Installing storcli command line tool on ESXi 6

As next step we will install storcli tool from Avago, this tool will help to configure disks on the controller. It can also be used to update firmware of the controller.

First we need to download it. You can find it on the same Download page on Avago website under Latest MegaRAID StorCLI.

Download the ZIP, unpack it and open the next ZIP inside. What we need is the VIB file from Vmware-NDS folder. In my case its called vmware-esx-storcli-1.19.04.vib.

Now, lets install it:
1. Upload the file to the local datastore of ESXI host,
2. Make sure ESXi is still in Maintenance mode,
3. Login to the ESXi host using SSH,
4. Execute the following command.

esxcli software vib install -v="/vmfs/volumes/DATASTORENAME/vmware-esx-storcli-1.19.04.vib" --no-sig-check

where DATASTORENAME is the name of your datastore.

Now you need to reboot your host to proceed with next steps:

reboot

Updating firmware of MegaRAID SAS 9361-8i on ESXi 6

There are two methods you can use to upgrade your firmware. You can either ask Softlayer to do it for you, or you can do it yourself.

Method 1

To request Softlayer to perform a firmware upgrade you will need to login to SoftLayer Portal, Navigate to server you need Firmware updated on. On the right side of the page you will see an Actions dropdown menu, click it and select Update Firmware.

You will be presented with some options of what to Update.

Select Raid Controller, or if you want you can select all to make sure all your FW is up to date.

This will send a request to Softlayer to perform firmware update, it can take up to one hour, and of course your server will be restarted.

Method 2

Second method is manual. First of all lets download the firmware file first. Once again visit Download page on Avago website in the Firmware section. We need the Latest MegaRAID Firmware. Check the ReadMe file first, it contains the actual version information. Looks for word Firmware in the file. Here is what I have there :

Firmware Package:24.15.0-0016 (MR 6.11)
Firmware 4.650.00-6121

As a next step, we need to check the Firmware version of the controller installed on the server. to do that we will use storcli utility we just installed. Here is the command you can use to do it fast.

[root@esx01:~] /opt/lsi/storcli/storcli /call show |grep FW
FW Package Build = 24.12.0-0020
FW Version = 4.620.00-5026

As you can see, in my case Firmware is not up to date, and we need update it.

Download the Firmware Zip file from Avago website, unpack it and upload to .rom file from inside to the datastore of your ESXi host. In my case the file is called mr3108fw.rom.

Once done, connect to your ESXi host with SSH. Execute the following command to flash the firmware:

/opt/lsi/storcli/storcli /c0 download file=/vmfs/volumes/DATASTORENAME/mr3108fw.rom

where DATASTORENAME is the name of your datastore.

and reboot the host once again:

reboot

Recreating RAID groups using storcli

Now, as I mentioned in Part 1 of these series, the way Softlayer creates RAID groups is not best for using with VSAN. I am sure this issue will be fixed with time, but for now, lets recreate those manually.

First of all we need to check what we have. we can do that using storcli.

/opt/lsi/storcli/storcli /c0 show

Command will show a lot of output, here is part of output i got on my system:

TOPOLOGY :
========

-----------------------------------------------------------------------------
DG Arr Row EID:Slot DID Type  State BT       Size PDC  PI SED DS3  FSpace TR
-----------------------------------------------------------------------------
 0 -   -   -        -   RAID1 Optl  N    931.0 GB dflt N  N   dflt N      N
 0 0   -   -        -   RAID1 Optl  N    931.0 GB dflt N  N   dflt N      N
 0 0   0   8:0      9   DRIVE Onln  N    931.0 GB dflt N  N   dflt -      N
 0 0   1   8:1      11  DRIVE Onln  N    931.0 GB dflt N  N   dflt -      N
 1 -   -   -        -   RAID0 Optl  N    1.818 TB dflt N  N   dflt N      N
 1 0   -   -        -   RAID0 Optl  N    1.818 TB dflt N  N   dflt N      N
 1 0   0   8:3      15  DRIVE Onln  N    1.818 TB dflt N  N   dflt -      N
 2 -   -   -        -   RAID0 Optl  N    1.818 TB dflt N  N   dflt N      N
 2 0   -   -        -   RAID0 Optl  N    1.818 TB dflt N  N   dflt N      N
 2 0   0   8:4      13  DRIVE Onln  N    1.818 TB dflt N  N   dflt -      N
 3 -   -   -        -   RAID0 Optl  N    1.818 TB dflt N  N   dflt N      N
 3 0   -   -        -   RAID0 Optl  N    1.818 TB dflt N  N   dflt N      N
 3 0   0   8:5      14  DRIVE Onln  N    1.818 TB dflt N  N   dflt -      N
 4 -   -   -        -   RAID0 Optl  N  744.687 GB dflt N  N   dflt N      N
 4 0   -   -        -   RAID0 Optl  N  744.687 GB dflt N  N   dflt N      N
 4 0   0   8:2      12  DRIVE Onln  N  744.687 GB dflt N  N   dflt -      N
 5 -   -   -        -   RAID0 Optl  N    1.818 TB dflt N  N   dflt N      N
 5 0   -   -        -   RAID0 Optl  N    1.818 TB dflt N  N   dflt N      N
 5 0   0   8:6      10  DRIVE Onln  N    1.818 TB dflt N  N   dflt -      N
-----------------------------------------------------------------------------


VD LIST :
=======

------------------------------------------------------------------
DG/VD TYPE  State Access Consist Cache Cac sCC       Size Name
------------------------------------------------------------------
0/0   RAID1 Optl  RW     Yes     RWBD  -   ON    931.0 GB RAID1-A
1/1   RAID0 Optl  RW     Yes     RWBD  -   ON    1.818 TB JBOD-C
2/2   RAID0 Optl  RW     Yes     RWBD  -   ON    1.818 TB JBOD-D
3/3   RAID0 Optl  RW     Yes     RWBD  -   ON    1.818 TB JBOD-E
4/4   RAID0 Optl  RW     Yes     NRWBD -   ON  744.687 GB JBOD-B
5/5   RAID0 Optl  RW     Yes     RWBD  -   ON    1.818 TB JBOD-F
------------------------------------------------------------------

PD LIST :
=======

-------------------------------------------------------------------------------
EID:Slt DID State DG       Size Intf Med SED PI SeSz Model                  Sp
-------------------------------------------------------------------------------
8:0       9 Onln   0   931.0 GB SATA HDD N   N  512B ST1000NM0033-9ZM173    U
8:1      11 Onln   0   931.0 GB SATA HDD N   N  512B ST1000NM0033-9ZM173    U
8:2      12 Onln   4 744.687 GB SATA SSD N   N  512B INTEL SSDSC2BA800G4    U
8:3      15 Onln   1   1.818 TB SATA HDD N   N  512B WDC WD2000FYYZ-01UL1B2 U
8:4      13 Onln   2   1.818 TB SATA HDD N   N  512B WDC WD2000FYYZ-01UL1B2 U
8:5      14 Onln   3   1.818 TB SATA HDD N   N  512B WDC WD2000FYYZ-01UL1B2 U
8:6      10 Onln   5   1.818 TB SATA HDD N   N  512B WDC WD2000FYYZ-01UL1B2 U
-------------------------------------------------------------------------------

As you can see in VD LIST, Softlayer created RAID0 per disk and the cache mode which is used is RWBD (read ahead always, write back, direct). we need to recreate those VDs with RWTD (read ahead always, write through, direct) cache mode.

A colleague of mine created a simple script to do this. You can use it, or you can run each command one by one:

#!/bin/sh
/opt/lsi/storcli/storcli /c0/v1 del
/opt/lsi/storcli/storcli /c0/v2 del
/opt/lsi/storcli/storcli /c0/v3 del
/opt/lsi/storcli/storcli /c0/v4 del
/opt/lsi/storcli/storcli /c0/v5 del
/opt/lsi/storcli/storcli /c0 add vd type=raid0 name=VSAN-SSD drive=8:2 nora wt direct strip=256
/opt/lsi/storcli/storcli /c0 add vd type=raid0 name=VSAN drive=8:3 ra wt direct strip=256
/opt/lsi/storcli/storcli /c0 add vd type=raid0 name=VSAN drive=8:4 ra wt direct strip=256
/opt/lsi/storcli/storcli /c0 add vd type=raid0 name=VSAN drive=8:5 ra wt direct strip=256
/opt/lsi/storcli/storcli /c0 add vd type=raid0 name=VSAN drive=8:6 ra wt direct strip=256

Note: Make sure you use correct drive number for disks. you can see those in PD LIST part of the output we saw before.

Once done run the /opt/lsi/storcli/storcli /c0 show to compare the output. Here is how it looks for me after executing the script:

VD LIST :
=======

-------------------------------------------------------------------
DG/VD TYPE  State Access Consist Cache Cac sCC       Size Name
-------------------------------------------------------------------
0/0   RAID1 Optl  RW     Yes     RWBD  -   ON    931.0 GB RAID1-A
1/1   RAID0 Optl  RW     Yes     NRWTD -   ON  744.687 GB VSAN-SSD
2/2   RAID0 Optl  RW     Yes     RWTD  -   ON    1.818 TB VSAN
3/3   RAID0 Optl  RW     Yes     RWTD  -   ON    1.818 TB VSAN
4/4   RAID0 Optl  RW     Yes     RWTD  -   ON    1.818 TB VSAN
5/5   RAID0 Optl  RW     Yes     RWTD  -   ON    1.818 TB VSAN
-------------------------------------------------------------------

Now we are ready to proceed with the remaining of the configuration.

Configuring Multicast for VSAN on Softlayer

Note: the following config should be done after VMkernel ports  for VSAN are already created.

We all know that VSAN requires multicast and there is a default configuration set by VMware. The t Default configuration are not supported in Softlayer. Softlayer supports the following multicast range 224.0.0.115-224.0.0.250 in their private network. So to work with this we need to change the multicast IPs used by VSAN.

Note: Softlayer multicast range 224.0.0.115-224.0.0.250 is per VLAN, which allows using of same address range in different VLANs. This also means that if you will configure another VSAN cluster in same Private VLAN, multicast addresses will need to be changed.

The procedure  is quite simple and is described in VMware KB2075451

But to make it faster here is the command which you can run:

esxcli vsan network ipv4 set -i VMK -d 224.0.0.220 -u 224.0.0.221

where VMK is the name of vmkernel port used by VSAN( for example vmk1 or vmk2). Run it on each ESXi.

To check if it’s OK, run:

esxcli vsan network list

That’s it for the controller and multicast part, now we are ready to login to vCenter and proceed with remaining configuration.

Configuring VSAN on vSphere 6 running on Softlayer.

On vSphere level, there is not much difference between configuring VSAN on Softlayer and configuring VSAN on whatever else.

Here are some Key point you need to remeber:
1. Create Separate Vmkernel portgroup for VSAN, make sure Virtual SAN traffic is selected.

2. In case your SSD is not identified by SSD in vSphere, select it in Storage Devices view and click the magic F icon.

The remaining configuration is as usual.

Small Update:

VMware recommends implementing the following KB, so take a look

That’s it for this series, I hope you enjoyed it, feel free to leave any comments or questions.

The following two tabs change content below.

• Bio
• Latest Posts

Aram Avetisyan

Aram Avetisyan is an IT specialist with more than 18 years experience. He has rich background in various IT related fields like Cloud, Virtualization and SDN. He holds several industry level certifications including but not limited to VCIX-DCV, VCIX-NV. He is also a vEXPERT in years 2014-2021.

Latest posts by Aram Avetisyan (see all)

• Make Youtube Videos About Technology? Why not… The Cross-Cloud Guy - October 7, 2021
• Automating (NSX-T) REST API using Ansible URI module - December 29, 2020
• Quick Reference: Create Security Policy with Firewall Rules using NSX-T Policy API - May 4, 2020