These past weeks, we have been pretty active in VSAN deployments on HP DL380 24SFF Gen9 based servers. Our HBA of choice is the P440ar and P440 (the P440ar being the embedded module and the P440’s being additional PCI adapters).
We choose to custom build VSAN nodes that specifically fit the designs we do for customers. One of our best practices is to use one HBA per group of 8 disks. In most cases, our custom nodes have between 8-16 disks per node so we typically use 2 HBAs (one embedded P440ar and an additional PCI P440). For customers who want room for future expansion, we actually add a third HBA ready to go, the customer would simply have to add disks to the disk bays.
The HP DL380 24FF Gen9 has 3 bays of 8SFF. Each HBA is wired to one 8SFF bay controlling 8 disks. We create one VSAN disk group per HBA (so we can create 3 disk groups of 8 disks per node).
These HBAs are supported in Pass-thru mode only.
Setting up VSAN is fairly simple, but when you start having ten’s of servers with 2-3 disk groups per host with each 8 disks, disk group creation can be a hassle.
Naturally, the first reaction is to let VSAN manage the disk groups by changing the setting “Add disks to storage” option from Manual to Automatic. This actually works, VSAN will automatically create disk groups based on the number of SSDs and HDDs (1 SSD per disk group).
The side effect with this automatic configuration is that VSAN will not necessarily pick the SDDs and HDDs on the same HBA. So if an HBA would fail, we would loose one SSD (thus one complete disk group) and HDDs from possibly other disk groups. In our case, we want disk groups to be aligned with specific HBAs. We want the SSD and HDDs from the same HBA to form a disk group.
So back to manual mode for the “Add disks to storage” setting (we created a script that scans all disks and HBAs and automatically creates the disk groups aligned to the HBAs).
Another issue with so many servers to build is that we need to change the mode of the HBAs from RAID to HBA Mode (this requires booting to Storage Array Utilities and changing the mode). This must be done for all servers serving storage. By default the HBAs are in RAID mode (not supported with these HBAs for VSAN).
We did find an easier way to make these changes to the HBAs. When installing ESXi with the HP Custom ISO, the hpssacli utility can be used to make changes to the HBA modes.
So we simply installed ESXi on all hosts, then simply executed the following command to change the HBA mode :
esxcli hpssacli cmd -q “controller slot=0 modify hbamode=on forced”
esxcli hpssacli cmd -q “controller slot=1 modify hbamode=on forced”
esxcli hpssacli cmd -q “controller slot=2 modify hbamode=on forced”
Where slot=x = the adapter instance. slot=0 = embedded HBA, slot=1 = first PCI HBA etc..
A reboot is required after these changes are made. Once rebooted, you can validate the HBA mode with the following command (look for HBA Mode)
esxcli hpssacli cmd -q “controller all show config”
A very important detail about the P440 and P440ar HBAs is that they do not support locator LED flashing when in HBA mode(Pass-thru). This means that you cannot locate a disk in one of the bays by clicking the “Turn On locator LED” option
Hope this helps anyone using the HP DL380 and P440 HBAs.
This Combo (VSAN + HP DL380 Gen9 servers) rocks!