Stage XXVI: vSAN with NVMe

After finishing the previous Stage XXV: New Nvidia GPUs for my HomeLab it was time to update my primary vSAN All Flash again…..

As I described in Stage XXI: vSAN 6.6 All Flash with 40GbE the “bottleneck” within my All Flash configuration seems to be the Fusion IO cards. I upgraded vSAN to 6.6.1 and decided to replace them with the latest enterprise stuff that is available:

Intel DC P3700 NVMe PCIe 800GB….

NVMe_P3700

Here are the specifications:

NVMe_Specs

Nice numbers….. The 1,6 TB model can achieve 150.000 IOPs write… Mine “only” 90.000.

The device is fully supported for vSAN All Flash caching….. I am using a Lenovo branded card.

NVMe_HCL2

My Fusion IO ioDrive2 Dual 365GB configuration results in 2 disk groups, each containing 2 x 2TB SanDisk Enterprise SSDs.

vSAN66_Diskgroups

I placed one host of my 2 Node cluster into maintenance mode and deleted both disk groups. After that, I shut down the server and replaced both Fusion IO cards with one new P3700 NVMe device.

Now I created one new disk group containing 4 x 2TB SanDisk 2TB SSDs…..

NVMe_DG_After

NVMe_DG_AfterDetail

The resync of the data from the other cluster node took several hours….

I repeated the same procedure at the other node….

Everything is up and running again and fully supported…..

NVMe_Health

I have customer installations where they reached a Dedupe and Compression factor of 4-5. (They all use 7 capacity SSDs, not 4 like I did at home)

NVMe_Dedupe

NVMe_Capacity

The performance of my primary vSAN All Flash is absolutely amazing. I have also 2 other vSAN cluster running at my two datacenter locations within my house. You can check the status here: HomeLab Actual State

My vSAN is now up to date, the datacenter Infrastructure upgraded and the datacenter network operating at 10/40 GbE.

What´s next?

The non-datacenter network needs an upgrade……

Check my next HomeLab episode: Stage XXVII: Ubiquiti Network Upgrade