VCDX #181 Marc Huppert

Stage XXXV: vSAN Expanding

After finishing the last Stage XXXIV: Site Recovery Manager, now it is time to write a little bit more details about my vSAN clusters….

I started using VMware vSAN very early, with the 5.5U1 release at customer sites and at home. Over the time the number of vSAN enabled clusters increased.

I have several SSDs from different vendors here ins use, what to choose for what kind of workload?

This explains the following list a little bit more:

Drive Capacity and Speed, MB/s Read, MB/s Write, IOPs Read, IOPs Write

SAS SSDs

  • 7,6 TB Samsung SAS 12G, 1400MB R, 1300MB W, 195k R, 31k W
  • 3,8 TB Toshiba SAS 12G, 2100MB R, 2100MB W, 370k R, 55k W
  • 2 TB SanDisk SAS 6G, 600MB R, 530MB W, 90k R, 35k W
  • 1,92 TB Toshiba SAS 12G, 1800MB R, 850MB W, 270k R, 25k W
  • 900 GB HGST SAS 12G, 1100MB R, 765MB W, 130k R, 100k W
  • 800 GB Samsung SAS 12G, 1000MB R, 1000MB W, 230k R, 70k W
  • 400 GB Toshiba SAS 12G, 1050MB R, 1050MB W, 200k R, 60k W
  • 200 GB Toshiba SAS 12G, 900MB R, 400MB W, 120k R, 25K W

NVMe Flash

  • 800 GB Intel P3700 NVMe, 2800MB R, 1900MB W, 460k R, 90k W

Fusion IO Flash

  • 365GB FusionIO ioDrive2, 3000MB R, 2500MB W, 480k R, 490k W

SATA SSDs

  • 1 TB SanDisk X400 SATA, 545MB R, 520MB W, 95k R, 75k W
  • 960 GB Intel S4500 SATA, 500MB R, 490MB W, 74k R, 36k W
  • 400 GB Intel S3700 SATA, 500MB R, 460MB W, 75k R, 36k W
  • 256 GB Samsung 840 Evo, 540MB R, 520MB W, 97k R, 67k W
  • 240 GB Micro 5100 M2 SATA, 540MB R, 520MB W, 78k R, 26K W
  • 200 GB IBM SATA, 150MB R, 35MB W, 20k R, 5k W

USB Flash

  • 16 GB SanDisk USB 2.0, 15MB R, 4MB W

vSAN Clusters

Here is a list of my 8 active clusters with the detailed vSAN resources and the SSD placement:

Apple Mac Mini (3 Nodes) All Flash with SATA/SATA

Storage Configuration per Node

  • 1 x 400GB Intel S3700 SATA SSDs for Caching
    • 0,5GB/s Read, 0,46GB/s Write, 75k IOPs Read, 36k IOPs Write
  • 1 x 256GB Samsung SATA III SSD for Capacity
    • 0,54GB/s Read, 0,52GB/s Write, 97k IOPs Read, 67k IOPs Wr.
  • 1 x 16GB SanDisk USB (ESXi)
    • 0,015GB/s Read, 0,004GB/s Write
  • vSAN Encrypted

Dell VRTX/M640 (4 Nodes) All Flash with SAS/SAS

Storage Configuration per Node

  • 1 x 800GB Samsung SAS 12G SSD for Caching Tier
    • 1GB/s Read, 1GB/s Write, 230k IOPs Read, 70k IOPs Write
  • 1 x 7,6TB Samsung SAS 12G Read Intensive for Capacity
    • 1,4GB/s Read, 1,3GB/s Write, 195k IOPs Read, 31k IOPs Write
  • 2 x 240GB M2 Micron Raid1 BOSS Device (ESXi)
    • 0,54GB/s Read, 0,52GB/s Write, 78k IOPs Read, 26k IOPs Wr.

Storage Configuration per Chassis

  • 8 x 3,8TB Toshiba SAS 12G Raid5 Non-vSAN
    • 2,1GB/s Read, 2,1GB/s Write, 370k IOPs Read, 55k IOPs Write
  • 4 x 900GB HGST SAS 12G Raid0 Non-vSAN
    • 1,1GB/s Read, 0,76GB/s Write, 130k IOPs Read, 100k IOPs Wr.

Fujitsu RX2540M2 (2 Nodes) All Flash with NVMe/SAS

Storage Configuration per Node

  • 1 x 800GB Intel P3700 NVMe for Caching
    • 2,8GB/s Read, 1,9GB/s Write, 460k IOPs Read, 90k IOPs Write
  • 4 x 2TB SanDisk SAS 6G SSD for Capacity
    • 0,6GB/s Read, 0,53GB/s Write, 90k IOPs Read, 35k IOPs Write
  • 1 x 200GB IBM SATA (ESXi)
    • 0,15GB/s Read, 0,03GB/s Write, 20k IOPs Read, 5k IOPs Write

IBM x3550M3 (Single Node) All Flash with FusionIO/SATA

Storage Configuration per Node

  • 1 x 365GB FusionIO ioDrive 2 for Caching
    • 3GB/s Read, 2,5GB/s Write, 480k IOPs Read, 490k IOPs Write
  • 2 x 1TB SanDisk X400 SATA SSD for Capacity
    • 0,54GB/s Read, 0,52GB/s Write, 95k IOPs Read, 75k IOPs Wr.
  • 1 x 16GB SanDisk USB (ESXi)
    • 0,015GB/s Read, 0,004GB/s Write

IBM x3650M3 (4 Nodes) Hybrid with FusionIO/HDD

Storage Configuration per Node

  • 1 x 365GB Fusion IO ioDrive2 for Caching
    • 3GB/s Read, 2,5GB/s Write, 480k IOPs Read, 490k IOPs Write
  • 5 x 146GB HGST SAS 10K HDD for Capacity
  • 1 x 16GB SanDisk USB (ESXi)
    • 0,015GB/s Read, 0,004GB/s Write

IBM x3650M4 (2 Nodes) All Flash with SATA/SATA

Storage Configuration per Node

  • 1 x 960GB Intel S4500 SATA SSD for Caching
    • 0,5GB/s Read, 0,49GB/s Write, 74k IOPs Read, 36k IOPs Write
  • 1 x 1TB Sandisk X400 SATA SSD for Capacity
    • 0,54GB/s Read, 0,52GB/s Write, 95k IOPs Read, 75k IOPs Wr.
  • 1 x 16GB SanDisk USB (ESXi)
    • 0,015GB/s Read, 0,004GB/s Write

IBM x3690X5 (2 Nodes) Hybrid with SATA/HDD

Storage Configuration per Node

  • 1 x 960GB Intel S4500 SATA SSD for Caching
    • 0,5GB/s Read, 0,49GB/s Write, 74k IOPs Read, 36k IOPs Write
  • 5 x 300GB HGST SAS 10K for Capacity
  • 1 x 16GB SanDisk USB (ESXi)
    • 0,015GB/s Read, 0,004GB/s Write
  • vSAN Encrypted

Stretched Cluster (30 Nodes) All Flash with NVMe/SAS

Storage Configuration per Node

  • 1 x 4GB Virtual NVMe for Caching
  • 1 x 10GB Virtual SAS for Capacity
  • 1 x 16GB Virtual SAS (ESXi)
  • Nested Environment

Yes, you read correctly: I have 8 different vSAN clusters running inside my HomeLab…. Captain vSAN is proud of me!

HomeLab vSAN Cluster Overview
My “I am Captain vSAN” flag hanging inside my HomeOffice

I hope you enjoy my HomeLab / HomeDC journey so far.

Here is the next episode: Stage XXXVI: NVIDIA GPUs Everywhere