HomeLab Stage L: 100GbE

This is my #HomeLab Blog Post number 50. Amazing to see the journey so far….

In the last episode I mentioned the awesome integrations from Dell into the VMware cosmos. Now it is time for a huge announcement:

I have successfully implemented 100GbE network inside my HomeLab environment. Yes you read it correct, 100 Gigabit at Home 🙂

Why? Because I wanted it… and I would like to max out my vMotion network.

After 1GbE, 10GbE and 40GbE

I am using my Ubiquiti network stuff since several years now and I am still very happy with it. Gigabit and 10GbE are working fine, the access points running without any problems. I really like their solution, but what about high performance when you need more than 10GbE?

In the past I invested into one Dell S6000 switch, which has 32 ports that supports 40GbE. That was a big improvment and I used it mainly for the vSAN and vMotion traffic.

Do you know that feeling when you want more????

It´s time for 100GbE

My entire network runs totally fine, so why change it? I really wanted to create the ultimate HomeLab…

Server side:

Starting point: 4 x Mellanox ConnectX-5 Dual Port 100GbE PCIe 3.0 x16

Bildergebnis für mellanox connectx-5

My new main vSAN cluster is equipped with 4 x Dell XR2 servers. Each server has 2 x PCIe x16 slots. One slot is used by my Nvidia T4 GPUs and the other one with these dual port NICs.

I ordered 8 x Mellanox QSFP28 cables supporting 100GbE for the servers.

Bildergebnis für mellanox 100gbe cables

Network side:

After some research I found my 100GbE canidate: Mellanox (Nvidia Networking) SN-2100 Switch. This tiny little beast is absolute amazing!

16 x 100GbE ports that can be splitted (later more on that topic)

Bildergebnis für sn2100
Tiny little 100GbE switch

Power consumption of the Mellanox SN2100 is 86 watts running in “production”. I also ordered the Rackmount Kit:

Bildergebnis für sn2100
My switch uses one slot of the rackmount kit. Enough space for future enhancements


The Mellanox switch got a Web UI and of course an CLI access. My switch is operating on the Mellanox Onyx Operating system.

The SN2100 is equipped with a redundant power supply

I really like the UI, but for several tasks you need the CLI

Most of my ports are connected via 100GbE, some with 40GbE and some are splitted to multiple 10GbE

QSFP-Split Mode:

For the switch uplink and some 10GbE ESXi connections I have configured some of the ports into split-mode for Breakout cables. This can be done via CLI on the port itself. First shutdown the port, enter module-type qsfp-split-4 and confirm. Now you have splitted your 100GbE port into 4 smaller one. They can operate at 1, 10 or 25GbE. Very flexible….

Bildergebnis für mellanox qsfp-split-4
I ordered some Mellanox compatible split cables from FS.com

Jumbo Frames:

I have enabled Jumbo Frames for several reasons: vMotion, vSAN and of course: NSX-T

Port information and configuration


I have configured an LACP portchannel as an uplink to my Ubiquiti US-16XG switch. Very easy and simple to configure:

LACP with various options


I have configured all my required VLANs on the Mellanox SN2100 switch and setup the ports:

Straight forward configuration


Main routing is done on the Ubiquiti USG-XG-8 machine, NSX-T TEP and NSX-T Edge TEP routing is done on the Mellanox SN2100. Why? Because Ubiquiti is not able to route Jumbo Frames between VLANs…..

VLAN24 and 25 are used within my NSX-T environment

Using 100GbE

The new Dell XR2 cluster is attached to the SN2100 via Dual 10GbE and Dual 100GbE connections.

Dual 10GbE and 100GbE at each server. Very cool

My new Monster Workstation is attached via 40GbE, also my Custom All Flash NAS system. I love that powerful performance inside my network.

vSAN is using the 100GbE vDS portgroup without any modification.

Each ESXi service is using only one vmkernel adapter. No Multi-NIC vMotion for example

vMotion is tuned to also support the 100GbE connection via 6 streams. This is done via an advanced setting inside ESXi:

A single vMotion stream has an average bandwidth utilization capability of 15 GbE. When we look at various NIC speeds, that lead to the following:
25 GbE : 1 stream = ~15 GbE
40 GbE : 2 streams = ~30 GbE
50 GbE : 3 streams = ~45 GbE
100 GbE : 6 streams = ~90 GbE
That means that you would need six vMotion VMkernel interfaces on a single ESXi host to be able to use the available bandwidth, using a 100GbE NIC, efficiently.
Reboot ESXi host after modifying these values!

The Dell S6000 switch is now only for test enviroments and setups. I can control each of my devices at both datacenters via IP Power / PDU.

Dynamic Routing

My next steps are the implementation of dynamic routing protocols around my entire network, including Mellanox, Ubiquiti and Velocloud. I had OSPF running in the past mainly for NSX-V. Maybe I will implement BGP or once again OSPF. We will see….

Mellanox supports both:


Ubiquiti´s USG-XG-8 can also do both (needs to be configured via CLI)

And finally my Veloclouds also supports both:

I wanted to create the ultimate HomeLab / HomeDC. What do you think?

But I am not finished, I think I will never be finished….. So stay tuned for the next episodes…..

Here is the next one: HomeLab Stage LI: Datacenter Resizing