VCDX #181 Marc Huppert

Stage XXXVI: Nvidia GPUs Everywhere

In the last episode I have written about Stage XXXV: vSAN Expanding

After working and playing some time with the Nvidia GPUs inside my environment it is time to expand…..

Here a quick drawing from one of my Horizon Trainings showing the Nvidia vGPU elements:

I already had some P4 running inside my HomeLab. Check out the Stage XXV: New Nvidia GPUs.

I increased the number of P4 to 6 and placed them inside my servers and I was able to get 3 x T4 as well…

That T4 is a Monster……

Here is the placement of my Nvidia GPUs inside my systems:

1 x P4 in x3650M4 Cluster

1 x T4 in x3650M4 Cluster

3 x P4 in VRTX Cluster

2 x T4 in RX2540M2 Cluster

2 x P4 in DL160 Cluster

When reviewing my HomeLab journey you will find information about nearly all of these servers, except the mentioned DL160 Cluster. Where is this cluster?

That cluster is running at my friend and co-workers HomeLab @DennisGoedde

Check out his blog under: https://dennisgoedde.blog

We configured the HPE DL160Gen8 Server in the following way

vSAN ROBO Cluster with Dual Nvidia P4 and 10GbE for vSAN and vMotion

As you can see, every system running in our HomeLabs have Nvidia GPUs configured. I really don’t want to run a VDI or Terminalserver environment without them.

Installation is very easy:

  • 2 x License Server
  • GPUs inside an ESXi Host
  • Nvidia vGPU Hypervisor driver
  • Advanced ESXi settings (for GPU support)
  • Advanced vCenter Server settings (for GPU vMotion)
  • vGPU Profile for VMs
  • Windows / Linux driver inside VMs
  • GPO Configuration for GPU settings
  • Tuning options

Step by Step:

Licensing

Diagram showing how NVIDIA vGPU software license files are downloaded from the NVIDIA Licensing Portal to the license server and how licensed clients borrow licenses from the server.

License Server Requirements:

  • The hosting platform may be a physical machine, an on-premises virtual machine (VM), or a VM on a supported cloud service. NVIDIA recommends using a host that is dedicated solely to running the license server.
  • The recommended minimum configuration is 2 CPU cores and 4 Gbytes of RAM. A high-end configuration of 4 or more CPU cores with 16 Gbytes of RAM is suitable for handling up to 150,000 licensed clients. 
  • At least 1 Gbyte of hard drive space is required.
  • The hosting platform must run a supported operating system.
  • On Window platforms, .NET Framework 4.5 or later is required.
  • TCP Ports 7070 and 8080
  • Java……. (I always choose OpenJDK)

Setup the VM, install OS, Patch it and configure OpenJDK

Screen capture showing the OpenJDK setup wizard.

Set the JAVA_HOME system variable to the full path to the jre folder of your JRE installation.

Ensure that the Path system variable contains the path to the java.exe executable file.

Screen capture showing license agreements for the license server on Windows.
Install License Server with Default Settings
Screen capture showing the home page for the license server management interface.

Add your license key to both license servers. How to create a license file? Check out the article from Dennis: https://dennisgoedde.blog/category/nvidia/

License Server HA Configuration:

Screen capture of the License Server Configuration page showing settings for enabling failover support.

Under License Generation, use the Backup URI and Main URI fields to set the URI to each server in a failover configuration.

  1. In the Backup field, enter http://secondary-server:7070/fne/bin/capability.
  2. In the Main field, enter http://primary-server:7070/fne/bin/capability.
  3. Set the options under Settings for server to sync between License servers as follows:Set the Synchronization to fne enabled option to true.
  4. Main FNE Server field, enter http://primary-server:7070/fne/bin/capability.

Hypervisor driver

Download the appropriate driver for your ESXi version from the Nvidia Website:

Install the vGPU driver via VMware Update Manager (VUM) or via CLI (esxcli) onto each of your ESXi hosts. Don’t forget the maintenance mode and the reboot….

Advanced ESXi Settings

Configure the following ESXi settings:

Advanced vCenter Server Settings

Configure vCenter GPU vMotion:

Configure vGPU Profile for VMs

Choose the right vGPU profile for your GPU. Here is an example of my P4 and T4 cards:

Configure your VMs with a PCI device and choose the profile. (Make sure you have enough licenses for the profile on your license server!)

Nvidia vGPU Driver Installation

Install the driver inside the VM and always choose the option “New Installation”

Reboot the VM after successful driver installation. The VMware Remote Console does no longer work from now!!! Make sure have enable a Remote Option…..

License your VM to use the Nvidia vGPU technology, without that step you are limited to 3 FPS!!!!!

Screen capture showing the Manage License option in NVIDIA Control Panel for a vGPU license

I really love the Power of NVIDIA GPUs inside my VMs.

Looking for the next episode?

Stage XXXVII: VeloCloud