In the last episode I have written about Stage XXXV: vSAN Expanding
After working and playing some time with the Nvidia GPUs inside my environment it is time to expand…..
Here a quick drawing from one of my Horizon Trainings showing the Nvidia vGPU elements:
I already had some P4 running inside my HomeLab. Check out the Stage XXV: New Nvidia GPUs.
I increased the number of P4 to 6 and placed them inside my servers and I was able to get 3 x T4 as well…
That T4 is a Monster……
Here is the placement of my Nvidia GPUs inside my systems:
1 x P4 in x3650M4 Cluster
1 x T4 in x3650M4 Cluster
3 x P4 in VRTX Cluster
2 x T4 in RX2540M2 Cluster
2 x P4 in DL160 Cluster
When reviewing my HomeLab journey you will find information about nearly all of these servers, except the mentioned DL160 Cluster. Where is this cluster?
That cluster is running at my friend and co-workers HomeLab @DennisGoedde
Check out his blog under: https://dennisgoedde.blog
We configured the HPE DL160Gen8 Server in the following way
As you can see, every system running in our HomeLabs have Nvidia GPUs configured. I really don’t want to run a VDI or Terminalserver environment without them.
Installation is very easy:
- 2 x License Server
- GPUs inside an ESXi Host
- Nvidia vGPU Hypervisor driver
- Advanced ESXi settings (for GPU support)
- Advanced vCenter Server settings (for GPU vMotion)
- vGPU Profile for VMs
- Windows / Linux driver inside VMs
- GPO Configuration for GPU settings
- Tuning options
Step by Step:
License Server Requirements:
- The hosting platform may be a physical machine, an on-premises virtual machine (VM), or a VM on a supported cloud service. NVIDIA recommends using a host that is dedicated solely to running the license server.
- The recommended minimum configuration is 2 CPU cores and 4 Gbytes of RAM. A high-end configuration of 4 or more CPU cores with 16 Gbytes of RAM is suitable for handling up to 150,000 licensed clients.
- At least 1 Gbyte of hard drive space is required.
- The hosting platform must run a supported operating system.
- On Window platforms, .NET Framework 4.5 or later is required.
- TCP Ports 7070 and 8080
- Java……. (I always choose OpenJDK)
Setup the VM, install OS, Patch it and configure OpenJDK
Set the JAVA_HOME system variable to the full path to the jre folder of your JRE installation.
Ensure that the Path system variable contains the path to the java.exe executable file.
Add your license key to both license servers. How to create a license file? Check out the article from Dennis: https://dennisgoedde.blog/category/nvidia/
License Server HA Configuration:
Under License Generation, use the Backup URI and Main URI fields to set the URI to each server in a failover configuration.
- In the Backup field, enter http://secondary-server:7070/fne/bin/capability.
- In the Main field, enter http://primary-server:7070/fne/bin/capability.
- Set the options under Settings for server to sync between License servers as follows:Set the Synchronization to fne enabled option to true.
- Main FNE Server field, enter http://primary-server:7070/fne/bin/capability.
Download the appropriate driver for your ESXi version from the Nvidia Website:
Install the vGPU driver via VMware Update Manager (VUM) or via CLI (esxcli) onto each of your ESXi hosts. Don’t forget the maintenance mode and the reboot….
Advanced ESXi Settings
Configure the following ESXi settings:
Advanced vCenter Server Settings
Configure vCenter GPU vMotion:
Configure vGPU Profile for VMs
Choose the right vGPU profile for your GPU. Here is an example of my P4 and T4 cards:
Configure your VMs with a PCI device and choose the profile. (Make sure you have enough licenses for the profile on your license server!)
Nvidia vGPU Driver Installation
Install the driver inside the VM and always choose the option “New Installation”
Reboot the VM after successful driver installation. The VMware Remote Console does no longer work from now!!! Make sure have enable a Remote Option…..
License your VM to use the Nvidia vGPU technology, without that step you are limited to 3 FPS!!!!!
I really love the Power of NVIDIA GPUs inside my VMs.
Looking for the next episode?