ESXi USB PCIe Passthrough Failing to Connect Connected Devices

passthroughvmware-esxi

I have a fresh install of ESXi 6.7U2 up and running on which I can successfully start VMs. To this setup I've added a StarTech.com 4 Port USB 3.0 Card Adapter (PEXUSB3S44V). This has been configured for pass through following https://kb.vmware.com/s/article/1010789. This can be configured for passthrough to a VM:

VM Configuration

The USB PCIe controllers are detected here as uPD720202 USB 3.0 Host Controller. A single PCIe adaptor has 4 independent controllers and I've chosen to pass then all through for simplicity while trying to debug this issue.

I have tried using both Amazon Linux 2 (on prem) and Ubuntu Server 18.04.2 for the VMs, what follows appears to be equivalent in both of them.

lspci output shows the expected pass through devices:

00:00.0 Host bridge: Intel Corporation 440BX/ZX/DX - 82443BX/ZX/DX Host bridge (rev 01)
00:01.0 PCI bridge: Intel Corporation 440BX/ZX/DX - 82443BX/ZX/DX AGP bridge (rev 01)
00:07.0 ISA bridge: Intel Corporation 82371AB/EB/MB PIIX4 ISA (rev 08)
00:07.1 IDE interface: Intel Corporation 82371AB/EB/MB PIIX4 IDE (rev 01)
00:07.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08)
00:07.7 System peripheral: VMware Virtual Machine Communication Interface (rev 10)
00:0f.0 VGA compatible controller: VMware SVGA II Adapter
00:10.0 SCSI storage controller: LSI Logic / Symbios Logic 53c1030 PCI-X Fusion-MPT Dual Ultra320 SCSI (rev 01)
00:11.0 PCI bridge: VMware PCI bridge (rev 02)
00:15.0 PCI bridge: VMware PCI Express Root Port (rev 01)
00:15.1 PCI bridge: VMware PCI Express Root Port (rev 01)
00:15.2 PCI bridge: VMware PCI Express Root Port (rev 01)
00:15.3 PCI bridge: VMware PCI Express Root Port (rev 01)
00:15.4 PCI bridge: VMware PCI Express Root Port (rev 01)
00:15.5 PCI bridge: VMware PCI Express Root Port (rev 01)
00:15.6 PCI bridge: VMware PCI Express Root Port (rev 01)
00:15.7 PCI bridge: VMware PCI Express Root Port (rev 01)
00:16.0 PCI bridge: VMware PCI Express Root Port (rev 01)
00:16.1 PCI bridge: VMware PCI Express Root Port (rev 01)
00:16.2 PCI bridge: VMware PCI Express Root Port (rev 01)
00:16.3 PCI bridge: VMware PCI Express Root Port (rev 01)
00:16.4 PCI bridge: VMware PCI Express Root Port (rev 01)
00:16.5 PCI bridge: VMware PCI Express Root Port (rev 01)
00:16.6 PCI bridge: VMware PCI Express Root Port (rev 01)
00:16.7 PCI bridge: VMware PCI Express Root Port (rev 01)
00:17.0 PCI bridge: VMware PCI Express Root Port (rev 01)
00:17.1 PCI bridge: VMware PCI Express Root Port (rev 01)
00:17.2 PCI bridge: VMware PCI Express Root Port (rev 01)
00:17.3 PCI bridge: VMware PCI Express Root Port (rev 01)
00:17.4 PCI bridge: VMware PCI Express Root Port (rev 01)
00:17.5 PCI bridge: VMware PCI Express Root Port (rev 01)
00:17.6 PCI bridge: VMware PCI Express Root Port (rev 01)
00:17.7 PCI bridge: VMware PCI Express Root Port (rev 01)
00:18.0 PCI bridge: VMware PCI Express Root Port (rev 01)
00:18.1 PCI bridge: VMware PCI Express Root Port (rev 01)
00:18.2 PCI bridge: VMware PCI Express Root Port (rev 01)
00:18.3 PCI bridge: VMware PCI Express Root Port (rev 01)
00:18.4 PCI bridge: VMware PCI Express Root Port (rev 01)
00:18.5 PCI bridge: VMware PCI Express Root Port (rev 01)
00:18.6 PCI bridge: VMware PCI Express Root Port (rev 01)
00:18.7 PCI bridge: VMware PCI Express Root Port (rev 01)
02:01.0 SATA controller: VMware SATA AHCI controller
03:00.0 USB controller: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller (rev 02)
04:00.0 USB controller: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller (rev 02)
0b:00.0 Ethernet controller: VMware VMXNET3 Ethernet Controller (rev 01)
13:00.0 USB controller: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller (rev 02)
1b:00.0 USB controller: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller (rev 02)

lsusb output before connecting a device to the adaptor:

Bus 008 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 007 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 006 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 005 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

Viewing dmesg output when trying to connect a device, depending on device (USB hub, USB data drive etc.), I either get no additional output or:

[  818.213414] xhci_hcd 0000:04:00.0: Abort failed to stop command ring: -110
[  818.213435] xhci_hcd 0000:04:00.0: xHCI host controller not responding, assume dead
[  818.213453] xhci_hcd 0000:04:00.0: HC died; cleaning up
[  818.213467] xhci_hcd 0000:04:00.0: WARNING: Host System Error
[  818.213497] xhci_hcd 0000:04:00.0: Error while assigning device slot ID
[  818.213511] xhci_hcd 0000:04:00.0: Max number of devices this xHCI host supports is 32.
[  818.213526] usb usb4-port1: couldn't allocate usb_device
[  818.213538] xhci_hcd 0000:04:00.0: Error while assigning device slot ID
[  818.213551] xhci_hcd 0000:04:00.0: Max number of devices this xHCI host supports is 32.
[  818.213565] usb usb3-port1: couldn't allocate usb_device

Clearly a device is being detected in the VM but something is preventing it from connecting successfully.

Things I tried (Ref https://bbs.archlinux.org/viewtopic.php?id=236536):

  1. Setting the kernel parameters iommu=soft pci=nomsi
  2. Setting the kernel parameter intel_iommu=off
  3. Different VMs (as mentioned above)
  4. Ensuring the USB devices are connected on VM boot
  5. Setting the ESXi host power management to 'High performance'

To confirm I wasn't dealing with a hardware/power issue I ditched the hypervisor on the same hardware by performing a boot from USB (LinuxMint). Here the PCIe adaptor worked correctly with all connected devices so I think that can be ruled out.

At the moment my suspicions are:

  1. This uPD720202 chipset isn't compatible with ESXi, even though many posts exist online suggesting it should work fine (which is why I bought it).
  2. I'm missing some critical configuration within the ESIx host.

If anyone spots anything obvious that I could try, or would like additional output that would be greatly appreciated.

Best Answer

Looks like it was option 1, hardware compatibility.

I swapped the StarTech.com 4 Port USB 3.0 Card Adapter (PEXUSB3S44V) for a 4 Port HighPoint RocketU RU1144D which now works as expected.

Related Topic