| Bug ID | 106111 |
|---|---|
| Summary | [GPU Passthrough]GPU (Polaris) not reinitialized with Linux V= M (Reset bug) |
| Product | DRI |
| Version | unspecified |
| Hardware | x86-64 (AMD64) |
| OS | All |
| Status | NEW |
| Severity | normal |
| Priority | medium |
| Component | DRM/AMDgpu |
| Assignee | dri-devel@lists.freedesktop.org |
| Reporter | berillions@gmail.com |
| CC | alexdeucher@gmail.com |
Created attachment 138887 [deta=
ils]
xorg.conf
Hi,
My Setup :
- AMD Ryzen 1600
- 16 Gb Memory RAM
- Host (Debian Stable, kernel 4.16.2) : AMD Rx560 4Gb
- Guest (Windows 10 / Archlinux Kernel 4.15.x-4.16.x) : AMD Rx580 - 8Gb
Years ago there was an issue on Windows virtual machine with Qemu/VFIO and =
AMD
GPU. It was impossible to reboot or use a 2nde time the Guest because the G=
PU
was not reinitialized when the Host was shutdown. The only solution to re-u=
se
the VM was to reboot the Host OR use a Nvidia GPU.
Actually, the issue is fixed on Windows VM + AMD GPU passed through (i don't
know how), i can use more times my VM without reboot the Host.=20
But if i use my Linux VM with my Rx580, the issue still exist. The first la=
unch
works, i can use the Rx580 to play without problem. But if i shutdown/reboot
the guest, the Rx580 is "blocked". I need to hard reboot because =
the system
hangs after ~2-3 minutes.
Thanks for your help,
Maxime=20
(Sorry for my English, i'm French)
Created attachment 138888 [=
details]
dmesg output after to launch the VM a second time
The IOMMU looks to be unhappy first: [ 40.201258] vfio_ecap_init: 0000:0a:00.0 hiding ecap 0x19@0x270 [ 40.201271] vfio_ecap_init: 0000:0a:00.0 hiding ecap 0x1b@0x2d0 [ 40.201279] vfio_ecap_init: 0000:0a:00.0 hiding ecap 0x1e@0x370 [ 159.958402] AMD-Vi: Completion-Wait loop timed out [ 160.118777] AMD-Vi: Completion-Wait loop timed out [ 160.799864] AMD-Vi: Event logged [ [ 160.799868] IOTLB_INV_TIMEOUT device=3D0a:00.0 address=3D0x000000043e8e8= 550] [ 160.799872] AMD-Vi: Event logged [ [ 160.799874] IOTLB_INV_TIMEOUT device=3D0a:00.0 address=3D0x000000043e8e8= 570] [ 160.799876] AMD-Vi: Event logged [ [ 160.799878] IOTLB_INV_TIMEOUT device=3D0a:00.0 address=3D0x000000043e8e8= 590] [ 161.801729] AMD-Vi: Event logged [ [ 161.801732] IOTLB_INV_TIMEOUT device=3D0a:00.0 address=3D0x000000043e8e8= 5e0] [ 180.096365] AMD-Vi: Completion-Wait loop timed out [ 180.256758] AMD-Vi: Completion-Wait loop timed out [ 180.417182] AMD-Vi: Completion-Wait loop timed out [ 180.577636] AMD-Vi: Completion-Wait loop timed out Can you try a v4.17-rc1 kernel? Specifically, these two updates: 6bd06f5a486c vfio/type1: Adopt fast IOTLB flush interface when unmap IOVAs eb5ecd1a40e2 iommu/amd: Add support for fast IOTLB flushing Something about AMD GPUs get unhappy if the IOMMU sends out too many invalidations and the above two patches can reduce the number of those invalidations by up to a factor of 512. https://git.kerne= l.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=3D6bd06f5a486c= 06023a618a86e8153b91d26f75f4 https://git.kerne= l.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=3Deb5ecd1a40e2= 098f805fb63cb07817ac48826e40
Created att=
achment 138893 [details]
dmesg after second launch + 4.17-rc1
Same problem with the Kernel 4.17-rc1. To be sure, i need to install this
kernel only on the Host, no need to install it on the Linux Guest ?
I use my own kernel 4.17 so maybe IOMMU/VFIO options are missing :
odelpasso@debian-desktop:~/Bureau$ cat /boot/config-4.17.0-rc1 | grep V=
FIO
CONFIG_VFIO_IOMMU_TYPE1=3Dm
CONFIG_VFIO_VIRQFD=3Dm
CONFIG_VFIO=3Dm
# CONFIG_VFIO_NOIOMMU is not set
CONFIG_VFIO_PCI=3Dm
CONFIG_VFIO_PCI_VGA=3Dy
CONFIG_VFIO_PCI_MMAP=3Dy
CONFIG_VFIO_PCI_INTX=3Dy
CONFIG_VFIO_PCI_IGD=3Dy
# CONFIG_VFIO_MDEV is not set
CONFIG_KVM_VFIO=3Dy
odelpasso@debian-desktop:~/Bureau$ cat /boot/config-4.17.0-rc1 | grep I=
OMMU
# CONFIG_GART_IOMMU is not set
# CONFIG_CALGARY_IOMMU is not set
CONFIG_IOMMU_HELPER=3Dy
CONFIG_VFIO_IOMMU_TYPE1=3Dm
# CONFIG_VFIO_NOIOMMU is not set
CONFIG_IOMMU_API=3Dy
CONFIG_IOMMU_SUPPORT=3Dy
# Generic IOMMU Pagetable Support
CONFIG_IOMMU_IOVA=3Dy
CONFIG_AMD_IOMMU=3Dy
CONFIG_AMD_IOMMU_V2=3Dy
# CONFIG_INTEL_IOMMU is not set
There is a difference, now we have: [ 84.997634] vfio_ecap_init: 0000:0a:00.0 hiding ecap 0x19@0x270 [ 84.997645] vfio_ecap_init: 0000:0a:00.0 hiding ecap 0x1b@0x2d0 [ 84.997653] vfio_ecap_init: 0000:0a:00.0 hiding ecap 0x1e@0x370 [ 145.518307] vfio_ecap_init: 0000:0a:00.0 hiding ecap 0x19@0x270 [ 145.518313] vfio_ecap_init: 0000:0a:00.0 hiding ecap 0x1b@0x2d0 [ 145.518318] vfio_ecap_init: 0000:0a:00.0 hiding ecap 0x1e@0x370 So prior to time 145.5 the VM was shutdown and started again and we could s= till read config space of the device. Previously we were already getting IOMMU faults before the second startup. But shortly after: [ 193.328586] AMD-Vi: Completion-Wait loop timed out [ 193.488711] AMD-Vi: Completion-Wait loop timed out [ 194.169913] iommu ivhd0: AMD-Vi: Event logged [ [ 194.169921] iommu ivhd0: IOTLB_INV_TIMEOUT device=3D0a:00.0 address=3D0x000000043e8aaca0] [ 194.169924] iommu ivhd0: AMD-Vi: Event logged [ [ 194.169928] iommu ivhd0: IOTLB_INV_TIMEOUT device=3D0a:00.0 address=3D0x000000043e8aacc0] And the stuck in D3 state is evidence that the device is no longer accessib= le on the bus. So that only delayed the issue, some interaction between the I= OMMU and GPU is still failing.
(In reply to Alex Williamson from comment #4) > There is a difference, now we have: >=20 > [ 84.997634] vfio_ecap_init: 0000:0a:00.0 hiding ecap 0x19@0x270 > [ 84.997645] vfio_ecap_init: 0000:0a:00.0 hiding ecap 0x1b@0x2d0 > [ 84.997653] vfio_ecap_init: 0000:0a:00.0 hiding ecap 0x1e@0x370 > [ 145.518307] vfio_ecap_init: 0000:0a:00.0 hiding ecap 0x19@0x270 > [ 145.518313] vfio_ecap_init: 0000:0a:00.0 hiding ecap 0x1b@0x2d0 > [ 145.518318] vfio_ecap_init: 0000:0a:00.0 hiding ecap 0x1e@0x370 >=20 > So prior to time 145.5 the VM was shutdown and started again and we co= uld > still read config space of the device. Previously we were already get= ting > IOMMU faults before the second startup. But shortly after: >=20 > [ 193.328586] AMD-Vi: Completion-Wait loop timed out > [ 193.488711] AMD-Vi: Completion-Wait loop timed out > [ 194.169913] iommu ivhd0: AMD-Vi: Event logged [ > [ 194.169921] iommu ivhd0: IOTLB_INV_TIMEOUT device=3D0a:00.0 > address=3D0x000000043e8aaca0] > [ 194.169924] iommu ivhd0: AMD-Vi: Event logged [ > [ 194.169928] iommu ivhd0: IOTLB_INV_TIMEOUT device=3D0a:00.0 > address=3D0x000000043e8aacc0] >=20 > And the stuck in D3 state is evidence that the device is no longer > accessible on the bus. So that only delayed the issue, some interacti= on > between the IOMMU and GPU is still failing. Thanks for the explaination Alex. Something could be done ?=20 By AMD or VFIO mainteners ?
This is still happening. It seems that these GPU need engine r= esets before bus reset, similar to what was done for Fury and Polaris, but more extensive. Temporary workaround (yeah sure) is to eject the driver - rmmod in guest or eject in Windows. This resets the engines. Windows did the resets on shutdown until version 18.5.1 where they broke shutdown sequence again - read release notes on Radeon Pro Vega FE drivers where they actually slightly care.
Another workaround that has worked for me with a Vega 56 is to= suspend-to-ram the host system before trying to start the guest again.
| What | Removed | Added |
|---|---|---|
| Resolution | --- | MOVED |
| Status | NEW | RESOLVED |
-- GitLab Migration Automatic Message -- This bug has been migrated to freedesktop.org's GitLab instance and has been closed from further activity. You can subscribe and participate further through the new bug through this = link to our GitLab instance: https://gitlab.freedesktop.org/drm/amd/issues/346.