[Bug 42635] New: PCIe passthrough broken with AMD iommu after s2disk / resume

[Bug 42635] New: PCIe passthrough broken with AMD iommu after s2disk / resume

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=42635

           Summary: PCIe passthrough broken with AMD iommu after s2disk /
                    resume
           Product: Virtualization
           Version: unspecified
    Kernel Version: 3.1
          Platform: All
        OS/Version: Linux
              Tree: Mainline
            Status: NEW
          Severity: normal
          Priority: P1
         Component: kvm
        AssignedTo: virtualization_kvm@kernel-bugs.osdl.org
        ReportedBy: kmueller@justmail.de
        Regression: No


A PCIe ethernet device is passed through to the guest and runs fine. Some time
later, the guest is shutdown (e.g. virsh shutdown guest) and the host is
suspended (s2disk) and resumed again. The guest is started again (and it starts
fine), but the device in the guest isn't working any more (it can be seen, it
can be pinged itself but nothing is reachable outside the device. The rx or tx
counters of the ifconfig output are always 0.

If the guest is shutdown and the device is bound to the host, it's working as
expected. If the device is afterwards bound to the guest again, it doesn't work
as before in the guest after resume.

kernel: 3.1 / 64bit / smp
kvm: 1.0
board: GA-990XA-UD3

If you need more information, I can provide them - feel free to ask!


Regards,
Klaus

-- 
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.

[Bug 42635] PCIe passthrough broken with AMD iommu after s2disk / resume

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=42635


Alex Williamson <alex.williamson@redhat.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |alex.williamson@redhat.com




--- Comment #1 from Alex Williamson <alex.williamson@redhat.com>  2012-01-23 15:01:29 ---
Between the first and second paragraph, you seem to be saying that the device
never works in the guest on the second invocation of the guest.  Is that true? 
What is the device?

-- 
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.

[Bug 42635] PCIe passthrough broken with AMD iommu after s2disk / resume

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=42635





--- Comment #2 from Klaus Mueller <kmueller@justmail.de>  2012-01-23 16:18:33 ---
The device never works in the guest after s2disk/resume cycle of the host. But
it always works if it's used by the host itself - even after a s2disk/resume
cycle.

The device is (as seen by the host):

03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI
Express Gigabit Ethernet controller (rev 06)
        Subsystem: Giga-byte Technology GA-EP45-DS5 Motherboard
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort-
<MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 45
        Region 0: I/O ports at ee00 [size=256]
        Region 2: Memory at fdcff000 (64-bit, prefetchable) [size=4K]
        Region 4: Memory at fdcf8000 (64-bit, prefetchable) [size=16K]
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA
PME(D0+,D1+,D2+,D3hot+,D3cold+)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
                Address: 00000000fee0f00c  Data: 4122
        Capabilities: [70] Express (v2) Endpoint, MSI 01
                DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <512ns,
L1 <64us
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
                DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
Unsupported-
                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop-
                        MaxPayload 128 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+
TransPend-
                LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Latency
L0 unlimited, L1 <64us
                        ClockPM+ Surprise- LLActRep- BwNot-
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+
DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range ABCD, TimeoutDis+
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
                LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance-
SpeedDis-, Selectable De-emphasis: -6dB
                         Transmit Margin: Normal Operating Range,
EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -6dB
        Capabilities: [b0] MSI-X: Enable- Count=4 Masked-
                Vector table: BAR=4 offset=00000000
                PBA: BAR=4 offset=00000800
        Capabilities: [d0] Vital Product Data
                Unknown small resource type 00, will not decode more.
        Capabilities: [100 v1] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
                AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
        Capabilities: [140 v1] Virtual Channel
                Caps:   LPEVC=0 RefClk=100ns PATEntryBits=1
                Arb:    Fixed- WRR32- WRR64- WRR128-
                Ctrl:   ArbSelect=Fixed
                Status: InProgress-
                VC0:    Caps:   PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
                        Arb:    Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
                        Ctrl:   Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
                        Status: NegoPending- InProgress-
        Capabilities: [160 v1] Device Serial Number 12-34-56-78-12-34-56-78
        Kernel driver in use: pci-stub


One more thing: the device can't be used in the guest anymore, too, after a
reboot of the guest by rebooting the guest itself. I have to shutdown the guest
with virsh shutdown and virsh start to get it working again. But this is
useless, if there was a suspend / resume cycle of the host between.

-- 
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.

[Bug 42635] PCIe passthrough broken with AMD iommu after s2disk / resume

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=42635





--- Comment #3 from Klaus Mueller <kmueller@justmail.de>  2012-04-03 07:27:43 ---
Meanwhile I detected, that using the boot option iommu=pt (or additional
iommu=1) shows the same problem (regardless of suspend/resume). If the VM is
started after the first virsh start / virsh shutdown cycle with virsh start
again, the device is dead in the VM. Host must be rebooted to get it working
again.

If the options iommu=pt (or additional iommu=1) are not used at all, the VM can
be restarted without any problem coming up.

1. Host booted w/ iommu=pt iommu=1
2. virsh start
3. virsh shutdown
4. virsh start
5. -> Passed through device in VM is dead.


1. Host booted w/o iommu=pt iommu=1
2. virsh start
3. virsh shutdown
4. virsh start
5. -> Passed through device in VM is fine.

-- 
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.