From: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
To: Sander Eikelenboom <linux@eikelenboom.it>
Cc: xen-devel <xen-devel@lists.xenproject.org>,
Jan Beulich <JBeulich@suse.com>
Subject: Re: Xen-unstable: pci-passthrough "irq 16: nobody cared" on HVM guest shutdown on irq of device not passed through.
Date: Tue, 7 Oct 2014 09:41:00 -0400 [thread overview]
Message-ID: <20141007134100.GD2604@laptop.dumpdata.com> (raw)
In-Reply-To: <4410685169.20141001155255@eikelenboom.it>
> Hi Jan,
>
> Just tested with about the same kernel (and same hardware) on linux + kvm:
>
> When i boot i see (on the host):
> [ 1.341556] pcieport 0000:00:02.0: irq 31 for MSI/MSI-X
> [ 1.341875] pcieport 0000:00:03.0: irq 32 for MSI/MSI-X
> [ 1.342145] pcieport 0000:00:05.0: irq 33 for MSI/MSI-X
> [ 1.342429] pcieport 0000:00:06.0: irq 34 for MSI/MSI-X
> [ 1.342588] pcieport 0000:00:09.0: irq 35 for MSI/MSI-X
> [ 1.342853] pcieport 0000:00:0a.0: irq 36 for MSI/MSI-X
> [ 1.343125] pcieport 0000:00:0b.0: irq 37 for MSI/MSI-X
> [ 1.343285] pcieport 0000:00:0c.0: irq 38 for MSI/MSI-X
> [ 1.343434] pcieport 0000:00:0d.0: irq 39 for MSI/MSI-X
> [ 1.343719] pcieport 0000:00:15.0: irq 40 for MSI/MSI-X
When this happend, did you see:
[ 66.819396] bus: 'pci': really_probe: bound device 0000:00:0d.0 to driver pcieport
[ 66.842286] bus: 'pci': driver_probe_device: matched device 0000:00:15.0 with driver pcieport
[ 66.868014] bus: 'pci': really_probe: probing driver pcieport with device 0000:00:15.0
-- some other code --
[ 67.757210] pcieport 0000:00:15.0: ?!?!? acpi_pci_irq_enable: PCI INT A -> GSI 16 (level, low) -> IRQ/rc 16
> [ 1.343897] pcieport 0000:05:00.0: irq 41 for MSI/MSI-X
> [ 1.344134] pcieport 0000:06:01.0: irq 42 for MSI/MSI-X
> [ 1.344398] pcieport 0000:06:02.0: irq 44 for MSI/MSI-X
>
> While the soundcard has still irq16.
>
> Passing through 09:00.0 to a KVM guest (as primary passthrough since that works
> with KVM) it works fine and after shutting down there is no "irq 16 nobody
> cared".
>
> So why would linux setup those irq/gsi's for the pcieport differently on Xen and on
> baremetal (and why can a irq/gsi be bound to 2 devices) ?
Right, that would be odd.
The IRQ can be bound to two devices if it is an 'level' type (aka, not an 'edge').
This is something that be extrapolated from the ACPI MADT tables
and the xen_register_gsi is the one that is told about that (polarity and level).
Looking back at your emails:
(XEN) [2014-09-25 13:29:04.111] IRQ: 16 affinity:01 vec:89 type=IO-APIC-level status=00000030 in-flight=0 domain-list=0: 16(--
It definitly is level, so that means it is shared.
Now there is a gotcha with shared interrupts - if there are _two_
devices and one is in one guest while the other is in another guest - both
domains have to Ack it.
That is where Xen PCIBack comes in - it has its own IRQ handler that will
determine (based on a hypercall) whether the device is shared and if so
Ack it. I am wondering if it is turned off (as in, Xen thinks it should not
be shared but Linux has the device shared).
You can fiddle with the Xen pciback parameters to turn on/off this 'fake'
irq handler.
But it all seems to point that we somehow are getting the wrong impression
about the 0000:00:0d.0 and 0000:00:15.0 and using legacy interrupts
instead of MSI ones. And I think that underlaying problem is causing
these secondary ones.
Could you attach also the full dmesg under baremetal with 'debug' and all
kinds of debug enabled ? That should help a bit in figuring out why
they get MSIs under baremetal but legacy interrupts under Xen.
Thank you!
>
> --
> Sander
>
>
> # cat /proc/interrupts
> CPU0 CPU1 CPU2 CPU3 CPU4 CPU5
> 0: 100420 0 0 0 0 0 IR-IO-APIC-edge timer
> 1: 1 0 0 0 0 2 IR-IO-APIC-edge i8042
> 7: 1 0 0 0 0 0 IR-IO-APIC-edge
> 8: 0 0 0 0 0 1 IR-IO-APIC-edge rtc0
> 9: 0 0 0 0 0 0 IR-IO-APIC-fasteoi acpi
> 12: 0 0 0 0 0 4 IR-IO-APIC-edge i8042
> 16: 0 0 0 0 0 796 IR-IO-APIC 16-fasteoi snd_hda_intel
> 17: 0 0 0 0 0 4 IR-IO-APIC 17-fasteoi ehci_hcd:usb1, ehci_hcd:usb2, ehci_hcd:usb3
> 18: 0 0 0 0 11 6414 IR-IO-APIC 18-fasteoi ohci_hcd:usb4, ohci_hcd:usb5, ohci_hcd:usb6, ohci_hcd:usb7
> 29: 0 0 0 0 0 0 IR-PCI-MSI-edge AMD-Vi
> 31: 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME
> 32: 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME
> 33: 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME
> 34: 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME
> 35: 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME
> 36: 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME
> 37: 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME
> 38: 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME
> 39: 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME
> 40: 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME
> 45: 0 0 0 1 79 95630 IR-PCI-MSI-edge ahci0
> 46: 0 0 0 0 0 0 IR-PCI-MSI-edge ahci1
> 47: 0 0 0 0 27 8643 IR-PCI-MSI-edge ahci2
> 48: 0 0 0 0 0 0 IR-PCI-MSI-edge ahci3
> 49: 0 0 0 0 0 0 IR-PCI-MSI-edge ahci4
> 50: 0 0 0 0 0 0 IR-PCI-MSI-edge ahci5
> 53: 0 0 0 0 0 0 IR-PCI-MSI-edge ahci
> 55: 0 0 0 0 184 22648 IR-PCI-MSI-edge eth0
> 57: 0 0 0 1 148 21232 IR-PCI-MSI-edge eth1
> 59: 0 0 0 0 0 1 IR-PCI-MSI-edge xhci_hcd
> 60: 0 0 0 0 0 0 IR-PCI-MSI-edge xhci_hcd
> 61: 0 0 0 0 0 0 IR-PCI-MSI-edge xhci_hcd
> 62: 0 0 0 0 0 0 IR-PCI-MSI-edge xhci_hcd
> 63: 0 0 0 0 0 0 IR-PCI-MSI-edge xhci_hcd
> 64: 0 0 0 0 0 0 IR-PCI-MSI-edge xhci_hcd
> 65: 0 0 0 0 0 0 IR-PCI-MSI-edge xhci_hcd
> 66: 0 0 0 0 0 23 IR-PCI-MSI-edge xhci_hcd
> 67: 0 0 0 0 0 0 IR-PCI-MSI-edge xhci_hcd
> 68: 0 0 0 0 0 0 IR-PCI-MSI-edge xhci_hcd
> 69: 0 0 0 0 0 0 IR-PCI-MSI-edge xhci_hcd
> 70: 0 0 0 0 0 0 IR-PCI-MSI-edge xhci_hcd
> 71: 0 0 0 0 0 0 IR-PCI-MSI-edge xhci_hcd
> 72: 0 0 0 0 0 0 IR-PCI-MSI-edge xhci_hcd
> 74: 0 0 0 0 23 22099 IR-PCI-MSI-edge xhci_hcd
> 75: 0 0 0 0 0 0 IR-PCI-MSI-edge xhci_hcd
> 76: 0 0 0 0 0 0 IR-PCI-MSI-edge xhci_hcd
> 77: 0 0 0 0 0 0 IR-PCI-MSI-edge xhci_hcd
> 78: 0 0 0 0 0 0 IR-PCI-MSI-edge xhci_hcd
> 79: 0 0 0 0 0 0 IR-PCI-MSI-edge xhci_hcd
> 80: 0 0 0 0 0 0 IR-PCI-MSI-edge xhci_hcd
> 81: 0 0 0 0 0 0 IR-IO-APIC 23-fasteoi cx25821[1]
> 83: 0 0 0 0 0 28 IR-PCI-MSI-edge snd_hda_intel
> 85: 0 0 0 0 3 652 IR-PCI-MSI-edge vfio-msi[0](0000:09:00.0)
> NMI: 3 3 3 4 4 3 Non-maskable interrupts
> LOC: 13423 25027 24547 30018 30575 46789 Local timer interrupts
> SPU: 0 0 0 0 0 0 Spurious interrupts
> PMI: 3 3 3 4 4 3 Performance monitoring interrupts
> IWI: 0 0 0 0 1 0 IRQ work interrupts
> RTR: 0 0 0 0 0 0 APIC ICR read retries
> RES: 88652 68934 52849 112934 76728 33317 Rescheduling interrupts
> CAL: 295 287 327 327 311 200 Function call interrupts
> TLB: 1165 1232 1122 1376 1248 1160 TLB shootdowns
> TRM: 0 0 0 0 0 0 Thermal event interrupts
> THR: 0 0 0 0 0 0 Threshold APIC interrupts
> MCE: 0 0 0 0 0 0 Machine check exceptions
> MCP: 4 4 4 4 4 4 Machine check polls
> THR: 0 0 0 0 0 0 Hypervisor callback interrupts
> ERR: 1
> MIS: 0
>
next prev parent reply other threads:[~2014-10-07 13:41 UTC|newest]
Thread overview: 30+ messages / expand[flat|nested] mbox.gz Atom feed top
2014-09-25 14:36 Xen-unstable: pci-passthrough "irq 16: nobody cared" on HVM guest shutdown on irq of device not passed through Sander Eikelenboom
2014-09-25 14:42 ` Andrew Cooper
2014-09-25 14:47 ` Sander Eikelenboom
2014-09-25 15:11 ` Jan Beulich
2014-09-25 15:49 ` Sander Eikelenboom
2014-09-25 16:14 ` Jan Beulich
2014-09-25 17:02 ` Sander Eikelenboom
2014-09-25 18:45 ` Sander Eikelenboom
2014-09-25 22:09 ` Sander Eikelenboom
2014-09-26 6:59 ` Jan Beulich
2014-09-26 9:18 ` Sander Eikelenboom
2014-09-26 9:43 ` Jan Beulich
2014-09-26 10:02 ` Sander Eikelenboom
2014-09-26 10:08 ` Jan Beulich
2014-09-27 14:00 ` Sander Eikelenboom
2014-09-27 18:02 ` Konrad Rzeszutek Wilk
2014-09-27 18:23 ` Sander Eikelenboom
2014-10-01 13:52 ` Sander Eikelenboom
2014-10-01 14:19 ` Jan Beulich
2014-10-07 13:41 ` Konrad Rzeszutek Wilk [this message]
2014-10-07 14:50 ` Jan Beulich
2014-10-08 12:56 ` Konrad Rzeszutek Wilk
2014-10-08 20:33 ` Sander Eikelenboom
2014-10-21 13:43 ` Sander Eikelenboom
2014-10-21 14:52 ` Jan Beulich
2014-09-26 6:54 ` Jan Beulich
2014-09-26 9:06 ` Sander Eikelenboom
2014-09-26 6:50 ` Jan Beulich
2014-09-26 9:00 ` Sander Eikelenboom
2014-09-26 9:09 ` Jan Beulich
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20141007134100.GD2604@laptop.dumpdata.com \
--to=konrad.wilk@oracle.com \
--cc=JBeulich@suse.com \
--cc=linux@eikelenboom.it \
--cc=xen-devel@lists.xenproject.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).