From mboxrd@z Thu Jan 1 00:00:00 1970 From: Boris Ostrovsky Subject: Re: xhci_hcd intterrupt affinity in Dom0/DomU limited to single interrupt Date: Tue, 01 Sep 2015 17:38:46 -0400 Message-ID: <55E61AE6.9060300@oracle.com> References: <1441121643.26292.63.camel@citrix.com> <800613365.4285959.1441128848192.JavaMail.yahoo@mail.yahoo.com> <631331126.1156575.1441129186888.JavaMail.yahoo@mail.yahoo.com> <20150901205657.GA18532@l.oracle.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; Format="flowed" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <20150901205657.GA18532@l.oracle.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Konrad Rzeszutek Wilk , Justin Acker Cc: "xen-devel@lists.xen.org" List-Id: xen-devel@lists.xenproject.org On 09/01/2015 04:56 PM, Konrad Rzeszutek Wilk wrote: > On Tue, Sep 01, 2015 at 05:39:46PM +0000, Justin Acker wrote: >> Taking this to the dev list from users. >> >> Is there a way to force or enable pirq delivery to a set of cpus as opposed to single device from being a assigned a single pirq so that its interrupt can be distributed across multiple cpus? I believe the device drivers do support multiple queues when run natively without the Dom0 loaded. The device in question is the xhci_hcd driver for which I/O transfers seem to be slowed when the Dom0 is loaded. The behavior seems to pass through to the DomU if pass through is enabled. I found some similar threads, but most relate to Ethernet controllers. I tried some of the x2apic and x2apic_phys dom0 kernel arguments, but none distributed the pirqs. Based on the reading relating to IRQs for Xen, I think pinning the pirqs to cpu0 is done to avoid an interrupt storm. I tried IRQ balance and when co nfigured/adjusted it will balance individual pirqs, but not multiple interrupts. > Yes. You can do it with smp affinity: > > https://cs.uwaterloo.ca/~brecht/servers/apic/SMP-affinity.txt > > I don't follow the "behavior seems to pass through to the DomU if pass through is enabled" ? > >> >> >> With irqbalance enabled in Dom0: > What version? There was a bug in it where it would never distribute the IRQs properly > across the CPUs. > > Boris (CC-ed) might remember the upstream patch that made this work properly? I think we ended up taking the latest version of irqbalance as the one that added support for Xen guests was still not quite working. Besides, that patch was for xen-dyn-events, not for pirqs. -boris >> CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 >> 76: 11304 0 149579 0 0 0 0 0 xen-pirq-msi 0000:00:1f.2 >> 77: 1243 0 0 35447 0 0 0 0 xen-pirq-msi radeon >> 78: 82521 0 0 0 0 0 0 0 xen-pirq-msi xhci_hcd >> 79: 23 0 0 0 0 0 0 0 xen-pirq-msi mei_me >> 80: 11 0 0 0 0 741 0 0 xen-pirq-msi em1 >> 81: 350 0 0 0 1671 0 0 0 xen-pirq-msi iwlwifi >> 82: 275 0 0 0 0 0 0 0 xen-pirq-msi snd_hda_intel >> >> With native 3.19 kernel: >> >> Without Dom0 for the same system from the first message: >> >> # cat /proc/interrupts >> CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 >> 0: 33 0 0 0 0 0 0 0 IR-IO-APIC-edge timer >> 8: 0 0 0 0 0 0 1 0 IR-IO-APIC-edge rtc0 >> 9: 20 0 0 0 0 1 1 1 IR-IO-APIC-fasteoi acpi >> 16: 15 0 8 1 4 1 1 1 IR-IO-APIC 16-fasteoi ehci_hcd:usb3 >> 18: 703940 5678 1426226 1303 3938243 111477 757871 510 IR-IO-APIC 18-fasteoi ath9k >> 23: 11 2 3 0 0 17 2 0 IR-IO-APIC 23-fasteoi ehci_hcd:usb4 >> 24: 0 0 0 0 0 0 0 0 DMAR_MSI-edge dmar0 >> 25: 0 0 0 0 0 0 0 0 DMAR_MSI-edge dmar1 >> 26: 20419 1609 26822 567 62281 5426 14928 395 IR-PCI-MSI-edge 0000:00:1f.2 >> 27: 17977230 628258 44247270 120391 1597809883 14440991 152189328 73322 IR-PCI-MSI-edge xhci_hcd >> 28: 563 0 0 0 1 0 6 0 IR-PCI-MSI-edge i915 >> 29: 14 0 0 4 2 4 0 0 IR-PCI-MSI-edge mei_me >> 30: 39514 1744 60339 157 129956 19702 72140 83 IR-PCI-MSI-edge eth0 >> 31: 3 0 0 1 54 0 0 2 IR-PCI-MSI-edge snd_hda_intel >> 32: 28145 284 53316 63 139165 4410 25760 27 IR-PCI-MSI-edge eth1-rx-0 >> 33: 1032 43 2392 5 1797 265 1507 20 IR-PCI-MSI-edge eth1-tx-0 >> 34: 0 1 0 0 0 1 2 0 IR-PCI-MSI-edge eth1 >> 35: 5 0 0 12 148 6 2 1 IR-PCI-MSI-edge snd_hda_intel >> >> >> The USB controller is an Intel C210: >> >> 00:14.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller (rev 04) (prog-if 30 [XHCI]) >> Subsystem: Dell Device 053e >> Flags: bus master, medium devsel, latency 0, IRQ 78 >> Memory at f7f20000 (64-bit, non-prefetchable) [size=64K] >> Capabilities: [70] Power Management version 2 >> Capabilities: [80] MSI: Enable+ Count=1/8 Maskable- 64bit+ >> Kernel driver in use: xhci_hcd >> Kernel modules: xhci_pci >> On Tuesday, September 1, 2015 11:50 AM, Ian Campbell wrote: >> >> >> On Tue, 2015-09-01 at 13:56 +0000, Justin Acker wrote: >>> Thanks Ian, >>> >>> I appreciate the explanation. I believe the device drivers do support >>> multiple queues when run natively without the Dom0 loaded. The device in >>> question is the xhci_hcd driver for which I/O transfers seem to be slowed >>> when the Dom0 is loaded. The behavior seems to pass through to the DomU >>> if pass through is enabled. I found some similar threads, but most relate >>> to Ethernet controllers. I tried some of the x2apic and x2apic_phys dom0 >>> kernel arguments, but none distributed the pirqs. Based on the reading >>> relating to IRQs for Xen, I think pinning the pirqs to cpu0 is done to >>> avoid an I/O storm. I tried IRQ balance and when configured/adjusted it >>> will balance individual pirqs, but not multiple interrupts. >>> >>> Is there a way to force or enable pirq delivery to a set of cpus as you >>> mentioned above or omit a single device from being a assigned a PIRQ so >>> that its interrupt can be distributed across all cpus? >> A PIRQ is the way an interrupt is exposed to a PV guest, without it there >> would be no interrupt at all. >> >> I'm afraid I'm out of my depth WRT how x86/MSIs and Xen x86/PV pirqs >> interact, in particular WRT configuring which set of CPUs can have the IRQ >> delivered. >> >> If no one else chimes in soon I'd suggest taking this to the dev list, at >> the very least someone who knows what they are talking about (i.e. other >> than me) might be able to help. >> >> Ian. >> >> >> >> >> _______________________________________________ >> Xen-devel mailing list >> Xen-devel@lists.xen.org >> http://lists.xen.org/xen-devel