From mboxrd@z Thu Jan 1 00:00:00 1970 From: Justin Acker Subject: Re: xhci_hcd intterrupt affinity in Dom0/DomU limited to single interrupt Date: Tue, 08 Sep 2015 12:02:21 -0400 Message-ID: <1441728141.8074.11.camel@localhost> References: <55E839E3020000780009F3CD@prv-mh.provo.novell.com> <1292756745.962760.1441281893582.JavaMail.yahoo@mail.yahoo.com> <55E87DA6020000780009F59D@prv-mh.provo.novell.com> <1441299125.30502.12.camel@localhost> <55E9675E020000780009F822@prv-mh.provo.novell.com> Reply-To: ackerj67@yahoo.com Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: Received: from mail6.bemta14.messagelabs.com ([193.109.254.103]) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1ZZLLj-0000it-9Q for xen-devel@lists.xenproject.org; Tue, 08 Sep 2015 16:02:23 +0000 In-Reply-To: <55E9675E020000780009F822@prv-mh.provo.novell.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Jan Beulich Cc: xen-devel List-Id: xen-devel@lists.xenproject.org On Fri, 2015-09-04 at 01:41 -0600, Jan Beulich wrote: > >>> On 03.09.15 at 18:52, wrote: > > On Thu, 2015-09-03 at 09:04 -0600, Jan Beulich wrote: > >> >>> On 03.09.15 at 14:04, wrote: > >> > I am still confused as to whether any device, or in this case xhci_hcd, > >> > can use more than one cpu at any given time. My understanding based on > >> > David's response is that it cannot due to the event channel mapping. The > >> > device interrupt can be pinned to a specific cpu by specifying the > >> > affinity. I was hoping there was a way to allow the driver's interrupt to be > >> > scheduled to use more than 1 CPU at any given time. > >> > >> The problem is that you're mixing up two things: devices and > >> interrupts. Any individual interrupt can only be serviced by a single > >> CPU at a time, due to the way event channels get bound. Any > >> individual device can have more than one interrupt (MSI or MSI-X), > >> and then each of these interrupts can be serviced on different > >> CPUs. > > > > Thanks for clarifying. To the original question, with respect to my > > limited understanding of the event channels and interrupts, each > > interrupt can be serviced on a different CPU using irqbalance or setting > > the affinity manually, but the same interrupt cannot be serviced by more > > than 1 CPU at a time? If so, is there a way around the 1:1 binding when > > loading the Dom0 kernel - a flag or option to use the native interrupt > > scheduling for some set of or all 8 CPUs that the device can schedule > > interrupts on when not loading the Dom0? The xhci_hcd, as one example, > > seems to perform better when it is able to have interrupts serviced by > > multiple CPUs. > > I don't follow - we tell you this doesn't work (multiple times and > different people), and you ask yet another time whether this can > be made work? Just to make this very clear once again: Under Xen > (and leaving aside pure HVM guests), interrupt load from a single > device can be spread across CPUs only when the device uses > multiple interrupts. > > Jan > I believe the driver does support use of multiple interrupts based on the previous explanation of the lspci output where it was established that the device could use up to 8 interrupts which is what I see on bare metal. What would cause the interrupt usage to be limited to a single CPU when the hypervisor is loaded? The device mentioned, xhci_hcd can use multiple interrupts on baremetal and the other devices including the network interfaces are also bound to a single interrupt when the hypervisor is loaded but can use multiple interrupts running on baremetal. 70: 0 0 0 0 0 0 0 133 xen-percpu-ipi callfuncsingle7 71: 0 0 0 0 0 0 0 0 xen-percpu-ipi irqwork7 72: 100 0 0 0 0 0 0 0 xen-dyn-event xenbus 73: 0 0 0 0 0 0 0 0 xen-dyn-virq xen-pcpu 74: 0 0 0 0 0 0 0 0 xen-dyn-virq hvc_console 75: 6809 0 0 0 0 0 0 0 xen-pirq-msi 0000:00:1f.2 76: 470660 0 0 0 0 0 0 0 xen-pirq-msi xhci_hcd 77: 608 0 0 0 0 0 0 0 xen-pirq-msi i915 78: 26 0 0 0 0 0 0 0 xen-pirq-msi mei_me 79: 243 0 0 0 0 0 0 0 xen-pirq-msi em1 80: 315 0 0 0 0 0 0 0 xen-pirq-msi-x p4p1-rx-0 81: 104 0 0 0 0 0 0 0 xen-pirq-msi-x p4p1-tx-0 82: 2 0 0 0 0 0 0 0 xen-pirq-msi-x p4p1 83: 59 0 0 0 0 0 0 0 xen-pirq-msi snd_hda_intel 84: 48 0 0 0 0 0 0 0 xen-dyn-event evtchn:xenstored 85: 0 0 0 0 0 0 0 0 xen-dyn-event evtchn:xenstored NMI: 0 0 0 0 0 0 0 0 Non-maskable interrupts LOC: 0 0 0 0 0 0 0 0 Local timer interrupts SPU: 0 0 0 0 0 0 0 0 Spurious interrupts PMI: 0 0 0 0 0 0 0 0 Performance monitoring interrupts IWI: 1 0 0 0 0 0 0 0 IRQ work interrupts RTR: 0 0 0 0 0 0 0 0 APIC ICR read retries RES: 44788 33111 267650 5641 35515 2312 10730 2479 Rescheduling interrupts CAL: 929 1202 1164 1314 1061 1296 1156 1097 Function call interrupts TLB: 0 0 0 0 0 0 0 0 TLB shootdowns TRM: 0 0 0 0 0 0 0 0 Thermal event interrupts THR: 0 0 0 0 0 0 0 0 Threshold APIC interrupts MCE: 0 0 0 0 0 0 0 0 Machine check exceptions MCP: 1 1 1 1 1 1 1 1 Machine check polls HYP: 538069 43568 273140 12112 44323 7871 18949 7924 Hypervisor callback interrupts Example of another device driver output with Hypervisor loaded: 04:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection Subsystem: Intel Corporation Device 0000 Physical Slot: 1-3 Flags: bus master, fast devsel, latency 0, IRQ 19 Memory at f5600000 (32-bit, non-prefetchable) [size=128K] I/O ports at b000 [size=32] Memory at f5620000 (32-bit, non-prefetchable) [size=16K] Capabilities: [c8] Power Management version 2 Capabilities: [d0] MSI: Enable- Count=1/1 Maskable- 64bit+ Capabilities: [e0] Express Endpoint, MSI 00 Capabilities: [a0] MSI-X: Enable+ Count=5 Masked- Capabilities: [100] Advanced Error Reporting Capabilities: [140] Device Serial Number Kernel driver in use: e1000e Kernel modules: e1000e Example of the same device driver output on Bare metal: 04:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection Subsystem: Intel Corporation Device 0000 Physical Slot: 1-3 Flags: bus master, fast devsel, latency 0, IRQ 19 Memory at f5600000 (32-bit, non-prefetchable) [size=128K] I/O ports at b000 [size=32] Memory at f5620000 (32-bit, non-prefetchable) [size=16K] Capabilities: [c8] Power Management version 2 Capabilities: [d0] MSI: Enable- Count=1/1 Maskable- 64bit+ Capabilities: [e0] Express Endpoint, MSI 00 Capabilities: [a0] MSI-X: Enable+ Count=5 Masked- Capabilities: [100] Advanced Error Reporting Capabilities: [140] Device Serial Number Kernel driver in use: e1000e Kernel modules: e1000e Bare metal: cat /proc/interrupts CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 0: 36 0 0 0 0 0 0 0 IR-IO-APIC-edge timer 8: 0 0 0 1 0 0 0 0 IR-IO-APIC-edge rtc0 9: 0 0 1 0 1 1 0 0 IR-IO-APIC-fasteoi acpi 16: 2 0 14 0 13 0 0 2 IR-IO-APIC 16-fasteoi ehci_hcd:usb3 18: 24420 225 39883 43 134488 1044 1085 46 IR-IO-APIC 18-fasteoi ath9k 23: 0 0 20 0 0 1 3 11 IR-IO-APIC 23-fasteoi ehci_hcd:usb4 24: 0 0 0 0 0 0 0 0 DMAR_MSI-edge dmar0 25: 0 0 0 0 0 0 0 0 DMAR_MSI-edge dmar1 26: 4119 1016 1074 519 4677 978 573 432 IR-PCI-MSI-edge 0000:00:1f.2 27: 337125 47893 708965 4049 53940667 263303 87847 4958 IR-PCI-MSI-edge xhci_hcd 28: 562 0 0 0 5 1 2 0 IR-PCI-MSI-edge i915 29: 7 0 0 0 4 8 0 6 IR-PCI-MSI-edge mei_me 30: 2222 23 2721 5 5810 181 149 5 IR-PCI-MSI-edge eth0 31: 1504 17 2196 6 6581 95 77 6 IR-PCI-MSI-edge eth1-rx-0 32: 2 0 1 0 3 0 0 0 IR-PCI-MSI-edge eth1-tx-0 33: 0 0 0 0 0 1 0 1 IR-PCI-MSI-edge eth1 34: 14 6 0 0 0 38 0 2 IR-PCI-MSI-edge snd_hda_intel 35: 4 3 0 0 0 178 0 3 IR-PCI-MSI-edge snd_hda_intel NMI: 11 105 87 1 75 2 9 1 Non-maskable interrupts LOC: 132282 1175126 905037 79133 428383 22748 118085 12513 Local timer interrupts SPU: 0 0 0 0 0 0 0 0 Spurious interrupts PMI: 11 105 87 1 75 2 9 1 Performance monitoring interrupts IWI: 1 0 0 0 0 0 0 0 IRQ work interrupts RTR: 0 0 0 0 0 0 0 0 APIC ICR read retries RES: 36810 28341 99815 953 60355 1142 878 823 Rescheduling interrupts CAL: 2134 1928 1793 1965 1917 1879 1877 1882 Function call interrupts TLB: 8064 365 20958 28575 37592 113 28495 445 TLB shootdowns TRM: 0 0 0 0 0 0 0 0 Thermal event interrupts THR: 0 0 0 0 0 0 0 0 Threshold APIC interrupts MCE: 0 0 0 0 0 0 0 0 Machine check exceptions MCP: 9 9 9 9 9 9 9 9 Machine check polls THR: 0 0 0 0 0 0 0 0 Hypervisor callback interrupts