From: Justin Acker <ackerj67@yahoo.com>
To: Jan Beulich <JBeulich@suse.com>
Cc: xen-devel <xen-devel@lists.xenproject.org>
Subject: Re: xhci_hcd intterrupt affinity in Dom0/DomU limited to single interrupt
Date: Tue, 08 Sep 2015 12:02:21 -0400 [thread overview]
Message-ID: <1441728141.8074.11.camel@localhost> (raw)
In-Reply-To: <55E9675E020000780009F822@prv-mh.provo.novell.com>
On Fri, 2015-09-04 at 01:41 -0600, Jan Beulich wrote:
> >>> On 03.09.15 at 18:52, <ackerj67@yahoo.com> wrote:
> > On Thu, 2015-09-03 at 09:04 -0600, Jan Beulich wrote:
> >> >>> On 03.09.15 at 14:04, <ackerj67@yahoo.com> wrote:
> >> > I am still confused as to whether any device, or in this case xhci_hcd,
> >> > can use more than one cpu at any given time. My understanding based on
> >> > David's response is that it cannot due to the event channel mapping. The
> >> > device interrupt can be pinned to a specific cpu by specifying the
> >> > affinity. I was hoping there was a way to allow the driver's interrupt to be
> >> > scheduled to use more than 1 CPU at any given time.
> >>
> >> The problem is that you're mixing up two things: devices and
> >> interrupts. Any individual interrupt can only be serviced by a single
> >> CPU at a time, due to the way event channels get bound. Any
> >> individual device can have more than one interrupt (MSI or MSI-X),
> >> and then each of these interrupts can be serviced on different
> >> CPUs.
> >
> > Thanks for clarifying. To the original question, with respect to my
> > limited understanding of the event channels and interrupts, each
> > interrupt can be serviced on a different CPU using irqbalance or setting
> > the affinity manually, but the same interrupt cannot be serviced by more
> > than 1 CPU at a time? If so, is there a way around the 1:1 binding when
> > loading the Dom0 kernel - a flag or option to use the native interrupt
> > scheduling for some set of or all 8 CPUs that the device can schedule
> > interrupts on when not loading the Dom0? The xhci_hcd, as one example,
> > seems to perform better when it is able to have interrupts serviced by
> > multiple CPUs.
>
> I don't follow - we tell you this doesn't work (multiple times and
> different people), and you ask yet another time whether this can
> be made work? Just to make this very clear once again: Under Xen
> (and leaving aside pure HVM guests), interrupt load from a single
> device can be spread across CPUs only when the device uses
> multiple interrupts.
>
> Jan
>
I believe the driver does support use of multiple interrupts based on
the previous explanation of the lspci output where it was established
that the device could use up to 8 interrupts which is what I see on bare
metal. What would cause the interrupt usage to be limited to a single
CPU when the hypervisor is loaded? The device mentioned, xhci_hcd can
use multiple interrupts on baremetal and the other devices including the
network interfaces are also bound to a single interrupt when the
hypervisor is loaded but can use multiple interrupts running on
baremetal.
70: 0 0 0 0 0 0
0 133 xen-percpu-ipi callfuncsingle7
71: 0 0 0 0 0 0
0 0 xen-percpu-ipi irqwork7
72: 100 0 0 0 0 0
0 0 xen-dyn-event xenbus
73: 0 0 0 0 0 0
0 0 xen-dyn-virq xen-pcpu
74: 0 0 0 0 0 0
0 0 xen-dyn-virq hvc_console
75: 6809 0 0 0 0 0
0 0 xen-pirq-msi 0000:00:1f.2
76: 470660 0 0 0 0 0
0 0 xen-pirq-msi xhci_hcd
77: 608 0 0 0 0 0
0 0 xen-pirq-msi i915
78: 26 0 0 0 0 0
0 0 xen-pirq-msi mei_me
79: 243 0 0 0 0 0
0 0 xen-pirq-msi em1
80: 315 0 0 0 0 0
0 0 xen-pirq-msi-x p4p1-rx-0
81: 104 0 0 0 0 0
0 0 xen-pirq-msi-x p4p1-tx-0
82: 2 0 0 0 0 0
0 0 xen-pirq-msi-x p4p1
83: 59 0 0 0 0 0
0 0 xen-pirq-msi snd_hda_intel
84: 48 0 0 0 0 0
0 0 xen-dyn-event evtchn:xenstored
85: 0 0 0 0 0 0
0 0 xen-dyn-event evtchn:xenstored
NMI: 0 0 0 0 0 0
0 0 Non-maskable interrupts
LOC: 0 0 0 0 0 0
0 0 Local timer interrupts
SPU: 0 0 0 0 0 0
0 0 Spurious interrupts
PMI: 0 0 0 0 0 0
0 0 Performance monitoring interrupts
IWI: 1 0 0 0 0 0
0 0 IRQ work interrupts
RTR: 0 0 0 0 0 0
0 0 APIC ICR read retries
RES: 44788 33111 267650 5641 35515 2312
10730 2479 Rescheduling interrupts
CAL: 929 1202 1164 1314 1061 1296
1156 1097 Function call interrupts
TLB: 0 0 0 0 0 0
0 0 TLB shootdowns
TRM: 0 0 0 0 0 0
0 0 Thermal event interrupts
THR: 0 0 0 0 0 0
0 0 Threshold APIC interrupts
MCE: 0 0 0 0 0 0
0 0 Machine check exceptions
MCP: 1 1 1 1 1 1
1 1 Machine check polls
HYP: 538069 43568 273140 12112 44323 7871
18949 7924 Hypervisor callback interrupts
Example of another device driver output with Hypervisor loaded:
04:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network
Connection
Subsystem: Intel Corporation Device 0000
Physical Slot: 1-3
Flags: bus master, fast devsel, latency 0, IRQ 19
Memory at f5600000 (32-bit, non-prefetchable) [size=128K]
I/O ports at b000 [size=32]
Memory at f5620000 (32-bit, non-prefetchable) [size=16K]
Capabilities: [c8] Power Management version 2
Capabilities: [d0] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [e0] Express Endpoint, MSI 00
Capabilities: [a0] MSI-X: Enable+ Count=5 Masked-
Capabilities: [100] Advanced Error Reporting
Capabilities: [140] Device Serial Number
Kernel driver in use: e1000e
Kernel modules: e1000e
Example of the same device driver output on Bare metal:
04:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network
Connection
Subsystem: Intel Corporation Device 0000
Physical Slot: 1-3
Flags: bus master, fast devsel, latency 0, IRQ 19
Memory at f5600000 (32-bit, non-prefetchable) [size=128K]
I/O ports at b000 [size=32]
Memory at f5620000 (32-bit, non-prefetchable) [size=16K]
Capabilities: [c8] Power Management version 2
Capabilities: [d0] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [e0] Express Endpoint, MSI 00
Capabilities: [a0] MSI-X: Enable+ Count=5 Masked-
Capabilities: [100] Advanced Error Reporting
Capabilities: [140] Device Serial Number
Kernel driver in use: e1000e
Kernel modules: e1000e
Bare metal:
cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3 CPU4 CPU5
CPU6 CPU7
0: 36 0 0 0 0 0
0 0 IR-IO-APIC-edge timer
8: 0 0 0 1 0 0
0 0 IR-IO-APIC-edge rtc0
9: 0 0 1 0 1 1
0 0 IR-IO-APIC-fasteoi acpi
16: 2 0 14 0 13 0
0 2 IR-IO-APIC 16-fasteoi ehci_hcd:usb3
18: 24420 225 39883 43 134488 1044
1085 46 IR-IO-APIC 18-fasteoi ath9k
23: 0 0 20 0 0 1
3 11 IR-IO-APIC 23-fasteoi ehci_hcd:usb4
24: 0 0 0 0 0 0
0 0 DMAR_MSI-edge dmar0
25: 0 0 0 0 0 0
0 0 DMAR_MSI-edge dmar1
26: 4119 1016 1074 519 4677 978
573 432 IR-PCI-MSI-edge 0000:00:1f.2
27: 337125 47893 708965 4049 53940667 263303
87847 4958 IR-PCI-MSI-edge xhci_hcd
28: 562 0 0 0 5 1
2 0 IR-PCI-MSI-edge i915
29: 7 0 0 0 4 8
0 6 IR-PCI-MSI-edge mei_me
30: 2222 23 2721 5 5810 181
149 5 IR-PCI-MSI-edge eth0
31: 1504 17 2196 6 6581 95
77 6 IR-PCI-MSI-edge eth1-rx-0
32: 2 0 1 0 3 0
0 0 IR-PCI-MSI-edge eth1-tx-0
33: 0 0 0 0 0 1
0 1 IR-PCI-MSI-edge eth1
34: 14 6 0 0 0 38
0 2 IR-PCI-MSI-edge snd_hda_intel
35: 4 3 0 0 0 178
0 3 IR-PCI-MSI-edge snd_hda_intel
NMI: 11 105 87 1 75 2
9 1 Non-maskable interrupts
LOC: 132282 1175126 905037 79133 428383 22748
118085 12513 Local timer interrupts
SPU: 0 0 0 0 0 0
0 0 Spurious interrupts
PMI: 11 105 87 1 75 2
9 1 Performance monitoring interrupts
IWI: 1 0 0 0 0 0
0 0 IRQ work interrupts
RTR: 0 0 0 0 0 0
0 0 APIC ICR read retries
RES: 36810 28341 99815 953 60355 1142
878 823 Rescheduling interrupts
CAL: 2134 1928 1793 1965 1917 1879
1877 1882 Function call interrupts
TLB: 8064 365 20958 28575 37592 113
28495 445 TLB shootdowns
TRM: 0 0 0 0 0 0
0 0 Thermal event interrupts
THR: 0 0 0 0 0 0
0 0 Threshold APIC interrupts
MCE: 0 0 0 0 0 0
0 0 Machine check exceptions
MCP: 9 9 9 9 9 9
9 9 Machine check polls
THR: 0 0 0 0 0 0
0 0 Hypervisor callback interrupts
next prev parent reply other threads:[~2015-09-08 16:02 UTC|newest]
Thread overview: 22+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <55E6C83402000078000D7CF5@prv-mh.provo.novell.com>
[not found] ` <1981596850.505327.1441214239184.JavaMail.yahoo@mail.yahoo.com>
2015-09-03 10:15 ` xhci_hcd intterrupt affinity in Dom0/DomU limited to single interrupt Jan Beulich
2015-09-03 12:04 ` Justin Acker
2015-09-03 15:04 ` Jan Beulich
2015-09-03 16:52 ` Justin Acker
2015-09-04 7:41 ` Jan Beulich
2015-09-08 16:02 ` Justin Acker [this message]
2015-09-09 6:48 ` Jan Beulich
2015-09-10 16:20 ` Justin Acker
2015-09-11 10:03 ` Jan Beulich
2015-09-16 20:31 ` Justin Acker
2015-09-21 12:53 ` Jan Beulich
[not found] <1441121643.26292.63.camel@citrix.com>
[not found] ` <800613365.4285959.1441128848192.JavaMail.yahoo@mail.yahoo.com>
2015-09-01 17:39 ` Justin Acker
2015-09-01 20:56 ` Konrad Rzeszutek Wilk
2015-09-01 21:38 ` Boris Ostrovsky
2015-09-01 23:09 ` Justin Acker
2015-09-02 12:53 ` Konrad Rzeszutek Wilk
2015-09-02 13:49 ` Ian Campbell
2015-09-02 17:12 ` Justin Acker
2015-09-02 17:02 ` Justin Acker
2015-09-02 13:47 ` David Vrabel
2015-09-02 17:25 ` Justin Acker
2015-09-02 17:35 ` David Vrabel
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1441728141.8074.11.camel@localhost \
--to=ackerj67@yahoo.com \
--cc=JBeulich@suse.com \
--cc=xen-devel@lists.xenproject.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.