From: Justin Acker <ackerj67@yahoo.com>
To: Jan Beulich <JBeulich@suse.com>
Cc: xen-devel <xen-devel@lists.xenproject.org>
Subject: Re: xhci_hcd intterrupt affinity in Dom0/DomU limited to single interrupt
Date: Tue, 08 Sep 2015 12:02:21 -0400 [thread overview]
Message-ID: <1441728141.8074.11.camel@localhost> (raw)
In-Reply-To: <55E9675E020000780009F822@prv-mh.provo.novell.com>
On Fri, 2015-09-04 at 01:41 -0600, Jan Beulich wrote:
> >>> On 03.09.15 at 18:52, <ackerj67@yahoo.com> wrote:
> > On Thu, 2015-09-03 at 09:04 -0600, Jan Beulich wrote:
> >> >>> On 03.09.15 at 14:04, <ackerj67@yahoo.com> wrote:
> >> > I am still confused as to whether any device, or in this case xhci_hcd,
> >> > can use more than one cpu at any given time. My understanding based on
> >> > David's response is that it cannot due to the event channel mapping. The
> >> > device interrupt can be pinned to a specific cpu by specifying the
> >> > affinity. I was hoping there was a way to allow the driver's interrupt to be
> >> > scheduled to use more than 1 CPU at any given time.
> >>
> >> The problem is that you're mixing up two things: devices and
> >> interrupts. Any individual interrupt can only be serviced by a single
> >> CPU at a time, due to the way event channels get bound. Any
> >> individual device can have more than one interrupt (MSI or MSI-X),
> >> and then each of these interrupts can be serviced on different
> >> CPUs.
> >
> > Thanks for clarifying. To the original question, with respect to my
> > limited understanding of the event channels and interrupts, each
> > interrupt can be serviced on a different CPU using irqbalance or setting
> > the affinity manually, but the same interrupt cannot be serviced by more
> > than 1 CPU at a time? If so, is there a way around the 1:1 binding when
> > loading the Dom0 kernel - a flag or option to use the native interrupt
> > scheduling for some set of or all 8 CPUs that the device can schedule
> > interrupts on when not loading the Dom0? The xhci_hcd, as one example,
> > seems to perform better when it is able to have interrupts serviced by
> > multiple CPUs.
>
> I don't follow - we tell you this doesn't work (multiple times and
> different people), and you ask yet another time whether this can
> be made work? Just to make this very clear once again: Under Xen
> (and leaving aside pure HVM guests), interrupt load from a single
> device can be spread across CPUs only when the device uses
> multiple interrupts.
>
> Jan
>
I believe the driver does support use of multiple interrupts based on
the previous explanation of the lspci output where it was established
that the device could use up to 8 interrupts which is what I see on bare
metal. What would cause the interrupt usage to be limited to a single
CPU when the hypervisor is loaded? The device mentioned, xhci_hcd can
use multiple interrupts on baremetal and the other devices including the
network interfaces are also bound to a single interrupt when the
hypervisor is loaded but can use multiple interrupts running on
baremetal.
70: 0 0 0 0 0 0
0 133 xen-percpu-ipi callfuncsingle7
71: 0 0 0 0 0 0
0 0 xen-percpu-ipi irqwork7
72: 100 0 0 0 0 0
0 0 xen-dyn-event xenbus
73: 0 0 0 0 0 0
0 0 xen-dyn-virq xen-pcpu
74: 0 0 0 0 0 0
0 0 xen-dyn-virq hvc_console
75: 6809 0 0 0 0 0
0 0 xen-pirq-msi 0000:00:1f.2
76: 470660 0 0 0 0 0
0 0 xen-pirq-msi xhci_hcd
77: 608 0 0 0 0 0
0 0 xen-pirq-msi i915
78: 26 0 0 0 0 0
0 0 xen-pirq-msi mei_me
79: 243 0 0 0 0 0
0 0 xen-pirq-msi em1
80: 315 0 0 0 0 0
0 0 xen-pirq-msi-x p4p1-rx-0
81: 104 0 0 0 0 0
0 0 xen-pirq-msi-x p4p1-tx-0
82: 2 0 0 0 0 0
0 0 xen-pirq-msi-x p4p1
83: 59 0 0 0 0 0
0 0 xen-pirq-msi snd_hda_intel
84: 48 0 0 0 0 0
0 0 xen-dyn-event evtchn:xenstored
85: 0 0 0 0 0 0
0 0 xen-dyn-event evtchn:xenstored
NMI: 0 0 0 0 0 0
0 0 Non-maskable interrupts
LOC: 0 0 0 0 0 0
0 0 Local timer interrupts
SPU: 0 0 0 0 0 0
0 0 Spurious interrupts
PMI: 0 0 0 0 0 0
0 0 Performance monitoring interrupts
IWI: 1 0 0 0 0 0
0 0 IRQ work interrupts
RTR: 0 0 0 0 0 0
0 0 APIC ICR read retries
RES: 44788 33111 267650 5641 35515 2312
10730 2479 Rescheduling interrupts
CAL: 929 1202 1164 1314 1061 1296
1156 1097 Function call interrupts
TLB: 0 0 0 0 0 0
0 0 TLB shootdowns
TRM: 0 0 0 0 0 0
0 0 Thermal event interrupts
THR: 0 0 0 0 0 0
0 0 Threshold APIC interrupts
MCE: 0 0 0 0 0 0
0 0 Machine check exceptions
MCP: 1 1 1 1 1 1
1 1 Machine check polls
HYP: 538069 43568 273140 12112 44323 7871
18949 7924 Hypervisor callback interrupts
Example of another device driver output with Hypervisor loaded:
04:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network
Connection
Subsystem: Intel Corporation Device 0000
Physical Slot: 1-3
Flags: bus master, fast devsel, latency 0, IRQ 19
Memory at f5600000 (32-bit, non-prefetchable) [size=128K]
I/O ports at b000 [size=32]
Memory at f5620000 (32-bit, non-prefetchable) [size=16K]
Capabilities: [c8] Power Management version 2
Capabilities: [d0] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [e0] Express Endpoint, MSI 00
Capabilities: [a0] MSI-X: Enable+ Count=5 Masked-
Capabilities: [100] Advanced Error Reporting
Capabilities: [140] Device Serial Number
Kernel driver in use: e1000e
Kernel modules: e1000e
Example of the same device driver output on Bare metal:
04:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network
Connection
Subsystem: Intel Corporation Device 0000
Physical Slot: 1-3
Flags: bus master, fast devsel, latency 0, IRQ 19
Memory at f5600000 (32-bit, non-prefetchable) [size=128K]
I/O ports at b000 [size=32]
Memory at f5620000 (32-bit, non-prefetchable) [size=16K]
Capabilities: [c8] Power Management version 2
Capabilities: [d0] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [e0] Express Endpoint, MSI 00
Capabilities: [a0] MSI-X: Enable+ Count=5 Masked-
Capabilities: [100] Advanced Error Reporting
Capabilities: [140] Device Serial Number
Kernel driver in use: e1000e
Kernel modules: e1000e
Bare metal:
cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3 CPU4 CPU5
CPU6 CPU7
0: 36 0 0 0 0 0
0 0 IR-IO-APIC-edge timer
8: 0 0 0 1 0 0
0 0 IR-IO-APIC-edge rtc0
9: 0 0 1 0 1 1
0 0 IR-IO-APIC-fasteoi acpi
16: 2 0 14 0 13 0
0 2 IR-IO-APIC 16-fasteoi ehci_hcd:usb3
18: 24420 225 39883 43 134488 1044
1085 46 IR-IO-APIC 18-fasteoi ath9k
23: 0 0 20 0 0 1
3 11 IR-IO-APIC 23-fasteoi ehci_hcd:usb4
24: 0 0 0 0 0 0
0 0 DMAR_MSI-edge dmar0
25: 0 0 0 0 0 0
0 0 DMAR_MSI-edge dmar1
26: 4119 1016 1074 519 4677 978
573 432 IR-PCI-MSI-edge 0000:00:1f.2
27: 337125 47893 708965 4049 53940667 263303
87847 4958 IR-PCI-MSI-edge xhci_hcd
28: 562 0 0 0 5 1
2 0 IR-PCI-MSI-edge i915
29: 7 0 0 0 4 8
0 6 IR-PCI-MSI-edge mei_me
30: 2222 23 2721 5 5810 181
149 5 IR-PCI-MSI-edge eth0
31: 1504 17 2196 6 6581 95
77 6 IR-PCI-MSI-edge eth1-rx-0
32: 2 0 1 0 3 0
0 0 IR-PCI-MSI-edge eth1-tx-0
33: 0 0 0 0 0 1
0 1 IR-PCI-MSI-edge eth1
34: 14 6 0 0 0 38
0 2 IR-PCI-MSI-edge snd_hda_intel
35: 4 3 0 0 0 178
0 3 IR-PCI-MSI-edge snd_hda_intel
NMI: 11 105 87 1 75 2
9 1 Non-maskable interrupts
LOC: 132282 1175126 905037 79133 428383 22748
118085 12513 Local timer interrupts
SPU: 0 0 0 0 0 0
0 0 Spurious interrupts
PMI: 11 105 87 1 75 2
9 1 Performance monitoring interrupts
IWI: 1 0 0 0 0 0
0 0 IRQ work interrupts
RTR: 0 0 0 0 0 0
0 0 APIC ICR read retries
RES: 36810 28341 99815 953 60355 1142
878 823 Rescheduling interrupts
CAL: 2134 1928 1793 1965 1917 1879
1877 1882 Function call interrupts
TLB: 8064 365 20958 28575 37592 113
28495 445 TLB shootdowns
TRM: 0 0 0 0 0 0
0 0 Thermal event interrupts
THR: 0 0 0 0 0 0
0 0 Threshold APIC interrupts
MCE: 0 0 0 0 0 0
0 0 Machine check exceptions
MCP: 9 9 9 9 9 9
9 9 Machine check polls
THR: 0 0 0 0 0 0
0 0 Hypervisor callback interrupts
next prev parent reply other threads:[~2015-09-08 16:02 UTC|newest]
Thread overview: 22+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <55E6C83402000078000D7CF5@prv-mh.provo.novell.com>
[not found] ` <1981596850.505327.1441214239184.JavaMail.yahoo@mail.yahoo.com>
2015-09-03 10:15 ` xhci_hcd intterrupt affinity in Dom0/DomU limited to single interrupt Jan Beulich
2015-09-03 12:04 ` Justin Acker
2015-09-03 15:04 ` Jan Beulich
2015-09-03 16:52 ` Justin Acker
2015-09-04 7:41 ` Jan Beulich
2015-09-08 16:02 ` Justin Acker [this message]
2015-09-09 6:48 ` Jan Beulich
2015-09-10 16:20 ` Justin Acker
2015-09-11 10:03 ` Jan Beulich
2015-09-16 20:31 ` Justin Acker
2015-09-21 12:53 ` Jan Beulich
[not found] <1441121643.26292.63.camel@citrix.com>
[not found] ` <800613365.4285959.1441128848192.JavaMail.yahoo@mail.yahoo.com>
2015-09-01 17:39 ` Justin Acker
2015-09-01 20:56 ` Konrad Rzeszutek Wilk
2015-09-01 21:38 ` Boris Ostrovsky
2015-09-01 23:09 ` Justin Acker
2015-09-02 12:53 ` Konrad Rzeszutek Wilk
2015-09-02 13:49 ` Ian Campbell
2015-09-02 17:12 ` Justin Acker
2015-09-02 17:02 ` Justin Acker
2015-09-02 13:47 ` David Vrabel
2015-09-02 17:25 ` Justin Acker
2015-09-02 17:35 ` David Vrabel
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1441728141.8074.11.camel@localhost \
--to=ackerj67@yahoo.com \
--cc=JBeulich@suse.com \
--cc=xen-devel@lists.xenproject.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).