From mboxrd@z Thu Jan 1 00:00:00 1970 From: Konrad Rzeszutek Wilk Subject: Re: xhci_hcd intterrupt affinity in Dom0/DomU limited to single interrupt Date: Wed, 2 Sep 2015 08:53:51 -0400 Message-ID: <20150902125351.GA30987@l.oracle.com> References: <1441121643.26292.63.camel@citrix.com> <800613365.4285959.1441128848192.JavaMail.yahoo@mail.yahoo.com> <631331126.1156575.1441129186888.JavaMail.yahoo@mail.yahoo.com> <20150901205657.GA18532@l.oracle.com> <1558077830.37674.1441148978455.JavaMail.yahoo@mail.yahoo.com> Mime-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Return-path: Content-Disposition: inline In-Reply-To: <1558077830.37674.1441148978455.JavaMail.yahoo@mail.yahoo.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Justin Acker Cc: "boris.ostrovsky@oracle.com" , "xen-devel@lists.xen.org" List-Id: xen-devel@lists.xenproject.org On Tue, Sep 01, 2015 at 11:09:38PM +0000, Justin Acker wrote: > = > From: Konrad Rzeszutek Wilk > To: Justin Acker = > Cc: "xen-devel@lists.xen.org" ; boris.ostrovsky@= oracle.com = > Sent: Tuesday, September 1, 2015 4:56 PM > Subject: Re: [Xen-devel] xhci_hcd intterrupt affinity in Dom0/DomU limit= ed to single interrupt > = > On Tue, Sep 01, 2015 at 05:39:46PM +0000, Justin Acker wrote: > > Taking this to the dev list from users. = > > = > > Is there a way to force or enable pirq delivery to a set of cpus as opp= osed to single device from being a assigned a single pirq so that its inter= rupt can be distributed across multiple cpus? I believe the device drivers = do support multiple queues when run natively without the Dom0 loaded. The d= evice in question is the xhci_hcd driver for which I/O transfers seem to be= slowed when the Dom0 is loaded. The behavior seems to pass through to the = DomU if pass through is enabled. I found some similar threads, but most rel= ate to Ethernet controllers. I tried some of the x2apic and x2apic_phys dom= 0 kernel arguments, but none distributed the pirqs. Based on the reading re= lating to IRQs for Xen, I think pinning the pirqs to cpu0 is done to avoid = an interrupt storm. I tried IRQ balance and when configured/adjusted it wil= l balance individual pirqs, but not multiple interrupts. > = > Yes. You can do it with smp affinity: > = > https://cs.uwaterloo.ca/~brecht/servers/apic/SMP-affinity.txt > Yes, this does allow for assigning a specific interrupt to a single cpu, = but it will not spread the interrupt load across a defined group or all cpu= s. Is it possible to define a range of CPUs or spread the interrupt load fo= r a device across all cpus as it does with a native kernel without the Dom0= loaded? It should be. Did you try giving it an mask that puts the interrupts on all= the CPUs? (0xf) ? > = > I don't follow the "behavior seems to pass through to the DomU if pass th= rough is enabled" ? > The device interrupts are limited to a single pirq if the device is used = directly in the Dom0. If the device is passed through to a DomU - i.e. the = xhci_hcd controller - then the DomU cannot spread the interrupt load across= the cpus in the VM. = Why? How are you seeing this? The method by which you use smp affinity shou= ld be exactly the same. And it looks to me that the device has a single pirq as well when booting a= s baremetal right? So the issue here is that you want to spread the interrupt delivery to happ= en across all of the CPUs. The smp_affinity should do it. Did you try modifying it by= hand (you may want to kill irqbalance when you do this just to make sure it does not writ= e its own values in)? > = > > = > > = > > = > > With irqbalance enabled in Dom0: > = > What version? There was a bug in it where it would never distribute the I= RQs properly > across the CPUs. > irqbalance version 1.0.7. > = > Boris (CC-ed) might remember the upstream patch that made this work prope= rly? > = > = > > = > > =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 CPU0=A0=A0=A0=A0=A0=A0 CPU1=A0=A0=A0=A0= =A0=A0 CPU2=A0=A0=A0=A0=A0=A0 CPU3=A0=A0=A0=A0=A0=A0 CPU4=A0=A0=A0=A0=A0=A0= CPU5=A0=A0=A0=A0=A0=A0 CPU6=A0=A0=A0=A0=A0=A0 CPU7=A0=A0=A0=A0=A0 = > > =A076:=A0=A0=A0=A0=A0 11304=A0=A0=A0=A0=A0=A0=A0=A0=A0 0=A0=A0=A0=A0 14= 9579=A0=A0=A0=A0=A0=A0=A0=A0=A0 0=A0=A0=A0=A0=A0=A0=A0=A0=A0 0=A0=A0=A0=A0= =A0=A0=A0=A0=A0 0=A0=A0=A0=A0=A0=A0=A0=A0=A0 0=A0=A0=A0=A0=A0=A0=A0=A0=A0 0= =A0 xen-pirq-msi=A0=A0=A0=A0=A0=A0 0000:00:1f.2 > > =A077:=A0=A0=A0=A0=A0=A0 1243=A0=A0=A0=A0=A0=A0=A0=A0=A0 0=A0=A0=A0=A0= =A0=A0=A0=A0=A0 0=A0=A0=A0=A0=A0 35447=A0=A0=A0=A0=A0=A0=A0=A0=A0 0=A0=A0= =A0=A0=A0=A0=A0=A0=A0 0=A0=A0=A0=A0=A0=A0=A0=A0=A0 0=A0=A0=A0=A0=A0=A0=A0= =A0=A0 0=A0 xen-pirq-msi=A0=A0=A0=A0=A0=A0 radeon > > =A078:=A0=A0=A0=A0=A0 82521=A0=A0=A0=A0=A0=A0=A0=A0=A0 0=A0=A0=A0=A0=A0= =A0=A0=A0=A0 0=A0=A0=A0=A0=A0=A0=A0=A0=A0 0=A0=A0=A0=A0=A0=A0=A0=A0=A0 0=A0= =A0=A0=A0=A0=A0=A0=A0=A0 0=A0=A0=A0=A0=A0=A0=A0=A0=A0 0=A0=A0=A0=A0=A0=A0= =A0=A0=A0 0=A0 xen-pirq-msi=A0=A0=A0=A0=A0=A0 xhci_hcd > > =A079:=A0=A0=A0=A0=A0=A0=A0=A0 23=A0=A0=A0=A0=A0=A0=A0=A0=A0 0=A0=A0=A0= =A0=A0=A0=A0=A0=A0 0=A0=A0=A0=A0=A0=A0=A0=A0=A0 0=A0=A0=A0=A0=A0=A0=A0=A0= =A0 0=A0=A0=A0=A0=A0=A0=A0=A0=A0 0=A0=A0=A0=A0=A0=A0=A0=A0=A0 0=A0=A0=A0=A0= =A0=A0=A0=A0=A0 0=A0 xen-pirq-msi=A0=A0=A0=A0=A0=A0 mei_me > > =A080:=A0=A0=A0=A0=A0=A0=A0=A0 11=A0=A0=A0=A0=A0=A0=A0=A0=A0 0=A0=A0=A0= =A0=A0=A0=A0=A0=A0 0=A0=A0=A0=A0=A0=A0=A0=A0=A0 0=A0=A0=A0=A0=A0=A0=A0=A0= =A0 0=A0=A0=A0=A0=A0=A0=A0 741=A0=A0=A0=A0=A0=A0=A0=A0=A0 0=A0=A0=A0=A0=A0= =A0=A0=A0=A0 0=A0 xen-pirq-msi=A0=A0=A0=A0=A0=A0 em1 > > =A081:=A0=A0=A0=A0=A0=A0=A0 350=A0=A0=A0=A0=A0=A0=A0=A0=A0 0=A0=A0=A0= =A0=A0=A0=A0=A0=A0 0=A0=A0=A0=A0=A0=A0=A0=A0=A0 0=A0=A0=A0=A0=A0=A0 1671=A0= =A0=A0=A0=A0=A0=A0=A0=A0 0=A0=A0=A0=A0=A0=A0=A0=A0=A0 0=A0=A0=A0=A0=A0=A0= =A0=A0=A0 0=A0 xen-pirq-msi=A0=A0=A0=A0=A0=A0 iwlwifi > > =A082:=A0=A0=A0=A0=A0=A0=A0 275=A0=A0=A0=A0=A0=A0=A0=A0=A0 0=A0=A0=A0= =A0=A0=A0=A0=A0=A0 0=A0=A0=A0=A0=A0=A0=A0=A0=A0 0=A0=A0=A0=A0=A0=A0=A0=A0= =A0 0=A0=A0=A0=A0=A0=A0=A0=A0=A0 0=A0=A0=A0=A0=A0=A0=A0=A0=A0 0=A0=A0=A0=A0= =A0=A0=A0=A0=A0 0=A0 xen-pirq-msi=A0=A0=A0=A0=A0=A0 snd_hda_intel > > = > > With native 3.19 kernel: > > = > > Without Dom0 for the same system from the first message: > > = > > # cat /proc/interrupts > > =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 CPU0=A0=A0=A0=A0=A0=A0 CPU1=A0=A0=A0=A0= =A0=A0 CPU2=A0=A0=A0=A0=A0=A0 CPU3=A0=A0=A0=A0=A0=A0 CPU4=A0=A0=A0=A0=A0=A0= CPU5=A0=A0=A0=A0=A0=A0 CPU6=A0=A0=A0=A0=A0=A0 CPU7=A0=A0=A0=A0=A0 = > > =A0 0:=A0=A0=A0=A0=A0=A0=A0=A0 33=A0=A0=A0=A0=A0=A0=A0=A0=A0 0=A0=A0=A0= =A0=A0=A0=A0=A0=A0 0=A0=A0=A0=A0=A0=A0=A0=A0=A0 0=A0=A0=A0=A0=A0=A0=A0=A0= =A0 0=A0=A0=A0=A0=A0=A0=A0=A0=A0 0=A0=A0=A0=A0=A0=A0=A0=A0=A0 0=A0=A0=A0=A0= =A0=A0=A0=A0=A0 0=A0 IR-IO-APIC-edge=A0=A0=A0=A0=A0 timer > > =A0 8:=A0=A0=A0=A0=A0=A0=A0=A0=A0 0=A0=A0=A0=A0=A0=A0=A0=A0=A0 0=A0=A0= =A0=A0=A0=A0=A0=A0=A0 0=A0=A0=A0=A0=A0=A0=A0=A0=A0 0=A0=A0=A0=A0=A0=A0=A0= =A0=A0 0=A0=A0=A0=A0=A0=A0=A0=A0=A0 0=A0=A0=A0=A0=A0=A0=A0=A0=A0 1=A0=A0=A0= =A0=A0=A0=A0=A0=A0 0=A0 IR-IO-APIC-edge=A0=A0=A0=A0=A0 rtc0 > > =A0 9:=A0=A0=A0=A0=A0=A0=A0=A0 20=A0=A0=A0=A0=A0=A0=A0=A0=A0 0=A0=A0=A0= =A0=A0=A0=A0=A0=A0 0=A0=A0=A0=A0=A0=A0=A0=A0=A0 0=A0=A0=A0=A0=A0=A0=A0=A0= =A0 0=A0=A0=A0=A0=A0=A0=A0=A0=A0 1=A0=A0=A0=A0=A0=A0=A0=A0=A0 1=A0=A0=A0=A0= =A0=A0=A0=A0=A0 1=A0 IR-IO-APIC-fasteoi=A0=A0 acpi > > =A016:=A0=A0=A0=A0=A0=A0=A0=A0 15=A0=A0=A0=A0=A0=A0=A0=A0=A0 0=A0=A0=A0= =A0=A0=A0=A0=A0=A0 8=A0=A0=A0=A0=A0=A0=A0=A0=A0 1=A0=A0=A0=A0=A0=A0=A0=A0= =A0 4=A0=A0=A0=A0=A0=A0=A0=A0=A0 1=A0=A0=A0=A0=A0=A0=A0=A0=A0 1=A0=A0=A0=A0= =A0=A0=A0=A0=A0 1=A0 IR-IO-APIC=A0 16-fasteoi=A0=A0 ehci_hcd:usb3 > > =A018:=A0=A0=A0=A0 703940=A0=A0=A0=A0=A0=A0 5678=A0=A0=A0 1426226=A0=A0= =A0=A0=A0=A0 1303=A0=A0=A0 3938243=A0=A0=A0=A0 111477=A0=A0=A0=A0 757871=A0= =A0=A0=A0=A0=A0=A0 510=A0 IR-IO-APIC=A0 18-fasteoi=A0=A0 ath9k > > =A023:=A0=A0=A0=A0=A0=A0=A0=A0 11=A0=A0=A0=A0=A0=A0=A0=A0=A0 2=A0=A0=A0= =A0=A0=A0=A0=A0=A0 3=A0=A0=A0=A0=A0=A0=A0=A0=A0 0=A0=A0=A0=A0=A0=A0=A0=A0= =A0 0=A0=A0=A0=A0=A0=A0=A0=A0 17=A0=A0=A0=A0=A0=A0=A0=A0=A0 2=A0=A0=A0=A0= =A0=A0=A0=A0=A0 0=A0 IR-IO-APIC=A0 23-fasteoi=A0=A0 ehci_hcd:usb4 > > =A024:=A0=A0=A0=A0=A0=A0=A0=A0=A0 0=A0=A0=A0=A0=A0=A0=A0=A0=A0 0=A0=A0= =A0=A0=A0=A0=A0=A0=A0 0=A0=A0=A0=A0=A0=A0=A0=A0=A0 0=A0=A0=A0=A0=A0=A0=A0= =A0=A0 0=A0=A0=A0=A0=A0=A0=A0=A0=A0 0=A0=A0=A0=A0=A0=A0=A0=A0=A0 0=A0=A0=A0= =A0=A0=A0=A0=A0=A0 0=A0 DMAR_MSI-edge=A0=A0=A0=A0=A0 dmar0 > > =A025:=A0=A0=A0=A0=A0=A0=A0=A0=A0 0=A0=A0=A0=A0=A0=A0=A0=A0=A0 0=A0=A0= =A0=A0=A0=A0=A0=A0=A0 0=A0=A0=A0=A0=A0=A0=A0=A0=A0 0=A0=A0=A0=A0=A0=A0=A0= =A0=A0 0=A0=A0=A0=A0=A0=A0=A0=A0=A0 0=A0=A0=A0=A0=A0=A0=A0=A0=A0 0=A0=A0=A0= =A0=A0=A0=A0=A0=A0 0=A0 DMAR_MSI-edge=A0=A0=A0=A0=A0 dmar1 > > =A026:=A0=A0=A0=A0=A0 20419=A0=A0=A0=A0=A0=A0 1609=A0=A0=A0=A0=A0 26822= =A0=A0=A0=A0=A0=A0=A0 567=A0=A0=A0=A0=A0 62281=A0=A0=A0=A0=A0=A0 5426=A0=A0= =A0=A0=A0 14928=A0=A0=A0=A0=A0=A0=A0 395=A0 IR-PCI-MSI-edge=A0=A0=A0=A0=A0 = 0000:00:1f.2 > > =A027:=A0=A0 17977230=A0=A0=A0=A0 628258=A0=A0 44247270=A0=A0=A0=A0 120= 391 1597809883=A0=A0 14440991=A0 152189328=A0=A0=A0=A0=A0 73322=A0 IR-PCI-M= SI-edge=A0=A0=A0=A0=A0 xhci_hcd > > =A028:=A0=A0=A0=A0=A0=A0=A0 563=A0=A0=A0=A0=A0=A0=A0=A0=A0 0=A0=A0=A0= =A0=A0=A0=A0=A0=A0 0=A0=A0=A0=A0=A0=A0=A0=A0=A0 0=A0=A0=A0=A0=A0=A0=A0=A0= =A0 1=A0=A0=A0=A0=A0=A0=A0=A0=A0 0=A0=A0=A0=A0=A0=A0=A0=A0=A0 6=A0=A0=A0=A0= =A0=A0=A0=A0=A0 0=A0 IR-PCI-MSI-edge=A0=A0=A0=A0=A0 i915 > > =A029:=A0=A0=A0=A0=A0=A0=A0=A0 14=A0=A0=A0=A0=A0=A0=A0=A0=A0 0=A0=A0=A0= =A0=A0=A0=A0=A0=A0 0=A0=A0=A0=A0=A0=A0=A0=A0=A0 4=A0=A0=A0=A0=A0=A0=A0=A0= =A0 2=A0=A0=A0=A0=A0=A0=A0=A0=A0 4=A0=A0=A0=A0=A0=A0=A0=A0=A0 0=A0=A0=A0=A0= =A0=A0=A0=A0=A0 0=A0 IR-PCI-MSI-edge=A0=A0=A0=A0=A0 mei_me > > =A030:=A0=A0=A0=A0=A0 39514=A0=A0=A0=A0=A0=A0 1744=A0=A0=A0=A0=A0 60339= =A0=A0=A0=A0=A0=A0=A0 157=A0=A0=A0=A0 129956=A0=A0=A0=A0=A0 19702=A0=A0=A0= =A0=A0 72140=A0=A0=A0=A0=A0=A0=A0=A0 83=A0 IR-PCI-MSI-edge=A0=A0=A0=A0=A0 e= th0 > > =A031:=A0=A0=A0=A0=A0=A0=A0=A0=A0 3=A0=A0=A0=A0=A0=A0=A0=A0=A0 0=A0=A0= =A0=A0=A0=A0=A0=A0=A0 0=A0=A0=A0=A0=A0=A0=A0=A0=A0 1=A0=A0=A0=A0=A0=A0=A0= =A0 54=A0=A0=A0=A0=A0=A0=A0=A0=A0 0=A0=A0=A0=A0=A0=A0=A0=A0=A0 0=A0=A0=A0= =A0=A0=A0=A0=A0=A0 2=A0 IR-PCI-MSI-edge=A0=A0=A0=A0=A0 snd_hda_intel > > =A032:=A0=A0=A0=A0=A0 28145=A0=A0=A0=A0=A0=A0=A0 284=A0=A0=A0=A0=A0 533= 16=A0=A0=A0=A0=A0=A0=A0=A0 63=A0=A0=A0=A0 139165=A0=A0=A0=A0=A0=A0 4410=A0= =A0=A0=A0=A0 25760=A0=A0=A0=A0=A0=A0=A0=A0 27=A0 IR-PCI-MSI-edge=A0=A0=A0= =A0=A0 eth1-rx-0 > > =A033:=A0=A0=A0=A0=A0=A0 1032=A0=A0=A0=A0=A0=A0=A0=A0 43=A0=A0=A0=A0=A0= =A0 2392=A0=A0=A0=A0=A0=A0=A0=A0=A0 5=A0=A0=A0=A0=A0=A0 1797=A0=A0=A0=A0=A0= =A0=A0 265=A0=A0=A0=A0=A0=A0 1507=A0=A0=A0=A0=A0=A0=A0=A0 20=A0 IR-PCI-MSI-= edge=A0=A0=A0=A0=A0 eth1-tx-0 > > =A034:=A0=A0=A0=A0=A0=A0=A0=A0=A0 0=A0=A0=A0=A0=A0=A0=A0=A0=A0 1=A0=A0= =A0=A0=A0=A0=A0=A0=A0 0=A0=A0=A0=A0=A0=A0=A0=A0=A0 0=A0=A0=A0=A0=A0=A0=A0= =A0=A0 0=A0=A0=A0=A0=A0=A0=A0=A0=A0 1=A0=A0=A0=A0=A0=A0=A0=A0=A0 2=A0=A0=A0= =A0=A0=A0=A0=A0=A0 0=A0 IR-PCI-MSI-edge=A0=A0=A0=A0=A0 eth1 > > =A035:=A0=A0=A0=A0=A0=A0=A0=A0=A0 5=A0=A0=A0=A0=A0=A0=A0=A0=A0 0=A0=A0= =A0=A0=A0=A0=A0=A0=A0 0=A0=A0=A0=A0=A0=A0=A0=A0 12=A0=A0=A0=A0=A0=A0=A0 148= =A0=A0=A0=A0=A0=A0=A0=A0=A0 6=A0=A0=A0=A0=A0=A0=A0=A0=A0 2=A0=A0=A0=A0=A0= =A0=A0=A0=A0 1=A0 IR-PCI-MSI-edge=A0=A0=A0=A0=A0 snd_hda_intel > > = > > = > > The USB controller is an Intel C210: > > = > > 00:14.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset = Family USB xHCI Host Controller (rev 04) (prog-if 30 [XHCI]) > > =A0=A0=A0 Subsystem: Dell Device 053e > > =A0=A0=A0 Flags: bus master, medium devsel, latency 0, IRQ 78 > > =A0=A0=A0 Memory at f7f20000 (64-bit, non-prefetchable) [size=3D64K] > > =A0=A0=A0 Capabilities: [70] Power Management version 2 > > =A0=A0=A0 Capabilities: [80] MSI: Enable+ Count=3D1/8 Maskable- 64bit+ > > =A0=A0=A0 Kernel driver in use: xhci_hcd > > =A0=A0=A0 Kernel modules: xhci_pci > >=A0 =A0 =A0 On Tuesday, September 1, 2015 11:50 AM, Ian Campbell wrote: > >=A0 =A0 = > > = > >=A0 On Tue, 2015-09-01 at 13:56 +0000, Justin Acker wrote: > > > Thanks Ian, > > > = > > > I appreciate the explanation. I believe the device drivers do support = > > > multiple queues when run natively without the Dom0 loaded. The device= in = > > > question is the xhci_hcd driver for which I/O transfers seem to be sl= owed = > > > when the Dom0 is loaded. The behavior seems to pass through to the Do= mU = > > > if pass through is enabled. I found some similar threads, but most re= late = > > > to Ethernet controllers. I tried some of the x2apic and x2apic_phys d= om0 = > > > kernel arguments, but none distributed the pirqs. Based on the readin= g = > > > relating to IRQs for Xen, I think pinning the pirqs to cpu0 is done t= o = > > > avoid an I/O storm. I tried IRQ balance and when configured/adjusted = it = > > > will balance individual pirqs, but not multiple interrupts. = > > > = > > > Is there a way to force or enable pirq delivery to a set of cpus as y= ou = > > > mentioned above or omit a single device from being a assigned a PIRQ = so = > > > that its interrupt can be distributed across all cpus? = > > = > > A PIRQ is the way an interrupt is exposed to a PV guest, without it the= re > > would be no interrupt at all. > > = > > I'm afraid I'm out of my depth WRT how x86/MSIs and Xen x86/PV pirqs > > interact, in particular WRT configuring which set of CPUs can have the = IRQ > > delivered. > > = > > If no one else chimes in soon I'd suggest taking this to the dev list, = at > > the very least someone who knows what they are talking about (i.e. other > > than me) might be able to help. > > = > > Ian. > > = > > = > > = > >=A0 = > = > > _______________________________________________ > > Xen-devel mailing list > > Xen-devel@lists.xen.org > > http://lists.xen.org/xen-devel > = > = > =