From mboxrd@z Thu Jan 1 00:00:00 1970 From: Justin Acker Subject: Re: xhci_hcd intterrupt affinity in Dom0/DomU limited to single interrupt Date: Wed, 2 Sep 2015 17:02:36 +0000 (UTC) Message-ID: <1517587364.479748.1441213356801.JavaMail.yahoo@mail.yahoo.com> References: <20150902125351.GA30987@l.oracle.com> Reply-To: Justin Acker Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============6714354145022443417==" Return-path: In-Reply-To: <20150902125351.GA30987@l.oracle.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Konrad Rzeszutek Wilk Cc: "boris.ostrovsky@oracle.com" , "xen-devel@lists.xen.org" List-Id: xen-devel@lists.xenproject.org --===============6714354145022443417== Content-Type: multipart/alternative; boundary="----=_Part_479747_808262945.1441213356790" Content-Length: 50427 ------=_Part_479747_808262945.1441213356790 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable From: Konrad Rzeszutek Wilk To: Justin Acker =20 Cc: "xen-devel@lists.xen.org" ; "boris.ostrovsky@o= racle.com" =20 Sent: Wednesday, September 2, 2015 8:53 AM Subject: Re: [Xen-devel] xhci_hcd intterrupt affinity in Dom0/DomU limited= to single interrupt =20 On Tue, Sep 01, 2015 at 11:09:38PM +0000, Justin Acker wrote: >=20 >=C2=A0 =C2=A0 =C2=A0 From: Konrad Rzeszutek Wilk >=C2=A0 To: Justin Acker =20 > Cc: "xen-devel@lists.xen.org" ; boris.ostrovsky@= oracle.com=20 >=C2=A0 Sent: Tuesday, September 1, 2015 4:56 PM >=C2=A0 Subject: Re: [Xen-devel] xhci_hcd intterrupt affinity in Dom0/DomU = limited to single interrupt >=C2=A0 =C2=A0=20 > On Tue, Sep 01, 2015 at 05:39:46PM +0000, Justin Acker wrote: > > Taking this to the dev list from users.=20 > >=20 > > Is there a way to force or enable pirq delivery to a set of cpus as opp= osed to single device from being a assigned a single pirq so that its inter= rupt can be distributed across multiple cpus? I believe the device drivers = do support multiple queues when run natively without the Dom0 loaded. The d= evice in question is the xhci_hcd driver for which I/O transfers seem to be= slowed when the Dom0 is loaded. The behavior seems to pass through to the = DomU if pass through is enabled. I found some similar threads, but most rel= ate to Ethernet controllers. I tried some of the x2apic and x2apic_phys dom= 0 kernel arguments, but none distributed the pirqs. Based on the reading re= lating to IRQs for Xen, I think pinning the pirqs to cpu0 is done to avoid = an interrupt storm. I tried IRQ balance and when configured/adjusted it wil= l balance individual pirqs, but not multiple interrupts. >=20 > Yes. You can do it with smp affinity: >=20 > https://cs.uwaterloo.ca/~brecht/servers/apic/SMP-affinity.txt > Yes, this does allow for assigning a specific interrupt to a single cpu, = but it will not spread the interrupt load across a defined group or all cpu= s. Is it possible to define a range of CPUs or spread the interrupt load fo= r a device across all cpus as it does with a native kernel without the Dom0= loaded? It should be. Did you try giving it an mask that puts the interrupts on all= the CPUs? (0xf) ? >=20 > I don't follow the "behavior seems to pass through to the DomU if pass th= rough is enabled" ? > The device interrupts are limited to a single pirq if the device is used = directly in the Dom0. If the device is passed through to a DomU - i.e. the = xhci_hcd controller - then the DomU cannot spread the interrupt load across= the cpus in the VM.=20 Why? How are you seeing this? The method by which you use smp affinity shou= ld be exactly the same. And it looks to me that the device has a single pirq as well when booting a= s baremetal right? On baremetal, it uses all 8 cpus for affinity as noted below (IRQ27) compar= ed to (IRQ78) in the Dom0. =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 CPU0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 CPU1=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 CPU2=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 CPU3=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 CPU4=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 CPU5=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 CPU6=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 CPU= 7=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 baremetal:=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0 27:=C2=A0=C2=A0 17977230=C2=A0= =C2=A0=C2=A0=C2=A0 628258=C2=A0=C2=A0 44247270=C2=A0=C2=A0=C2=A0=C2=A0 1203= 91 1597809883=C2=A0=C2=A0 14440991=C2=A0 152189328=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 73322=C2=A0 IR-PCI-MSI-edge=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 xhci_hcdDo= m0 or DomU passed through: 78:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 82521=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 0=C2=A0 xen-pirq-msi=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 xhci_= hcd So the issue here is that you want to spread the interrupt delivery to happ= en across all of the CPUs. The smp_affinity should do it. Did you try modifying it by= hand (you may want to kill irqbalance when you do this just to make sure it does not writ= e its own values in)? Yes, this would be great if there is a way to spread the affinity across al= l cpus or a specified set of CPUs similar to the native kernel behavior. I = guess it would be spread across a set or all pirqs? With irqbalance disable= d, I did adjust try the interrupt affinity manually (i.e. echo ff /prox/irq= /78/smp_affinity). The interrupt will move to the specified CPU (0 through = 7). Without specifying the affinity manually, it does look like it's mapped= to all cpus by default. With the Dom0 loaded, cat /proc/irq/78/smp_affinit= y returns ff, but the interrupt never appears to be scheduled on more than = cpu.=20 >=20 > >=20 > >=20 > >=20 > > With irqbalance enabled in Dom0: >=20 > What version? There was a bug in it where it would never distribute the I= RQs properly > across the CPUs. > irqbalance version 1.0.7. >=20 > Boris (CC-ed) might remember the upstream patch that made this work prope= rly? >=20 >=20 > >=20 > > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 CPU0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 CPU1=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 CPU= 2=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 CPU3=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 CPU4=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 CPU5=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 CPU6=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 CPU7=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=20 > > =C2=A076:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 11304=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0 149579=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0 xen-pirq-msi=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 0000:00:1f.2 > > =C2=A077:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 1243=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 35447=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0 xen-pirq-msi= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 radeon > > =C2=A078:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 82521=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0= =C2=A0 xen-pirq-msi=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 xhci_hcd > > =C2=A079:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 23=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 0=C2=A0 xen-pirq-msi=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 mei_m= e > > =C2=A080:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 11=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 741=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 0=C2=A0 xen-pirq-msi=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 em1 > > =C2=A081:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 350=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 1671=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0 x= en-pirq-msi=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 iwlwifi > > =C2=A082:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 275=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 0=C2=A0 xen-pirq-msi=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 snd_hda_int= el > >=20 > > With native 3.19 kernel: > >=20 > > Without Dom0 for the same system from the first message: > >=20 > > # cat /proc/interrupts > > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 CPU0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 CPU1=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 CPU= 2=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 CPU3=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 CPU4=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 CPU5=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 CPU6=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 CPU7=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=20 > > =C2=A0 0:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 33=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 0=C2=A0 IR-IO-APIC-edge=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 timer > > =C2=A0 8:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 1=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 0=C2=A0 IR-IO-APIC-edge=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 rtc0 > > =C2=A0 9:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 20=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 1=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 1=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 1=C2=A0 IR-IO-APIC-fasteoi=C2=A0=C2=A0 acpi > > =C2=A016:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 15=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 8=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0 1=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 4=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 1=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 1=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 1=C2=A0 IR-IO-APIC=C2=A0 16-fasteoi=C2=A0=C2=A0 ehci_hcd:usb3 > > =C2=A018:=C2=A0=C2=A0=C2=A0=C2=A0 703940=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 5678=C2=A0=C2=A0=C2=A0 1426226=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 1= 303=C2=A0=C2=A0=C2=A0 3938243=C2=A0=C2=A0=C2=A0=C2=A0 111477=C2=A0=C2=A0=C2= =A0=C2=A0 757871=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 510=C2=A0 IR-IO-= APIC=C2=A0 18-fasteoi=C2=A0=C2=A0 ath9k > > =C2=A023:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 11=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 2=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 3=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 17=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 2=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 0=C2=A0 IR-IO-APIC=C2=A0 23-fasteoi=C2=A0=C2=A0 ehci_hcd:usb4 > > =C2=A024:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 0=C2=A0 DMAR_MSI-edge=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 dmar0 > > =C2=A025:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 0=C2=A0 DMAR_MSI-edge=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 dmar1 > > =C2=A026:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 20419=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 1609=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 26822=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 567=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 62281=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 5426=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 14928=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 395=C2=A0 IR-PCI-MSI-edge=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 0000:00:1f.2 > > =C2=A027:=C2=A0=C2=A0 17977230=C2=A0=C2=A0=C2=A0=C2=A0 628258=C2=A0=C2= =A0 44247270=C2=A0=C2=A0=C2=A0=C2=A0 120391 1597809883=C2=A0=C2=A0 14440991= =C2=A0 152189328=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 73322=C2=A0 IR-PCI-MSI-edge= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 xhci_hcd > > =C2=A028:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 563=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 1=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 6=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 0=C2=A0 IR-PCI-MSI-edge=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 i915 > > =C2=A029:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 14=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0 4=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 2=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 4=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 0=C2=A0 IR-PCI-MSI-edge=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 mei_me > > =C2=A030:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 39514=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 1744=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 60339=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 157=C2=A0=C2=A0=C2=A0=C2=A0 129956=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 19702=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 72140=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 83=C2=A0 IR-PCI-MSI-edge=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0 eth0 > > =C2=A031:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 3=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 1=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 54=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0 2=C2=A0 IR-PCI-MSI-edge=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 snd_hda_int= el > > =C2=A032:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 28145=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 284=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 53316=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 63=C2=A0=C2=A0=C2=A0=C2=A0 139165=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 4410=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 25760=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 27=C2=A0 IR-PCI-MSI-edge=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 eth1-rx-0 > > =C2=A033:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 1032=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 43=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 2392=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 5=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 1797=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 265=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 1507=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 20=C2=A0 IR-PCI-MSI-edge=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 eth1-tx-0 > > =C2=A034:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 1=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 1=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 2=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 0=C2=A0 IR-PCI-MSI-edge=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 eth1 > > =C2=A035:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 5=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 12=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 148=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 6=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 2=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = 1=C2=A0 IR-PCI-MSI-edge=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 snd_hda_intel > >=20 > >=20 > > The USB controller is an Intel C210: > >=20 > > 00:14.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset = Family USB xHCI Host Controller (rev 04) (prog-if 30 [XHCI]) > > =C2=A0=C2=A0=C2=A0 Subsystem: Dell Device 053e > > =C2=A0=C2=A0=C2=A0 Flags: bus master, medium devsel, latency 0, IRQ 78 > > =C2=A0=C2=A0=C2=A0 Memory at f7f20000 (64-bit, non-prefetchable) [size= =3D64K] > > =C2=A0=C2=A0=C2=A0 Capabilities: [70] Power Management version 2 > > =C2=A0=C2=A0=C2=A0 Capabilities: [80] MSI: Enable+ Count=3D1/8 Maskable= - 64bit+ > > =C2=A0=C2=A0=C2=A0 Kernel driver in use: xhci_hcd > > =C2=A0=C2=A0=C2=A0 Kernel modules: xhci_pci > >=C2=A0 =C2=A0 =C2=A0 On Tuesday, September 1, 2015 11:50 AM, Ian Campbel= l wrote: > >=C2=A0 =C2=A0=20 > >=20 > >=C2=A0 On Tue, 2015-09-01 at 13:56 +0000, Justin Acker wrote: > > > Thanks Ian, > > >=20 > > > I appreciate the explanation. I believe the device drivers do support= =20 > > > multiple queues when run natively without the Dom0 loaded. The device= in=20 > > > question is the xhci_hcd driver for which I/O transfers seem to be sl= owed=20 > > > when the Dom0 is loaded. The behavior seems to pass through to the Do= mU=20 > > > if pass through is enabled. I found some similar threads, but most re= late=20 > > > to Ethernet controllers. I tried some of the x2apic and x2apic_phys d= om0=20 > > > kernel arguments, but none distributed the pirqs. Based on the readin= g=20 > > > relating to IRQs for Xen, I think pinning the pirqs to cpu0 is done t= o=20 > > > avoid an I/O storm. I tried IRQ balance and when configured/adjusted = it=20 > > > will balance individual pirqs, but not multiple interrupts.=20 > > >=20 > > > Is there a way to force or enable pirq delivery to a set of cpus as y= ou=20 > > > mentioned above or omit a single device from being a assigned a PIRQ = so=20 > > > that its interrupt can be distributed across all cpus?=20 > >=20 > > A PIRQ is the way an interrupt is exposed to a PV guest, without it the= re > > would be no interrupt at all. > >=20 > > I'm afraid I'm out of my depth WRT how x86/MSIs and Xen x86/PV pirqs > > interact, in particular WRT configuring which set of CPUs can have the = IRQ > > delivered. > >=20 > > If no one else chimes in soon I'd suggest taking this to the dev list, = at > > the very least someone who knows what they are talking about (i.e. othe= r > > than me) might be able to help. > >=20 > > Ian. > >=20 > >=20 > >=20 > >=C2=A0=20 >=20 > > _______________________________________________ > > Xen-devel mailing list > > Xen-devel@lists.xen.org > > http://lists.xen.org/xen-devel >=20 >=20 >=C2=A0=20 ------=_Part_479747_808262945.1441213356790 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable

From: Konrad Rzeszutek Wilk <konra= d.wilk@oracle.com>
To:<= /b> Justin Acker <ackerj67@yahoo.com>
Cc: "xen-devel@lists.xen.org" <xen-devel@lists.xen= .org>; "boris.ostrovsky@oracle.com" <boris.ostrovsky@oracle.com> <= br> Sent: Wednesday, Septe= mber 2, 2015 8:53 AM
Subject: Re: [Xen-devel] xhci_hcd intterrupt affinity in Dom0/DomU limited t= o single interrupt

On Tue, Sep 01, 2015 at 11:09:38PM +00= 00, Justin Acker wrote:
>
> = ;     From: Konrad Rzeszutek Wilk <k= onrad.wilk@oracle.com>
>  To: Justin Acker= <ackerj67@yahoo.com>
> Cc:= "xen-devel@lists.xen.org" <xen-devel@lists.xen.org>; boris.ostrovsky@oracle.com
>  Sent: Tuesda= y, September 1, 2015 4:56 PM
>  Subject: Re: [Xen= -devel] xhci_hcd intterrupt affinity in Dom0/DomU limited to single interru= pt
>   
> On Tue, Sep = 01, 2015 at 05:39:46PM +0000, Justin Acker wrote:
> &g= t; Taking this to the dev list from users.
> > > > Is there a way to force or enable pirq delivery t= o a set of cpus as opposed to single device from being a assigned a single = pirq so that its interrupt can be distributed across multiple cpus? I belie= ve the device drivers do support multiple queues when run natively without = the Dom0 loaded. The device in question is the xhci_hcd driver for which I/= O transfers seem to be slowed when the Dom0 is loaded. The behavior seems t= o pass through to the DomU if pass through is enabled. I found some similar= threads, but most relate to Ethernet controllers. I tried some of the x2ap= ic and x2apic_phys dom0 kernel arguments, but none distributed the pirqs. B= ased on the reading relating to IRQs for Xen, I think pinning the pirqs to = cpu0 is done to avoid an interrupt storm. I tried IRQ balance and when conf= igured/adjusted it will balance individual pirqs, but not multiple interrup= ts.
>
> Yes. You can do it with = smp affinity:
>
> https://cs.uwaterl= oo.ca/~brecht/servers/apic/SMP-affinity.txt
> Yes,= this does allow for assigning a specific interrupt to a single cpu, but it= will not spread the interrupt load across a defined group or all cpus. Is = it possible to define a range of CPUs or spread the interrupt load for a de= vice across all cpus as it does with a native kernel without the Dom0 loade= d?

It should be. Did you try giving it= an mask that puts the interrupts on all the CPUs?
(0xf) = ?
>
> I don't follow the "behavi= or seems to pass through to the DomU if pass through is enabled" ?
> The device interrupts are limited to a single pirq if the de= vice is used directly in the Dom0. If the device is passed through to a Dom= U - i.e. the xhci_hcd controller - then the DomU cannot spread the interrup= t load across the cpus in the VM.

Why= ? How are you seeing this? The method by which you use smp affinity should<= br clear=3D"none">be exactly the same.

And it looks to me that the de= vice has a single pirq as well when booting as baremetal right?

On baremetal, it uses all 8 cpus for affinity = as noted below (IRQ27) compared to (IRQ78) in the Dom0.
    &nb= sp;             = ;            &n= bsp;            = ;            &n= bsp;            = ;         CPU0   &nb= sp;   CPU1       CPU2  &n= bsp;    CPU3       CPU4 &= nbsp;     CPU5       CPU6=        CPU7     
baremetal: =             &n= bsp;            = ;             27:&nb= sp;  17977230     628258   44247270 = ;    120391 1597809883  =20 14440991  152189328      73322  IR-PCI-M= SI-edge      xhci_hcd
Dom0 or DomU passed through: 78: &nbs= p;    82521        &= nbsp; 0          0  =         0     &= nbsp;   =20 0          0   =        0      &= nbsp;   0  xen-pirq-msi       = xhci_hcd

<= /div>So the issue here is that you want to spread the interrupt delivery to= happen across
all of the CPUs. The smp_affinity should d= o it. Did you try modifying it by hand (you may
want to k= ill irqbalance when you do this just to make sure it does not write its own= values in)?



Y= es, this would be great if there is a way to spread the affinity across all= cpus or a specified set of CPUs similar to the native kernel behavior. I g= uess it would be spread across a set or all pirqs? With irqbalance disabled= , I did adjust try the interrupt affinity manually (i.e. echo ff /prox/irq/= 78/smp_affinity). The interrupt will move to the specified CPU (0 through 7= ). Without specifying the affinity manually, it does look like it's mapped = to all cpus by default. With the Dom0 loaded, cat /proc/irq/78/smp_affinity= returns ff, but the interrupt never appears to be scheduled on more than c= pu.




<= /div>


>
> >
> >
> >
> > With irqbalance enabled in= Dom0:
>
> What version? There w= as a bug in it where it would never distribute the IRQs properly
> across the CPUs.
> irqbalance version 1= .0.7.
>
> Boris (CC-ed) might re= member the upstream patch that made this work properly?
&= gt;
>
> >
> >            CPU= 0       CPU1     &nb= sp; CPU2       CPU3    &n= bsp;  CPU4       CPU5   &= nbsp;   CPU6       CPU7  =    
> >  76:   =    11304          0&= nbsp;    149579       &nb= sp;  0          0 &n= bsp;        0    &nb= sp;     0       &nbs= p;  0  xen-pirq-msi       0000:00:1= f.2
> >  77:     &nbs= p; 1243          0  =         0      = 35447          0  &n= bsp;       0     &nb= sp;    0        &nbs= p; 0  xen-pirq-msi       radeon
> >  78:      82521 &nbs= p;        0     = ;     0        =   0          0  = ;        0     =      0        &= nbsp; 0  xen-pirq-msi       xhci_hcd
> >  79:       = ;  23          0 &nb= sp;        0    &nbs= p;     0        = ;  0          0 &nbs= p;        0     = ;     0  xen-pirq-msi     = ;  mei_me
> >  80:    = ;     11        = ;  0          0 &nbs= p;        0     = ;     0        741&n= bsp;         0   &nb= sp;      0  xen-pirq-msi   &nb= sp;   em1
> >  81:   =      350        = ;  0          0 &nbs= p;        0     = ;  1671          0 &= nbsp;        0    &n= bsp;     0  xen-pirq-msi    &n= bsp;  iwlwifi
> >  82:   &= nbsp;    275        =   0          0  = ;        0     =      0        &= nbsp; 0          0  =         0  xen-pirq-msi  =      snd_hda_intel
> >
> > With native 3.19 kernel:
> > <= br clear=3D"none">> > Without Dom0 for the same system from the first= message:
> >
> > # cat /p= roc/interrupts
> >      &n= bsp;     CPU0       CPU1&= nbsp;      CPU2      = ; CPU3       CPU4    &nbs= p;  CPU5       CPU6   &nb= sp;   CPU7     
> = >   0:         33 &nbs= p;        0     = ;     0        =   0          0  = ;        0     =      0        &= nbsp; 0  IR-IO-APIC-edge      timer
> >   8:       &nbs= p;  0          0 &nb= sp;        0    &nbs= p;     0        = ;  0          0 &nbs= p;        1     = ;     0  IR-IO-APIC-edge    &n= bsp; rtc0
> >   9:    &nbs= p;    20        &nbs= p; 0          0  &nb= sp;       0     &nbs= p;    0         = ; 1          1  &nbs= p;       1  IR-IO-APIC-fasteoi &nbs= p; acpi
> >  16:     =     15         = 0          8   = ;       1      =     4          = 1          1   =        1  IR-IO-APIC  16-fasteoi&nb= sp;  ehci_hcd:usb3
> >  18:  &n= bsp;  703940       5678   = ; 1426226       1303    393824= 3     111477     757871  =       510  IR-IO-APIC  18-fasteoi &= nbsp; ath9k
> >  23:    &n= bsp;    11        &n= bsp; 2          3  &= nbsp;       0     &n= bsp;    0         17=           2   &= nbsp;      0  IR-IO-APIC  23-fasteoi&nbs= p;  ehci_hcd:usb4
> >  24:  &nb= sp;       0     &nbs= p;    0         = ; 0          0  &nbs= p;       0      = ;    0         = 0          0  DMAR_MSI-e= dge      dmar0
> >  2= 5:          0   = ;       0      =     0          = 0          0   =        0      &= nbsp;   0          0=   DMAR_MSI-edge      dmar1
= > >  26:      20419   &n= bsp;   1609      26822   =      567      62281  = ;     5426      14928 &nb= sp;      395  IR-PCI-MSI-edge  &nbs= p;   0000:00:1f.2
> >  27: &nbs= p; 17977230     628258   44247270  =    120391 1597809883   14440991  152189328 &n= bsp;    73322  IR-PCI-MSI-edge    &= nbsp; xhci_hcd
> >  28:    = ;    563        &nbs= p; 0          0  &nb= sp;       0     &nbs= p;    1         = ; 0          6  &nbs= p;       0  IR-PCI-MSI-edge  &= nbsp;   i915
> >  29:  &nb= sp;      14      &nb= sp;   0          0&n= bsp;         4   &nb= sp;      2      &nbs= p;   4          0&nb= sp;         0  IR-PCI-MSI-edge=       mei_me
> >  30:=       39514       17= 44      60339      &= nbsp; 157     129956      1970= 2      72140      &n= bsp;  83  IR-PCI-MSI-edge      eth0
> >  31:       =    3          0 = ;         0    =       1       &= nbsp; 54          0  = ;        0     =      2  IR-PCI-MSI-edge    &nb= sp; snd_hda_intel
> >  32:   &n= bsp;  28145        284  &= nbsp;   53316         63&= nbsp;    139165       4410&nbs= p;     25760       &= nbsp; 27  IR-PCI-MSI-edge      eth1-rx-0
> >  33:       1032&= nbsp;        43    &= nbsp;  2392          5&nb= sp;      1797      &= nbsp; 265       1507    &= nbsp;    20  IR-PCI-MSI-edge    &nb= sp; eth1-tx-0
> >  34:    =       0       &= nbsp;  1          0 =          0    &= nbsp;     0       &n= bsp;  1          2 &= nbsp;        0  IR-PCI-MSI-edge&nbs= p;     eth1
> >  35: =          5    &= nbsp;     0       &n= bsp;  0         12  =       148       = ;   6          2&nbs= p;         1  IR-PCI-MSI-edge&= nbsp;     snd_hda_intel
> > > >
> > The USB controller i= s an Intel C210:
> >
> > 0= 0:14.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Famil= y USB xHCI Host Controller (rev 04) (prog-if 30 [XHCI])
&= gt; >     Subsystem: Dell Device 053e
&= gt; >     Flags: bus master, medium devsel, latency 0, IR= Q 78
> >     Memory at f7f20000 (64-= bit, non-prefetchable) [size=3D64K]
> >   = ;  Capabilities: [70] Power Management version 2
>= ; >     Capabilities: [80] MSI: Enable+ Count=3D1/8 Maska= ble- 64bit+
> >     Kernel driver in= use: xhci_hcd
> >     Kernel module= s: xhci_pci
> >      On Tuesday, Sep= tember 1, 2015 11:50 AM, Ian Campbell <ian.cam= pbell@citrix.com> wrote:
> >    > >
> >  On Tue, 2015-= 09-01 at 13:56 +0000, Justin Acker wrote:
> > > = Thanks Ian,
> > >
> > &= gt; I appreciate the explanation. I believe the device drivers do support <= br clear=3D"none">> > > multiple queues when run natively without = the Dom0 loaded. The device in
> > > question i= s the xhci_hcd driver for which I/O transfers seem to be slowed
> > > when the Dom0 is loaded. The behavior seems to pas= s through to the DomU
> > > if pass through is = enabled. I found some similar threads, but most relate
&= gt; > > to Ethernet controllers. I tried some of the x2apic and x2api= c_phys dom0
> > > kernel arguments, but none di= stributed the pirqs. Based on the reading
> > >= relating to IRQs for Xen, I think pinning the pirqs to cpu0 is done to > > > avoid an I/O storm. I tried IRQ balance and = when configured/adjusted it
> > > will balance = individual pirqs, but not multiple interrupts.
> >= >
> > > Is there a way to force or enable p= irq delivery to a set of cpus as you
> > > ment= ioned above or omit a single device from being a assigned a PIRQ so
> > > that its interrupt can be distributed across all= cpus?
> >
> > A PIRQ is = the way an interrupt is exposed to a PV guest, without it there
> > would be no interrupt at all.
> > =
> > I'm afraid I'm out of my depth WRT how x86/MSI= s and Xen x86/PV pirqs
> > interact, in particular = WRT configuring which set of CPUs can have the IRQ
> &= gt; delivered.
> >
> > If = no one else chimes in soon I'd suggest taking this to the dev list, at
> > the very least someone who knows what they are talk= ing about (i.e. other
> > than me) might be able to= help.
> >
> > Ian.
> >
> >
> = >
> > 
>
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xen.org
> &= gt; http://lists.xen.org/xen-devel
>
>
> 


------=_Part_479747_808262945.1441213356790-- --===============6714354145022443417== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel --===============6714354145022443417==--