From mboxrd@z Thu Jan 1 00:00:00 1970 From: Justin Acker Subject: Re: xhci_hcd intterrupt affinity in Dom0/DomU limited to single interrupt Date: Tue, 1 Sep 2015 23:09:38 +0000 (UTC) Message-ID: <1558077830.37674.1441148978455.JavaMail.yahoo@mail.yahoo.com> References: <1441121643.26292.63.camel@citrix.com> <800613365.4285959.1441128848192.JavaMail.yahoo@mail.yahoo.com> <631331126.1156575.1441129186888.JavaMail.yahoo@mail.yahoo.com> <20150901205657.GA18532@l.oracle.com> Reply-To: Justin Acker Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============7533861469222575905==" Return-path: In-Reply-To: <20150901205657.GA18532@l.oracle.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Konrad Rzeszutek Wilk Cc: "boris.ostrovsky@oracle.com" , "xen-devel@lists.xen.org" List-Id: xen-devel@lists.xenproject.org --===============7533861469222575905== Content-Type: multipart/alternative; boundary="----=_Part_37673_1036646890.1441148978442" Content-Length: 40856 ------=_Part_37673_1036646890.1441148978442 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable From: Konrad Rzeszutek Wilk To: Justin Acker =20 Cc: "xen-devel@lists.xen.org" ; boris.ostrovsky@or= acle.com=20 Sent: Tuesday, September 1, 2015 4:56 PM Subject: Re: [Xen-devel] xhci_hcd intterrupt affinity in Dom0/DomU limited= to single interrupt =20 On Tue, Sep 01, 2015 at 05:39:46PM +0000, Justin Acker wrote: > Taking this to the dev list from users.=20 >=20 > Is there a way to force or enable pirq delivery to a set of cpus as oppos= ed to single device from being a assigned a single pirq so that its interru= pt can be distributed across multiple cpus? I believe the device drivers do= support multiple queues when run natively without the Dom0 loaded. The dev= ice in question is the xhci_hcd driver for which I/O transfers seem to be s= lowed when the Dom0 is loaded. The behavior seems to pass through to the Do= mU if pass through is enabled. I found some similar threads, but most relat= e to Ethernet controllers. I tried some of the x2apic and x2apic_phys dom0 = kernel arguments, but none distributed the pirqs. Based on the reading rela= ting to IRQs for Xen, I think pinning the pirqs to cpu0 is done to avoid an= interrupt storm. I tried IRQ balance and when configured/adjusted it will = balance individual pirqs, but not multiple interrupts. Yes. You can do it with smp affinity: https://cs.uwaterloo.ca/~brecht/servers/apic/SMP-affinity.txt Yes, this does allow for assigning a specific interrupt to a single cpu, bu= t it will not spread the interrupt load across a defined group or all cpus.= Is it possible to define a range of CPUs or spread the interrupt load for = a device across all cpus as it does with a native kernel without the Dom0 l= oaded? I don't follow the "behavior seems to pass through to the DomU if pass thro= ugh is enabled" ? The device interrupts are limited to a single pirq if the device is used di= rectly in the Dom0. If the device is passed through to a DomU - i.e. the xh= ci_hcd controller - then the DomU cannot spread the interrupt load across t= he cpus in the VM.=20 >=20 >=20 >=20 > With irqbalance enabled in Dom0: What version? There was a bug in it where it would never distribute the IRQ= s properly across the CPUs. irqbalance version 1.0.7. Boris (CC-ed) might remember the upstream patch that made this work properl= y? >=20 > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 CPU0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 CPU1=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 CPU= 2=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 CPU3=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 CPU4=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 CPU5=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 CPU6=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 CPU7=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=20 > =C2=A076:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 11304=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0 149579=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0 xen-pirq-msi=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 0000:00:1f.2 > =C2=A077:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 1243=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 35447=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0 xen-pirq-msi= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 radeon > =C2=A078:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 82521=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2= =A0 xen-pirq-msi=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 xhci_hcd > =C2=A079:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 23=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0 0=C2=A0 xen-pirq-msi=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 mei_me > =C2=A080:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 11=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 741=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = 0=C2=A0 xen-pirq-msi=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 em1 > =C2=A081:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 350=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 1671=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0 xen-pir= q-msi=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 iwlwifi > =C2=A082:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 275=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = 0=C2=A0 xen-pirq-msi=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 snd_hda_intel >=20 > With native 3.19 kernel: >=20 > Without Dom0 for the same system from the first message: >=20 > # cat /proc/interrupts > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 CPU0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 CPU1=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 CPU= 2=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 CPU3=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 CPU4=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 CPU5=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 CPU6=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 CPU7=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=20 > =C2=A0 0:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 33=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0 0=C2=A0 IR-IO-APIC-edge=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 timer > =C2=A0 8:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 1=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 0=C2=A0 IR-IO-APIC-edge=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 rtc0 > =C2=A0 9:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 20=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 1=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 1=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0 1=C2=A0 IR-IO-APIC-fasteoi=C2=A0=C2=A0 acpi > =C2=A016:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 15=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 8=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 1=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 4=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 1=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 1=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0 1=C2=A0 IR-IO-APIC=C2=A0 16-fasteoi=C2=A0=C2=A0 ehci_hcd:usb3 > =C2=A018:=C2=A0=C2=A0=C2=A0=C2=A0 703940=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 5678=C2=A0=C2=A0=C2=A0 1426226=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 1303= =C2=A0=C2=A0=C2=A0 3938243=C2=A0=C2=A0=C2=A0=C2=A0 111477=C2=A0=C2=A0=C2=A0= =C2=A0 757871=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 510=C2=A0 IR-IO-API= C=C2=A0 18-fasteoi=C2=A0=C2=A0 ath9k > =C2=A023:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 11=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 2=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 3=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 17=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 2=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 0=C2=A0 IR-IO-APIC=C2=A0 23-fasteoi=C2=A0=C2=A0 ehci_hcd:usb4 > =C2=A024:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 0=C2=A0 DMAR_MSI-edge=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 dmar0 > =C2=A025:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 0=C2=A0 DMAR_MSI-edge=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 dmar1 > =C2=A026:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 20419=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0 1609=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 26822=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 567=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 62281=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 5426=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 14928=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 395=C2=A0 IR-PCI-MSI-edge=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 0000:00:1f.2 > =C2=A027:=C2=A0=C2=A0 17977230=C2=A0=C2=A0=C2=A0=C2=A0 628258=C2=A0=C2=A0= 44247270=C2=A0=C2=A0=C2=A0=C2=A0 120391 1597809883=C2=A0=C2=A0 14440991=C2= =A0 152189328=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 73322=C2=A0 IR-PCI-MSI-edge=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 xhci_hcd > =C2=A028:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 563=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 1=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 6=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = 0=C2=A0 IR-PCI-MSI-edge=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 i915 > =C2=A029:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 14=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 4=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 2=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 4=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0 0=C2=A0 IR-PCI-MSI-edge=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 mei_me > =C2=A030:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 39514=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0 1744=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 60339=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 157=C2=A0=C2=A0=C2=A0=C2=A0 129956=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0 19702=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 72140=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 83=C2=A0 IR-PCI-MSI-edge=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 eth0 > =C2=A031:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 3=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 1=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 54=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0 2=C2=A0 IR-PCI-MSI-edge=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 snd_hda_int= el > =C2=A032:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 28145=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 284=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 53316=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 63=C2=A0=C2=A0=C2=A0=C2=A0 139165=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 4410=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 25760=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 27=C2=A0 IR-PCI-MSI-edge=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 eth1-rx-0 > =C2=A033:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 1032=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 43=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 2392=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 5=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 1797=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 265=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 1507=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 20=C2=A0 IR-PCI-MSI-edge=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 eth1-tx-0 > =C2=A034:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 1=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 1=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 2=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 0=C2=A0 IR-PCI-MSI-edge=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 eth1 > =C2=A035:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 5=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 12=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 148=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 6=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 2=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = 1=C2=A0 IR-PCI-MSI-edge=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 snd_hda_intel >=20 >=20 > The USB controller is an Intel C210: >=20 > 00:14.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Fa= mily USB xHCI Host Controller (rev 04) (prog-if 30 [XHCI]) > =C2=A0=C2=A0=C2=A0 Subsystem: Dell Device 053e > =C2=A0=C2=A0=C2=A0 Flags: bus master, medium devsel, latency 0, IRQ 78 > =C2=A0=C2=A0=C2=A0 Memory at f7f20000 (64-bit, non-prefetchable) [size=3D= 64K] > =C2=A0=C2=A0=C2=A0 Capabilities: [70] Power Management version 2 > =C2=A0=C2=A0=C2=A0 Capabilities: [80] MSI: Enable+ Count=3D1/8 Maskable- = 64bit+ > =C2=A0=C2=A0=C2=A0 Kernel driver in use: xhci_hcd > =C2=A0=C2=A0=C2=A0 Kernel modules: xhci_pci >=C2=A0 =C2=A0 =C2=A0 On Tuesday, September 1, 2015 11:50 AM, Ian Campbell = wrote: >=C2=A0 =C2=A0=20 >=20 >=C2=A0 On Tue, 2015-09-01 at 13:56 +0000, Justin Acker wrote: > > Thanks Ian, > >=20 > > I appreciate the explanation. I believe the device drivers do support= =20 > > multiple queues when run natively without the Dom0 loaded. The device i= n=20 > > question is the xhci_hcd driver for which I/O transfers seem to be slow= ed=20 > > when the Dom0 is loaded. The behavior seems to pass through to the DomU= =20 > > if pass through is enabled. I found some similar threads, but most rela= te=20 > > to Ethernet controllers. I tried some of the x2apic and x2apic_phys dom= 0=20 > > kernel arguments, but none distributed the pirqs. Based on the reading= =20 > > relating to IRQs for Xen, I think pinning the pirqs to cpu0 is done to= =20 > > avoid an I/O storm. I tried IRQ balance and when configured/adjusted it= =20 > > will balance individual pirqs, but not multiple interrupts.=20 > >=20 > > Is there a way to force or enable pirq delivery to a set of cpus as you= =20 > > mentioned above or omit a single device from being a assigned a PIRQ so= =20 > > that its interrupt can be distributed across all cpus?=20 >=20 > A PIRQ is the way an interrupt is exposed to a PV guest, without it there > would be no interrupt at all. >=20 > I'm afraid I'm out of my depth WRT how x86/MSIs and Xen x86/PV pirqs > interact, in particular WRT configuring which set of CPUs can have the IR= Q > delivered. >=20 > If no one else chimes in soon I'd suggest taking this to the dev list, at > the very least someone who knows what they are talking about (i.e. other > than me) might be able to help. >=20 > Ian. >=20 >=20 >=20 >=C2=A0=20 > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel ------=_Part_37673_1036646890.1441148978442 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable

From:= Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
To: Justin Acker <ackerj67@yahoo.com= >
Cc: "xen-devel@li= sts.xen.org" <xen-devel@lists.xen.org>; boris.ostrovsky@oracle.com Sent: Tuesday, Septembe= r 1, 2015 4:56 PM
Subject:= Re: [Xen-devel] xhci_hcd intterrupt affinity in Dom0/DomU limited to s= ingle interrupt

On Tue, Sep 01, 2015 at 05:39:46PM +0000,= Justin Acker wrote:
> Taking this to the dev list fro= m users.
>
> Is there a way to = force or enable pirq delivery to a set of cpus as opposed to single device = from being a assigned a single pirq so that its interrupt can be distribute= d across multiple cpus? I believe the device drivers do support multiple qu= eues when run natively without the Dom0 loaded. The device in question is t= he xhci_hcd driver for which I/O transfers seem to be slowed when the Dom0 = is loaded. The behavior seems to pass through to the DomU if pass through i= s enabled. I found some similar threads, but most relate to Ethernet contro= llers. I tried some of the x2apic and x2apic_phys dom0 kernel arguments, bu= t none distributed the pirqs. Based on the reading relating to IRQs for Xen= , I think pinning the pirqs to cpu0 is done to avoid an interrupt storm. I = tried IRQ balance and when configured/adjusted it will balance individual p= irqs, but not multiple interrupts.

Yes= . You can do it with smp affinity:

h= ttps://cs.uwaterloo.ca/~brecht/servers/apic/SMP-affinity.txt

Yes, this does allow for assigning a specific interrupt t= o a single cpu, but it will not spread the interrupt load across a defined = group or all cpus. Is it possible to define a range of CPUs or spread the i= nterrupt load for a device across all cpus as it does with a native kernel = without the Dom0 loaded?

I don't follow the "behavior seems to pass through to t= he DomU if pass through is enabled" ?

The devic= e interrupts are limited to a single pirq if the device is used directly in= the Dom0. If the device is passed through to a DomU - i.e. the xhci_hcd co= ntroller - then the DomU cannot spread the interrupt load across the cpus i= n the VM.

>
>
>
> With irqbalance enabled in Dom0:
What version? There was a bug in it whe= re it would never distribute the IRQs properly
across the CPUs.

irqbalance version 1.0.7.

Boris (CC-ed) mig= ht remember the upstream patch that made this work properly?


=
>
>     &nb= sp;      CPU0       = CPU1       CPU2     =   CPU3       CPU4    = ;   CPU5       CPU6  &nbs= p;    CPU7     
= >  76:      11304    &= nbsp;     0     149579  &= nbsp;       0     &n= bsp;    0        &nb= sp; 0          0  &n= bsp;       0  xen-pirq-msi  &n= bsp;    0000:00:1f.2
>  77: &= nbsp;     1243       = ;   0          0&nbs= p;     35447       &= nbsp;  0          0 =          0    &= nbsp;     0  xen-pirq-msi    &= nbsp;  radeon
>  78:    =   82521          0 &= nbsp;        0    &n= bsp;     0       &nb= sp;  0          0 &n= bsp;        0    &nb= sp;     0  xen-pirq-msi    &nb= sp;  xhci_hcd
>  79:    =      23        =   0          0  = ;        0     =      0        &= nbsp; 0          0  =         0  xen-pirq-msi  =      mei_me
>  80:  = ;       11      = ;    0         = 0          0   = ;       0      =   741          0 &nb= sp;        0  xen-pirq-msi &nb= sp;     em1
>  81:  = ;      350      &nbs= p;   0          0&nb= sp;         0   &nbs= p;   1671          0=           0   &= nbsp;      0  xen-pirq-msi   &= nbsp;   iwlwifi
>  82:   = ;     275       &nbs= p;  0          0 &nb= sp;        0    &nbs= p;     0        = ;  0          0 &nbs= p;        0  xen-pirq-msi &nbs= p;     snd_hda_intel
>
> With native 3.19 kernel:
>
> Without Dom0 for the same system from the first message:
>
> # cat /proc/interrupts
>            CP= U0       CPU1     &n= bsp; CPU2       CPU3    &= nbsp;  CPU4       CPU5   =     CPU6       CPU7  = ;   
>   0:   &nbs= p;     33       &nbs= p;  0          0 &nb= sp;        0    &nbs= p;     0        = ;  0          0 &nbs= p;        0  IR-IO-APIC-edge &= nbsp;    timer
>   8:  &= nbsp;       0     &n= bsp;    0        &nb= sp; 0          0  &n= bsp;       0     &nb= sp;    0        &nbs= p; 1          0  IR-IO-AP= IC-edge      rtc0
>   9:=          20    =       0       &= nbsp;  0          0 =          0    &= nbsp;     1       &n= bsp;  1          1  = IR-IO-APIC-fasteoi   acpi
>  16: &= nbsp;       15     &= nbsp;    0        &n= bsp; 8          1  &= nbsp;       4     &n= bsp;    1        &nb= sp; 1          1  IR-IO-A= PIC  16-fasteoi   ehci_hcd:usb3
>  = ;18:     703940       567= 8    1426226       1303 &= nbsp;  3938243     111477     = 757871        510  IR-IO-APIC = 18-fasteoi   ath9k
>  23:  &= nbsp;      11      &= nbsp;   2          3=           0   &= nbsp;      0      &n= bsp;  17          2 =          0  IR-IO-APIC  2= 3-fasteoi   ehci_hcd:usb4
>  24: &= nbsp;        0    &n= bsp;     0       &nb= sp;  0          0 &n= bsp;        0    &nb= sp;     0       &nbs= p;  0          0  DM= AR_MSI-edge      dmar0
> &nbs= p;25:          0  &n= bsp;       0     &nb= sp;    0        &nbs= p; 0          0  &nb= sp;       0     &nbs= p;    0         = ; 0  DMAR_MSI-edge      dmar1
>  26:      20419   &nbs= p;   1609      26822   &n= bsp;    567      62281  &= nbsp;    5426      14928  = ;      395  IR-PCI-MSI-edge   =    0000:00:1f.2
>  27:   1797= 7230     628258   44247270   &= nbsp; 120391 1597809883   14440991  152189328  &nb= sp;   73322  IR-PCI-MSI-edge      x= hci_hcd
>  28:      = ;  563          0 &n= bsp;        0    &nb= sp;     0       &nbs= p;  1          0 &nb= sp;        6    &nbs= p;     0  IR-PCI-MSI-edge    &= nbsp; i915
>  29:     &n= bsp;   14          0=           0   &= nbsp;      4      &n= bsp;   2          4&= nbsp;         0   &n= bsp;      0  IR-PCI-MSI-edge   = ;   mei_me
>  30:   &nbs= p;  39514       1744   &n= bsp;  60339        157  &= nbsp;  129956      19702   &nb= sp;  72140         83  IR= -PCI-MSI-edge      eth0
> &nb= sp;31:          3  &= nbsp;       0     &n= bsp;    0        &nb= sp; 1         54   &= nbsp;      0      &n= bsp;   0          2&= nbsp; IR-PCI-MSI-edge      snd_hda_intel
>  32:      28145  &nb= sp;     284      53316 &n= bsp;       63     139165&= nbsp;      4410      2576= 0         27  IR-PCI-MSI-edge&= nbsp;     eth1-rx-0
>  33:&nb= sp;      1032      &= nbsp;  43       2392   &n= bsp;      5       17= 97        265    &nb= sp;  1507         20  IR-= PCI-MSI-edge      eth1-tx-0
>=  34:          0 &nb= sp;        1    &nbs= p;     0        = ;  0          0 &nbs= p;        1     = ;     2        =   0  IR-PCI-MSI-edge      eth1
>  35:        &nb= sp; 5          0  &n= bsp;       0     &nb= sp;   12        148  = ;        6     =      2        &= nbsp; 1  IR-PCI-MSI-edge      snd_hda_intel>
>
> The U= SB controller is an Intel C210:
>
&= gt; 00:14.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset = Family USB xHCI Host Controller (rev 04) (prog-if 30 [XHCI])
>     Subsystem: Dell Device 053e
&= gt;     Flags: bus master, medium devsel, latency 0, IRQ 78<= br clear=3D"none">>     Memory at f7f20000 (64-bit, non-p= refetchable) [size=3D64K]
>     Capabil= ities: [70] Power Management version 2
>   &= nbsp; Capabilities: [80] MSI: Enable+ Count=3D1/8 Maskable- 64bit+
>     Kernel driver in use: xhci_hcd
>     Kernel modules: xhci_pci
>      On Tuesday, September 1, 2015 11:50 AM, Ian Camp= bell <ian.campbell@citrix.com> wrote:
>   
>
>=   On Tue, 2015-09-01 at 13:56 +0000, Justin Acker wrote:
> > Thanks Ian,
> >
&= gt; > I appreciate the explanation. I believe the device drivers do supp= ort
> > multiple queues when run natively without = the Dom0 loaded. The device in
> > question is the= xhci_hcd driver for which I/O transfers seem to be slowed
> > when the Dom0 is loaded. The behavior seems to pass through to= the DomU
> > if pass through is enabled. I found = some similar threads, but most relate
> > to Ether= net controllers. I tried some of the x2apic and x2apic_phys dom0
> > kernel arguments, but none distributed the pirqs. Based= on the reading
> > relating to IRQs for Xen, I th= ink pinning the pirqs to cpu0 is done to
> > avoid= an I/O storm. I tried IRQ balance and when configured/adjusted it
> > will balance individual pirqs, but not multiple interr= upts.
> >
> > Is there a = way to force or enable pirq delivery to a set of cpus as you
> > mentioned above or omit a single device from being a assigne= d a PIRQ so
> > that its interrupt can be distribu= ted across all cpus?
>
> A PIRQ= is the way an interrupt is exposed to a PV guest, without it there
> would be no interrupt at all.
>
> I'm afraid I'm out of my depth WRT how x86/MSIs and Xen x= 86/PV pirqs
> interact, in particular WRT configuring = which set of CPUs can have the IRQ
> delivered.
>
> If no one else chimes in soon I'd = suggest taking this to the dev list, at
> the very lea= st someone who knows what they are talking about (i.e. other
> than me) might be able to help.
>
> Ian.

>
> >
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xe= n-devel



------=_Part_37673_1036646890.1441148978442-- --===============7533861469222575905== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel --===============7533861469222575905==--