From mboxrd@z Thu Jan 1 00:00:00 1970 From: Justin Acker Subject: xhci_hcd intterrupt affinity in Dom0/DomU limited to single interrupt Date: Tue, 1 Sep 2015 17:39:46 +0000 (UTC) Message-ID: <631331126.1156575.1441129186888.JavaMail.yahoo@mail.yahoo.com> References: <1441121643.26292.63.camel@citrix.com> <800613365.4285959.1441128848192.JavaMail.yahoo@mail.yahoo.com> Reply-To: Justin Acker Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============1205095371090951997==" Return-path: In-Reply-To: <800613365.4285959.1441128848192.JavaMail.yahoo@mail.yahoo.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: "xen-devel@lists.xen.org" List-Id: xen-devel@lists.xenproject.org --===============1205095371090951997== Content-Type: multipart/alternative; boundary="----=_Part_1156574_1230247901.1441129186877" Content-Length: 38365 ------=_Part_1156574_1230247901.1441129186877 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Taking this to the dev list from users.=20 Is there a way to force or enable pirq delivery to a set of cpus as opposed= to single device from being a assigned a single pirq so that its interrupt= can be distributed across multiple cpus? I believe the device drivers do s= upport multiple queues when run natively without the Dom0 loaded. The devic= e in question is the xhci_hcd driver for which I/O transfers seem to be slo= wed when the Dom0 is loaded. The behavior seems to pass through to the DomU= if pass through is enabled. I found some similar threads, but most relate = to Ethernet controllers. I tried some of the x2apic and x2apic_phys dom0 ke= rnel arguments, but none distributed the pirqs. Based on the reading relati= ng to IRQs for Xen, I think pinning the pirqs to cpu0 is done to avoid an i= nterrupt storm. I tried IRQ balance and when configured/adjusted it will ba= lance individual pirqs, but not multiple interrupts. With irqbalance enabled in Dom0: =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 CPU0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 CPU1=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 CPU2= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 CPU3=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 CPU4=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 CPU5=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 CPU6=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 CPU7=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=20 =C2=A076:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 11304=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0 149579=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0 xen-pirq-msi=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 0000:00:1f.2 =C2=A077:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 1243=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 35447=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0 xen-pirq-msi=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 radeon =C2=A078:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 82521=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0 x= en-pirq-msi=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 xhci_hcd =C2=A079:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 23=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 0=C2=A0 xen-pirq-msi=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 mei_me =C2=A080:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 11=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 741=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0= =C2=A0 xen-pirq-msi=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 em1 =C2=A081:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 350=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 1671=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0 xen-pir= q-msi=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 iwlwifi =C2=A082:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 275=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = 0=C2=A0 xen-pirq-msi=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 snd_hda_intel With native 3.19 kernel: Without Dom0 for the same system from the first message: # cat /proc/interrupts =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 CPU0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 CPU1=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 CPU2= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 CPU3=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 CPU4=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 CPU5=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 CPU6=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 CPU7=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=20 =C2=A0 0:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 33=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 0=C2=A0 IR-IO-APIC-edge=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 timer =C2=A0 8:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 1=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 0=C2=A0 IR-IO-APIC-edge=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 rtc0 =C2=A0 9:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 20=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 1=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 1=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 1=C2=A0 IR-IO-APIC-fasteoi=C2=A0=C2=A0 acpi =C2=A016:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 15=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 8=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 1=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 4=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 1=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 1=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 1=C2=A0 IR-IO-APIC=C2=A0 16-fasteoi=C2=A0=C2=A0 ehci_hcd:usb3 =C2=A018:=C2=A0=C2=A0=C2=A0=C2=A0 703940=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 5678=C2=A0=C2=A0=C2=A0 1426226=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 1303= =C2=A0=C2=A0=C2=A0 3938243=C2=A0=C2=A0=C2=A0=C2=A0 111477=C2=A0=C2=A0=C2=A0= =C2=A0 757871=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 510=C2=A0 IR-IO-API= C=C2=A0 18-fasteoi=C2=A0=C2=A0 ath9k =C2=A023:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 11=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 2=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 3=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 17=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 2=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = 0=C2=A0 IR-IO-APIC=C2=A0 23-fasteoi=C2=A0=C2=A0 ehci_hcd:usb4 =C2=A024:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 0=C2=A0 DMAR_MSI-edge=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 dmar0 =C2=A025:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 0=C2=A0 DMAR_MSI-edge=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 dmar1 =C2=A026:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 20419=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 1609=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 26822=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 567=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 62281=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 5426=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 14928=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 395=C2=A0 IR-PCI-MSI-edge=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0 0000:00:1f.2 =C2=A027:=C2=A0=C2=A0 17977230=C2=A0=C2=A0=C2=A0=C2=A0 628258=C2=A0=C2=A0 4= 4247270=C2=A0=C2=A0=C2=A0=C2=A0 120391 1597809883=C2=A0=C2=A0 14440991=C2= =A0 152189328=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 73322=C2=A0 IR-PCI-MSI-edge=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 xhci_hcd =C2=A028:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 563=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 1=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 6=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = 0=C2=A0 IR-PCI-MSI-edge=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 i915 =C2=A029:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 14=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 4=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 2=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 4=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 0=C2=A0 IR-PCI-MSI-edge=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 mei_me =C2=A030:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 39514=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 1744=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 60339=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 157=C2=A0=C2=A0=C2=A0=C2=A0 129956=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 19702=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 72140=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 83=C2=A0 IR-PCI-MSI-edge=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 eth0 =C2=A031:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 3=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0 1=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 54=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 2=C2=A0 IR-PCI-MSI-edge=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 snd_hda_intel =C2=A032:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 28145=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 284=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 53316=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 63=C2=A0=C2=A0=C2=A0=C2=A0 139165=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 4410=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 25760=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 27=C2=A0 IR-PCI-MSI-edge=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 eth1-rx-0 =C2=A033:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 1032=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 43=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 2392=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 5=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 1797=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 265=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 1507=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 20=C2=A0 IR-PCI-MSI-edge=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 eth1-tx-0 =C2=A034:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 1=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 1=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 2=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 0=C2=A0 IR-PCI-MSI-edge=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 eth1 =C2=A035:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 5=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 12=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 148=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 6=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 2=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 1= =C2=A0 IR-PCI-MSI-edge=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 snd_hda_intel The USB controller is an Intel C210: 00:14.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Fami= ly USB xHCI Host Controller (rev 04) (prog-if 30 [XHCI]) =C2=A0=C2=A0=C2=A0 Subsystem: Dell Device 053e =C2=A0=C2=A0=C2=A0 Flags: bus master, medium devsel, latency 0, IRQ 78 =C2=A0=C2=A0=C2=A0 Memory at f7f20000 (64-bit, non-prefetchable) [size=3D64= K] =C2=A0=C2=A0=C2=A0 Capabilities: [70] Power Management version 2 =C2=A0=C2=A0=C2=A0 Capabilities: [80] MSI: Enable+ Count=3D1/8 Maskable- 64= bit+ =C2=A0=C2=A0=C2=A0 Kernel driver in use: xhci_hcd =C2=A0=C2=A0=C2=A0 Kernel modules: xhci_pci On Tuesday, September 1, 2015 11:50 AM, Ian Campbell wrote: =20 On Tue, 2015-09-01 at 13:56 +0000, Justin Acker wrote: > Thanks Ian, >=20 > I appreciate the explanation. I believe the device drivers do support=20 > multiple queues when run natively without the Dom0 loaded. The device in= =20 > question is the xhci_hcd driver for which I/O transfers seem to be slowed= =20 > when the Dom0 is loaded. The behavior seems to pass through to the DomU= =20 > if pass through is enabled. I found some similar threads, but most relate= =20 > to Ethernet controllers. I tried some of the x2apic and x2apic_phys dom0= =20 > kernel arguments, but none distributed the pirqs. Based on the reading=20 > relating to IRQs for Xen, I think pinning the pirqs to cpu0 is done to=20 > avoid an I/O storm. I tried IRQ balance and when configured/adjusted it= =20 > will balance individual pirqs, but not multiple interrupts.=20 >=20 > Is there a way to force or enable pirq delivery to a set of cpus as you= =20 > mentioned above or omit a single device from being a assigned a PIRQ so= =20 > that its interrupt can be distributed across all cpus?=20 A PIRQ is the way an interrupt is exposed to a PV guest, without it there would be no interrupt at all. I'm afraid I'm out of my depth WRT how x86/MSIs and Xen x86/PV pirqs interact, in particular WRT configuring which set of CPUs can have the IRQ delivered. If no one else chimes in soon I'd suggest taking this to the dev list, at the very least someone who knows what they are talking about (i.e. other than me) might be able to help. Ian. ------=_Part_1156574_1230247901.1441129186877 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Taking this to the dev list from use= rs.

Is there a way to force or enable pirq delivery to a s= et of cpus as opposed to single device from being a assigned a single pirq = so=20 that its interrupt can be distributed across multiple cpus? I believe the d= evice drivers do support multiple queues when run natively without the Dom0= loaded. The device in question is the xhci_hcd driver for which I/O transf= ers seem to be slowed when the Dom0 is loaded. The behavior seems to pass t= hrough to the DomU if pass through is enabled. I found some similar threads= , but most relate to Ethernet controllers. I tried some of the x2apic and x= 2apic_phys dom0 kernel arguments, but none distributed the pirqs. Based on = the reading relating to IRQs for Xen, I think pinning the pirqs to cpu0 is = done to avoid an interrupt storm. I tried IRQ balance and when configured/a= djusted it will balance individual pirqs, but not multiple interrupts.



Wi= th irqbalance enabled in Dom0:

 &nbs= p;         CPU0   &n= bsp;   CPU1       CPU2  &= nbsp;    CPU3       CPU4 =       CPU5       CPU= 6       CPU7       76:  &= nbsp;   11304        &nbs= p; 0     149579      &nbs= p;   0          0&nb= sp;         0   &nbs= p;      0       = ;   0  xen-pirq-msi       0000= :00:1f.2
 77:&nb= sp;      1243      &= nbsp;   0          0=       35447      &nb= sp;   0          0&n= bsp;         0   &nb= sp;      0  xen-pirq-msi   &nb= sp;   radeon
 78:      82521    &nbs= p;     0        = ;  0          0 &nbs= p;        0     = ;     0        =   0          0  xen-= pirq-msi       xhci_hcd
 79:     &nb= sp;   23          0&= nbsp;         0   &n= bsp;      0      &nb= sp;   0          0&n= bsp;         0   &nb= sp;      0  xen-pirq-msi   &nb= sp;   mei_me
 80:         11  &= nbsp;       0     &n= bsp;    0        &nb= sp; 0          0  &n= bsp;     741       &= nbsp;  0          0 = xen-pirq-msi       em1
 81:     &nb= sp;  350          0 =          0    &= nbsp;     0       1671&nb= sp;         0   &nbs= p;      0       = ;   0  xen-pirq-msi       iwlw= ifi
 82: &n= bsp;      275      &= nbsp;   0          0=           0   &= nbsp;      0      &n= bsp;   0          0&= nbsp;         0  xen-pirq-msi&= nbsp;      snd_hda_intel

With native 3.19 kernel:

Without= Dom0 for the same system from the first message:

# cat /proc/interrupts
           CPU= 0       CPU1     &nb= sp; CPU2       CPU3    &n= bsp;  CPU4       CPU5   &= nbsp;   CPU6       CPU7  =    
&= nbsp; 0:         33  &nbs= p;       0      = ;    0         = 0          0   = ;       0      =     0          = 0  IR-IO-APIC-edge      timer
  8:    &nb= sp;     0       &nbs= p;  0          0 &nb= sp;        0    &nbs= p;     0        = ;  0          1 &nbs= p;        0  IR-IO-APIC-edge &= nbsp;    rtc0
  9:         20 &= nbsp;        0    &n= bsp;     0       &nb= sp;  0          0 &n= bsp;        1    &nb= sp;     1       &nbs= p;  1  IR-IO-APIC-fasteoi   acpi
 16:     &nbs= p;   15          0&n= bsp;         8   &nb= sp;      1      &nbs= p;   4          1&nb= sp;         1   &nbs= p;      1  IR-IO-APIC  16-fasteoi &= nbsp; ehci_hcd:usb3
&= nbsp;18:     703940      = 5678    1426226       1303&nb= sp;   3938243     111477   &nb= sp; 757871        510  IR-IO-APIC&n= bsp; 18-fasteoi   ath9k
 23:         11&n= bsp;         2   &nb= sp;      3      &nbs= p;   0          0&nb= sp;        17    &nb= sp;     2       &nbs= p;  0  IR-IO-APIC  23-fasteoi   ehci_hcd:usb4
 24:  &nbs= p;       0      = ;    0         = 0          0   = ;       0      =     0          = 0          0  DMAR_MSI-ed= ge      dmar0
 25:        =   0          0  = ;        0     =      0        &= nbsp; 0          0  =         0     &= nbsp;    0  DMAR_MSI-edge     = dmar1
 26: = ;     20419       1609&nb= sp;     26822       = 567      62281      = ; 5426      14928     &nb= sp;  395  IR-PCI-MSI-edge      0000:00:1= f.2
 27: &n= bsp; 17977230     628258   44247270 &nbs= p;   120391 1597809883   14440991  152189328 =      73322  IR-PCI-MSI-edge    = ;  xhci_hcd
&nbs= p;28:        563    =       0       &= nbsp;  0          0 =          1    &= nbsp;     0       &n= bsp;  6          0  = IR-PCI-MSI-edge      i915
 29:      =    14          0&nbs= p;         0    = ;      4       =    2          4 = ;         0    =       0  IR-PCI-MSI-edge   &nb= sp;  mei_me
&nbs= p;30:      39514     &nbs= p; 1744      60339     &n= bsp;  157     129956     = 19702      72140     &nb= sp;   83  IR-PCI-MSI-edge      eth0=
 31:  = ;        3     =      0        &= nbsp; 0          1  =        54      =     0          = 0          2  IR-PCI-MSI-= edge      snd_hda_intel
 32:      28145&n= bsp;       284      = 53316         63   &= nbsp; 139165       4410   &nbs= p;  25760         27  IR-= PCI-MSI-edge      eth1-rx-0
 33:     &nbs= p; 1032         43   = ;    2392        &nb= sp; 5       1797     = ;   265       1507   = ;      20  IR-PCI-MSI-edge   &= nbsp;  eth1-tx-0
 34:          0 &nb= sp;        1    &nbs= p;     0        = ;  0          0 &nbs= p;        1     = ;     2        =   0  IR-PCI-MSI-edge      eth1
 35:   &n= bsp;      5      &nb= sp;   0          0&n= bsp;        12    &n= bsp;   148          = 6          2   =        1  IR-PCI-MSI-edge  &nb= sp;   snd_hda_intel


The USB controller is an Inte= l C210:

00:14.0 USB controller: Intel Cor= poration 7 Series/C210 Series Chipset Family USB xHCI Host Controller (rev = 04) (prog-if 30 [XHCI])
    Subsystem: Dell Device 053e
    Flags: bus master, medium d= evsel, latency 0, IRQ 78
    Memory at f7f20000 (64-bit, non-prefetchable) [size= =3D64K]
  &= nbsp; Capabilities: [70] Power Management version 2
    Capabilities: [80] MSI: E= nable+ Count=3D1/8 Maskable- 64bit+
    Kernel driver in use: xhci_hcd
  =   Kernel modules: xhci_pci

=
On Tuesday, September = 1, 2015 11:50 AM, Ian Campbell <ian.campbell@citrix.com> wrote:


On Tue, 2015-09-01 at 13:56 +0000, Justin Acker wrote:
> Thanks Ian,
>
>= ; I appreciate the explanation. I believe the device drivers do support > multiple queues when run natively without the Dom0 loa= ded. The device in
> question is the xhci_hcd driver = for which I/O transfers seem to be slowed
> when the = Dom0 is loaded. The behavior seems to pass through to the DomU
> if pass through is enabled. I found some similar threads, but m= ost relate
> to Ethernet controllers. I tried some of= the x2apic and x2apic_phys dom0
> kernel arguments, = but none distributed the pirqs. Based on the reading
>= ; relating to IRQs for Xen, I think pinning the pirqs to cpu0 is done to > avoid an I/O storm. I tried IRQ balance and when conf= igured/adjusted it
> will balance individual pirqs, b= ut not multiple interrupts.
>
>= Is there a way to force or enable pirq delivery to a set of cpus as you > mentioned above or omit a single device from being a = assigned a PIRQ so
> that its interrupt can be distri= buted across all cpus?

A PIRQ is the = way an interrupt is exposed to a PV guest, without it there
would be no interrupt at all.

I'm a= fraid I'm out of my depth WRT how x86/MSIs and Xen x86/PV pirqs
interact, in particular WRT configuring which set of CPUs can have t= he IRQ
delivered.

If= no one else chimes in soon I'd suggest taking this to the dev list, at
the very least someone who knows what they are talking about= (i.e. other
than me) might be able to help.


Ian.



=
------=_Part_1156574_1230247901.1441129186877-- --===============1205095371090951997== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel --===============1205095371090951997==--