From mboxrd@z Thu Jan 1 00:00:00 1970 From: Justin Acker Subject: Re: xhci_hcd intterrupt affinity in Dom0/DomU limited to single interrupt Date: Thu, 3 Sep 2015 12:04:53 +0000 (UTC) Message-ID: <1292756745.962760.1441281893582.JavaMail.yahoo@mail.yahoo.com> References: <55E839E3020000780009F3CD@prv-mh.provo.novell.com> Reply-To: Justin Acker Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============6007155289817719173==" Return-path: Received: from mail6.bemta14.messagelabs.com ([193.109.254.103]) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1ZXTGE-0001nm-K3 for xen-devel@lists.xenproject.org; Thu, 03 Sep 2015 12:04:59 +0000 In-Reply-To: <55E839E3020000780009F3CD@prv-mh.provo.novell.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Jan Beulich Cc: xen-devel List-Id: xen-devel@lists.xenproject.org --===============6007155289817719173== Content-Type: multipart/alternative; boundary="----=_Part_962758_613991074.1441281893545" Content-Length: 17750 ------=_Part_962758_613991074.1441281893545 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable From: Jan Beulich To: Justin Acker =20 Cc: xen-devel =20 Sent: Thursday, September 3, 2015 6:15 AM Subject: Re: [Xen-devel] xhci_hcd intterrupt affinity in Dom0/DomU limited= to single interrupt =20 (re-adding xen-devel) >>> On 02.09.15 at 19:17, wrote: >=C2=A0 =C2=A0 =C2=A0 From: Jan Beulich >=C2=A0 Sent: Wednesday, September 2, 2015 4:58 AM >>>> Justin Acker 09/02/15 1:14 AM >>> >> 00:14.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset F= amily USB xHCI Host Controller (rev 04) (prog-if 30 [XHCI]) >>=C2=A0 =C2=A0 Subsystem: Dell Device 053e >>=C2=A0 =C2=A0 Flags: bus master, medium devsel, latency 0, IRQ 78 >>=C2=A0 =C2=A0 Memory at f7f20000 (64-bit, non-prefetchable) [size=3D64K] >>=C2=A0 =C2=A0 Capabilities: [70] Power Management version 2 >>=C2=A0 =C2=A0 Capabilities: [80] MSI: Enable+ Count=3D1/8 Maskable- 64bit= + >=20 > This shows that the driver could use up to 8 MSI IRQs, but chose to use j= ust=20 > one. If > this is the same under Xen and the native kernel, the driver likely doesn= 't=20 > know any > better. If under native more interrupts are being used, there might be an= =20 > issue with > Xen specific code in the kernel or hypervisor code. We'd need to see deta= ils=20 > to be > able to tell. >=20 > Please let me know what details I should provide.=20 >=20 > Jan Please, first of all, get you reply style fixed. Just look at the above and tell me how a reader should figure which parts of the text were written by whom. Together with other replies you sent, I first of all wonder whether you've understood what you've been told: Any interrupt delivered via the event channel mechanism can't be delivered to more than one CPU unless it gets moved around them by a tool or manually. When you set the affinity to more than on (v)CPU, the kernel will pick one (usually the first) out of the provided set and bind the event channel to that vCPU. =C2=A0=C2=A0 =C2=A0 I am still confused as to whether any device, or in thi= s case xhci_hcd, can use more than one cpu at any given time. My =C2=A0=C2= =A0=C2=A0=C2=A0understanding based on David's response is that it cannot du= e to the event channel mapping. The device interrupt can be pinned =C2=A0= =C2=A0=C2=A0=C2=A0to a specific cpu by specifying the affinity. I was hopin= g there was a way to allow the driver's interrupt to be scheduled to =C2=A0= =C2=A0=C2=A0=C2=A0use more than 1 CPU at any given time.=20 As to, in the XHCI case, using multi-vector MSI: Please tell use whether the lspci output still left in context above was with a kernel running natively or under Xen. In the former case, the driver may need improving. In the latter case we'd need to see, for comparison, the same output with a natively running kernel. If it matches the Xen one, same thing (driver may need improving). If it doesn't match, maximum verbosity hypervisor and kernel logs would be what we'd need to start with. Jan Above driver context was from a native kernel. However, the driver appears = to load the same in both cases.=20 Native kernel: 00:14.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Fami= ly USB xHCI (rev 05) (prog-if 30 [XHCI]) =C2=A0=C2=A0=C2=A0 Subsystem: Intel Corporation 8 Series/C220 Series Chipse= t Family USB xHCI =C2=A0=C2=A0=C2=A0 Flags: bus master, medium devsel, latency 0, IRQ 27 =C2=A0=C2=A0=C2=A0 Memory at f7e20000 (64-bit, non-prefetchable) [size=3D64= K] =C2=A0=C2=A0=C2=A0 Capabilities: [70] Power Management version 2 =C2=A0=C2=A0=C2=A0 Capabilities: [80] MSI: Enable+ Count=3D1/8 Maskable- 64= bit+ =C2=A0=C2=A0=C2=A0 Kernel driver in use: xhci_hcd With Dom0 loaded: 00:14.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Fami= ly USB xHCI (rev 05) (prog-if 30 [XHCI]) =C2=A0=C2=A0=C2=A0 Subsystem: Intel Corporation 8 Series/C220 Series Chipse= t Family USB xHCI =C2=A0=C2=A0=C2=A0 Flags: bus master, medium devsel, latency 0, IRQ 78 =C2=A0=C2=A0=C2=A0 Memory at f7e20000 (64-bit, non-prefetchable) [size=3D64= K]=C2=A0=C2=A0=C2=A0 Capabilities: [70] Power Management version 2 =C2=A0=C2=A0=C2=A0 Capabilities: [80] MSI: Enable+ Count=3D1/8 Maskable- 64= bit+=C2=A0=C2=A0=C2=A0 Kernel driver in use: xhci_hcd ------=_Part_962758_613991074.1441281893545 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable

From: Jan Beulich <JBeulich@suse.com><= br clear=3D"none"> To: Just= in Acker <ackerj67@yahoo.com>
Cc: xen-devel <xen-devel@lists.xenproject.or= g>
Sent:= Thursday, September 3, 2015 6:15 AM
Subject: Re: [Xen-devel] xhci_hcd intterr= upt affinity in Dom0/DomU limited to single interrupt

(re-adding xen-devel)=

>>> On 02.09.15 at 19:17, &l= t;ackerj67@yahoo.com= > wrote:
>      From: Jan Beulich &= lt;jbeulich@suse.com&g= t;
>  Sent: Wednesday, September 2, 2015 4:58 AM<= br clear=3D"none">>>>> Justin Acker <ackerj67@yahoo.com> 09/02/15 1:14 AM >= ;>>
>> 00:14.0 USB controller: Intel Corporat= ion 7 Series/C210 Series Chipset Family USB xHCI Host Controller (rev 04) (= prog-if 30 [XHCI])
>>    Subsystem: Dell = Device 053e
>>    Flags: bus master, medi= um devsel, latency 0, IRQ 78
>>    Memory= at f7f20000 (64-bit, non-prefetchable) [size=3D64K]
>= >    Capabilities: [70] Power Management version 2
>>    Capabilities: [80] MSI: Enable+ Count=3D1/8 Ma= skable- 64bit+
>
> This shows th= at the driver could use up to 8 MSI IRQs, but chose to use just
> one. If
> this is the same under Xen an= d the native kernel, the driver likely doesn't
> know= any
> better. If under native more interrupts are bei= ng used, there might be an
> issue with
> Xen specific code in the kernel or hypervisor code. We'd need to = see details
> to be
> able to te= ll.
>
> Please let me know what = details I should provide.
>
> J= an

Please, first of all, get you reply= style fixed. Just look at the above
and tell me how a re= ader should figure which parts of the text were
written b= y whom.

Together with other replies yo= u sent, I first of all wonder whether
you've understood w= hat you've been told: Any interrupt delivered
via the eve= nt channel mechanism can't be delivered to more than
one = CPU unless it gets moved around them by a tool or manually.
When you set the affinity to more than on (v)CPU, the kernel will
pick one (usually the first) out of the provided set and bind t= he
event channel to that vCPU.

     I am still confused as= to whether any device, or in this case xhci_hcd, can use more than one cpu= at any given time. My     understanding based on David= 's response is that it cannot due to the event channel mapping. The device = interrupt can be pinned     to a specific cpu by specif= ying the affinity. I was hoping there was a way to allow the driver's inter= rupt to be scheduled to     use more than 1 CPU at any = given time.

As to, in = the XHCI case, using multi-vector MSI: Please tell use
wh= ether the lspci output still left in context above was with a
kernel running natively or under Xen. In the former case, the
driver may need improving. In the latter case we'd need to see,for comparison, the same output with a natively running ke= rnel. If
it matches the Xen one, same thing (driver may n= eed improving).
If it doesn't match, maximum verbosity hy= pervisor and kernel logs
would be what we'd need to start= with.




Jan

Abo= ve driver context was from a native kernel. However, the driver appears to = load the same in both cases.

Native kernel:

00:14.0 USB controller: Intel Corporation 8 Ser= ies/C220 Series Chipset Family USB xHCI (rev 05) (prog-if 30 [XHCI])
    Subsystem: Intel Corporation 8 Series/C220= Series Chipset Family USB xHCI
    Flag= s: bus master, medium devsel, latency 0, IRQ 27
 &= nbsp;  Memory at f7e20000 (64-bit, non-prefetchable) [size=3D64K]
    Capabilities: [70] Power Management vers= ion 2
    Capabilities: [80] MSI: Enable= + Count=3D1/8 Maskable- 64bit+
    Kerne= l driver in use: xhci_hcd

With Dom= 0 loaded:

00:14.0 USB controller: = Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI (rev 05) (pr= og-if 30 [XHCI])
    Subsystem: Intel Co= rporation 8 Series/C220 Series Chipset Family USB xHCI
=     Flags: bus master, medium devsel, latency 0, IRQ 78
    Memory at f7e20000 (64-bit, non-prefetch= able) [size=3D64K]
&= nbsp;   Capabilities: [70] Power Management version 2
    Capabilities: [80] MSI: Enable+ Count=3D1/8 Ma= skable- 64bit+
    Kernel driver in use: xhci_hcd=


=

=
------=_Part_962758_613991074.1441281893545-- --===============6007155289817719173== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel --===============6007155289817719173==--