Possible bug on ARM with irq passthrough

Possible bug on ARM with irq passthrough

[Ivan Pavić2]

Hello,

I'm testing IRQ latency on exynos5422. I'm using Xen 4.10-unstable. I used device passthrough for SPI irq 64. For guest domain I used  FreeRTOS. When I use credit scheduler after destruction of domain, IRQ is released. I changed scheduler to credit2 and rtds at boot time and IRQ wasn't released after destruction. xl create dom-free-rtos.cfg fails with:

libxl: error: libxl_create.c:1278:domcreate_launch_dm: Domain 5:failed give domain access to irq 64: Device or resource busy

This is very strange because it doesn't happen with credit scheduler. This is output of xl dmesg:

(XEN) IRQ 64 is already used by domain 1

Everything works first time but when I destroy domain and try to create again it fails. It works normally with credit scheduler. In dom-free-rtos.cfg:
...
irqs = [64]
....

Regards,
Ivan Pavic

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Possible bug on ARM with irq passthrough

[Julien Grall]

On 04/07/17 21:20, Ivan Pavić2 wrote:
> Hello,

Hi Ivan,

> I'm testing IRQ latency on exynos5422. I'm using Xen 4.10-unstable. I used device passthrough for SPI irq 64. For guest domain I used  FreeRTOS. When I use credit scheduler after destruction of domain, IRQ is released. I changed scheduler to credit2 and rtds at boot time and IRQ wasn't released after destruction. xl create dom-free-rtos.cfg fails with:
>
> libxl: error: libxl_create.c:1278:domcreate_launch_dm: Domain 5:failed give domain access to irq 64: Device or resource busy
>
> This is very strange because it doesn't happen with credit scheduler. This is output of xl dmesg:
>
> (XEN) IRQ 64 is already used by domain 1
>
> Everything works first time but when I destroy domain and try to create again it fails. It works normally with credit scheduler. In dom-free-rtos.cfg:
> ...
> irqs = [64]
> ....

The Domain 1 still seems to be alive, you can confirm by via "xl list". 
You said it works with credit but not credit2. I think you hit the bug 
described in [1].

Dario, do you have any status on this bug? I think we should consider it 
as a blocker for Xen 4.10.

Cheers,

[1] 
https://lists.xenproject.org/archives/html/xen-devel/2017-02/msg00138.html

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Possible bug on ARM with irq passthrough

[Ivan Pavic]

On 07/05/2017 01:27 PM, Julien Grall wrote:
> On 04/07/17 21:20, Ivan Pavić2 wrote:
>> Hello,
>
> Hi Ivan,
>
>> I'm testing IRQ latency on exynos5422. I'm using Xen 4.10-unstable. I 
>> used device passthrough for SPI irq 64. For guest domain I used  
>> FreeRTOS. When I use credit scheduler after destruction of domain, 
>> IRQ is released. I changed scheduler to credit2 and rtds at boot time 
>> and IRQ wasn't released after destruction. xl create 
>> dom-free-rtos.cfg fails with:
>>
>> libxl: error: libxl_create.c:1278:domcreate_launch_dm: Domain 
>> 5:failed give domain access to irq 64: Device or resource busy
>>
>> This is very strange because it doesn't happen with credit scheduler. 
>> This is output of xl dmesg:
>>
>> (XEN) IRQ 64 is already used by domain 1
>>
>> Everything works first time but when I destroy domain and try to 
>> create again it fails. It works normally with credit scheduler. In 
>> dom-free-rtos.cfg:
>> ...
>> irqs = [64]
>> ....
>
> The Domain 1 still seems to be alive, you can confirm by via "xl 
> list". You said it works with credit but not credit2. I think you hit 
> the bug described in [1].
I checked xl-list. There is no domain with id 1, only dom0. Also I tried 
xenctx 1 which resulted in
     xc_domain_getinfo: No such process

Same problem is with rtds scheduler. As I said, no problem with credit 
scheduler.
> Dario, do you have any status on this bug? I think we should consider 
> it as a blocker for Xen 4.10.
>
> Cheers,
>
> [1] 
> https://lists.xenproject.org/archives/html/xen-devel/2017-02/msg00138.html
>
Regards,

Ivan Pavic


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Possible bug on ARM with irq passthrough

[Julien Grall]

Hi Ivan,

On 05/07/17 13:36, Ivan Pavic wrote:
> On 07/05/2017 01:27 PM, Julien Grall wrote:
>> On 04/07/17 21:20, Ivan Pavić2 wrote:
>>> Hello,
>>
>> Hi Ivan,
>>
>>> I'm testing IRQ latency on exynos5422. I'm using Xen 4.10-unstable. I
>>> used device passthrough for SPI irq 64. For guest domain I used
>>> FreeRTOS. When I use credit scheduler after destruction of domain,
>>> IRQ is released. I changed scheduler to credit2 and rtds at boot time
>>> and IRQ wasn't released after destruction. xl create
>>> dom-free-rtos.cfg fails with:
>>>
>>> libxl: error: libxl_create.c:1278:domcreate_launch_dm: Domain
>>> 5:failed give domain access to irq 64: Device or resource busy
>>>
>>> This is very strange because it doesn't happen with credit scheduler.
>>> This is output of xl dmesg:
>>>
>>> (XEN) IRQ 64 is already used by domain 1
>>>
>>> Everything works first time but when I destroy domain and try to
>>> create again it fails. It works normally with credit scheduler. In
>>> dom-free-rtos.cfg:
>>> ...
>>> irqs = [64]
>>> ....
>>
>> The Domain 1 still seems to be alive, you can confirm by via "xl
>> list". You said it works with credit but not credit2. I think you hit
>> the bug described in [1].
> I checked xl-list. There is no domain with id 1, only dom0. Also I tried
> xenctx 1 which resulted in
>     xc_domain_getinfo: No such process
>
> Same problem is with rtds scheduler. As I said, no problem with credit
> scheduler.

IRQs assigned to a domain will be released by domain_vgic_free() when 
calling release_guest_irq(). This is called by Xen when the domain is 
fully destroyed.

It might be possible the domain is waiting to be fully destroyed, and 
therefore hidden from the toolstack. Can you check if the domain is been 
destroyed by Xen. You can add printk in domain_destroy and 
complete_domain_destroy.

  * domain_destroy is called when there is no more reference on the 
domain (see put_domain)
  * complete_domain_destroy is called when RCU is quiescent

If you got a message in the former, but not the latter. Then you hit the 
bug described in my previous e-mail.

This would happen if some of the pCPUs are idle states (using wfi).r.
The main difference between credit1 and credit2 (I am not sure for RTDS) 
is there are IPI traffic with credit1 on idle pCPU because of the ticker.

>> Dario, do you have any status on this bug? I think we should consider
>> it as a blocker for Xen 4.10.
>>
>> Cheers,
>>
>> [1]
>> https://lists.xenproject.org/archives/html/xen-devel/2017-02/msg00138.html
>>
>>

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Possible bug on ARM with irq passthrough

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 958 bytes --]

On Wed, 2017-07-05 at 13:55 +0100, Julien Grall wrote:
> 
> If you got a message in the former, but not the latter. Then you hit
> the 
> bug described in my previous e-mail.
> 
> This would happen if some of the pCPUs are idle states (using wfi).r.
> The main difference between credit1 and credit2 (I am not sure for
> RTDS) 
> is there are IPI traffic with credit1 on idle pCPU because of the
> ticker.
> 
Exactly,

It's a bug in RCU, which happens to be exposed (or, at least, it turns
out to be this severe) only by certain schedulers, and only on ARM.

I think it makes sense that this is visible with RTDS as well, because
it also does not have a periodic timer.

Dario
-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

[-- Attachment #2: Type: text/plain, Size: 127 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Possible bug on ARM with irq passthrough

[Ivan Pavic]

On 07/05/2017 02:55 PM, Julien Grall wrote:
> Hi Ivan,
>
> On 05/07/17 13:36, Ivan Pavic wrote:
>> On 07/05/2017 01:27 PM, Julien Grall wrote:
>>> On 04/07/17 21:20, Ivan Pavić2 wrote:
>>>> Hello,
>>>
>>> Hi Ivan,
>>>
>>>> I'm testing IRQ latency on exynos5422. I'm using Xen 4.10-unstable. I
>>>> used device passthrough for SPI irq 64. For guest domain I used
>>>> FreeRTOS. When I use credit scheduler after destruction of domain,
>>>> IRQ is released. I changed scheduler to credit2 and rtds at boot time
>>>> and IRQ wasn't released after destruction. xl create
>>>> dom-free-rtos.cfg fails with:
>>>>
>>>> libxl: error: libxl_create.c:1278:domcreate_launch_dm: Domain
>>>> 5:failed give domain access to irq 64: Device or resource busy
>>>>
>>>> This is very strange because it doesn't happen with credit scheduler.
>>>> This is output of xl dmesg:
>>>>
>>>> (XEN) IRQ 64 is already used by domain 1
>>>>
>>>> Everything works first time but when I destroy domain and try to
>>>> create again it fails. It works normally with credit scheduler. In
>>>> dom-free-rtos.cfg:
>>>> ...
>>>> irqs = [64]
>>>> ....
>>>
>>> The Domain 1 still seems to be alive, you can confirm by via "xl
>>> list". You said it works with credit but not credit2. I think you hit
>>> the bug described in [1].
>> I checked xl-list. There is no domain with id 1, only dom0. Also I tried
>> xenctx 1 which resulted in
>>     xc_domain_getinfo: No such process
>>
>> Same problem is with rtds scheduler. As I said, no problem with credit
>> scheduler.
>
> IRQs assigned to a domain will be released by domain_vgic_free() when 
> calling release_guest_irq(). This is called by Xen when the domain is 
> fully destroyed.
>
> It might be possible the domain is waiting to be fully destroyed, and 
> therefore hidden from the toolstack. Can you check if the domain is 
> been destroyed by Xen. You can add printk in domain_destroy and 
> complete_domain_destroy.
>
>  * domain_destroy is called when there is no more reference on the 
> domain (see put_domain)
>  * complete_domain_destroy is called when RCU is quiescent
>
> If you got a message in the former, but not the latter. Then you hit 
> the bug described in my previous e-mail.
     Yes, I've got message only in domain_destroy.
>
> This would happen if some of the pCPUs are idle states (using wfi).r.
> The main difference between credit1 and credit2 (I am not sure for 
> RTDS) is there are IPI traffic with credit1 on idle pCPU because of 
> the ticker.
>
>>> Dario, do you have any status on this bug? I think we should consider
>>> it as a blocker for Xen 4.10.
>>>
>>> Cheers,
>>>
>>> [1]
>>> https://lists.xenproject.org/archives/html/xen-devel/2017-02/msg00138.html 
>>>
>>>
>>>
>
> Cheers,
>
     Regards,

     Ivan Pavic


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Possible bug on ARM with irq passthrough

[Julien Grall]

Hi Ivan,

On 05/07/17 14:50, Ivan Pavic wrote:
>
>
> On 07/05/2017 02:55 PM, Julien Grall wrote:
>> Hi Ivan,
>>
>> On 05/07/17 13:36, Ivan Pavic wrote:
>>> On 07/05/2017 01:27 PM, Julien Grall wrote:
>>>> On 04/07/17 21:20, Ivan Pavić2 wrote:
>>>>> Hello,
>>>>
>>>> Hi Ivan,
>>>>
>>>>> I'm testing IRQ latency on exynos5422. I'm using Xen 4.10-unstable. I
>>>>> used device passthrough for SPI irq 64. For guest domain I used
>>>>> FreeRTOS. When I use credit scheduler after destruction of domain,
>>>>> IRQ is released. I changed scheduler to credit2 and rtds at boot time
>>>>> and IRQ wasn't released after destruction. xl create
>>>>> dom-free-rtos.cfg fails with:
>>>>>
>>>>> libxl: error: libxl_create.c:1278:domcreate_launch_dm: Domain
>>>>> 5:failed give domain access to irq 64: Device or resource busy
>>>>>
>>>>> This is very strange because it doesn't happen with credit scheduler.
>>>>> This is output of xl dmesg:
>>>>>
>>>>> (XEN) IRQ 64 is already used by domain 1
>>>>>
>>>>> Everything works first time but when I destroy domain and try to
>>>>> create again it fails. It works normally with credit scheduler. In
>>>>> dom-free-rtos.cfg:
>>>>> ...
>>>>> irqs = [64]
>>>>> ....
>>>>
>>>> The Domain 1 still seems to be alive, you can confirm by via "xl
>>>> list". You said it works with credit but not credit2. I think you hit
>>>> the bug described in [1].
>>> I checked xl-list. There is no domain with id 1, only dom0. Also I tried
>>> xenctx 1 which resulted in
>>>     xc_domain_getinfo: No such process
>>>
>>> Same problem is with rtds scheduler. As I said, no problem with credit
>>> scheduler.
>>
>> IRQs assigned to a domain will be released by domain_vgic_free() when
>> calling release_guest_irq(). This is called by Xen when the domain is
>> fully destroyed.
>>
>> It might be possible the domain is waiting to be fully destroyed, and
>> therefore hidden from the toolstack. Can you check if the domain is
>> been destroyed by Xen. You can add printk in domain_destroy and
>> complete_domain_destroy.
>>
>>  * domain_destroy is called when there is no more reference on the
>> domain (see put_domain)
>>  * complete_domain_destroy is called when RCU is quiescent
>>
>> If you got a message in the former, but not the latter. Then you hit
>> the bug described in my previous e-mail.
>     Yes, I've got message only in domain_destroy.

Ok, so you are hitting the bug described in my previous e-mail. I would 
recommend to stick on credit1 until this bug is fixed (I will mark it as 
a blocker).

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Possible bug on ARM with irq passthrough

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 938 bytes --]

On Fri, 2017-07-07 at 17:19 +0100, Julien Grall wrote:
> On 05/07/17 14:50, Ivan Pavic wrote:
> > On 07/05/2017 02:55 PM, Julien Grall wrote:
> > > 
> > > If you got a message in the former, but not the latter. Then you
> > > hit
> > > the bug described in my previous e-mail.
> > 
> >     Yes, I've got message only in domain_destroy.
> 
> Ok, so you are hitting the bug described in my previous e-mail.
>
Indeed.

>  I would 
> recommend to stick on credit1 until this bug is fixed (I will mark it
> as 
> a blocker).
> 
Right. As said, it's on my priority list when back from Budapest.

Ivan, I'll ping you when I'll be done. :-)

Regards,
Dario
-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

[-- Attachment #2: Type: text/plain, Size: 127 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Possible bug on ARM with irq passthrough

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 940 bytes --]

Hello again,

On Sat, 2017-07-08 at 09:25 +0200, Dario Faggioli wrote:
> On Fri, 2017-07-07 at 17:19 +0100, Julien Grall wrote:
> >  I would 
> > recommend to stick on credit1 until this bug is fixed (I will mark
> > it
> > as 
> > a blocker).
> > 
> 
> Right. As said, it's on my priority list when back from Budapest.
> 
> Ivan, I'll ping you when I'll be done. :-)
> 
So, Ivan, patches are here:

 https://lists.xen.org/archives/html/xen-devel/2017-07/msg02770.html

They're not tested on ARM (the problem is generic, it's only than on
ARM has more serious symptoms) so, if you could give them a try, that
would be really helpful. :-)

Regards,
Dario
-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

[-- Attachment #2: Type: text/plain, Size: 127 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Possible bug on ARM with irq passthrough

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 1723 bytes --]

On Wed, 2017-07-05 at 12:27 +0100, Julien Grall wrote:
> On 04/07/17 21:20, Ivan Pavić2 wrote:
> > 
> > This is very strange because it doesn't happen with credit
> > scheduler. This is output of xl dmesg:
> > 
> > (XEN) IRQ 64 is already used by domain 1
> > 
> > Everything works first time but when I destroy domain and try to
> > create again it fails. It works normally with credit scheduler. In
> > dom-free-rtos.cfg:
> > ...
> > irqs = [64]
> > ....
> 
> The Domain 1 still seems to be alive, you can confirm by via "xl
> list". 
> You said it works with credit but not credit2. I think you hit the
> bug 
> described in [1].
> 
> Dario, do you have any status on this bug?
>
I have a patch in the works (different than the draft I sent you back
then). It needs some more thinking and testing though, because I am
still not sure it does the correct thing.

As you may have noticed, in fact, the tracing work I've also been doing
recently, includes adding tracing for RCUs. That was to help me
understand what actually happen, how things should work and why they
don't.

I had to stop working on the patch and switch to other things during
the last phases of 4.9 freeze and release, but I will get back to it
right after the Summit.

>  I think we should consider it 
> as a blocker for Xen 4.10.
> 
FWIW, I do agree. And I don't think there should be much problem fixing
this on time for that.

Regards,
Dario
-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

[-- Attachment #2: Type: text/plain, Size: 127 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel