Re: Question regarding RT patches for ARM

Re: Question regarding RT patches for ARM

[Richard Weinberger]

Fawad,

[CC'ing RT experts]

On Mon, May 15, 2017 at 2:44 PM, Fawad Lateef <fawadlateef@gmail.com> wrote:
> Hi All,
>
> I am working on custom i.MX6 quad hardware and using RT patches for
> almost latest stable kernel 4.9 and facing some weird system stall OR
> 'unhandled page fault - exceptions'.
>
> Just want to confirm that "are RT patches for ARM are stable OR they
> are still experimental?" As I heard from my other colleagues that RT
> patches are not stable for ARM and should be avoided.

-- 
Thanks,
//richard

Re: Question regarding RT patches for ARM

[Fawad Lateef]

Hi,

I am talking in general about RT patches. Like downloaded 4.9.20 from
kernel.org and applied patch "4.9.20-rt16.patch".

Thanks

-- Fawad Lateef


On 15 May 2017 at 14:51, GuJiangfei <qiledexin@163.com> wrote:
> Which rt patch do you mean?
>
>
> 发自网易邮箱大师
> On 05/15/2017 20:44, Fawad Lateef wrote:
>
> Hi All,
>
> I am working on custom i.MX6 quad hardware and using RT patches for
> almost latest stable kernel 4.9 and facing some weird system stall OR
> 'unhandled page fault - exceptions'.
>
> Just want to confirm that "are RT patches for ARM are stable OR they
> are still experimental?" As I heard from my other colleagues that RT
> patches are not stable for ARM and should be avoided.
>
> Thanks in advance,
>
> -- Fawad Lateef
>
> _______________________________________________
> Kernelnewbies mailing list
> Kernelnewbies@kernelnewbies.org
> https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: Question regarding RT patches for ARM

[Sebastian Andrzej Siewior]

On 2017-05-15 15:09:45 [+0200], Fawad Lateef wrote:
> Hi,
> 
> I am talking in general about RT patches. Like downloaded 4.9.20 from
> kernel.org and applied patch "4.9.20-rt16.patch".

understood

> On 15 May 2017 at 14:51, GuJiangfei <qiledexin@163.com> wrote:
> > Which rt patch do you mean?
> >
> >
> > 发自网易邮箱大师
> > On 05/15/2017 20:44, Fawad Lateef wrote:
> >
> > Hi All,
> >
> > I am working on custom i.MX6 quad hardware and using RT patches for
> > almost latest stable kernel 4.9 and facing some weird system stall OR
> > 'unhandled page fault - exceptions'.

and you are sure that this problems are caused by the RT patch or is
there a chance that this is unrelated to the RT patch?

> > Just want to confirm that "are RT patches for ARM are stable OR they
> > are still experimental?" As I heard from my other colleagues that RT
> > patches are not stable for ARM and should be avoided.

It is stable and I am not aware of any ARM / IMX specific problems.

Since this got here from kernelnewbies: I would not recommend the
Preempt-RT patch set to add to any kernel just to have fancy patches.
People use it because they need to fulfill specific requirements which
would not work without it. You might want to check some of the
documentation in the RT-wiki
	https://wiki.linuxfoundation.org/realtime/documentation/start

> > Thanks in advance,
> >
> > -- Fawad Lateef

Sebastian

Re: Question regarding RT patches for ARM

[Fawad Lateef]

Hi Sebastian,


On 15 May 2017 at 15:47, Sebastian Andrzej Siewior
<sebastian.siewior@linutronix.de> wrote:
> On 2017-05-15 15:09:45 [+0200], Fawad Lateef wrote:
>> Hi,
>>
>> I am talking in general about RT patches. Like downloaded 4.9.20 from
>> kernel.org and applied patch "4.9.20-rt16.patch".
>
> understood
>
>> On 15 May 2017 at 14:51, GuJiangfei <qiledexin@163.com> wrote:
>> > Which rt patch do you mean?
>> >
>> >
>> > 发自网易邮箱大师
>> > On 05/15/2017 20:44, Fawad Lateef wrote:
>> >
>> > Hi All,
>> >
>> > I am working on custom i.MX6 quad hardware and using RT patches for
>> > almost latest stable kernel 4.9 and facing some weird system stall OR
>> > 'unhandled page fault - exceptions'.
>
> and you are sure that this problems are caused by the RT patch or is
> there a chance that this is unrelated to the RT patch?
>

The hardware platform is used by many other projects/products _but_
none of them is using RT patches.
So really not sure if its hardware issue, though problem can be seen
only in RT version of kernel which is needed for project.

>> > Just want to confirm that "are RT patches for ARM are stable OR they
>> > are still experimental?" As I heard from my other colleagues that RT
>> > patches are not stable for ARM and should be avoided.
>
> It is stable and I am not aware of any ARM / IMX specific problems.
>

Ok, thanks. Good to know. I actually want this sort of confirmation as
till now "platform/hardware manufacturer" says RT is unstable and
should not be used (as newer kernel already have low-latency)

> Since this got here from kernelnewbies: I would not recommend the
> Preempt-RT patch set to add to any kernel just to have fancy patches.
> People use it because they need to fulfill specific requirements which
> would not work without it. You might want to check some of the
> documentation in the RT-wiki
>         https://wiki.linuxfoundation.org/realtime/documentation/start
>

In our case we need RT because few other hardware components (external
devices) needs really strict hard-limit and we can't miss that.

(BTW I should have not included kernelnewbies in the CC list, but as
was asking question in general that's why included).

>> > Thanks in advance,
>> >
>> > -- Fawad Lateef
>
> Sebastian

Thanks,

Fawad Lateef

Re: Question regarding RT patches for ARM

[Eric Nelson]

Hi Fawad,

> 发自网易邮箱大师
> On 05/15/2017 20:44, Fawad Lateef wrote:
>
> Hi All,
>
> I am working on custom i.MX6 quad hardware and using RT patches for
> almost latest stable kernel 4.9 and facing some weird system stall OR
> 'unhandled page fault - exceptions'.
>

I'm not sure about the unhandled page fault, but the stalls may be
because of the SDMA driver.

See this patch in the Freescale/NXP community for kernel 4.1:
https://github.com/Freescale/meta-freescale/blob/master/recipes-kernel/linux/linux-fslc-imx-rt/0003-Work-around-CPU-stalls-in-the-imx-sdma-driver.patch

Regards,


Eric

Re: Question regarding RT patches for ARM

[Fawad Lateef]

Hi Eric,


On 15 May 2017 at 16:12, Eric Nelson <eric@nelint.com> wrote:
> Hi Fawad,
>
>> 发自网易邮箱大师
>> On 05/15/2017 20:44, Fawad Lateef wrote:
>>
>> Hi All,
>>
>> I am working on custom i.MX6 quad hardware and using RT patches for
>> almost latest stable kernel 4.9 and facing some weird system stall OR
>> 'unhandled page fault - exceptions'.
>>
>
> I'm not sure about the unhandled page fault, but the stalls may be
> because of the SDMA driver.
>
> See this patch in the Freescale/NXP community for kernel 4.1:
> https://github.com/Freescale/meta-freescale/blob/master/recipes-kernel/linux/linux-fslc-imx-rt/0003-Work-around-CPU-stalls-in-the-imx-sdma-driver.patch
>

Thanks for the patch link. I will give it a try.

> Regards,
>
>
> Eric

Regards,

Fawad Lateef

Re: Question regarding RT patches for ARM

[Fawad Lateef]

Hi,


On 15 May 2017 at 16:20, Fawad Lateef <fawadlateef@gmail.com> wrote:
> Hi Eric,
>
>
> On 15 May 2017 at 16:12, Eric Nelson <eric@nelint.com> wrote:
>> Hi Fawad,
>>
>>> 发自网易邮箱大师
>>> On 05/15/2017 20:44, Fawad Lateef wrote:
>>>
>>> Hi All,
>>>
>>> I am working on custom i.MX6 quad hardware and using RT patches for
>>> almost latest stable kernel 4.9 and facing some weird system stall OR
>>> 'unhandled page fault - exceptions'.
>>>
>>
>> I'm not sure about the unhandled page fault, but the stalls may be
>> because of the SDMA driver.
>>
>> See this patch in the Freescale/NXP community for kernel 4.1:
>> https://github.com/Freescale/meta-freescale/blob/master/recipes-kernel/linux/linux-fslc-imx-rt/0003-Work-around-CPU-stalls-in-the-imx-sdma-driver.patch
>>
>
> Thanks for the patch link. I will give it a try.

Talked to hardware manufacturer and they applied the patch.

Still that SDMA is only used for NOR flash and only u-boot and its
environment are stored in nor flash. So not sure if that is going to
help in any case.

So I would really like few more suggestions OR confirmations about ARM
(cortex-A9 based imx6) (preemept)-RT support/patches are stable and
can be used for final products.

Thanks

>
>> Regards,
>>
>>
>> Eric
>
> Regards,
>
> Fawad Lateef

Re: Question regarding RT patches for ARM

[Eric Nelson]

Hi Fawad,

On 05/17/2017 06:40 AM, Fawad Lateef wrote:
> On 15 May 2017 at 16:20, Fawad Lateef <fawadlateef@gmail.com> wrote:
>> Hi Eric,
>> On 15 May 2017 at 16:12, Eric Nelson <eric@nelint.com> wrote:
>>> Hi Fawad,
>>>
>>>> 发自网易邮箱大师
>>>> On 05/15/2017 20:44, Fawad Lateef wrote:
>>>>
>>>> Hi All,
>>>>
>>>> I am working on custom i.MX6 quad hardware and using RT patches for
>>>> almost latest stable kernel 4.9 and facing some weird system stall OR
>>>> 'unhandled page fault - exceptions'.
>>>>
>>>
>>> I'm not sure about the unhandled page fault, but the stalls may be
>>> because of the SDMA driver.
>>>
>>> See this patch in the Freescale/NXP community for kernel 4.1:
>>> https://github.com/Freescale/meta-freescale/blob/master/recipes-kernel/linux/linux-fslc-imx-rt/0003-Work-around-CPU-stalls-in-the-imx-sdma-driver.patch
>>>
>>
>> Thanks for the patch link. I will give it a try.
> 
> Talked to hardware manufacturer and they applied the patch.
> 
> Still that SDMA is only used for NOR flash and only u-boot and its
> environment are stored in nor flash. So not sure if that is going to
> help in any case.
> 

SDMA is used for much more than NOR flash (SSI, USB, and sometimes
UARTs).

> So I would really like few more suggestions OR confirmations about ARM
> (cortex-A9 based imx6) (preemept)-RT support/patches are stable and
> can be used for final products.
> 

I don't have any experience with the RT patches on 4.9 or later
kernels, but do know that they're being used on i.MX6 with the
vendor kernels (3.10.x, 4.1.x).

I believe Alison Chaiken mentioned using them with 4.9 kernels:
http://elinux.org/images/4/42/IRQs-_the_Hard%2C_the_Soft%2C_the_Threaded_and_the_Preemptible.pdf
https://www.youtube.com/watch?v=-pehAzaP1eg&feature=youtu.be&list=PLbzoR-pLrL6pRFP6SOywVJWdEHlmQE51q

Regards,


Eric

Re: Question regarding RT patches for ARM

[Sebastian Andrzej Siewior]

On 2017-05-17 07:45:39 [-0700], Eric Nelson wrote:
> Hi Fawad,
Hi Fawad,

> On 05/17/2017 06:40 AM, Fawad Lateef wrote:
> > > > I'm not sure about the unhandled page fault, but the stalls may be
> > > > because of the SDMA driver.
> > > > 
> > > > See this patch in the Freescale/NXP community for kernel 4.1:
> > > > https://github.com/Freescale/meta-freescale/blob/master/recipes-kernel/linux/linux-fslc-imx-rt/0003-Work-around-CPU-stalls-in-the-imx-sdma-driver.patch

That is something I see for the first time. The description says that it
leads to this stall but it fails to explain _how_ it got there.

> > > Thanks for the patch link. I will give it a try.
> > 
> > Talked to hardware manufacturer and they applied the patch.
> > 
> > Still that SDMA is only used for NOR flash and only u-boot and its
> > environment are stored in nor flash. So not sure if that is going to
> > help in any case.
> > 
> 
> SDMA is used for much more than NOR flash (SSI, USB, and sometimes
> UARTs).
> 
> > So I would really like few more suggestions OR confirmations about ARM
> > (cortex-A9 based imx6) (preemept)-RT support/patches are stable and
> > can be used for final products.
> > 
> 
> I don't have any experience with the RT patches on 4.9 or later
> kernels, but do know that they're being used on i.MX6 with the
> vendor kernels (3.10.x, 4.1.x).

I just looked into a few boot logs and it seems the Phyflex board (imx6
quad) bootet v4.9.27-rt18, run cyclictest including some tests and
nothing produced this "unhandled page fault" you mention here. This
includes the previous RT version in the v4.9 line.

Of this "page fault" happens on boot, it has to do something without
your .config + hw combination. If it happens at rune-time it is probably
triggered by a driver or an application is triggering this.

> Regards,
> 
> 
> Eric

Sebastian

Re: Question regarding RT patches for ARM

[Fawad Lateef]

Hi Sebastian,


On 17 May 2017 at 18:02, Sebastian Andrzej Siewior
<bigeasy@linutronix.de> wrote:
> On 2017-05-17 07:45:39 [-0700], Eric Nelson wrote:
>> Hi Fawad,
> Hi Fawad,
>
>> On 05/17/2017 06:40 AM, Fawad Lateef wrote:
>> > > > I'm not sure about the unhandled page fault, but the stalls may be
>> > > > because of the SDMA driver.
>> > > >
>> > > > See this patch in the Freescale/NXP community for kernel 4.1:
>> > > > https://github.com/Freescale/meta-freescale/blob/master/recipes-kernel/linux/linux-fslc-imx-rt/0003-Work-around-CPU-stalls-in-the-imx-sdma-driver.patch
>
> That is something I see for the first time. The description says that it
> leads to this stall but it fails to explain _how_ it got there.
>
>> > > Thanks for the patch link. I will give it a try.
>> >
>> > Talked to hardware manufacturer and they applied the patch.
>> >
>> > Still that SDMA is only used for NOR flash and only u-boot and its
>> > environment are stored in nor flash. So not sure if that is going to
>> > help in any case.
>> >
>>
>> SDMA is used for much more than NOR flash (SSI, USB, and sometimes
>> UARTs).
>>
>> > So I would really like few more suggestions OR confirmations about ARM
>> > (cortex-A9 based imx6) (preemept)-RT support/patches are stable and
>> > can be used for final products.
>> >
>>
>> I don't have any experience with the RT patches on 4.9 or later
>> kernels, but do know that they're being used on i.MX6 with the
>> vendor kernels (3.10.x, 4.1.x).
>
> I just looked into a few boot logs and it seems the Phyflex board (imx6
> quad) bootet v4.9.27-rt18, run cyclictest including some tests and
> nothing produced this "unhandled page fault" you mention here. This
> includes the previous RT version in the v4.9 line.
>

Thanks, seems like RT patches are quite stable for ARM then.

> Of this "page fault" happens on boot, it has to do something without
> your .config + hw combination. If it happens at rune-time it is probably
> triggered by a driver or an application is triggering this.
>

Yes, this happens at run-time and not always. Sometimes hardware just
hangs/stalls.

As now RT is widely used on ARM so to me seems like it something
related to hardware design/setting OR application.

Another question: Is RT user-space application can make kernel/system
crash? As to me kernel should kill the app but let itself running
_but_ not sure in the case of RT patched kernel.

>> Regards,
>>
>>
>> Eric
>
> Sebastian

Thanks,

Fawad Lateef

Re: Question regarding RT patches for ARM

[Sebastian Andrzej Siewior]

On 2017-05-18 13:08:32 [+0200], Fawad Lateef wrote:
> Hi Sebastian,
Hi Fawad,

> > Of this "page fault" happens on boot, it has to do something without
> > your .config + hw combination. If it happens at rune-time it is probably
> > triggered by a driver or an application is triggering this.
> >
> 
> Yes, this happens at run-time and not always. Sometimes hardware just
> hangs/stalls.

So this sounds like a race condition.

> As now RT is widely used on ARM so to me seems like it something
> related to hardware design/setting OR application.
> 
> Another question: Is RT user-space application can make kernel/system
> crash? As to me kernel should kill the app but let itself running
> _but_ not sure in the case of RT patched kernel.

This is correct. With RT or without it: a use space should not be able
to crash the kernel.
It is possible that a race within a driver triggers more reliably on RT.
I would recommend to strip down the system to the minimum where the
issue should not trigger. And then slowly increase the "enabled"
hardware components so you might get an idea when or what could cause
it.

> Thanks,
> 
> Fawad Lateef

Sebastian

Re: Question regarding RT patches for ARM

[Fawad Lateef]

Hi Eric,


On 17 May 2017 at 16:45, Eric Nelson <eric@nelint.com> wrote:
> Hi Fawad,
>
> On 05/17/2017 06:40 AM, Fawad Lateef wrote:
>>
>> On 15 May 2017 at 16:20, Fawad Lateef <fawadlateef@gmail.com> wrote:
>>>
>>> Hi Eric,
>>> On 15 May 2017 at 16:12, Eric Nelson <eric@nelint.com> wrote:
>>>>
>>>> Hi Fawad,
>>>>
>>>>> 发自网易邮箱大师
>>>>> On 05/15/2017 20:44, Fawad Lateef wrote:
>>>>>
>>>>> Hi All,
>>>>>
>>>>> I am working on custom i.MX6 quad hardware and using RT patches for
>>>>> almost latest stable kernel 4.9 and facing some weird system stall OR
>>>>> 'unhandled page fault - exceptions'.
>>>>>
>>>>
>>>> I'm not sure about the unhandled page fault, but the stalls may be
>>>> because of the SDMA driver.
>>>>
>>>> See this patch in the Freescale/NXP community for kernel 4.1:
>>>>
>>>> https://github.com/Freescale/meta-freescale/blob/master/recipes-kernel/linux/linux-fslc-imx-rt/0003-Work-around-CPU-stalls-in-the-imx-sdma-driver.patch
>>>>
>>>
>>> Thanks for the patch link. I will give it a try.
>>
>>
>> Talked to hardware manufacturer and they applied the patch.
>>
>> Still that SDMA is only used for NOR flash and only u-boot and its
>> environment are stored in nor flash. So not sure if that is going to
>> help in any case.
>>
>
> SDMA is used for much more than NOR flash (SSI, USB, and sometimes
> UARTs).
>

Yes SDMA can be used for many devices but in our case its only set to
use for NOR (to confirm I also checked sdma interrupts in system and
they just 21 or so which happened during boot time).

>> So I would really like few more suggestions OR confirmations about ARM
>> (cortex-A9 based imx6) (preemept)-RT support/patches are stable and
>> can be used for final products.
>>
>
> I don't have any experience with the RT patches on 4.9 or later
> kernels, but do know that they're being used on i.MX6 with the
> vendor kernels (3.10.x, 4.1.x).
>
> I believe Alison Chaiken mentioned using them with 4.9 kernels:
> http://elinux.org/images/4/42/IRQs-_the_Hard%2C_the_Soft%2C_the_Threaded_and_the_Preemptible.pdf
> https://www.youtube.com/watch?v=-pehAzaP1eg&feature=youtu.be&list=PLbzoR-pLrL6pRFP6SOywVJWdEHlmQE51q
>

Thanks for the video/link. Seems like very informative. Will check it soon.

> Regards,
>
>
> Eric

Regards,

Fawad Lateef