Linux system got freezing when "Xenomai App" started (or terminated).

Linux system got freezing when "Xenomai App" started (or terminated).

[임준수(Junesoo Im) 수석 두산로보틱스]

hello. I am a Linux Application Developer.


Application uses Xenomai 3.1x.
The machine environment is 64bit and kernel 5.4.255.

Repeat machine reboot at 55 second intervals.
Most of the time it works fine, but sometimes the system freezes.
The mouse doesn't move and the keyboard doesn't work.
The screen also freezes.
There are no special messages in the kernel log.
The same goes for syslog.



So I'm running some tests.
If you test without running the application, there is no problem with the system.
Problems occur when the application runs.
However, I don't see any special messages in the application log.


I haven't found the cause of Freezing yet.
I am currently viewing the “https://v3.xenomai.org/troubleshooting/” site.
I would like to test some of the things mentioned on this site.
If possible, I would like you to look at this site together.


This site is talking about Kernel configuration that needs to be removed.
"CONFIG_KGDB"
This option is currently enabled, so I removed it and tested it.
But Freezing occurred.
A problem has occurred, but should this option be removed?


This site talks about "Freeze or machine lockup".
1. “Stack space overflow issue now biting sim real-time kernel thread?”
2. “Spurious delay/timeout values computed by the application (specifically: too short)”
3. “A case of freeze is a system call called in a loop which fails without its return value being properly checked.”


There are three explanations: I have a question about the first one.
How many threads can the xenomai program create?
What is the maximum stack size per thread in the xenomai program?
Our program has at least 22 xenomai threads.
The stack size is different for each thread. It may have a maximum of 8 MB and a minimum of 2 MB.



The question is complicated and long, but I desperately need help.
Please, reply.
Thank you.

Linux system got freezing when "Xenomai App" started (or terminated).

[임준수(Junesoo Im) 수석 두산로보틱스]

Hello. I am a Linux Application Developer.


Application uses Xenomai 3.1x.
The machine environment is 64bit and kernel 5.4.255.

Repeat machine reboot at 55 second intervals.
Most of the time it works fine, but sometimes the system freezes.
The mouse doesn't move and the keyboard doesn't work.
The screen also freezes.
There are no special messages in the kernel log.
The same goes for syslog.



So I'm running some tests.
If you test without running the application, there is no problem with the system.
Problems occur when the application runs.
However, I don't see any special messages in the application log.


I haven't found the cause of Freezing yet.
I am currently viewing the “https://v3.xenomai.org/troubleshooting/” site.
I would like to test some of the things mentioned on this site.
If possible, I would like you to look at this site together.


This site is talking about Kernel configuration that needs to be removed.
"CONFIG_KGDB"
This option is currently enabled, so I removed it and tested it.
But Freezing occurred.
A problem has occurred, but should this option be removed?


This site talks about "Freeze or machine lockup".
1. “Stack space overflow issue now biting sim real-time kernel thread?”
2. “Spurious delay/timeout values computed by the application (specifically: too short)”
3. “A case of freeze is a system call called in a loop which fails without its return value being properly checked.”


There are three explanations: I have a question about the first one.
How many threads can the xenomai program create?
What is the maximum stack size per thread in the xenomai program?
Our program has at least 22 xenomai threads.
The stack size is different for each thread. It may have a maximum of 8 MB and a minimum of 2 MB.



The question is complicated and long, but I desperately need help.
Please, reply.
Thank you.

Re: Linux system got freezing when "Xenomai App" started (or terminated).

[François Legal]

Le Lundi, Avril 15, 2024 11:23 CEST, ÀÓÁؼö(Junesoo Im) ¼ö¼® µÎ»ê·Îº¸Æ½½º	<junesoo.im@doosan.com> a écrit:

> Hello. I am a Linux Application Developer.
>
>
> Application uses Xenomai 3.1x.
> The machine environment is 64bit and kernel 5.4.255.
>
> Repeat machine reboot at 55 second intervals.
> Most of the time it works fine, but sometimes the system freezes.
> The mouse doesn't move and the keyboard doesn't work.
> The screen also freezes.
> There are no special messages in the kernel log.
> The same goes for syslog.
>
>
>
> So I'm running some tests.
> If you test without running the application, there is no problem with the system.
> Problems occur when the application runs.
> However, I don't see any special messages in the application log.
>
>
> I haven't found the cause of Freezing yet.
> I am currently viewing the ¡°https://v3.xenomai.org/troubleshooting/¡± site.
> I would like to test some of the things mentioned on this site.
> If possible, I would like you to look at this site together.
>
>
> This site is talking about Kernel configuration that needs to be removed.
> "CONFIG_KGDB"
> This option is currently enabled, so I removed it and tested it.
> But Freezing occurred.
> A problem has occurred, but should this option be removed?
>
>
> This site talks about "Freeze or machine lockup".
> 1. ¡°Stack space overflow issue now biting sim real-time kernel thread?¡±
> 2. ¡°Spurious delay/timeout values computed by the application (specifically: too short)¡±
> 3. ¡°A case of freeze is a system call called in a loop which fails without its return value being properly checked.¡±
>
>
> There are three explanations: I have a question about the first one.
> How many threads can the xenomai program create?
> What is the maximum stack size per thread in the xenomai program?
> Our program has at least 22 xenomai threads.
> The stack size is different for each thread. It may have a maximum of 8 MB and a minimum of 2 MB.
>
>
>
> The question is complicated and long, but I desperately need help.
> Please, reply.
> Thank you.


 Hi

have you tried enabling CONFIG_XENO_OPT_WATCHDOG to enable detection of runaway RT threads.
This might help you diagnosing the problem.

François

RE: Linux system got freezing when "Xenomai App" started (or terminated).

[임준수(Junesoo Im) 수석 두산로보틱스]

Dear François Legal.


Thank you so much for your quick reply!!


I checked "CONFIG_XENO_OPT_WATCHDOG" in your message.


(This e-mail type is not HTML. So, I write down some informations.)
I search "CONFIG_XENO_OPT_WATCHDOG" option in kernel's ".config" file
   -------------------------------------------------
   $KERNEL_ROOT) grep CONFIG_XENO_OPT_WATCHDOG .config
   Output> CONFIG_XENO_OPT_WATCHDOG=y
   Output> CONFIG_XENO_OPT_WATCHDOG_TIMEOUT=4

Also, Xenomai's watchdog works well.
Because Robot program is sometimes terminated by the watchdog.

Xenomai watchdog is also a problem, but right now we are focusing on system freezing.


Is there anything else you can tell me?

Thanks.
Best Regards.


-----Original Message-----
From: François Legal <devel@thom.fr.eu.org> 
Sent: Monday, April 15, 2024 6:49 PM
To: 임준수(Junesoo Im) 수석 두산로보틱스 <junesoo.im@doosan.com>
Cc: xenomai@lists.linux.dev; 박경훈(Kyunghoon Park) 팀장 두산로보틱스 <kyunghoon2.park@doosan.com>
Subject: Re: Linux system got freezing when "Xenomai App" started (or terminated).

주의: 이 메일은 외부에서 발송 되었습니다. 링크 혹은 첨부파일 열람 시 주의하시기 바랍니다.

CAUTION: This email is sent by an external account. Please take extra caution when clicking the link or opening the attachment.

Le Lundi, Avril 15, 2024 11:23 CEST, ÀÓÁؼö(Junesoo Im) ¼ö¼® µÎ»ê·Îº¸Æ½½º       <junesoo.im@doosan.com> a écrit:

> Hello. I am a Linux Application Developer.
>
>
> Application uses Xenomai 3.1x.
> The machine environment is 64bit and kernel 5.4.255.
>
> Repeat machine reboot at 55 second intervals.
> Most of the time it works fine, but sometimes the system freezes.
> The mouse doesn't move and the keyboard doesn't work.
> The screen also freezes.
> There are no special messages in the kernel log.
> The same goes for syslog.
>
>
>
> So I'm running some tests.
> If you test without running the application, there is no problem with the system.
> Problems occur when the application runs.
> However, I don't see any special messages in the application log.
>
>
> I haven't found the cause of Freezing yet.
> I am currently viewing the ¡°https://secure-web.cisco.com/1XLvVz-8MEZDdYZfDgbDD6YEaNOjJeaQ6a47KjHffv0KQbTypxooAIDbLz8urSXEI6vRlttlL27KSW_mfF0ENueummSZ38ucNEvjty5-ZVBy60tljLbXzBNlB5cgnIzhT8moSYlzkUphFAcsDj-ghcCQ1QnKn1xIcEJho6xMExZ2wArvZaBvOJ0fcuxDro1q4C6PwSin_iyNBt03iXUJuhHltWbGv9K-oth9mI-QTIiNDu3wG7Xej8rdBFNijChtrgaOlUBzUBAxaipEmzDHdBpRA1HEHvIm523iNHg7XsdjgKftK0Od1fpZmEZRtSn1N/https%3A%2F%2Fv3.xenomai.org%2Ftroubleshooting%2F¡± site.
> I would like to test some of the things mentioned on this site.
> If possible, I would like you to look at this site together.
>
>
> This site is talking about Kernel configuration that needs to be removed.
> "CONFIG_KGDB"
> This option is currently enabled, so I removed it and tested it.
> But Freezing occurred.
> A problem has occurred, but should this option be removed?
>
>
> This site talks about "Freeze or machine lockup".
> 1. ¡°Stack space overflow issue now biting sim real-time kernel 
> thread?¡± 2. ¡°Spurious delay/timeout values computed by the 
> application (specifically: too short)¡± 3. ¡°A case of freeze is a 
> system call called in a loop which fails without its return value 
> being properly checked.¡±
>
>
> There are three explanations: I have a question about the first one.
> How many threads can the xenomai program create?
> What is the maximum stack size per thread in the xenomai program?
> Our program has at least 22 xenomai threads.
> The stack size is different for each thread. It may have a maximum of 8 MB and a minimum of 2 MB.
>
>
>
> The question is complicated and long, but I desperately need help.
> Please, reply.
> Thank you.


 Hi

have you tried enabling CONFIG_XENO_OPT_WATCHDOG to enable detection of runaway RT threads.
This might help you diagnosing the problem.

François

RE: Linux system got freezing when "Xenomai App" started (or terminated).

[François Legal]

Le Lundi, Avril 15, 2024 12:07 CEST, 임준수(Junesoo Im) 수석 두산로보틱스 <junesoo.im@doosan.com> a écrit:

> Dear François Legal.
>
>
> Thank you so much for your quick reply!!
>
>
> I checked "CONFIG_XENO_OPT_WATCHDOG" in your message.
>
>
> (This e-mail type is not HTML. So, I write down some informations.)
> I search "CONFIG_XENO_OPT_WATCHDOG" option in kernel's ".config" file
>    -------------------------------------------------
>    $KERNEL_ROOT) grep CONFIG_XENO_OPT_WATCHDOG .config
>    Output> CONFIG_XENO_OPT_WATCHDOG=y
>    Output> CONFIG_XENO_OPT_WATCHDOG_TIMEOUT=4
>
> Also, Xenomai's watchdog works well.
> Because Robot program is sometimes terminated by the watchdog.
>
> Xenomai watchdog is also a problem, but right now we are focusing on system freezing.
>
>
> Is there anything else you can tell me?
>
> Thanks.
> Best Regards.
>
>
> -----Original Message-----
> From: François Legal <devel@thom.fr.eu.org>
> Sent: Monday, April 15, 2024 6:49 PM
> To: 임준수(Junesoo Im) 수석 두산로보틱스 <junesoo.im@doosan.com>
> Cc: xenomai@lists.linux.dev; 박경훈(Kyunghoon Park) 팀장 두산로보틱스 <kyunghoon2.park@doosan.com>
> Subject: Re: Linux system got freezing when "Xenomai App" started (or terminated).
>
> 주의: 이 메일은 외부에서 발송 되었습니다. 링크 혹은 첨부파일 열람 시 주의하시기 바랍니다.
>
> CAUTION: This email is sent by an external account. Please take extra caution when clicking the link or opening the attachment.
>
> Le Lundi, Avril 15, 2024 11:23 CEST, ÀÓÁؼö(Junesoo Im) ¼ö¼® µÎ»ê·Îº¸Æ½½º       <junesoo.im@doosan.com> a écrit:
>
> > Hello. I am a Linux Application Developer.
> >
> >
> > Application uses Xenomai 3.1x.
> > The machine environment is 64bit and kernel 5.4.255.
> >
> > Repeat machine reboot at 55 second intervals.
> > Most of the time it works fine, but sometimes the system freezes.
> > The mouse doesn't move and the keyboard doesn't work.
> > The screen also freezes.
> > There are no special messages in the kernel log.
> > The same goes for syslog.
> >
> >
> >
> > So I'm running some tests.
> > If you test without running the application, there is no problem with the system.
> > Problems occur when the application runs.
> > However, I don't see any special messages in the application log.
> >
> >
> > I haven't found the cause of Freezing yet.
> > I am currently viewing the ¡°https://secure-web.cisco.com/1XLvVz-8MEZDdYZfDgbDD6YEaNOjJeaQ6a47KjHffv0KQbTypxooAIDbLz8urSXEI6vRlttlL27KSW_mfF0ENueummSZ38ucNEvjty5-ZVBy60tljLbXzBNlB5cgnIzhT8moSYlzkUphFAcsDj-ghcCQ1QnKn1xIcEJho6xMExZ2wArvZaBvOJ0fcuxDro1q4C6PwSin_iyNBt03iXUJuhHltWbGv9K-oth9mI-QTIiNDu3wG7Xej8rdBFNijChtrgaOlUBzUBAxaipEmzDHdBpRA1HEHvIm523iNHg7XsdjgKftK0Od1fpZmEZRtSn1N/https%3A%2F%2Fv3.xenomai.org%2Ftroubleshooting%2F¡± site.
> > I would like to test some of the things mentioned on this site.
> > If possible, I would like you to look at this site together.
> >
> >
> > This site is talking about Kernel configuration that needs to be removed.
> > "CONFIG_KGDB"
> > This option is currently enabled, so I removed it and tested it.
> > But Freezing occurred.
> > A problem has occurred, but should this option be removed?
> >
> >
> > This site talks about "Freeze or machine lockup".
> > 1. ¡°Stack space overflow issue now biting sim real-time kernel
> > thread?¡± 2. ¡°Spurious delay/timeout values computed by the
> > application (specifically: too short)¡± 3. ¡°A case of freeze is a
> > system call called in a loop which fails without its return value
> > being properly checked.¡±
> >
> >
> > There are three explanations: I have a question about the first one.
> > How many threads can the xenomai program create?
> > What is the maximum stack size per thread in the xenomai program?
> > Our program has at least 22 xenomai threads.
> > The stack size is different for each thread. It may have a maximum of 8 MB and a minimum of 2 MB.
> >
> >
> >
> > The question is complicated and long, but I desperately need help.
> > Please, reply.
> > Thank you.
>
>
>  Hi
>
> have you tried enabling CONFIG_XENO_OPT_WATCHDOG to enable detection of runaway RT threads.
> This might help you diagnosing the problem.
>
> François
>
>

So you mean xenomai watchdog is already active, but still the system hangs ?
And the system hangs only when you run the RT application ?
Do your application rely on custom kernel drivers ?
Is JTAG debugging an option ?

François

RE: Linux system got freezing when "Xenomai App" started (or terminated).

[임준수(Junesoo Im) 수석 두산로보틱스]

Okay, I answered your question below.


So you mean xenomai watchdog is already active, but still the system hangs ?
-> Yes. One thing is certain, when testing the device reboot, only system hang occurs.
    At this time, the watchdog did not occur.

And the system hangs only when you run the RT application ?
-> Yes. System hang does not occur unless you run the robot program (with Xenomai).
    For reference, the robot program is automatically executed by Linux rc.local. at start.

Do your application rely on custom kernel drivers ?
-> Yes. The robot program has EtherCAT functionality. And the EtherCAT function uses "atemsys.ko".

Is JTAG debugging an option ?
-> Unfortunately, there isn't any.


I'll wait for your opinion.
Thanks.
Best Regards.

-----Original Message-----
From: François Legal <devel@thom.fr.eu.org> 
Sent: Monday, April 15, 2024 7:16 PM
To: 임준수(Junesoo Im) 수석 두산로보틱스 <junesoo.im@doosan.com>
Cc: xenomai@lists.linux.dev; 박경훈(Kyunghoon Park) 팀장 두산로보틱스 <kyunghoon2.park@doosan.com>
Subject: RE: Linux system got freezing when "Xenomai App" started (or terminated).

주의: 이 메일은 외부에서 발송 되었습니다. 링크 혹은 첨부파일 열람 시 주의하시기 바랍니다.

CAUTION: This email is sent by an external account. Please take extra caution when clicking the link or opening the attachment.

Le Lundi, Avril 15, 2024 12:07 CEST, 임준수(Junesoo Im) 수석 두산로보틱스 <junesoo.im@doosan.com> a écrit:

> Dear François Legal.
>
>
> Thank you so much for your quick reply!!
>
>
> I checked "CONFIG_XENO_OPT_WATCHDOG" in your message.
>
>
> (This e-mail type is not HTML. So, I write down some informations.) I 
> search "CONFIG_XENO_OPT_WATCHDOG" option in kernel's ".config" file
>    -------------------------------------------------
>    $KERNEL_ROOT) grep CONFIG_XENO_OPT_WATCHDOG .config
>    Output> CONFIG_XENO_OPT_WATCHDOG=y
>    Output> CONFIG_XENO_OPT_WATCHDOG_TIMEOUT=4
>
> Also, Xenomai's watchdog works well.
> Because Robot program is sometimes terminated by the watchdog.
>
> Xenomai watchdog is also a problem, but right now we are focusing on system freezing.
>
>
> Is there anything else you can tell me?
>
> Thanks.
> Best Regards.
>
>
> -----Original Message-----
> From: François Legal <devel@thom.fr.eu.org>
> Sent: Monday, April 15, 2024 6:49 PM
> To: 임준수(Junesoo Im) 수석 두산로보틱스 <junesoo.im@doosan.com>
> Cc: xenomai@lists.linux.dev; 박경훈(Kyunghoon Park) 팀장 두산로보틱스 
> <kyunghoon2.park@doosan.com>
> Subject: Re: Linux system got freezing when "Xenomai App" started (or terminated).
>
> 주의: 이 메일은 외부에서 발송 되었습니다. 링크 혹은 첨부파일 열람 시 주의하시기 바랍니다.
>
> CAUTION: This email is sent by an external account. Please take extra caution when clicking the link or opening the attachment.
>
> Le Lundi, Avril 15, 2024 11:23 CEST, ÀÓÁؼö(Junesoo Im) ¼ö¼® µÎ»ê·Îº¸Æ½½º       <junesoo.im@doosan.com> a écrit:
>
> > Hello. I am a Linux Application Developer.
> >
> >
> > Application uses Xenomai 3.1x.
> > The machine environment is 64bit and kernel 5.4.255.
> >
> > Repeat machine reboot at 55 second intervals.
> > Most of the time it works fine, but sometimes the system freezes.
> > The mouse doesn't move and the keyboard doesn't work.
> > The screen also freezes.
> > There are no special messages in the kernel log.
> > The same goes for syslog.
> >
> >
> >
> > So I'm running some tests.
> > If you test without running the application, there is no problem with the system.
> > Problems occur when the application runs.
> > However, I don't see any special messages in the application log.
> >
> >
> > I haven't found the cause of Freezing yet.
> > I am currently viewing the ¡°https://secure-web.cisco.com/1XLvVz-8MEZDdYZfDgbDD6YEaNOjJeaQ6a47KjHffv0KQbTypxooAIDbLz8urSXEI6vRlttlL27KSW_mfF0ENueummSZ38ucNEvjty5-ZVBy60tljLbXzBNlB5cgnIzhT8moSYlzkUphFAcsDj-ghcCQ1QnKn1xIcEJho6xMExZ2wArvZaBvOJ0fcuxDro1q4C6PwSin_iyNBt03iXUJuhHltWbGv9K-oth9mI-QTIiNDu3wG7Xej8rdBFNijChtrgaOlUBzUBAxaipEmzDHdBpRA1HEHvIm523iNHg7XsdjgKftK0Od1fpZmEZRtSn1N/https%3A%2F%2Fv3.xenomai.org%2Ftroubleshooting%2F¡± site.
> > I would like to test some of the things mentioned on this site.
> > If possible, I would like you to look at this site together.
> >
> >
> > This site is talking about Kernel configuration that needs to be removed.
> > "CONFIG_KGDB"
> > This option is currently enabled, so I removed it and tested it.
> > But Freezing occurred.
> > A problem has occurred, but should this option be removed?
> >
> >
> > This site talks about "Freeze or machine lockup".
> > 1. ¡°Stack space overflow issue now biting sim real-time kernel 
> > thread?¡± 2. ¡°Spurious delay/timeout values computed by the 
> > application (specifically: too short)¡± 3. ¡°A case of freeze is a 
> > system call called in a loop which fails without its return value 
> > being properly checked.¡±
> >
> >
> > There are three explanations: I have a question about the first one.
> > How many threads can the xenomai program create?
> > What is the maximum stack size per thread in the xenomai program?
> > Our program has at least 22 xenomai threads.
> > The stack size is different for each thread. It may have a maximum of 8 MB and a minimum of 2 MB.
> >
> >
> >
> > The question is complicated and long, but I desperately need help.
> > Please, reply.
> > Thank you.
>
>
>  Hi
>
> have you tried enabling CONFIG_XENO_OPT_WATCHDOG to enable detection of runaway RT threads.
> This might help you diagnosing the problem.
>
> François
>
>

So you mean xenomai watchdog is already active, but still the system hangs ?
And the system hangs only when you run the RT application ?
Do your application rely on custom kernel drivers ?
Is JTAG debugging an option ?

François

Re: Linux system got freezing when "Xenomai App" started (or terminated).

[Jan Kiszka]

On 15.04.24 11:23, 임준수(Junesoo Im) 수석 두산로보틱스 wrote:
> Hello. I am a Linux Application Developer.
> 
> 
> Application uses Xenomai 3.1x.
> The machine environment is 64bit and kernel 5.4.255.
> 
> Repeat machine reboot at 55 second intervals.
> Most of the time it works fine, but sometimes the system freezes.
> The mouse doesn't move and the keyboard doesn't work.
> The screen also freezes.
> There are no special messages in the kernel log.
> The same goes for syslog.
> 

Does the system already have a debug UART attach so that the kernel can
dump potential crash messages there? This is too often essential for
diagnosis.

> 
> 
> So I'm running some tests.
> If you test without running the application, there is no problem with the system.
> Problems occur when the application runs.
> However, I don't see any special messages in the application log.
> 
> 
> I haven't found the cause of Freezing yet.
> I am currently viewing the “https://v3.xenomai.org/troubleshooting/” site.
> I would like to test some of the things mentioned on this site.
> If possible, I would like you to look at this site together.
> 
> 
> This site is talking about Kernel configuration that needs to be removed.
> "CONFIG_KGDB"
> This option is currently enabled, so I removed it and tested it.
> But Freezing occurred.
> A problem has occurred, but should this option be removed?
> 
> 
> This site talks about "Freeze or machine lockup".
> 1. “Stack space overflow issue now biting sim real-time kernel thread?”
> 2. “Spurious delay/timeout values computed by the application (specifically: too short)”
> 3. “A case of freeze is a system call called in a loop which fails without its return value being properly checked.”
> 
> 
> There are three explanations: I have a question about the first one.
> How many threads can the xenomai program create?

There are some internal resource limits, configurable during build-time,
but you will get errors on creation when you hit them.

> What is the maximum stack size per thread in the xenomai program?

Only limited by available memory and memory layout.

> Our program has at least 22 xenomai threads.
> The stack size is different for each thread. It may have a maximum of 8 MB and a minimum of 2 MB.
> 

This is not the problem when it comes to freezes like you are seeing.

> 
> 
> The question is complicated and long, but I desperately need help.

Did you already compare / try the kernel configs for x86 that we are
using in xenomai-images [1]? Sometimes, user run into interesting corner
cases that your tests didn't cover because we can't test all the
possible config variations.

Jan

[1] https://source.denx.de/Xenomai/xenomai-images/

-- 
Siemens AG, Technology
Linux Expert Center

RE: Linux system got freezing when "Xenomai App" started (or terminated).

[François Legal]

Le Lundi, Avril 15, 2024 12:37 CEST, 임준수(Junesoo Im) 수석 두산로보틱스 <junesoo.im@doosan.com> a écrit:

> Okay, I answered your question below.
>
>
> So you mean xenomai watchdog is already active, but still the system hangs ?
> -> Yes. One thing is certain, when testing the device reboot, only system hang occurs.
>     At this time, the watchdog did not occur.
>
> And the system hangs only when you run the RT application ?
> -> Yes. System hang does not occur unless you run the robot program (with Xenomai).
>     For reference, the robot program is automatically executed by Linux rc.local. at start.
>
> Do your application rely on custom kernel drivers ?
> -> Yes. The robot program has EtherCAT functionality. And the EtherCAT function uses "atemsys.ko".
>
> Is JTAG debugging an option ?
> -> Unfortunately, there isn't any.
>
>
> I'll wait for your opinion.
> Thanks.
> Best Regards.
>
> -----Original Message-----
> From: François Legal <devel@thom.fr.eu.org>
> Sent: Monday, April 15, 2024 7:16 PM
> To: 임준수(Junesoo Im) 수석 두산로보틱스 <junesoo.im@doosan.com>
> Cc: xenomai@lists.linux.dev; 박경훈(Kyunghoon Park) 팀장 두산로보틱스 <kyunghoon2.park@doosan.com>
> Subject: RE: Linux system got freezing when "Xenomai App" started (or terminated).
>
> 주의: 이 메일은 외부에서 발송 되었습니다. 링크 혹은 첨부파일 열람 시 주의하시기 바랍니다.
>
> CAUTION: This email is sent by an external account. Please take extra caution when clicking the link or opening the attachment.
>
> Le Lundi, Avril 15, 2024 12:07 CEST, 임준수(Junesoo Im) 수석 두산로보틱스 <junesoo.im@doosan.com> a écrit:
>
> > Dear François Legal.
> >
> >
> > Thank you so much for your quick reply!!
> >
> >
> > I checked "CONFIG_XENO_OPT_WATCHDOG" in your message.
> >
> >
> > (This e-mail type is not HTML. So, I write down some informations.) I
> > search "CONFIG_XENO_OPT_WATCHDOG" option in kernel's ".config" file
> >    -------------------------------------------------
> >    $KERNEL_ROOT) grep CONFIG_XENO_OPT_WATCHDOG .config
> >    Output> CONFIG_XENO_OPT_WATCHDOG=y
> >    Output> CONFIG_XENO_OPT_WATCHDOG_TIMEOUT=4
> >
> > Also, Xenomai's watchdog works well.
> > Because Robot program is sometimes terminated by the watchdog.
> >
> > Xenomai watchdog is also a problem, but right now we are focusing on system freezing.
> >
> >
> > Is there anything else you can tell me?
> >
> > Thanks.
> > Best Regards.
> >
> >
> > -----Original Message-----
> > From: François Legal <devel@thom.fr.eu.org>
> > Sent: Monday, April 15, 2024 6:49 PM
> > To: 임준수(Junesoo Im) 수석 두산로보틱스 <junesoo.im@doosan.com>
> > Cc: xenomai@lists.linux.dev; 박경훈(Kyunghoon Park) 팀장 두산로보틱스
> > <kyunghoon2.park@doosan.com>
> > Subject: Re: Linux system got freezing when "Xenomai App" started (or terminated).
> >
> > 주의: 이 메일은 외부에서 발송 되었습니다. 링크 혹은 첨부파일 열람 시 주의하시기 바랍니다.
> >
> > CAUTION: This email is sent by an external account. Please take extra caution when clicking the link or opening the attachment.
> >
> > Le Lundi, Avril 15, 2024 11:23 CEST, ÀÓÁؼö(Junesoo Im) ¼ö¼® µÎ»ê·Îº¸Æ½½º       <junesoo.im@doosan.com> a écrit:
> >
> > > Hello. I am a Linux Application Developer.
> > >
> > >
> > > Application uses Xenomai 3.1x.
> > > The machine environment is 64bit and kernel 5.4.255.
> > >
> > > Repeat machine reboot at 55 second intervals.
> > > Most of the time it works fine, but sometimes the system freezes.
> > > The mouse doesn't move and the keyboard doesn't work.
> > > The screen also freezes.
> > > There are no special messages in the kernel log.
> > > The same goes for syslog.
> > >
> > >
> > >
> > > So I'm running some tests.
> > > If you test without running the application, there is no problem with the system.
> > > Problems occur when the application runs.
> > > However, I don't see any special messages in the application log.
> > >
> > >
> > > I haven't found the cause of Freezing yet.
> > > I am currently viewing the ¡°https://secure-web.cisco.com/1XLvVz-8MEZDdYZfDgbDD6YEaNOjJeaQ6a47KjHffv0KQbTypxooAIDbLz8urSXEI6vRlttlL27KSW_mfF0ENueummSZ38ucNEvjty5-ZVBy60tljLbXzBNlB5cgnIzhT8moSYlzkUphFAcsDj-ghcCQ1QnKn1xIcEJho6xMExZ2wArvZaBvOJ0fcuxDro1q4C6PwSin_iyNBt03iXUJuhHltWbGv9K-oth9mI-QTIiNDu3wG7Xej8rdBFNijChtrgaOlUBzUBAxaipEmzDHdBpRA1HEHvIm523iNHg7XsdjgKftK0Od1fpZmEZRtSn1N/https%3A%2F%2Fv3.xenomai.org%2Ftroubleshooting%2F¡± site.
> > > I would like to test some of the things mentioned on this site.
> > > If possible, I would like you to look at this site together.
> > >
> > >
> > > This site is talking about Kernel configuration that needs to be removed.
> > > "CONFIG_KGDB"
> > > This option is currently enabled, so I removed it and tested it.
> > > But Freezing occurred.
> > > A problem has occurred, but should this option be removed?
> > >
> > >
> > > This site talks about "Freeze or machine lockup".
> > > 1. ¡°Stack space overflow issue now biting sim real-time kernel
> > > thread?¡± 2. ¡°Spurious delay/timeout values computed by the
> > > application (specifically: too short)¡± 3. ¡°A case of freeze is a
> > > system call called in a loop which fails without its return value
> > > being properly checked.¡±
> > >
> > >
> > > There are three explanations: I have a question about the first one.
> > > How many threads can the xenomai program create?
> > > What is the maximum stack size per thread in the xenomai program?
> > > Our program has at least 22 xenomai threads.
> > > The stack size is different for each thread. It may have a maximum of 8 MB and a minimum of 2 MB.
> > >
> > >
> > >
> > > The question is complicated and long, but I desperately need help.
> > > Please, reply.
> > > Thank you.
> >
> >
> >  Hi
> >
> > have you tried enabling CONFIG_XENO_OPT_WATCHDOG to enable detection of runaway RT threads.
> > This might help you diagnosing the problem.
> >
> > François
> >
> >
>
> So you mean xenomai watchdog is already active, but still the system hangs ?
> And the system hangs only when you run the RT application ?
> Do your application rely on custom kernel drivers ?
> Is JTAG debugging an option ?
>
> François
>

Well, I can't say for sure, but IMHO, if xenomai watchdog does not trigger, I guess we can say it's not a userland runaway thread that causes the hang.

So this probably means problemsat kernel level. I don't know this kernel module your mentionning, but if it is proprietary, could you try to exercise it's functions with a simple test program (that would idealy reproduce the real application behaviour regarding driver access) and see if the hang can occur in that configuration.

Apart from that, and without JTAG debug possibility, I can not give any more advice on this. Sorry.

François

RE: Linux system got freezing when "Xenomai App" started (or terminated).

[임준수(Junesoo Im) 수석 두산로보틱스]

Dear François Legal.


I succeeded in getting the kernel log through serial.
If I attach it to the email, can you look at it together?
Or should I post a log of the question I'm asking?

I have something to ask about the kernel log.

Please reply.
Thanks.
Best Regards.


-----Original Message-----
From: François Legal <devel@thom.fr.eu.org> 
Sent: Monday, April 15, 2024 10:21 PM
To: 임준수(Junesoo Im) 수석 두산로보틱스 <junesoo.im@doosan.com>
Cc: xenomai@lists.linux.dev; 박경훈(Kyunghoon Park) 팀장 두산로보틱스 <kyunghoon2.park@doosan.com>
Subject: RE: Linux system got freezing when "Xenomai App" started (or terminated).

주의: 이 메일은 외부에서 발송 되었습니다. 링크 혹은 첨부파일 열람 시 주의하시기 바랍니다.

CAUTION: This email is sent by an external account. Please take extra caution when clicking the link or opening the attachment.

Le Lundi, Avril 15, 2024 12:37 CEST, 임준수(Junesoo Im) 수석 두산로보틱스 <junesoo.im@doosan.com> a écrit:

> Okay, I answered your question below.
>
>
> So you mean xenomai watchdog is already active, but still the system hangs ?
> -> Yes. One thing is certain, when testing the device reboot, only system hang occurs.
>     At this time, the watchdog did not occur.
>
> And the system hangs only when you run the RT application ?
> -> Yes. System hang does not occur unless you run the robot program (with Xenomai).
>     For reference, the robot program is automatically executed by Linux rc.local. at start.
>
> Do your application rely on custom kernel drivers ?
> -> Yes. The robot program has EtherCAT functionality. And the EtherCAT function uses "atemsys.ko".
>
> Is JTAG debugging an option ?
> -> Unfortunately, there isn't any.
>
>
> I'll wait for your opinion.
> Thanks.
> Best Regards.
>
> -----Original Message-----
> From: François Legal <devel@thom.fr.eu.org>
> Sent: Monday, April 15, 2024 7:16 PM
> To: 임준수(Junesoo Im) 수석 두산로보틱스 <junesoo.im@doosan.com>
> Cc: xenomai@lists.linux.dev; 박경훈(Kyunghoon Park) 팀장 두산로보틱스 
> <kyunghoon2.park@doosan.com>
> Subject: RE: Linux system got freezing when "Xenomai App" started (or terminated).
>
> 주의: 이 메일은 외부에서 발송 되었습니다. 링크 혹은 첨부파일 열람 시 주의하시기 바랍니다.
>
> CAUTION: This email is sent by an external account. Please take extra caution when clicking the link or opening the attachment.
>
> Le Lundi, Avril 15, 2024 12:07 CEST, 임준수(Junesoo Im) 수석 두산로보틱스 <junesoo.im@doosan.com> a écrit:
>
> > Dear François Legal.
> >
> >
> > Thank you so much for your quick reply!!
> >
> >
> > I checked "CONFIG_XENO_OPT_WATCHDOG" in your message.
> >
> >
> > (This e-mail type is not HTML. So, I write down some informations.) 
> > I search "CONFIG_XENO_OPT_WATCHDOG" option in kernel's ".config" file
> >    -------------------------------------------------
> >    $KERNEL_ROOT) grep CONFIG_XENO_OPT_WATCHDOG .config
> >    Output> CONFIG_XENO_OPT_WATCHDOG=y
> >    Output> CONFIG_XENO_OPT_WATCHDOG_TIMEOUT=4
> >
> > Also, Xenomai's watchdog works well.
> > Because Robot program is sometimes terminated by the watchdog.
> >
> > Xenomai watchdog is also a problem, but right now we are focusing on system freezing.
> >
> >
> > Is there anything else you can tell me?
> >
> > Thanks.
> > Best Regards.
> >
> >
> > -----Original Message-----
> > From: François Legal <devel@thom.fr.eu.org>
> > Sent: Monday, April 15, 2024 6:49 PM
> > To: 임준수(Junesoo Im) 수석 두산로보틱스 <junesoo.im@doosan.com>
> > Cc: xenomai@lists.linux.dev; 박경훈(Kyunghoon Park) 팀장 두산로보틱스 
> > <kyunghoon2.park@doosan.com>
> > Subject: Re: Linux system got freezing when "Xenomai App" started (or terminated).
> >
> > 주의: 이 메일은 외부에서 발송 되었습니다. 링크 혹은 첨부파일 열람 시 주의하시기 바랍니다.
> >
> > CAUTION: This email is sent by an external account. Please take extra caution when clicking the link or opening the attachment.
> >
> > Le Lundi, Avril 15, 2024 11:23 CEST, ÀÓÁؼö(Junesoo Im) ¼ö¼® µÎ»ê·Îº¸Æ½½º       <junesoo.im@doosan.com> a écrit:
> >
> > > Hello. I am a Linux Application Developer.
> > >
> > >
> > > Application uses Xenomai 3.1x.
> > > The machine environment is 64bit and kernel 5.4.255.
> > >
> > > Repeat machine reboot at 55 second intervals.
> > > Most of the time it works fine, but sometimes the system freezes.
> > > The mouse doesn't move and the keyboard doesn't work.
> > > The screen also freezes.
> > > There are no special messages in the kernel log.
> > > The same goes for syslog.
> > >
> > >
> > >
> > > So I'm running some tests.
> > > If you test without running the application, there is no problem with the system.
> > > Problems occur when the application runs.
> > > However, I don't see any special messages in the application log.
> > >
> > >
> > > I haven't found the cause of Freezing yet.
> > > I am currently viewing the ¡°https://secure-web.cisco.com/1XLvVz-8MEZDdYZfDgbDD6YEaNOjJeaQ6a47KjHffv0KQbTypxooAIDbLz8urSXEI6vRlttlL27KSW_mfF0ENueummSZ38ucNEvjty5-ZVBy60tljLbXzBNlB5cgnIzhT8moSYlzkUphFAcsDj-ghcCQ1QnKn1xIcEJho6xMExZ2wArvZaBvOJ0fcuxDro1q4C6PwSin_iyNBt03iXUJuhHltWbGv9K-oth9mI-QTIiNDu3wG7Xej8rdBFNijChtrgaOlUBzUBAxaipEmzDHdBpRA1HEHvIm523iNHg7XsdjgKftK0Od1fpZmEZRtSn1N/https%3A%2F%2Fv3.xenomai.org%2Ftroubleshooting%2F¡± site.
> > > I would like to test some of the things mentioned on this site.
> > > If possible, I would like you to look at this site together.
> > >
> > >
> > > This site is talking about Kernel configuration that needs to be removed.
> > > "CONFIG_KGDB"
> > > This option is currently enabled, so I removed it and tested it.
> > > But Freezing occurred.
> > > A problem has occurred, but should this option be removed?
> > >
> > >
> > > This site talks about "Freeze or machine lockup".
> > > 1. ¡°Stack space overflow issue now biting sim real-time kernel 
> > > thread?¡± 2. ¡°Spurious delay/timeout values computed by the 
> > > application (specifically: too short)¡± 3. ¡°A case of freeze is a 
> > > system call called in a loop which fails without its return value 
> > > being properly checked.¡±
> > >
> > >
> > > There are three explanations: I have a question about the first one.
> > > How many threads can the xenomai program create?
> > > What is the maximum stack size per thread in the xenomai program?
> > > Our program has at least 22 xenomai threads.
> > > The stack size is different for each thread. It may have a maximum of 8 MB and a minimum of 2 MB.
> > >
> > >
> > >
> > > The question is complicated and long, but I desperately need help.
> > > Please, reply.
> > > Thank you.
> >
> >
> >  Hi
> >
> > have you tried enabling CONFIG_XENO_OPT_WATCHDOG to enable detection of runaway RT threads.
> > This might help you diagnosing the problem.
> >
> > François
> >
> >
>
> So you mean xenomai watchdog is already active, but still the system hangs ?
> And the system hangs only when you run the RT application ?
> Do your application rely on custom kernel drivers ?
> Is JTAG debugging an option ?
>
> François
>

Well, I can't say for sure, but IMHO, if xenomai watchdog does not trigger, I guess we can say it's not a userland runaway thread that causes the hang.

So this probably means problemsat kernel level. I don't know this kernel module your mentionning, but if it is proprietary, could you try to exercise it's functions with a simple test program (that would idealy reproduce the real application behaviour regarding driver access) and see if the hang can occur in that configuration.

Apart from that, and without JTAG debug possibility, I can not give any more advice on this. Sorry.

François

RE: Linux system got freezing when "Xenomai App" started (or terminated).

[임준수(Junesoo Im) 수석 두산로보틱스]

Dear Jan,


I am validating our Xenomai Application with following setup:

   - Xenomai version: https://source.denx.de/Xenomai/xenomai/-/tree/v3.1.4
   - Ipipe: https://source.denx.de/Xenomai/ipipe-x86/-/tree/ipipe-core-5.4.255-x86-15



During Reliability Test in which we do the loop:
    {
        1. Boot up machine, start Xenomai application
        2. Wait for 1 min
        3. Reboot Machine
    }

Sometime we face the double-unlock issue right after starting Xenomai Application as below:
    1. [   14.046242] [Xenomai] lock 00000000733490c8 already unlocked on CPU #1
        [   14.046242]           last owner = kernel/xenomai/posix/process.c:1243 (handle_schedule_event(), CPU #-1)
    2. [   13.787715] [Xenomai] lock 0000000054aa4b33 already unlocked on CPU #3
        [   13.787715]           last owner = kernel/xenomai/posix/thread.c:91 (thread_hash(), CPU #0)
In consequence, System hung up (with Soft lock/Hard lock printed) and cannot recover.



I have done some searching and found this commit:
https://lore.kernel.org/all/9de20ac1-f92f-693e-f205-c965a2c6b434@xenomai.org/

As my understanding, this issue is caused by "xnlock_get()" which does not dissable the interrupt.
So I have some questions:

    1. Do you think our issue is related to "xnlock_get()"?
        As I checked the kernel code, there are some places still using that API, while mostly "xnlock_get_irqsave()" is used.
    2. Is "nklock" could also be accessed by POSIX kernel?
    3. Is migrating back and forth between Primary and Secondary mode safe? As we previously experienced live-lock when calling Linux VDSO gettimeofday(),
        we don't know if other Linux nonRT APIs is safe or not.



I am looking forward to hearing from you.

Thanks and BRs,
June.


-----Original Message-----
From: Jan Kiszka <jan.kiszka@siemens.com> 
Sent: Monday, April 15, 2024 10:17 PM
To: 임준수(Junesoo Im) 수석 두산로보틱스 <junesoo.im@doosan.com>; xenomai@lists.linux.dev
Cc: 박경훈(Kyunghoon Park) 팀장 두산로보틱스 <kyunghoon2.park@doosan.com>
Subject: Re: Linux system got freezing when "Xenomai App" started (or terminated).

주의: 이 메일은 외부에서 발송 되었습니다. 링크 혹은 첨부파일 열람 시 주의하시기 바랍니다.

CAUTION: This email is sent by an external account. Please take extra caution when clicking the link or opening the attachment.

On 15.04.24 11:23, 임준수(Junesoo Im) 수석 두산로보틱스 wrote:
> Hello. I am a Linux Application Developer.
>
>
> Application uses Xenomai 3.1x.
> The machine environment is 64bit and kernel 5.4.255.
>
> Repeat machine reboot at 55 second intervals.
> Most of the time it works fine, but sometimes the system freezes.
> The mouse doesn't move and the keyboard doesn't work.
> The screen also freezes.
> There are no special messages in the kernel log.
> The same goes for syslog.
>

Does the system already have a debug UART attach so that the kernel can dump potential crash messages there? This is too often essential for diagnosis.

>
>
> So I'm running some tests.
> If you test without running the application, there is no problem with the system.
> Problems occur when the application runs.
> However, I don't see any special messages in the application log.
>
>
> I haven't found the cause of Freezing yet.
> I am currently viewing the “https://secure-web.cisco.com/1oCU8v81ZI68sTkUFYDXWzRodL03_u4-nIhubM3eqgc8MgUKGls76sG7WAeTomVSko-jFUeALgmonHqzRYuOAGV4Njb4lD1WQ694xIhnUfhiUmQc4N2EWIgxb69Udsl6XvK2zDdmh5gAQ6puhHZt1A7v-Fm3ZsVF-3VyEDLbDCi6G4rMgmhzS-QkPf8zVzJtS2k_OwViBzDyECFtNzh88L9Lr3mlHcmblBG9hFQNN59PUWzHNxovqA1wMjsn4TEU7Uhw-TlKwDymYoC77G22McqdR-xRhJkvEmFnn68SyQ9sq9gk4uzDCN3xUzrqsKM26/https%3A%2F%2Fv3.xenomai.org%2Ftroubleshooting%2F” site.
> I would like to test some of the things mentioned on this site.
> If possible, I would like you to look at this site together.
>
>
> This site is talking about Kernel configuration that needs to be removed.
> "CONFIG_KGDB"
> This option is currently enabled, so I removed it and tested it.
> But Freezing occurred.
> A problem has occurred, but should this option be removed?
>
>
> This site talks about "Freeze or machine lockup".
> 1. “Stack space overflow issue now biting sim real-time kernel thread?”
> 2. “Spurious delay/timeout values computed by the application (specifically: too short)”
> 3. “A case of freeze is a system call called in a loop which fails without its return value being properly checked.”
>
>
> There are three explanations: I have a question about the first one.
> How many threads can the xenomai program create?

There are some internal resource limits, configurable during build-time, but you will get errors on creation when you hit them.

> What is the maximum stack size per thread in the xenomai program?

Only limited by available memory and memory layout.

> Our program has at least 22 xenomai threads.
> The stack size is different for each thread. It may have a maximum of 8 MB and a minimum of 2 MB.
>

This is not the problem when it comes to freezes like you are seeing.

>
>
> The question is complicated and long, but I desperately need help.

Did you already compare / try the kernel configs for x86 that we are using in xenomai-images [1]? Sometimes, user run into interesting corner cases that your tests didn't cover because we can't test all the possible config variations.

Jan

[1] https://secure-web.cisco.com/1GEj1iNRJAlhlMzCkNZht04buQGzCYUwdToCvZgFcq1QBhV6arj1eIz__RDCBOfd2J8NJEtasNeqgaFpgeT7iNdLVZvGIdkBBcX3oK-yvxutoI3qnUAV6-BFaDxMyxb4DZhfkV5RWaqRAsypHs92ZQYfAm1tlOa9hgVJQgStEb_4qfl50UQ1rXLaMQdy3EQgDpNW44h2jwIfOSVHkPLhrEypoNNlL0y9QaGj_UUmwvU1LLLY194S7YMwmRRAirtbMLU020w4m7r4twwRVykjGq8J_JJEVHS8gZvriwTaLFl-r1kOsR6qj44tTSlwr3OqG/https%3A%2F%2Fsource.denx.de%2FXenomai%2Fxenomai-images%2F

--
Siemens AG, Technology
Linux Expert Center

Re: Linux system got freezing when "Xenomai App" started (or terminated).

[Jan Kiszka]

On 31.07.24 11:35, 임준수(Junesoo Im) 수석 두산로보틱스 wrote:
> Dear Jan,
>
>
> I am validating our Xenomai Application with following setup:
>
>    - Xenomai version: https://source.denx.de/Xenomai/xenomai/-/tree/v3.1.4
>    - Ipipe: https://source.denx.de/Xenomai/ipipe-x86/-/tree/ipipe-core-5.4.255-x86-15
>
>
>
> During Reliability Test in which we do the loop:
>     {
>         1. Boot up machine, start Xenomai application
>         2. Wait for 1 min
>         3. Reboot Machine
>     }
>
> Sometime we face the double-unlock issue right after starting Xenomai Application as below:
>     1. [   14.046242] [Xenomai] lock 00000000733490c8 already unlocked on CPU #1
>         [   14.046242]           last owner = kernel/xenomai/posix/process.c:1243 (handle_schedule_event(), CPU #-1)
>     2. [   13.787715] [Xenomai] lock 0000000054aa4b33 already unlocked on CPU #3
>         [   13.787715]           last owner = kernel/xenomai/posix/thread.c:91 (thread_hash(), CPU #0)
> In consequence, System hung up (with Soft lock/Hard lock printed) and cannot recover.
>
>
>
> I have done some searching and found this commit:
> https://lore.kernel.org/all/9de20ac1-f92f-693e-f205-c965a2c6b434@xenomai.org/
>
> As my understanding, this issue is caused by "xnlock_get()" which does not dissable the interrupt.
> So I have some questions:
>
>     1. Do you think our issue is related to "xnlock_get()"?
>         As I checked the kernel code, there are some places still using that API, while mostly "xnlock_get_irqsave()" is used.

We may have a corner case around the reboot code - or can you also
reproduce without rebooting?

Furthermore, please test with recent versions as well. I-pipe kernels
are in maintenance mode, receiving less attention and minimal updates
these days now.

>     2. Is "nklock" could also be accessed by POSIX kernel?
>     3. Is migrating back and forth between Primary and Secondary mode safe? As we previously experienced live-lock when calling Linux VDSO gettimeofday(),

This is architectural with legacy I-pipe. Another reason to move to >
5.4, thus dovetail.

Jan