* Re: Re:
@ 2022-02-11 15:06 Caine Chen
0 siblings, 0 replies; 6+ messages in thread
From: Caine Chen @ 2022-02-11 15:06 UTC (permalink / raw)
To: neelx.g@gmail.com; +Cc: linux-rt-users@vger.kernel.org
Hi Daniel:
Thanks for your reply.
> Not really. I guess there's one misunderstanding in your description.
> Disabling the bottom half is local to running thread and not to the
> CPU which executes that thread. As an effect, preemption practically
> enables the bottom half again (as long as the new thread did not have
> it already disabled before, of course...).
It's a bit confused for me why disabling the bottom half is local to thread
and not to the CPU. From my humble perspective, every forced threaded
irq_threads will invoke local_bh_disable( ) and try to get bh_lock before they
enter irq handler. If bh_lock(now is softirq_ctrl.lock) is held by other thread,
all forced-threaded irq_threads on this CPU will wait until the lock is released.
So how does preemption enable the bottom half again?
To test this, I did an experiment in v5.4 kernel.
First, I created a kthread and bound it to CPU0:
int test_init( )
{
......
p = kthread_create(my_debug_func, NULL, "my_test");
kthread_bind(p, 0);
wake_up_process(p);
......
}
This kthread will invoke local_bh_disable()/local_bh_enable() periodically:
int my_debug_func(void *arg)
{
......
while(!kthread_should_stop()) {
......
local_bh_disable();
/* just do some busy work, such as memcpy, kmalloc and so on */
do_some_work();
local_bh_enable();
}
......
}
What'more, I added some logs in some forced-threaded irq handlers to find out when they was excuted.
After "my_test" thread disabled local bh, there were no forced-threaded irq threads running on CPU0.
But after "my_test" thread enabled local bh, forced-threaded irqs came again.
It seems that disabling the bottom half is local to CPU.
> That said, the irq_thread will _not_ be blocked as bottom half is not
> disabled in it's context. From your chart, it's disabled only in
> thread_3 context and thread_1 context. But these two are independent
> (due to the different thread contexts and not the different CPU
> contexts as you misassumed) and they do not block each other either,
> it's the rw_lock serializing these threads, right?
> You should be able to see this with tracing. There should be no issue
> or the issue is different than you think it is and different than you
> described here.
> Hopefully the above helps you,
> Daniel
Thanks
Caine
This email and any attachments thereto may contain private, confidential, and privileged material for the sole use of the intended recipient. Any review, copying, or distribution of this email (or any attachments thereto) by others is strictly prohibited. If you are not the intended recipient, please contact the sender immediately and permanently delete the original and any copies of this email and any attachments thereto.
此电子邮件及附件所包含内容具有机密性，且仅限于接收人使用。未经允许，禁止第三人阅读、复制或传播该电子邮件中的任何信息。如果您不属于以上电子邮件的目标接收者，请您立即通知发送人并删除原电子邮件及其相关的附件。
^ permalink raw reply [flat|nested] 6+ messages in thread[parent not found: <10b1995b392e490aaa2db645f219015e@dji.com>]
* 转发:
[not found] <10b1995b392e490aaa2db645f219015e@dji.com>
@ 2022-01-17 12:54 ` Caine Chen
2022-02-03 11:49 ` Daniel Vacek
0 siblings, 1 reply; 6+ messages in thread
From: Caine Chen @ 2022-01-17 12:54 UTC (permalink / raw)
To: linux-rt-users@vger.kernel.org
Hi guys:
We found that some IRQ threads will block in local_bh_disable( ) for
long time in some situation and we hope to get your valuable suggestions.
My kernel version is 5.4 and the irq-delay is caused by the use of
write_lock_bh().
It can be described in the following figure:
(1) Thread_1 which is a SCHED_NORMAL thread runs on CPU1,
and it uses read_lock_bh() to protect some data.
(2) Thread_2 which is a SCHED_RR thread runs on CPU1 and it preempts thread_1
after thread_1 invoked read_lock_bh(). Thread_2 may run 60 ms in my system.
(3) Thread_3 which is a SCHED_NORMAL thread runs on CPU0. This thread acquires
writer's lock by invoking write_lock_bh(). This function will disable
button-half firstly by invoking local_bh_disable( ). But it will block in
rt_write_lock() , because read lock is held by thread_1.
(4) At this time, if irq thread without IRQF_NO_THREAD flag on CPU0 trys to
acquire bh_lock(it has been renamed as softirq_ctrl.lock now), irq
thread will block because this lock is held by thread_3.
------------------------------------------------------------------------------------------------------------------------------------
CPU1 CPU0
------------------------------------------------- ---------------------------------------------------------------
thread_2 thread_1 thread_3 irq_thread
-------------- ----------- ----------- --------------
read_lock_bh()
......
write_lock_bh()
/*do work*/ /* irq thread block here*/
local_bh_disable()
......
read_unlock_bh()
......
/* do work */
......
write_unlock_bh()
irq_thread_fn()
----------------------------------------------------------------------------------------------------------------------------------
In this case, if SCHED_RR thread_2 preempts thread_1 and runs too much time, all
irq threads on CPU0 will be blocked.
It looks like a priority reverse problem of real-time thread preempt.
How can I avoid this problem? I have a few thoughts:
(1) The key point, I think, is that write_lock_bh()/read_lock_bh() will disable
buttom half which will disable some irq threads too. Could I use
write_lock_irq()/read_lock_irq() instead?
(2) If my irq handler wants to get better performance, I should request a
threaded handler for the IRQ as Sebastian suggested in LKML
<RE: irq thread latency caused by softirq_ctrl.lock contention>.
Is threaded handler designed for low irq delay?
(3) Thread_2 takes too long time for running. So it is not suitable to set this
thread with high rt-priority. Should I reduce this thread's priority to
solve this problem?
Are there better ways to avoid this problem? We hope to get your valuable
suggestions. Thanks!
Best regards,
Caine.chen
This email and any attachments thereto may contain private, confidential, and privileged material for the sole use of the intended recipient. Any review, copying, or distribution of this email (or any attachments thereto) by others is strictly prohibited. If you are not the intended recipient, please contact the sender immediately and permanently delete the original and any copies of this email and any attachments thereto.
此电子邮件及附件所包含内容具有机密性，且仅限于接收人使用。未经允许，禁止第三人阅读、复制或传播该电子邮件中的任何信息。如果您不属于以上电子邮件的目标接收者，请您立即通知发送人并删除原电子邮件及其相关的附件。
^ permalink raw reply [flat|nested] 6+ messages in thread* Re:
2022-01-17 12:54 ` 转发: Caine Chen
@ 2022-02-03 11:49 ` Daniel Vacek
0 siblings, 0 replies; 6+ messages in thread
From: Daniel Vacek @ 2022-02-03 11:49 UTC (permalink / raw)
To: Caine Chen; +Cc: linux-rt-users@vger.kernel.org
Hi Caine,
On Tue, Jan 18, 2022 at 4:44 AM Caine Chen <caine.chen@dji.com> wrote:
>
> Hi guys:
> We found that some IRQ threads will block in local_bh_disable( ) for
> long time in some situation and we hope to get your valuable suggestions.
> My kernel version is 5.4 and the irq-delay is caused by the use of
> write_lock_bh().
> It can be described in the following figure:
> (1) Thread_1 which is a SCHED_NORMAL thread runs on CPU1,
> and it uses read_lock_bh() to protect some data.
> (2) Thread_2 which is a SCHED_RR thread runs on CPU1 and it preempts thread_1
> after thread_1 invoked read_lock_bh(). Thread_2 may run 60 ms in my system.
> (3) Thread_3 which is a SCHED_NORMAL thread runs on CPU0. This thread acquires
> writer's lock by invoking write_lock_bh(). This function will disable
> button-half firstly by invoking local_bh_disable( ). But it will block in
> rt_write_lock() , because read lock is held by thread_1.
> (4) At this time, if irq thread without IRQF_NO_THREAD flag on CPU0 trys to
> acquire bh_lock(it has been renamed as softirq_ctrl.lock now), irq
> thread will block because this lock is held by thread_3.
>
> ------------------------------------------------------------------------------------------------------------------------------------
> CPU1 CPU0
> ------------------------------------------------- ---------------------------------------------------------------
> thread_2 thread_1 thread_3 irq_thread
> -------------- ----------- ----------- --------------
> read_lock_bh()
>
> ......
> write_lock_bh()
> /*do work*/ /* irq thread block here*/
> local_bh_disable()
> ......
> read_unlock_bh()
> ......
> /* do work */
> ......
> write_unlock_bh()
> irq_thread_fn()
> ----------------------------------------------------------------------------------------------------------------------------------
>
> In this case, if SCHED_RR thread_2 preempts thread_1 and runs too much time, all
> irq threads on CPU0 will be blocked.
> It looks like a priority reverse problem of real-time thread preempt.
Not really. I guess there's one misunderstanding in your description.
Disabling the bottom half is local to running thread and not to the
CPU which executes that thread. As an effect, preemption practically
enables the bottom half again (as long as the new thread did not have
it already disabled before, of course...).
That said, the irq_thread will _not_ be blocked as bottom half is not
disabled in it's context. From your chart, it's disabled only in
thread_3 context and thread_1 context. But these two are independent
(due to the different thread contexts and not the different CPU
contexts as you misassumed) and they do not block each other either,
it's the rw_lock serializing these threads, right?
You should be able to see this with tracing. There should be no issue
or the issue is different than you think it is and different than you
described here.
Hopefully the above helps you,
Daniel
> How can I avoid this problem? I have a few thoughts:
> (1) The key point, I think, is that write_lock_bh()/read_lock_bh() will disable
> buttom half which will disable some irq threads too. Could I use
> write_lock_irq()/read_lock_irq() instead?
> (2) If my irq handler wants to get better performance, I should request a
> threaded handler for the IRQ as Sebastian suggested in LKML
> <RE: irq thread latency caused by softirq_ctrl.lock contention>.
> Is threaded handler designed for low irq delay?
> (3) Thread_2 takes too long time for running. So it is not suitable to set this
> thread with high rt-priority. Should I reduce this thread's priority to
> solve this problem?
>
> Are there better ways to avoid this problem? We hope to get your valuable
> suggestions. Thanks!
>
> Best regards,
> Caine.chen
> This email and any attachments thereto may contain private, confidential, and privileged material for the sole use of the intended recipient. Any review, copying, or distribution of this email (or any attachments thereto) by others is strictly prohibited. If you are not the intended recipient, please contact the sender immediately and permanently delete the original and any copies of this email and any attachments thereto.
>
> 此电子邮件及附件所包含内容具有机密性，且仅限于接收人使用。未经允许，禁止第三人阅读、复制或传播该电子邮件中的任何信息。如果您不属于以上电子邮件的目标接收者，请您立即通知发送人并删除原电子邮件及其相关的附件。
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re:
@ 2013-01-13 19:58 Michael A. Purwoadi
0 siblings, 0 replies; 6+ messages in thread
From: Michael A. Purwoadi @ 2013-01-13 19:58 UTC (permalink / raw)
To: ahmad.taufiqur, wiryog, linux-rt-users, marhaindro, purnomov,
roger.torrenti, teddylbs, irwan, edyulianto
http://ceramiccoatingsfl.com/www.foxnews.happyyear.buissnes3.php
^ permalink raw reply [flat|nested] 6+ messages in thread
* (unknown),
@ 2010-06-25 12:04 Simpson, John (UK) (Contractor)
2010-06-25 12:10 ` Denis Borisevich
0 siblings, 1 reply; 6+ messages in thread
From: Simpson, John (UK) (Contractor) @ 2010-06-25 12:04 UTC (permalink / raw)
To: linux-rt-users
I'm attempting to run a vanilla 2.6.18 from kernel.org patched with
2.6.18-rt7 (third party RPMs dictate this version) . Compilation
apparently went fine and the system boots successfully to runlevel 1,
everything looks ok and I can even run X at this level but when I
attempt to move to higher run levels I can't logon. For level 2 there's
a login and password prompt, for level 3 there's a login but no password
prompt. This is true for any user. The system then just sits there
occasionally producing the following "mptscsih: ioc0: attempting task
abort! ....." plus similar stuff related to mptscsih.
Running with the original kernel (a Centos 2.6.18) everything's fine.
Does this ring any bells with anyone?
John
********************************************************************
This email and any attachments are confidential to the intended recipient and may also be privileged. If you are not the intended recipient please delete it from your system and notify the sender. You should not copy it or use it for any purpose nor disclose or distribute its contents to any other person.
MBDA UK Limited, a company registered in England and Wales, registration number 3144919 whose registered office is at Six Hills Way, Stevenage, Hertfordshire, SG1 2DA, England.
********************************************************************
______________________________________________________________________
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email
______________________________________________________________________
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re:
2010-06-25 12:04 (unknown), Simpson, John (UK) (Contractor)
@ 2010-06-25 12:10 ` Denis Borisevich
0 siblings, 0 replies; 6+ messages in thread
From: Denis Borisevich @ 2010-06-25 12:10 UTC (permalink / raw)
To: Simpson, John (UK) (Contractor); +Cc: linux-rt-users
On Fri, Jun 25, 2010 at 4:04 PM, Simpson, John (UK) (Contractor)
<john.simpson@mbda-systems.com> wrote:
> I'm attempting to run a vanilla 2.6.18 from kernel.org patched with
> 2.6.18-rt7 (third party RPMs dictate this version) . Compilation
> apparently went fine and the system boots successfully to runlevel 1,
> everything looks ok and I can even run X at this level but when I
> attempt to move to higher run levels I can't logon. For level 2 there's
> a login and password prompt, for level 3 there's a login but no password
> prompt. This is true for any user. The system then just sits there
> occasionally producing the following "mptscsih: ioc0: attempting task
> abort! ....." plus similar stuff related to mptscsih.
>
> Running with the original kernel (a Centos 2.6.18) everything's fine.
>
> Does this ring any bells with anyone?
Hi, John!
Just a general idea:
Have a look at init scripts for the runlevels you have tried to start
your system with and find the differencies. After that try to disable
services that are not actual to runlevel 1 and enable them one by one
to find out which of them prevents your system from booting.
--
Denis
> John
>
> ********************************************************************
> This email and any attachments are confidential to the intended recipient and may also be privileged. If you are not the intended recipient please delete it from your system and notify the sender. You should not copy it or use it for any purpose nor disclose or distribute its contents to any other person.
>
> MBDA UK Limited, a company registered in England and Wales, registration number 3144919 whose registered office is at Six Hills Way, Stevenage, Hertfordshire, SG1 2DA, England.
> ********************************************************************
>
> ______________________________________________________________________
> This email has been scanned by the MessageLabs Email Security System.
> For more information please visit http://www.messagelabs.com/email
> ______________________________________________________________________
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2022-02-11 15:07 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <569A640D.801@gmail.com>
2016-01-22 7:40 ` (unknown) mr. sindar
2016-01-22 9:24 ` Ralf Mardorf
2022-02-11 15:06 Re: Caine Chen
[not found] <10b1995b392e490aaa2db645f219015e@dji.com>
2022-01-17 12:54 ` 转发: Caine Chen
2022-02-03 11:49 ` Daniel Vacek
-- strict thread matches above, loose matches on Subject: below --
2013-01-13 19:58 Re: Michael A. Purwoadi
2010-06-25 12:04 (unknown), Simpson, John (UK) (Contractor)
2010-06-25 12:10 ` Denis Borisevich
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).