Re: rcu stalls during fstests runs for xfs

Re: rcu stalls during fstests runs for xfs

[Matthieu Baerts]

Hi Shinichiro,

Sorry to jump in, but I *think* our CI for the MPTCP subsystem is
hitting the same issue.

On 30/01/2026 12:16, Shinichiro Kawasaki wrote:
> On Jan 29, 2026 / 15:19, Paul E. McKenney wrote:
> [...]
>>>>> I have seen the static-key pattern called out by Dave Chinner when running
>>>>> KASAN on large systems.  We worked around this by disabling KASAN's use
>>>>> of static keys.  In case you were running KASAN in these tests.
>>>>
>>>> As to KASAN, yes, I enable it in my test runs. I find three static-keys under
>>>> mm/kasan/*. I will think if they can be disabled in my test runs. Thanks.
>>>
>>> There is a set of Kconfig options that disables static branches.  If you
>>> cannot find them quickly, please let me know and I can look them up.
> 
> Thank you. But now I know the fix series by Thomas is available. I prioritize
> the evaluation of the fix series. Later on, I will try disabling the static-keys
> if it is required.
> 
>>
>> And Thomas Gleixner posted an alleged fix to the CID issue here:
>>
>> https://lore.kernel.org/lkml/20260129210219.452851594@kernel.org/
>>
>> Please let him know whether or not it helps.
> 
> Good to see this fix candidate series, thanks :) I have set up the patches and
> started my regular test runs. So far, the hangs have been observed once or twice
> a week. To confirm the effect of the fix series, I think two weeks runs will be
> required. Once I get the result, will share it on this thread and with Thomas.

I know it is only one week now, but did you see any effects so far? On
my side, I applied the v2 series -- which has been applied in
tip/sched/urgent -- but I still have issues, and it looks like it is
even more frequent. Maybe what I see is different. If you no longer see
the issues on your side after one week, I'm going to start a new thread
with my issues not to mix them.

Note that in my case, the issue is visible on a system where nested VMs
are used, with and without KASAN (enabled via debug.config), just after
having started a VSOCK listening socket via socat.

Cheers,
Matt
-- 
Sponsored by the NGI0 Core fund.

Re: rcu stalls during fstests runs for xfs

[Shinichiro Kawasaki]

On Feb 06, 2026 / 10:33, Matthieu Baerts wrote:
> Hi Shinichiro,
> 
> Sorry to jump in, but I *think* our CI for the MPTCP subsystem is
> hitting the same issue.

Hi Matthieu,

> On 30/01/2026 12:16, Shinichiro Kawasaki wrote:
> > On Jan 29, 2026 / 15:19, Paul E. McKenney wrote:
> > [...]
> >>>>> I have seen the static-key pattern called out by Dave Chinner when running
> >>>>> KASAN on large systems.  We worked around this by disabling KASAN's use
> >>>>> of static keys.  In case you were running KASAN in these tests.
> >>>>
> >>>> As to KASAN, yes, I enable it in my test runs. I find three static-keys under
> >>>> mm/kasan/*. I will think if they can be disabled in my test runs. Thanks.
> >>>
> >>> There is a set of Kconfig options that disables static branches.  If you
> >>> cannot find them quickly, please let me know and I can look them up.
> > 
> > Thank you. But now I know the fix series by Thomas is available. I prioritize
> > the evaluation of the fix series. Later on, I will try disabling the static-keys
> > if it is required.
> > 
> >>
> >> And Thomas Gleixner posted an alleged fix to the CID issue here:
> >>
> >> https://lore.kernel.org/lkml/20260129210219.452851594@kernel.org/
> >>
> >> Please let him know whether or not it helps.
> > 
> > Good to see this fix candidate series, thanks :) I have set up the patches and
> > started my regular test runs. So far, the hangs have been observed once or twice
> > a week. To confirm the effect of the fix series, I think two weeks runs will be
> > required. Once I get the result, will share it on this thread and with Thomas.
> 
> I know it is only one week now, but did you see any effects so far?

No, I do not see any hang so far. And I hope there will be no hang in the
next week either. Fingers crossed...

> On
> my side, I applied the v2 series -- which has been applied i
> tip/sched/urgent -- but I still have issues, and it looks like it is
> even more frequent. Maybe what I see is different. If you no longer see
> the issues on your side after one week, I'm going to start a new thread
> with my issues not to mix them.
> 
> Note that in my case, the issue is visible on a system where nested VMs
> are used, with and without KASAN (enabled via debug.config), just after
> having started a VSOCK listening socket via socat.

I applied the v1 series on top of my test target xfs kernel branches enabling
KASAN.

Re: rcu stalls during fstests runs for xfs

[Matthieu Baerts]

On 06/02/2026 11:02, Shinichiro Kawasaki wrote:
> On Feb 06, 2026 / 10:33, Matthieu Baerts wrote:
>> Hi Shinichiro,
>>
>> Sorry to jump in, but I *think* our CI for the MPTCP subsystem is
>> hitting the same issue.
> 
> Hi Matthieu,
> 
>> On 30/01/2026 12:16, Shinichiro Kawasaki wrote:
>>> On Jan 29, 2026 / 15:19, Paul E. McKenney wrote:
>>> [...]
>>>>>>> I have seen the static-key pattern called out by Dave Chinner when running
>>>>>>> KASAN on large systems.  We worked around this by disabling KASAN's use
>>>>>>> of static keys.  In case you were running KASAN in these tests.
>>>>>>
>>>>>> As to KASAN, yes, I enable it in my test runs. I find three static-keys under
>>>>>> mm/kasan/*. I will think if they can be disabled in my test runs. Thanks.
>>>>>
>>>>> There is a set of Kconfig options that disables static branches.  If you
>>>>> cannot find them quickly, please let me know and I can look them up.
>>>
>>> Thank you. But now I know the fix series by Thomas is available. I prioritize
>>> the evaluation of the fix series. Later on, I will try disabling the static-keys
>>> if it is required.
>>>
>>>>
>>>> And Thomas Gleixner posted an alleged fix to the CID issue here:
>>>>
>>>> https://lore.kernel.org/lkml/20260129210219.452851594@kernel.org/
>>>>
>>>> Please let him know whether or not it helps.
>>>
>>> Good to see this fix candidate series, thanks :) I have set up the patches and
>>> started my regular test runs. So far, the hangs have been observed once or twice
>>> a week. To confirm the effect of the fix series, I think two weeks runs will be
>>> required. Once I get the result, will share it on this thread and with Thomas.
>>
>> I know it is only one week now, but did you see any effects so far?
> 
> No, I do not see any hang so far. And I hope there will be no hang in the
> next week either. Fingers crossed...

Thank you for your reply!

>> On
>> my side, I applied the v2 series -- which has been applied i
>> tip/sched/urgent -- but I still have issues, and it looks like it is
>> even more frequent. Maybe what I see is different. If you no longer see
>> the issues on your side after one week, I'm going to start a new thread
>> with my issues not to mix them.
>>
>> Note that in my case, the issue is visible on a system where nested VMs
>> are used, with and without KASAN (enabled via debug.config), just after
>> having started a VSOCK listening socket via socat.
> 
> I applied the v1 series on top of my test target xfs kernel branches enabling
> KASAN.

Sorry for the noise, I guess I have a different issue, even if the
traces look similar [1]. Hopefully someone can help me find the root
cause :)

[1]
https://github.com/multipath-tcp/mptcp_net-next/actions/runs/21723325004/job/62658752123#step:7:7288

Cheers,
Matt
-- 
Sponsored by the NGI0 Core fund.