Suspicious RCU usage in enable_drhd_fault_handling()

Suspicious RCU usage in enable_drhd_fault_handling()

[Ido Schimmel]

Hi,

I have recently enabled CONFIG_RCU_EXPERT and CONFIG_PROVE_RCU_LIST in
our debug configuration file and started observing the following splat
[1] on one of our machines during boot. Searched the archives, but did
not find a similar report.

Not sure what is the right fix as I am not familiar with this code, but
I can easily test patches.

Thanks

[1]
smp: Bringing up secondary CPUs ...
smpboot: x86: Booting SMP configuration:
.... node  #0, CPUs:      #1 #2 #3
numa_add_cpu cpu 1 node 0: mask now 0-1

=============================
WARNING: suspicious RCU usage
6.12.0-rc4-custom-ge7e8d9b44a86 #1 Not tainted
-----------------------------
drivers/iommu/intel/dmar.c:2059 RCU-list traversed in non-reader section!!

other info that might help us debug this:

rcu_scheduler_active = 1, debug_locks = 1
2 locks held by cpuhp/1/22:
 #0: ffffffff8454e340 (cpu_hotplug_lock){++++}-{0:0}, at: cpuhp_thread_fun+0x17e/0x7b0
 #1: ffffffff84550ae0 (cpuhp_state-up){+.+.}-{0:0}, at: cpuhp_thread_fun+0x17e/0x7b0

stack backtrace:
CPU: 1 UID: 0 PID: 22 Comm: cpuhp/1 Not tainted 6.12.0-rc4-custom-ge7e8d9b44a86 #1
Hardware name: Mellanox Technologies Ltd. MSN3800/VMOD0007, BIOS 5.11 01/06/2019
Call Trace:
 <TASK>
 dump_stack_lvl+0xf4/0x120
 lockdep_rcu_suspicious+0x215/0x3c0
 enable_drhd_fault_handling+0x316/0x3d0
 cpuhp_invoke_callback+0x3c3/0x9c0
 cpuhp_thread_fun+0x500/0x7b0
 smpboot_thread_fn+0x680/0xa80
 kthread+0x2c9/0x3b0
 ret_from_fork+0x4d/0x80
 ret_from_fork_asm+0x1a/0x30
 </TASK>
numa_add_cpu cpu 2 node 0: mask now 0-2
numa_add_cpu cpu 3 node 0: mask now 0-3
 #4 #5 #6 #7
numa_add_cpu cpu 4 node 0: mask now 0-4
[...]
numa_add_cpu cpu 5 node 0: mask now 0-5
numa_add_cpu cpu 6 node 0: mask now 0-6
numa_add_cpu cpu 7 node 0: mask now 0-7
smp: Brought up 1 node, 8 CPUs
smpboot: Total of 8 processors activated (35120.16 BogoMIPS)

Re: Suspicious RCU usage in enable_drhd_fault_handling()

[Baolu Lu]

On 2024/10/28 16:43, Ido Schimmel wrote:
> Hi,
> 
> I have recently enabled CONFIG_RCU_EXPERT and CONFIG_PROVE_RCU_LIST in
> our debug configuration file and started observing the following splat
> [1] on one of our machines during boot. Searched the archives, but did
> not find a similar report.
> 
> Not sure what is the right fix as I am not familiar with this code, but
> I can easily test patches.

Thanks for reporting this issue. I've been able to reproduce it locally
and will work on a fix. I'll let you know as soon as I have an update.

--
baolu

Re: Suspicious RCU usage in enable_drhd_fault_handling()

[Breno Leitao]

Hello Baolu,

On Tue, Oct 29, 2024 at 09:15:55AM +0800, Baolu Lu wrote:
> On 2024/10/28 16:43, Ido Schimmel wrote:
> > Hi,
> > 
> > I have recently enabled CONFIG_RCU_EXPERT and CONFIG_PROVE_RCU_LIST in
> > our debug configuration file and started observing the following splat
> > [1] on one of our machines during boot. Searched the archives, but did
> > not find a similar report.
> > 
> > Not sure what is the right fix as I am not familiar with this code, but
> > I can easily test patches.
> 
> Thanks for reporting this issue. I've been able to reproduce it locally
> and will work on a fix. I'll let you know as soon as I have an update.

Have you had a chance to look at this problem? I am still seeing it in
6.13-rc7.

Thanks
--breno

Re: Suspicious RCU usage in enable_drhd_fault_handling()

[Baolu Lu]

On 1/16/25 21:34, Breno Leitao wrote:
> On Tue, Oct 29, 2024 at 09:15:55AM +0800, Baolu Lu wrote:
>> On 2024/10/28 16:43, Ido Schimmel wrote:
>>> Hi,
>>>
>>> I have recently enabled CONFIG_RCU_EXPERT and CONFIG_PROVE_RCU_LIST in
>>> our debug configuration file and started observing the following splat
>>> [1] on one of our machines during boot. Searched the archives, but did
>>> not find a similar report.
>>>
>>> Not sure what is the right fix as I am not familiar with this code, but
>>> I can easily test patches.
>> Thanks for reporting this issue. I've been able to reproduce it locally
>> and will work on a fix. I'll let you know as soon as I have an update.
> Have you had a chance to look at this problem? I am still seeing it in
> 6.13-rc7.

Yes. I have spent some time on this, but I haven't yet figured out a
simple way to fix it.

I tried to add the rcu lock or use the dmar_global_lock instead, but
neither proved successful. Perhaps it's time to reconsider the locking
scheme for those lists.

Thanks,
baolu