resctrl mount fail on v6.13-rc1

resctrl mount fail on v6.13-rc1

[Luck, Tony]

Anyone better a decoding lockdep dumps then me make sense of this?

All I did was build v6.13-rc1 with (among others)

CONFIG_PROVE_LOCKING=y
CONFIG_PROVE_RAW_LOCK_NESTING=y
CONFIG_PROVE_RCU=y

and then mount the resctrl filesystem:

$ sudo mount -t resctrl resctrl /sys/fs/resctrl

There are only trivial changes to the resctrl code between
v6.12 (which works) and v6.13-rc1:

$ git log --oneline v6.13-rc1 ^v6.12 -- arch/x86/kernel/cpu/resctrl
5a4b3fbb4849 Merge tag 'x86_cache_for_v6.13' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
9bce6e94c4b3 x86/resctrl: Support Sub-NUMA cluster mode SNC6
29eaa7958367 x86/resctrl: Slightly clean-up mbm_config_show()

So something in kernfs? Or the way resctrl uses kernfs?

-Tony


[  124.435545]
[  124.437062] ======================================================
[  124.443245] WARNING: possible circular locking dependency detected
[  124.449425] 6.13.0-rc1 #208 Not tainted
[  124.453264] ------------------------------------------------------
[  124.459443] mount/4268 is trying to acquire lock:
[  124.464149] ff20974596218178 (&sbsec->lock){+.+.}-{4:4}, at: selinux_set_mnt_opts+0x6d/0x7e0
[  124.472598]
[  124.472598] but task is already holding lock:
[  124.478427] ff2097460518f0e0 (&type->s_umount_key#75/1){+.+.}-{4:4}, at: alloc_super+0xd9/0x3d0
[  124.487128]
[  124.487128] which lock already depends on the new lock.
[  124.487128]
[  124.495300]
[  124.495300] the existing dependency chain (in reverse order) is:
[  124.502771]
[  124.502771] -> #5 (&type->s_umount_key#75/1){+.+.}-{4:4}:
[  124.509651]        lock_release+0x135/0x2c0
[  124.513846]        __mutex_unlock_slowpath+0x3a/0x2c0
[  124.518908]        rdt_get_tree+0x1b5/0x590
[  124.523103]        vfs_get_tree+0x25/0x100
[  124.527208]        path_mount+0x4c6/0xbd0
[  124.531231]        __x64_sys_mount+0x113/0x150
[  124.535675]        do_syscall_64+0x93/0x180
[  124.539871]        entry_SYSCALL_64_after_hwframe+0x76/0x7e
[  124.545449]
[  124.545449] -> #4 (cpu_hotplug_lock){++++}-{0:0}:
[  124.551631]        __cpuhp_state_add_instance+0x37/0x1e0
[  124.556952]        blk_mq_alloc_and_init_hctx+0x450/0x4d0
[  124.562358]        blk_mq_realloc_hw_ctxs+0x1df/0x230
[  124.567411]        blk_mq_init_allocated_queue+0x138/0x480
[  124.572906]        blk_mq_alloc_queue+0x7a/0xd0
[  124.577437]        scsi_alloc_sdev+0x283/0x3c0
[  124.581892]        scsi_probe_and_add_lun+0x1c0/0x450
[  124.586952]        __scsi_add_device+0x10b/0x130
[  124.591569]        ata_scsi_scan_host+0x98/0x1c0
[  124.596190]        async_run_entry_fn+0x2d/0x130
[  124.600818]        process_one_work+0x212/0x700
[  124.605350]        worker_thread+0x1ca/0x380
[  124.609622]        kthread+0xdd/0x110
[  124.613298]        ret_from_fork+0x2d/0x50
[  124.617405]        ret_from_fork_asm+0x1a/0x30
[  124.621859]
[  124.621859] -> #3 (&q->sysfs_lock){+.+.}-{4:4}:
[  124.627864]        __mutex_lock+0xd1/0xcc0
[  124.631962]        queue_attr_store+0x5d/0xa0
[  124.636330]        kernfs_fop_write_iter+0x158/0x210
[  124.641306]        vfs_write+0x2bb/0x550
[  124.645240]        ksys_write+0x70/0xf0
[  124.649078]        do_syscall_64+0x93/0x180
[  124.653265]        entry_SYSCALL_64_after_hwframe+0x76/0x7e
[  124.658834]
[  124.658834] -> #2 (&q->q_usage_counter(io)){++++}-{0:0}:
[  124.665622]        blk_mq_submit_bio+0x929/0xb20
[  124.670240]        __submit_bio+0x10d/0x1f0
[  124.674427]        submit_bio_noacct_nocheck+0x300/0x400
[  124.679739]        btrfs_submit_chunk+0x1a9/0x660
[  124.684453]        btrfs_submit_bbio+0x16/0x30
[  124.688899]        read_extent_buffer_pages+0x17e/0x1f0
[  124.694132]        btrfs_read_extent_buffer+0x8f/0x180
[  124.699278]        read_block_for_search+0x224/0x390
[  124.704246]        btrfs_search_slot+0x2f0/0xd10
[  124.708865]        btrfs_init_root_free_objectid+0x8e/0x130
[  124.714446]        btrfs_get_root_ref+0x20d/0x3a0
[  124.719149]        btrfs_lookup_dentry+0x58d/0x610
[  124.723942]        btrfs_lookup+0xe/0x30
[  124.727870]        __lookup_slow+0xfc/0x1b0
[  124.732063]        walk_component+0xdb/0x150
[  124.736336]        path_lookupat+0x6a/0x1a0
[  124.740519]        filename_lookup+0xee/0x1f0
[  124.744880]        vfs_path_lookup+0x4e/0x80
[  124.749150]        mount_subtree+0x87/0x130
[  124.753337]        btrfs_get_tree+0x32b/0x790
[  124.757696]        vfs_get_tree+0x25/0x100
[  124.761795]        path_mount+0x4c6/0xbd0
[  124.765807]        __x64_sys_mount+0x113/0x150
[  124.770252]        do_syscall_64+0x93/0x180
[  124.774438]        entry_SYSCALL_64_after_hwframe+0x76/0x7e
[  124.780009]
[  124.780009] -> #1 (btrfs-tree-00){++++}-{4:4}:
[  124.785931]        down_read_nested+0x30/0x170
[  124.790383]        btrfs_tree_read_lock_nested+0x21/0xd0
[  124.795696]        btrfs_read_lock_root_node+0x40/0xd0
[  124.800833]        btrfs_search_slot+0x144/0xd10
[  124.805454]        btrfs_lookup_xattr+0x8b/0xf0
[  124.809985]        btrfs_getxattr+0x55/0x110
[  124.814257]        __vfs_getxattr+0x7b/0xb0
[  124.818446]        sb_finish_set_opts+0x1ad/0x340
[  124.823151]        selinux_set_mnt_opts+0x408/0x7e0
[  124.828030]        iterate_supers+0x77/0xe0
[  124.832214]        selinux_policy_commit+0x2d1/0x2f0
[  124.837179]        sel_write_load+0x506/0xb90
[  124.841539]        vfs_write+0xf5/0x550
[  124.845379]        ksys_write+0x70/0xf0
[  124.849218]        do_syscall_64+0x93/0x180
[  124.853403]        entry_SYSCALL_64_after_hwframe+0x76/0x7e
[  124.858975]
[  124.858975] -> #0 (&sbsec->lock){+.+.}-{4:4}:
[  124.864806]        __lock_acquire+0x1425/0x21d0
[  124.869339]        lock_acquire+0xd5/0x300
[  124.873439]        __mutex_lock+0xd1/0xcc0
[  124.877538]        selinux_set_mnt_opts+0x6d/0x7e0
[  124.882330]        security_sb_set_mnt_opts+0x50/0x80
[  124.887389]        vfs_get_tree+0x7d/0x100
[  124.891489]        path_mount+0x4c6/0xbd0
[  124.895502]        __x64_sys_mount+0x113/0x150
[  124.899947]        do_syscall_64+0x93/0x180
[  124.904134]        entry_SYSCALL_64_after_hwframe+0x76/0x7e
[  124.909703]
[  124.909703] other info that might help us debug this:
[  124.909703]
[  124.917695] Chain exists of:
[  124.917695]   &sbsec->lock --> cpu_hotplug_lock --> &type->s_umount_key#75/1
[  124.917695]
[  124.929081]  Possible unsafe locking scenario:
[  124.929081]
[  124.934994]        CPU0                    CPU1
[  124.939523]        ----                    ----
[  124.944049]   lock(&type->s_umount_key#75/1);
[  124.948406]                                lock(cpu_hotplug_lock);
[  124.954587]                                lock(&type->s_umount_key#75/1);
[  124.961458]   lock(&sbsec->lock);
[  124.964778]
[  124.964778]  *** DEADLOCK ***
[  124.964778]
[  124.970695] 1 lock held by mount/4268:
[  124.974446]  #0: ff2097460518f0e0 (&type->s_umount_key#75/1){+.+.}-{4:4}, at: alloc_super+0xd9/0x3d0
[  124.983583]
[  124.983583] stack backtrace:
[  124.987943] CPU: 85 UID: 0 PID: 4268 Comm: mount Not tainted 6.13.0-rc1 #208
[  124.994995] Hardware name: Intel Corporation WilsonCity/WilsonCity, BIOS WLYDCRB1.SYS.0021.P06.2104260458 04/26/2021
[  125.005508] Call Trace:
[  125.007962]  <TASK>
[  125.010067]  dump_stack_lvl+0x73/0xb0
[  125.013742]  print_circular_bug+0x26e/0x340
[  125.017927]  check_noncircular+0x148/0x160
[  125.022025]  __lock_acquire+0x1425/0x21d0
[  125.026036]  lock_acquire+0xd5/0x300
[  125.029616]  ? selinux_set_mnt_opts+0x6d/0x7e0
[  125.034064]  __mutex_lock+0xd1/0xcc0
[  125.037639]  ? selinux_set_mnt_opts+0x6d/0x7e0
[  125.042085]  ? reacquire_held_locks+0xd1/0x1f0
[  125.046531]  ? selinux_set_mnt_opts+0x6d/0x7e0
[  125.050975]  ? reacquire_held_locks+0xd1/0x1f0
[  125.055425]  ? selinux_set_mnt_opts+0x6d/0x7e0
[  125.059877]  selinux_set_mnt_opts+0x6d/0x7e0
[  125.064152]  security_sb_set_mnt_opts+0x50/0x80
[  125.068691]  vfs_get_tree+0x7d/0x100
[  125.072270]  ? capable+0x32/0x60
[  125.075511]  path_mount+0x4c6/0xbd0
[  125.079005]  __x64_sys_mount+0x113/0x150
[  125.082937]  do_syscall_64+0x93/0x180
[  125.086603]  ? do_syscall_64+0x9f/0x180
[  125.090440]  ? lockdep_hardirqs_on+0x7b/0x100
[  125.094799]  ? do_syscall_64+0x9f/0x180
[  125.098638]  ? do_syscall_64+0x9f/0x180
[  125.102478]  ? lockdep_hardirqs_on+0x7b/0x100
[  125.106838]  ? do_syscall_64+0x9f/0x180
[  125.110676]  ? do_user_addr_fault+0x359/0x790
[  125.115043]  ? exc_page_fault+0x126/0x280
[  125.119057]  ? clear_bhb_loop+0x45/0xa0
[  125.122893]  ? clear_bhb_loop+0x45/0xa0
[  125.126736]  entry_SYSCALL_64_after_hwframe+0x76/0x7e
[  125.131788] RIP: 0033:0x7fead6d8cc9e
[  125.135374] Code: 48 8b 0d 6d 11 0c 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 49 89 ca b8 a5 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 3a 11 0c 00 f7 d8 64 89 01 48
[  125.154119] RSP: 002b:00007ffcd6f80798 EFLAGS: 00000246 ORIG_RAX: 00000000000000a5
[  125.161684] RAX: ffffffffffffffda RBX: 000056371f46f550 RCX: 00007fead6d8cc9e
[  125.168815] RDX: 000056371f46f780 RSI: 000056371f46f7c0 RDI: 000056371f46f7a0
[  125.175949] RBP: 00007ffcd6f808c0 R08: 0000000000000000 R09: 0000000000000001
[  125.183080] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
[  125.190210] R13: 000056371f46f7a0 R14: 000056371f46f780 R15: 00007fead6ebd076
[  125.197339]  </TASK>

Re: resctrl mount fail on v6.13-rc1

[Reinette Chatre]

Hi Tony,

On 12/2/24 1:42 PM, Luck, Tony wrote:
> Anyone better a decoding lockdep dumps then me make sense of this?
> 
> All I did was build v6.13-rc1 with (among others)
> 
> CONFIG_PROVE_LOCKING=y
> CONFIG_PROVE_RAW_LOCK_NESTING=y
> CONFIG_PROVE_RCU=y
> 
> and then mount the resctrl filesystem:
> 
> $ sudo mount -t resctrl resctrl /sys/fs/resctrl
> 
> There are only trivial changes to the resctrl code between
> v6.12 (which works) and v6.13-rc1:
> 
> $ git log --oneline v6.13-rc1 ^v6.12 -- arch/x86/kernel/cpu/resctrl
> 5a4b3fbb4849 Merge tag 'x86_cache_for_v6.13' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
> 9bce6e94c4b3 x86/resctrl: Support Sub-NUMA cluster mode SNC6
> 29eaa7958367 x86/resctrl: Slightly clean-up mbm_config_show()
> 
> So something in kernfs? Or the way resctrl uses kernfs?

I am not seeing this but that may be because I am not testing with
selinux enabled. My test kernel has:
# CONFIG_SECURITY_SELINUX is not set

I am also not running with any btrfs filesystems. 

Is this your usual setup in which you are seeing this the first time? Is it
perhaps possible for you to bisect?

The subject states "resctrl mount fail" - could you please confirm if
resctrl cannot be mounted in addition to the lockdep warning?

Reinette

Re: resctrl mount fail on v6.13-rc1

[Luck, Tony]

On Mon, Dec 02, 2024 at 02:26:48PM -0800, Reinette Chatre wrote:
> Hi Tony,
> 
> On 12/2/24 1:42 PM, Luck, Tony wrote:
> > Anyone better a decoding lockdep dumps then me make sense of this?
> > 
> > All I did was build v6.13-rc1 with (among others)
> > 
> > CONFIG_PROVE_LOCKING=y
> > CONFIG_PROVE_RAW_LOCK_NESTING=y
> > CONFIG_PROVE_RCU=y
> > 
> > and then mount the resctrl filesystem:
> > 
> > $ sudo mount -t resctrl resctrl /sys/fs/resctrl
> > 
> > There are only trivial changes to the resctrl code between
> > v6.12 (which works) and v6.13-rc1:
> > 
> > $ git log --oneline v6.13-rc1 ^v6.12 -- arch/x86/kernel/cpu/resctrl
> > 5a4b3fbb4849 Merge tag 'x86_cache_for_v6.13' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
> > 9bce6e94c4b3 x86/resctrl: Support Sub-NUMA cluster mode SNC6
> > 29eaa7958367 x86/resctrl: Slightly clean-up mbm_config_show()
> > 
> > So something in kernfs? Or the way resctrl uses kernfs?
> 
> I am not seeing this but that may be because I am not testing with
> selinux enabled. My test kernel has:
> # CONFIG_SECURITY_SELINUX is not set

I did have SELINUX configured. Disabling this CONFIG option
makes the lockdep splat go away.
> 
> I am also not running with any btrfs filesystems. 

My root and home filesystems are btrfs, so difficult to
check if this is also connected.
> 
> Is this your usual setup in which you are seeing this the first time? Is it
> perhaps possible for you to bisect?
> 
> The subject states "resctrl mount fail" - could you please confirm if
> resctrl cannot be mounted in addition to the lockdep warning?

The filesystem did get mounted despite all the lockdep noise.

> 
> Reinette

-Tony

Re: resctrl mount fail on v6.13-rc1

[Fenghua Yu]

Hi, Tony and Reinette,

On 12/2/24 14:26, Reinette Chatre wrote:
> Hi Tony,
> 
> On 12/2/24 1:42 PM, Luck, Tony wrote:
>> Anyone better a decoding lockdep dumps then me make sense of this?
>>
>> All I did was build v6.13-rc1 with (among others)
>>
>> CONFIG_PROVE_LOCKING=y
>> CONFIG_PROVE_RAW_LOCK_NESTING=y
>> CONFIG_PROVE_RCU=y
>>
>> and then mount the resctrl filesystem:
>>
>> $ sudo mount -t resctrl resctrl /sys/fs/resctrl
>>
>> There are only trivial changes to the resctrl code between
>> v6.12 (which works) and v6.13-rc1:
>>
>> $ git log --oneline v6.13-rc1 ^v6.12 -- arch/x86/kernel/cpu/resctrl
>> 5a4b3fbb4849 Merge tag 'x86_cache_for_v6.13' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
>> 9bce6e94c4b3 x86/resctrl: Support Sub-NUMA cluster mode SNC6
>> 29eaa7958367 x86/resctrl: Slightly clean-up mbm_config_show()
>>
>> So something in kernfs? Or the way resctrl uses kernfs?
> 
> I am not seeing this but that may be because I am not testing with
> selinux enabled. My test kernel has:
> # CONFIG_SECURITY_SELINUX is not set
> 
> I am also not running with any btrfs filesystems.
> 
> Is this your usual setup in which you are seeing this the first time? Is it
> perhaps possible for you to bisect?
> 
> The subject states "resctrl mount fail" - could you please confirm if
> resctrl cannot be mounted in addition to the lockdep warning?
> 
> Reinette
> 

I can reproduce the issue with
CONFIG_PROVE_LOCKING=y
CONFIG_PROVE_RAW_LOCK_NESTING=y
CONFIG_PROVE_RCU=y
and SELINUX enabled.

Thanks.

-Fenghua

Re: resctrl mount fail on v6.13-rc1

[Fenghua Yu]

Hi, Tony and Reinette,

On 12/2/24 14:46, Luck, Tony wrote:
> On Mon, Dec 02, 2024 at 02:26:48PM -0800, Reinette Chatre wrote:
>> Hi Tony,
>>
>> On 12/2/24 1:42 PM, Luck, Tony wrote:
>>> Anyone better a decoding lockdep dumps then me make sense of this?
>>>
>>> All I did was build v6.13-rc1 with (among others)
>>>
>>> CONFIG_PROVE_LOCKING=y
>>> CONFIG_PROVE_RAW_LOCK_NESTING=y
>>> CONFIG_PROVE_RCU=y
>>>
>>> and then mount the resctrl filesystem:
>>>
>>> $ sudo mount -t resctrl resctrl /sys/fs/resctrl
>>>
>>> There are only trivial changes to the resctrl code between
>>> v6.12 (which works) and v6.13-rc1:
>>>
>>> $ git log --oneline v6.13-rc1 ^v6.12 -- arch/x86/kernel/cpu/resctrl
>>> 5a4b3fbb4849 Merge tag 'x86_cache_for_v6.13' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
>>> 9bce6e94c4b3 x86/resctrl: Support Sub-NUMA cluster mode SNC6
>>> 29eaa7958367 x86/resctrl: Slightly clean-up mbm_config_show()
>>>
>>> So something in kernfs? Or the way resctrl uses kernfs?
>>
>> I am not seeing this but that may be because I am not testing with
>> selinux enabled. My test kernel has:
>> # CONFIG_SECURITY_SELINUX is not set
> 
> I did have SELINUX configured. Disabling this CONFIG option
> makes the lockdep splat go away.
>>
>> I am also not running with any btrfs filesystems.
> 
> My root and home filesystems are btrfs, so difficult to
> check if this is also connected.
>>
>> Is this your usual setup in which you are seeing this the first time? Is it
>> perhaps possible for you to bisect?
>>
>> The subject states "resctrl mount fail" - could you please confirm if
>> resctrl cannot be mounted in addition to the lockdep warning?
> 
> The filesystem did get mounted despite all the lockdep noise.
> 
>>
>> Reinette
> 
> -Tony

I did see a similar warning on ksoftirqd before mount resctrl during 
boot time:

[   65.611812] WARNING: possible circular locking dependency detected
[   65.611813] 6.13.0-rc1-dsa+ #62 Not tainted
[   65.611814] ------------------------------------------------------
[   65.611815] ksoftirqd/4/44 is trying to acquire lock:
[   65.611817] ffffffffa787fbb8 ((console_sem).lock){....}-{2:2}, at: 
down_trylock+0x13/0x40
[   65.611829]
                but task is already holding lock:
[   65.611830] ff4ed95eefb32898 (&rq->__lock){-.-.}-{2:2}, at: 
raw_spin_rq_lock_nested+0x27/0x40
[   65.643035]
                which lock already depends on the new lock.

But I did not see the locking warning when mount resctrl. I don't use brtfs.

Thanks.

-Fenghua

Re: resctrl mount fail on v6.13-rc1

[Luck, Tony]

On Mon, Dec 02, 2024 at 02:26:48PM -0800, Reinette Chatre wrote:
> Hi Tony,
> 
> On 12/2/24 1:42 PM, Luck, Tony wrote:
> > Anyone better a decoding lockdep dumps then me make sense of this?
> > 
> > All I did was build v6.13-rc1 with (among others)
> > 
> > CONFIG_PROVE_LOCKING=y
> > CONFIG_PROVE_RAW_LOCK_NESTING=y
> > CONFIG_PROVE_RCU=y
> > 
> > and then mount the resctrl filesystem:
> > 
> > $ sudo mount -t resctrl resctrl /sys/fs/resctrl
> > 
> > There are only trivial changes to the resctrl code between
> > v6.12 (which works) and v6.13-rc1:
> > 
> > $ git log --oneline v6.13-rc1 ^v6.12 -- arch/x86/kernel/cpu/resctrl
> > 5a4b3fbb4849 Merge tag 'x86_cache_for_v6.13' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
> > 9bce6e94c4b3 x86/resctrl: Support Sub-NUMA cluster mode SNC6
> > 29eaa7958367 x86/resctrl: Slightly clean-up mbm_config_show()
> > 
> > So something in kernfs? Or the way resctrl uses kernfs?
> 
> I am not seeing this but that may be because I am not testing with
> selinux enabled. My test kernel has:
> # CONFIG_SECURITY_SELINUX is not set
> 
> I am also not running with any btrfs filesystems. 
> 
> Is this your usual setup in which you are seeing this the first time? Is it
> perhaps possible for you to bisect?

Bisection says:

$ git bisect bad
f1be1788a32e8fa63416ad4518bbd1a85a825c9d is the first bad commit
commit f1be1788a32e8fa63416ad4518bbd1a85a825c9d
Author: Ming Lei <ming.lei@redhat.com>
Date:   Fri Oct 25 08:37:20 2024 +0800

    block: model freeze & enter queue as lock for supporting lockdep

> 
> The subject states "resctrl mount fail" - could you please confirm if
> resctrl cannot be mounted in addition to the lockdep warning?
> 
> Reinette
> 

Re: resctrl mount fail on v6.13-rc1

[Ming Lei]

On Mon, Dec 02, 2024 at 06:47:38PM -0800, Luck, Tony wrote:
> On Mon, Dec 02, 2024 at 02:26:48PM -0800, Reinette Chatre wrote:
> > Hi Tony,
> > 
> > On 12/2/24 1:42 PM, Luck, Tony wrote:
> > > Anyone better a decoding lockdep dumps then me make sense of this?
> > > 
> > > All I did was build v6.13-rc1 with (among others)
> > > 
> > > CONFIG_PROVE_LOCKING=y
> > > CONFIG_PROVE_RAW_LOCK_NESTING=y
> > > CONFIG_PROVE_RCU=y
> > > 
> > > and then mount the resctrl filesystem:
> > > 
> > > $ sudo mount -t resctrl resctrl /sys/fs/resctrl

[linux]# make -C tools/testing/selftests TARGETS=resctrl run_tests
make: Entering directory '/root/git/linux/tools/testing/selftests'
make[1]: Nothing to be done for 'all'.
TAP version 13
1..1
# timeout set to 120
# selftests: resctrl: resctrl_tests
# TAP version 13
# # Fail: Check kernel supports resctrl filesystem
# 1..0 # SKIP resctrl FS does not exist. Enable X86_CPU_RESCTRL config option.
ok 1 selftests: resctrl: resctrl_tests # SKIP
make: Leaving directory '/root/git/linux/tools/testing/selftests'

[linux]# grep X86_CPU_RESCTRL .config
CONFIG_X86_CPU_RESCTRL=y

Can you share how to make /sys/fs/resctrl so that I can check if the
recent changes in block tree can avoid this warning? 

> > > 
> > > There are only trivial changes to the resctrl code between
> > > v6.12 (which works) and v6.13-rc1:
> > > 
> > > $ git log --oneline v6.13-rc1 ^v6.12 -- arch/x86/kernel/cpu/resctrl
> > > 5a4b3fbb4849 Merge tag 'x86_cache_for_v6.13' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
> > > 9bce6e94c4b3 x86/resctrl: Support Sub-NUMA cluster mode SNC6
> > > 29eaa7958367 x86/resctrl: Slightly clean-up mbm_config_show()
> > > 
> > > So something in kernfs? Or the way resctrl uses kernfs?
> > 
> > I am not seeing this but that may be because I am not testing with
> > selinux enabled. My test kernel has:
> > # CONFIG_SECURITY_SELINUX is not set
> > 
> > I am also not running with any btrfs filesystems. 
> > 
> > Is this your usual setup in which you are seeing this the first time? Is it
> > perhaps possible for you to bisect?
> 
> Bisection says:
> 
> $ git bisect bad
> f1be1788a32e8fa63416ad4518bbd1a85a825c9d is the first bad commit
> commit f1be1788a32e8fa63416ad4518bbd1a85a825c9d
> Author: Ming Lei <ming.lei@redhat.com>
> Date:   Fri Oct 25 08:37:20 2024 +0800
> 
>     block: model freeze & enter queue as lock for supporting lockdep
> 
> > 
> > The subject states "resctrl mount fail" - could you please confirm if
> > resctrl cannot be mounted in addition to the lockdep warning?

It seems one lockdep false positive, but it shouldn't cause 'resctrl
mount fail', I will take a look at the lock chains and see if some of
them can be cut.


Thanks,
Ming

Re: resctrl mount fail on v6.13-rc1

[Reinette Chatre]

+x86 folks

On 12/2/24 7:35 PM, Ming Lei wrote:
> On Mon, Dec 02, 2024 at 06:47:38PM -0800, Luck, Tony wrote:
>> On Mon, Dec 02, 2024 at 02:26:48PM -0800, Reinette Chatre wrote:
>>> Hi Tony,
>>>
>>> On 12/2/24 1:42 PM, Luck, Tony wrote:
>>>> Anyone better a decoding lockdep dumps then me make sense of this?
>>>>
>>>> All I did was build v6.13-rc1 with (among others)
>>>>
>>>> CONFIG_PROVE_LOCKING=y
>>>> CONFIG_PROVE_RAW_LOCK_NESTING=y
>>>> CONFIG_PROVE_RCU=y
>>>>
>>>> and then mount the resctrl filesystem:
>>>>
>>>> $ sudo mount -t resctrl resctrl /sys/fs/resctrl
> 
> [linux]# make -C tools/testing/selftests TARGETS=resctrl run_tests
> make: Entering directory '/root/git/linux/tools/testing/selftests'
> make[1]: Nothing to be done for 'all'.
> TAP version 13
> 1..1
> # timeout set to 120
> # selftests: resctrl: resctrl_tests
> # TAP version 13
> # # Fail: Check kernel supports resctrl filesystem
> # 1..0 # SKIP resctrl FS does not exist. Enable X86_CPU_RESCTRL config option.
> ok 1 selftests: resctrl: resctrl_tests # SKIP
> make: Leaving directory '/root/git/linux/tools/testing/selftests'
> 
> [linux]# grep X86_CPU_RESCTRL .config
> CONFIG_X86_CPU_RESCTRL=y
> 
> Can you share how to make /sys/fs/resctrl so that I can check if the
> recent changes in block tree can avoid this warning? 

Thank you very much for taking a look.

You may not be testing on hardware supported by resctrl. A /proc/cpuinfo
flag bit that indicates hardware supported by resctrl is "rdt_a". When booting a kernel
compiled with CONFIG_X86_CPU_RESCTRL=y on such hardware dmesg will contain
information like:
	resctrl: L3 allocation detected
	resctrl: MB allocation detected
	resctrl: L3 monitoring detected                           

> 
>>>>
>>>> There are only trivial changes to the resctrl code between
>>>> v6.12 (which works) and v6.13-rc1:
>>>>
>>>> $ git log --oneline v6.13-rc1 ^v6.12 -- arch/x86/kernel/cpu/resctrl
>>>> 5a4b3fbb4849 Merge tag 'x86_cache_for_v6.13' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
>>>> 9bce6e94c4b3 x86/resctrl: Support Sub-NUMA cluster mode SNC6
>>>> 29eaa7958367 x86/resctrl: Slightly clean-up mbm_config_show()
>>>>
>>>> So something in kernfs? Or the way resctrl uses kernfs?
>>>
>>> I am not seeing this but that may be because I am not testing with
>>> selinux enabled. My test kernel has:
>>> # CONFIG_SECURITY_SELINUX is not set
>>>
>>> I am also not running with any btrfs filesystems. 
>>>
>>> Is this your usual setup in which you are seeing this the first time? Is it
>>> perhaps possible for you to bisect?
>>
>> Bisection says:
>>
>> $ git bisect bad
>> f1be1788a32e8fa63416ad4518bbd1a85a825c9d is the first bad commit
>> commit f1be1788a32e8fa63416ad4518bbd1a85a825c9d
>> Author: Ming Lei <ming.lei@redhat.com>
>> Date:   Fri Oct 25 08:37:20 2024 +0800
>>
>>     block: model freeze & enter queue as lock for supporting lockdep
>>
>>>
>>> The subject states "resctrl mount fail" - could you please confirm if
>>> resctrl cannot be mounted in addition to the lockdep warning?
> 
> It seems one lockdep false positive, but it shouldn't cause 'resctrl
> mount fail', I will take a look at the lock chains and see if some of
> them can be cut.

Tony confirmed [1] in follow-up message that resctrl mount succeeded despite
the lockdep notice.

Reinette


[1] https://lore.kernel.org/all/Z0441XN_KoCP-fNz@agluck-desk3/

Re: resctrl mount fail on v6.13-rc1

[Reinette Chatre]

On 12/2/24 6:47 PM, Luck, Tony wrote:
> On Mon, Dec 02, 2024 at 02:26:48PM -0800, Reinette Chatre wrote:
>> Hi Tony,
>>
>> On 12/2/24 1:42 PM, Luck, Tony wrote:
>>> Anyone better a decoding lockdep dumps then me make sense of this?
>>>
>>> All I did was build v6.13-rc1 with (among others)
>>>
>>> CONFIG_PROVE_LOCKING=y
>>> CONFIG_PROVE_RAW_LOCK_NESTING=y
>>> CONFIG_PROVE_RCU=y
>>>
>>> and then mount the resctrl filesystem:
>>>
>>> $ sudo mount -t resctrl resctrl /sys/fs/resctrl
>>>
>>> There are only trivial changes to the resctrl code between
>>> v6.12 (which works) and v6.13-rc1:
>>>
>>> $ git log --oneline v6.13-rc1 ^v6.12 -- arch/x86/kernel/cpu/resctrl
>>> 5a4b3fbb4849 Merge tag 'x86_cache_for_v6.13' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
>>> 9bce6e94c4b3 x86/resctrl: Support Sub-NUMA cluster mode SNC6
>>> 29eaa7958367 x86/resctrl: Slightly clean-up mbm_config_show()
>>>
>>> So something in kernfs? Or the way resctrl uses kernfs?
>>
>> I am not seeing this but that may be because I am not testing with
>> selinux enabled. My test kernel has:
>> # CONFIG_SECURITY_SELINUX is not set
>>
>> I am also not running with any btrfs filesystems. 
>>
>> Is this your usual setup in which you are seeing this the first time? Is it
>> perhaps possible for you to bisect?
> 
> Bisection says:
> 
> $ git bisect bad
> f1be1788a32e8fa63416ad4518bbd1a85a825c9d is the first bad commit
> commit f1be1788a32e8fa63416ad4518bbd1a85a825c9d
> Author: Ming Lei <ming.lei@redhat.com>
> Date:   Fri Oct 25 08:37:20 2024 +0800
> 
>     block: model freeze & enter queue as lock for supporting lockdep
> 

Thank you very much Tony. Since you did not respond to the question about
bisect I assumed that you would not do it. I ended up duplicating the bisect
effort after getting an environment in which I can reproduce the issue. Doing so
I am able to confirm the commit pointed to by bisect. 
The commit cannot be reverted cleanly so I could not test v6.13-rc1 with it
reverted.

Ming Lei: I'd be happy to help with testing if you do not have hardware with
which you can reproduce the issue.

Reinette

Re: resctrl mount fail on v6.13-rc1

[Reinette Chatre]

On 12/2/24 8:54 PM, Reinette Chatre wrote:
> 
> 
> On 12/2/24 6:47 PM, Luck, Tony wrote:
>> On Mon, Dec 02, 2024 at 02:26:48PM -0800, Reinette Chatre wrote:
>>> Hi Tony,
>>>
>>> On 12/2/24 1:42 PM, Luck, Tony wrote:
>>>> Anyone better a decoding lockdep dumps then me make sense of this?
>>>>
>>>> All I did was build v6.13-rc1 with (among others)
>>>>
>>>> CONFIG_PROVE_LOCKING=y
>>>> CONFIG_PROVE_RAW_LOCK_NESTING=y
>>>> CONFIG_PROVE_RCU=y
>>>>
>>>> and then mount the resctrl filesystem:
>>>>
>>>> $ sudo mount -t resctrl resctrl /sys/fs/resctrl
>>>>
>>>> There are only trivial changes to the resctrl code between
>>>> v6.12 (which works) and v6.13-rc1:
>>>>
>>>> $ git log --oneline v6.13-rc1 ^v6.12 -- arch/x86/kernel/cpu/resctrl
>>>> 5a4b3fbb4849 Merge tag 'x86_cache_for_v6.13' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
>>>> 9bce6e94c4b3 x86/resctrl: Support Sub-NUMA cluster mode SNC6
>>>> 29eaa7958367 x86/resctrl: Slightly clean-up mbm_config_show()
>>>>
>>>> So something in kernfs? Or the way resctrl uses kernfs?
>>>
>>> I am not seeing this but that may be because I am not testing with
>>> selinux enabled. My test kernel has:
>>> # CONFIG_SECURITY_SELINUX is not set
>>>
>>> I am also not running with any btrfs filesystems. 
>>>
>>> Is this your usual setup in which you are seeing this the first time? Is it
>>> perhaps possible for you to bisect?
>>
>> Bisection says:
>>
>> $ git bisect bad
>> f1be1788a32e8fa63416ad4518bbd1a85a825c9d is the first bad commit
>> commit f1be1788a32e8fa63416ad4518bbd1a85a825c9d
>> Author: Ming Lei <ming.lei@redhat.com>
>> Date:   Fri Oct 25 08:37:20 2024 +0800
>>
>>     block: model freeze & enter queue as lock for supporting lockdep
>>
> 
> Thank you very much Tony. Since you did not respond to the question about
> bisect I assumed that you would not do it. I ended up duplicating the bisect
> effort after getting an environment in which I can reproduce the issue. Doing so
> I am able to confirm the commit pointed to by bisect. 
> The commit cannot be reverted cleanly so I could not test v6.13-rc1 with it
> reverted.
> 
> Ming Lei: I'd be happy to help with testing if you do not have hardware with
> which you can reproduce the issue.

One datapoint that I neglected to mention: btrfs does not seem to be required. The system
I tested on used ext4 filesystem resulting in trace below:

[   67.598375] ======================================================
[   67.604562] WARNING: possible circular locking dependency detected
[   67.610741] 6.12.0-rc4-00037-gf1be1788a32e #1 Not tainted
[   67.616146] ------------------------------------------------------
[   67.622326] mount/2738 is trying to acquire lock:
[   67.627031] ff2f341adb984578 (&sbsec->lock){+.+.}-{4:4}, at: selinux_set_mnt_opts+0x71/0x730
[   67.635481] 
               but task is already holding lock:
[   67.641312] ff2f341ad746a0e0 (&type->s_umount_key#65/1){+.+.}-{4:4}, at: alloc_super+0xd9/0x3d0
[   67.650013] 
               which lock already depends on the new lock.

[   67.658183] 
               the existing dependency chain (in reverse order) is:
[   67.665654] 
               -> #5 (&type->s_umount_key#65/1){+.+.}-{4:4}:
[   67.672525]        reacquire_held_locks+0xce/0x1f0
[   67.677317]        lock_release+0x11e/0x2b0
[   67.681503]        __mutex_unlock_slowpath+0x3b/0x290
[   67.686558]        rdt_get_tree+0x1b9/0x5c0
[   67.690750]        vfs_get_tree+0x29/0xf0
[   67.694762]        vfs_cmd_create+0x59/0xe0
[   67.698948]        __do_sys_fsconfig+0x4f3/0x6c0
[   67.703567]        do_syscall_64+0xc5/0x210
[   67.707763]        entry_SYSCALL_64_after_hwframe+0x77/0x7f
[   67.713341] 
               -> #4 (cpu_hotplug_lock){++++}-{0:0}:
[   67.719523]        lock_acquire.part.0+0x69/0x1b0
[   67.724226]        __cpuhp_state_add_instance+0x3b/0x1c0
[   67.729546]        blk_mq_alloc_and_init_hctx+0x44d/0x4c0
[   67.734945]        blk_mq_realloc_hw_ctxs+0x15e/0x190
[   67.739998]        blk_mq_init_allocated_queue+0x138/0x480
[   67.745485]        blk_mq_alloc_queue+0x7b/0xe0
[   67.750015]        scsi_alloc_sdev+0x281/0x3c0
[   67.754463]        scsi_probe_and_add_lun+0x1f5/0x450
[   67.759514]        __scsi_add_device+0x10f/0x130
[   67.764133]        ata_scsi_scan_host+0x9c/0x1b0
[   67.768753]        async_run_entry_fn+0x31/0x130
[   67.773378]        process_one_work+0x204/0x590
[   67.777912]        worker_thread+0x180/0x340
[   67.782182]        kthread+0xd0/0x100
[   67.785848]        ret_from_fork+0x31/0x50
[   67.789947]        ret_from_fork_asm+0x1a/0x30
[   67.794394] 
               -> #3 (&q->sysfs_lock){+.+.}-{4:4}:
[   67.800399]        lock_acquire.part.0+0x69/0x1b0
[   67.805103]        __mutex_lock+0xaf/0x740
[   67.809205]        queue_attr_store+0x61/0xb0
[   67.813573]        kernfs_fop_write_iter+0x13a/0x200
[   67.818546]        vfs_write+0x29c/0x540
[   67.822470]        ksys_write+0x73/0xf0
[   67.826311]        do_syscall_64+0xc5/0x210
[   67.830496]        entry_SYSCALL_64_after_hwframe+0x77/0x7f
[   67.836069] 
               -> #2 (&q->q_usage_counter(io)){++++}-{0:0}:
[   67.842852]        lock_acquire.part.0+0x69/0x1b0
[   67.847557]        blk_mq_submit_bio+0x81b/0xab0
[   67.852179]        __submit_bio+0x97/0x180
[   67.856277]        submit_bio_noacct_nocheck+0x324/0x430
[   67.861589]        ext4_read_bh+0x51/0x90
[   67.865601]        ext4_sb_bread+0x75/0x90
[   67.869699]        ext4_xattr_get+0xf5/0x250
[   67.873973]        __vfs_getxattr+0x7f/0xb0
[   67.878159]        inode_doinit_use_xattr+0x63/0x170
[   67.883124]        inode_doinit_with_dentry+0x36b/0x530
[   67.888348]        security_d_instantiate+0x39/0x50
[   67.893228]        d_splice_alias+0x52/0x4e0
[   67.897499]        ext4_lookup+0x1e1/0x270
[   67.901599]        lookup_one_qstr_excl+0x6f/0xa0
[   67.906306]        filename_create+0xcb/0x1b0
[   67.910665]        do_mkdirat+0x5c/0x140
[   67.914589]        __x64_sys_mkdir+0x46/0x70
[   67.918863]        do_syscall_64+0xc5/0x210
[   67.923056]        entry_SYSCALL_64_after_hwframe+0x77/0x7f
[   67.928629] 
               -> #1 (&ei->xattr_sem){++++}-{4:4}:
[   67.934633]        lock_acquire.part.0+0x69/0x1b0
[   67.939340]        down_read+0x31/0x150
[   67.943178]        ext4_xattr_get+0x71/0x250
[   67.947448]        __vfs_getxattr+0x7f/0xb0
[   67.951634]        sb_finish_set_opts+0x1ab/0x290
[   67.956341]        selinux_set_mnt_opts+0x4d9/0x730
[   67.961220]        iterate_supers+0x7b/0xf0
[   67.965405]        selinux_policy_commit+0x24d/0x2b0
[   67.970373]        sel_write_load+0x4ee/0xbc0
[   67.974739]        vfs_write+0xe5/0x540
[   67.978577]        ksys_write+0x73/0xf0
[   67.982415]        do_syscall_64+0xc5/0x210
[   67.986604]        entry_SYSCALL_64_after_hwframe+0x77/0x7f
[   67.992175] 
               -> #0 (&sbsec->lock){+.+.}-{4:4}:
[   67.998006]        check_prev_add+0xeb/0xd80
[   68.002280]        __lock_acquire+0x11b6/0x1640
[   68.006811]        lock_acquire.part.0+0x69/0x1b0
[   68.011515]        __mutex_lock+0xaf/0x740
[   68.015615]        selinux_set_mnt_opts+0x71/0x730
[   68.020407]        security_sb_set_mnt_opts+0x54/0x90
[   68.025460]        vfs_get_tree+0x81/0xf0
[   68.029472]        vfs_cmd_create+0x59/0xe0
[   68.033656]        __do_sys_fsconfig+0x4f3/0x6c0
[   68.038276]        do_syscall_64+0xc5/0x210
[   68.042461]        entry_SYSCALL_64_after_hwframe+0x77/0x7f
[   68.048036] 
               other info that might help us debug this:

[   68.056033] Chain exists of:
                 &sbsec->lock --> cpu_hotplug_lock --> &type->s_umount_key#65/1

[   68.067419]  Possible unsafe locking scenario:

[   68.073331]        CPU0                    CPU1
[   68.077863]        ----                    ----
[   68.082393]   lock(&type->s_umount_key#65/1);
[   68.086752]                                lock(cpu_hotplug_lock);
[   68.092931]                                lock(&type->s_umount_key#65/1);
[   68.099806]   lock(&sbsec->lock);
[   68.103122] 
                *** DEADLOCK ***

[   68.109033] 2 locks held by mount/2738:
[   68.112873]  #0: ff2f341adbb5b070 (&fc->uapi_mutex){+.+.}-{4:4}, at: __do_sys_fsconfig+0x4bf/0x6c0
[   68.121831]  #1: ff2f341ad746a0e0 (&type->s_umount_key#65/1){+.+.}-{4:4}, at: alloc_super+0xd9/0x3d0
[   68.130966] 
               stack backtrace:
[   68.135327] CPU: 109 UID: 0 PID: 2738 Comm: mount Not tainted 6.12.0-rc4-00037-gf1be1788a32e #1
[   68.144017] Hardware name: Intel Corporation WilsonCity/WilsonCity, BIOS WLYDCRB1.SYS.0027.P82.2204080829 04/08/2022
[   68.154528] Call Trace:
[   68.156973]  <TASK>
[   68.159078]  dump_stack_lvl+0x5d/0x80
[   68.162744]  print_circular_bug.cold+0x178/0x1be
[   68.167364]  check_noncircular+0x14e/0x170
[   68.171466]  check_prev_add+0xeb/0xd80
[   68.175223]  __lock_acquire+0x11b6/0x1640
[   68.179238]  lock_acquire.part.0+0x69/0x1b0
[   68.183423]  ? selinux_set_mnt_opts+0x71/0x730
[   68.187866]  ? selinux_set_mnt_opts+0x71/0x730
[   68.192313]  __mutex_lock+0xaf/0x740
[   68.195892]  ? selinux_set_mnt_opts+0x71/0x730
[   68.200338]  ? selinux_set_mnt_opts+0x71/0x730
[   68.204782]  ? selinux_set_mnt_opts+0x71/0x730
[   68.209228]  ? lock_is_held_type+0x85/0xf0
[   68.213328]  selinux_set_mnt_opts+0x71/0x730
[   68.217599]  security_sb_set_mnt_opts+0x54/0x90
[   68.222134]  vfs_get_tree+0x81/0xf0
[   68.225623]  ? capable+0x3a/0x60
[   68.228855]  vfs_cmd_create+0x59/0xe0
[   68.232524]  __do_sys_fsconfig+0x4f3/0x6c0
[   68.236623]  do_syscall_64+0xc5/0x210
[   68.240296]  entry_SYSCALL_64_after_hwframe+0x77/0x7f
[   68.245348] RIP: 0033:0x7f1fcd171d5e
[   68.248928] Code: 73 01 c3 48 8b 0d b2 40 0f 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 49 89 ca b8 af 01 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 82 40 0f 00 f7 d8 64 89 01 48
[   68.267672] RSP: 002b:00007ffde8bb54b8 EFLAGS: 00000246 ORIG_RAX: 00000000000001af
[   68.275245] RAX: ffffffffffffffda RBX: 0000557854ef36c0 RCX: 00007f1fcd171d5e
[   68.282377] RDX: 0000000000000000 RSI: 0000000000000006 RDI: 0000000000000003
[   68.289507] RBP: 00007ffde8bb5600 R08: 0000000000000000 R09: 0000000000000001
[   68.296641] R10: 0000000000000000 R11: 0000000000000246 R12: 00007f1fcd2ecb00
[   68.303770] R13: 0000000000000000 R14: 0000557854ef47a0 R15: 00007f1fcd2e1561
[   68.310899]  </TASK>

Re: resctrl mount fail on v6.13-rc1

[Ming Lei]

On Mon, Dec 02, 2024 at 09:02:45PM -0800, Reinette Chatre wrote:
> 
> 
> On 12/2/24 8:54 PM, Reinette Chatre wrote:
> > 
> > 
> > On 12/2/24 6:47 PM, Luck, Tony wrote:
> >> On Mon, Dec 02, 2024 at 02:26:48PM -0800, Reinette Chatre wrote:
> >>> Hi Tony,
> >>>
> >>> On 12/2/24 1:42 PM, Luck, Tony wrote:
> >>>> Anyone better a decoding lockdep dumps then me make sense of this?
> >>>>
> >>>> All I did was build v6.13-rc1 with (among others)
> >>>>
> >>>> CONFIG_PROVE_LOCKING=y
> >>>> CONFIG_PROVE_RAW_LOCK_NESTING=y
> >>>> CONFIG_PROVE_RCU=y
> >>>>
> >>>> and then mount the resctrl filesystem:
> >>>>
> >>>> $ sudo mount -t resctrl resctrl /sys/fs/resctrl
> >>>>
> >>>> There are only trivial changes to the resctrl code between
> >>>> v6.12 (which works) and v6.13-rc1:
> >>>>
> >>>> $ git log --oneline v6.13-rc1 ^v6.12 -- arch/x86/kernel/cpu/resctrl
> >>>> 5a4b3fbb4849 Merge tag 'x86_cache_for_v6.13' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
> >>>> 9bce6e94c4b3 x86/resctrl: Support Sub-NUMA cluster mode SNC6
> >>>> 29eaa7958367 x86/resctrl: Slightly clean-up mbm_config_show()
> >>>>
> >>>> So something in kernfs? Or the way resctrl uses kernfs?
> >>>
> >>> I am not seeing this but that may be because I am not testing with
> >>> selinux enabled. My test kernel has:
> >>> # CONFIG_SECURITY_SELINUX is not set
> >>>
> >>> I am also not running with any btrfs filesystems. 
> >>>
> >>> Is this your usual setup in which you are seeing this the first time? Is it
> >>> perhaps possible for you to bisect?
> >>
> >> Bisection says:
> >>
> >> $ git bisect bad
> >> f1be1788a32e8fa63416ad4518bbd1a85a825c9d is the first bad commit
> >> commit f1be1788a32e8fa63416ad4518bbd1a85a825c9d
> >> Author: Ming Lei <ming.lei@redhat.com>
> >> Date:   Fri Oct 25 08:37:20 2024 +0800
> >>
> >>     block: model freeze & enter queue as lock for supporting lockdep
> >>
> > 
> > Thank you very much Tony. Since you did not respond to the question about
> > bisect I assumed that you would not do it. I ended up duplicating the bisect
> > effort after getting an environment in which I can reproduce the issue. Doing so
> > I am able to confirm the commit pointed to by bisect. 
> > The commit cannot be reverted cleanly so I could not test v6.13-rc1 with it
> > reverted.
> > 
Gi> > Ming Lei: I'd be happy to help with testing if you do not have hardware with
> > which you can reproduce the issue.
> 
> One datapoint that I neglected to mention: btrfs does not seem to be required. The system
> I tested on used ext4 filesystem resulting in trace below:

Hi Reinette and Tony,

The warning is triggered because the two subsystems are connected with
&cpu_hotplug_lock.

rdt_get_tree():
	cpus_read_lock();
    mutex_lock(&rdtgroup_mutex);
	...

blk_mq_realloc_hw_ctxs()
	mutex_lock(&q->sysfs_lock);
	...
	blk_mq_alloc_and_init_hctx()
		blk_mq_init_hctx
			cpuhp_state_add_instance_nocalls
				__cpuhp_state_add_instance
					cpus_read_lock();

Given cpus_read_lock() is often implied in cpuhp APIs, I feel rdt_get_tree()
may re-order the two locks for avoiding the dependency.


Thanks,
Ming

Re: resctrl mount fail on v6.13-rc1

[Reinette Chatre]

Hi Ming,

On 12/3/24 7:27 PM, Ming Lei wrote:
> On Mon, Dec 02, 2024 at 09:02:45PM -0800, Reinette Chatre wrote:
>>
>>
>> On 12/2/24 8:54 PM, Reinette Chatre wrote:
>>>
>>>
>>> On 12/2/24 6:47 PM, Luck, Tony wrote:
>>>> On Mon, Dec 02, 2024 at 02:26:48PM -0800, Reinette Chatre wrote:
>>>>> Hi Tony,
>>>>>
>>>>> On 12/2/24 1:42 PM, Luck, Tony wrote:
>>>>>> Anyone better a decoding lockdep dumps then me make sense of this?
>>>>>>
>>>>>> All I did was build v6.13-rc1 with (among others)
>>>>>>
>>>>>> CONFIG_PROVE_LOCKING=y
>>>>>> CONFIG_PROVE_RAW_LOCK_NESTING=y
>>>>>> CONFIG_PROVE_RCU=y
>>>>>>
>>>>>> and then mount the resctrl filesystem:
>>>>>>
>>>>>> $ sudo mount -t resctrl resctrl /sys/fs/resctrl
>>>>>>
>>>>>> There are only trivial changes to the resctrl code between
>>>>>> v6.12 (which works) and v6.13-rc1:
>>>>>>
>>>>>> $ git log --oneline v6.13-rc1 ^v6.12 -- arch/x86/kernel/cpu/resctrl
>>>>>> 5a4b3fbb4849 Merge tag 'x86_cache_for_v6.13' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
>>>>>> 9bce6e94c4b3 x86/resctrl: Support Sub-NUMA cluster mode SNC6
>>>>>> 29eaa7958367 x86/resctrl: Slightly clean-up mbm_config_show()
>>>>>>
>>>>>> So something in kernfs? Or the way resctrl uses kernfs?
>>>>>
>>>>> I am not seeing this but that may be because I am not testing with
>>>>> selinux enabled. My test kernel has:
>>>>> # CONFIG_SECURITY_SELINUX is not set
>>>>>
>>>>> I am also not running with any btrfs filesystems. 
>>>>>
>>>>> Is this your usual setup in which you are seeing this the first time? Is it
>>>>> perhaps possible for you to bisect?
>>>>
>>>> Bisection says:
>>>>
>>>> $ git bisect bad
>>>> f1be1788a32e8fa63416ad4518bbd1a85a825c9d is the first bad commit
>>>> commit f1be1788a32e8fa63416ad4518bbd1a85a825c9d
>>>> Author: Ming Lei <ming.lei@redhat.com>
>>>> Date:   Fri Oct 25 08:37:20 2024 +0800
>>>>
>>>>     block: model freeze & enter queue as lock for supporting lockdep
>>>>
>>>
>>> Thank you very much Tony. Since you did not respond to the question about
>>> bisect I assumed that you would not do it. I ended up duplicating the bisect
>>> effort after getting an environment in which I can reproduce the issue. Doing so
>>> I am able to confirm the commit pointed to by bisect. 
>>> The commit cannot be reverted cleanly so I could not test v6.13-rc1 with it
>>> reverted.
>>>
> Gi> > Ming Lei: I'd be happy to help with testing if you do not have hardware with
>>> which you can reproduce the issue.
>>
>> One datapoint that I neglected to mention: btrfs does not seem to be required. The system
>> I tested on used ext4 filesystem resulting in trace below:
> 
> Hi Reinette and Tony,
> 
> The warning is triggered because the two subsystems are connected with
> &cpu_hotplug_lock.
> 
> rdt_get_tree():
> 	cpus_read_lock();
>     mutex_lock(&rdtgroup_mutex);
> 	...
> 
> blk_mq_realloc_hw_ctxs()
> 	mutex_lock(&q->sysfs_lock);
> 	...
> 	blk_mq_alloc_and_init_hctx()
> 		blk_mq_init_hctx
> 			cpuhp_state_add_instance_nocalls
> 				__cpuhp_state_add_instance
> 					cpus_read_lock();
> 
> Given cpus_read_lock() is often implied in cpuhp APIs, I feel rdt_get_tree()
> may re-order the two locks for avoiding the dependency.

This is not possible for exactly the reason you provide ("cpus_read_lock() is
often implied in cpuhp APIs").

resctrl relies on hotplug state callbacks for its initialization. You can find
the callback setup in:

arch/x86/kernel/cpu/resctrl/core.c:

static int __init resctrl_late_init(void)
{

	...
	state = cpuhp_setup_state(CPUHP_AP_ONLINE_DYN,
				  "x86/resctrl/cat:online:",
				  resctrl_arch_online_cpu,
				  resctrl_arch_offline_cpu);
	...
}

Since resctrl code is called by the CPU hotplug subsystem with cpu_hotplug_lock
already held it is not possible for resctrl to change the lock ordering.

Reinette

Re: resctrl mount fail on v6.13-rc1

[Ming Lei]

On Wed, Dec 04, 2024 at 08:48:14AM -0800, Reinette Chatre wrote:
> Hi Ming,
> 
> On 12/3/24 7:27 PM, Ming Lei wrote:
> > On Mon, Dec 02, 2024 at 09:02:45PM -0800, Reinette Chatre wrote:
> >>
> >>
> >> On 12/2/24 8:54 PM, Reinette Chatre wrote:
> >>>
> >>>
> >>> On 12/2/24 6:47 PM, Luck, Tony wrote:
> >>>> On Mon, Dec 02, 2024 at 02:26:48PM -0800, Reinette Chatre wrote:
> >>>>> Hi Tony,
> >>>>>
> >>>>> On 12/2/24 1:42 PM, Luck, Tony wrote:
> >>>>>> Anyone better a decoding lockdep dumps then me make sense of this?
> >>>>>>
> >>>>>> All I did was build v6.13-rc1 with (among others)
> >>>>>>
> >>>>>> CONFIG_PROVE_LOCKING=y
> >>>>>> CONFIG_PROVE_RAW_LOCK_NESTING=y
> >>>>>> CONFIG_PROVE_RCU=y
> >>>>>>
> >>>>>> and then mount the resctrl filesystem:
> >>>>>>
> >>>>>> $ sudo mount -t resctrl resctrl /sys/fs/resctrl
> >>>>>>
> >>>>>> There are only trivial changes to the resctrl code between
> >>>>>> v6.12 (which works) and v6.13-rc1:
> >>>>>>
> >>>>>> $ git log --oneline v6.13-rc1 ^v6.12 -- arch/x86/kernel/cpu/resctrl
> >>>>>> 5a4b3fbb4849 Merge tag 'x86_cache_for_v6.13' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
> >>>>>> 9bce6e94c4b3 x86/resctrl: Support Sub-NUMA cluster mode SNC6
> >>>>>> 29eaa7958367 x86/resctrl: Slightly clean-up mbm_config_show()
> >>>>>>
> >>>>>> So something in kernfs? Or the way resctrl uses kernfs?
> >>>>>
> >>>>> I am not seeing this but that may be because I am not testing with
> >>>>> selinux enabled. My test kernel has:
> >>>>> # CONFIG_SECURITY_SELINUX is not set
> >>>>>
> >>>>> I am also not running with any btrfs filesystems. 
> >>>>>
> >>>>> Is this your usual setup in which you are seeing this the first time? Is it
> >>>>> perhaps possible for you to bisect?
> >>>>
> >>>> Bisection says:
> >>>>
> >>>> $ git bisect bad
> >>>> f1be1788a32e8fa63416ad4518bbd1a85a825c9d is the first bad commit
> >>>> commit f1be1788a32e8fa63416ad4518bbd1a85a825c9d
> >>>> Author: Ming Lei <ming.lei@redhat.com>
> >>>> Date:   Fri Oct 25 08:37:20 2024 +0800
> >>>>
> >>>>     block: model freeze & enter queue as lock for supporting lockdep
> >>>>
> >>>
> >>> Thank you very much Tony. Since you did not respond to the question about
> >>> bisect I assumed that you would not do it. I ended up duplicating the bisect
> >>> effort after getting an environment in which I can reproduce the issue. Doing so
> >>> I am able to confirm the commit pointed to by bisect. 
> >>> The commit cannot be reverted cleanly so I could not test v6.13-rc1 with it
> >>> reverted.
> >>>
> > Gi> > Ming Lei: I'd be happy to help with testing if you do not have hardware with
> >>> which you can reproduce the issue.
> >>
> >> One datapoint that I neglected to mention: btrfs does not seem to be required. The system
> >> I tested on used ext4 filesystem resulting in trace below:
> > 
> > Hi Reinette and Tony,
> > 
> > The warning is triggered because the two subsystems are connected with
> > &cpu_hotplug_lock.
> > 
> > rdt_get_tree():
> > 	cpus_read_lock();
> >     mutex_lock(&rdtgroup_mutex);
> > 	...
> > 
> > blk_mq_realloc_hw_ctxs()
> > 	mutex_lock(&q->sysfs_lock);
> > 	...
> > 	blk_mq_alloc_and_init_hctx()
> > 		blk_mq_init_hctx
> > 			cpuhp_state_add_instance_nocalls
> > 				__cpuhp_state_add_instance
> > 					cpus_read_lock();
> > 
> > Given cpus_read_lock() is often implied in cpuhp APIs, I feel rdt_get_tree()
> > may re-order the two locks for avoiding the dependency.
> 
> This is not possible for exactly the reason you provide ("cpus_read_lock() is
> often implied in cpuhp APIs").
> 
> resctrl relies on hotplug state callbacks for its initialization. You can find
> the callback setup in:
> 
> arch/x86/kernel/cpu/resctrl/core.c:
> 
> static int __init resctrl_late_init(void)
> {
> 
> 	...
> 	state = cpuhp_setup_state(CPUHP_AP_ONLINE_DYN,
> 				  "x86/resctrl/cat:online:",
> 				  resctrl_arch_online_cpu,
> 				  resctrl_arch_offline_cpu);
> 	...
> }
> 
> Since resctrl code is called by the CPU hotplug subsystem with cpu_hotplug_lock
> already held it is not possible for resctrl to change the lock ordering.

OK, I see now, and thanks for the explanation.

I will try to figure out moving cpuhp_state_add_instance_nocalls out of
q->sysfs_lock, and it should be fine in case that queue is live.

Thanks,
Ming