From: Kyle Sanderson <kyle.leet@gmail.com>
To: Nilay Shroff <nilay@linux.ibm.com>,
linux-block@vger.kernel.org,
Linus Torvalds <torvalds@linux-foundation.org>,
Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: axboe@kernel.dk, hch@lst.de, ming.lei@redhat.com, hare@suse.de,
sth@linux.ibm.com, gjoyce@ibm.com, linux-fsdevel@vger.kernel.org,
linux-kernel@vger.kernel.org
Subject: [6.16.9 / 6.17.0 PANIC REGRESSION] block: fix lockdep warning caused by lock dependency in elv_iosched_store
Date: Tue, 30 Sep 2025 22:20:44 -0700 [thread overview]
Message-ID: <25a87311-70fd-4248-86e4-dd5fecf6cc99@gmail.com> (raw)
In-Reply-To: <20250730074614.2537382-3-nilay@linux.ibm.com>
On 7/30/2025 12:46 AM, Nilay Shroff wrote:
> To address this, move all sched_tags allocations and deallocations outside
> of both the ->elevator_lock and the ->freeze_lock. Since the lifetime of
> the elevator queue and its associated sched_tags is closely tied, the
> allocated sched_tags are now stored in the elevator queue structure. Then,
> during the actual elevator switch (which runs under ->freeze_lock and
> ->elevator_lock), the pre-allocated sched_tags are assigned to the
> appropriate q->hctx. Once the elevator switch is complete and the locks
> are released, the old elevator queue and its associated sched_tags are
> freed.
> ...
>
> [1] https://lore.kernel.org/all/0659ea8d-a463-47c8-9180-43c719e106eb@linux.ibm.com/
>
> Reported-by: Stefan Haberland <sth@linux.ibm.com>
> Closes: https://lore.kernel.org/all/0659ea8d-a463-47c8-9180-43c719e106eb@linux.ibm.com/
> Reviewed-by: Ming Lei <ming.lei@redhat.com>
> Reviewed-by: Christoph Hellwig <hch@lst.de>
> Reviewed-by: Hannes Reinecke <hare@suse.de>
> Signed-off-by: Nilay Shroff <nilay@linux.ibm.com>
Hi Nilay,
I am coming off of a 36 hour travel stint, and 6.16.7 (I do not have
that log, and it mightily messed up my xfs root requiring offline
repair), 6.16.9, and 6.17.0 simply do not boot on my system. After
unlocking with LUKS I get this panic consistently and immediately, and I
believe this is the problematic commit which was unfortunately carried
to the previous and current stable. I am using this udev rule:
`ACTION=="add|change", KERNEL=="sd*[!0-9]|sr*|nvme*",
ATTR{queue/scheduler}="bfq"`
> Sep 30 21:19:39 moon kernel: io scheduler bfq registered
> Sep 30 21:19:39 moon kernel: ------------[ cut here ]------------
> Sep 30 21:19:39 moon kernel: kernel BUG at mm/slub.c:563!
> Sep 30 21:19:39 moon kernel: Oops: general protection fault, probably
for non-canonical address 0x2cdf52296eacb08: 0000 [#1] SMP NOPTI
> Sep 30 21:19:39 moon kernel: CPU: 2 UID: 0 PID: 791 Comm:
(udev-worker) Not tainted 6.17.0-061700-generic #202509282239
PREEMPT(voluntary)
> Sep 30 21:19:39 moon kernel: Hardware name: Supermicro Super
Server/A2SDi-8C-HLN4F, BIOS 2.0 03/08/2024
> Sep 30 21:19:39 moon kernel: RIP: 0010:kfree+0x6b/0x360
> Sep 30 21:19:39 moon kernel: Code: 80 48 01 d8 0f 82 f6 02 00 00 48
c7 c2 00 00 00 80 48 2b 15 af 3f 61 01 48 01 d0 48 c1 e8 0c 48 c1 e0 06
48 03 05 8d 3f 61 01 <48> 8b 50 08 49 89 c4 f6 c2 01 0f 85 2f 02 00 00
0f 1f 44 00 00 41
> Sep 30 21:19:39 moon kernel: RSP: 0018:ffffc9e804257930 EFLAGS: 00010207
> Sep 30 21:19:39 moon kernel: RAX: 02cdf52296eacb00 RBX:
b37de27a3ab2cae5 RCX: 0000000000000000
> Sep 30 21:19:39 moon kernel: RDX: 000076bb00000000 RSI:
ffffffff983b7c31 RDI: b37de27a3ab2cae5
> Sep 30 21:19:39 moon kernel: RBP: ffffc9e804257978 R08:
0000000000000000 R09: 0000000000000000
> Sep 30 21:19:39 moon kernel: R10: 0000000000000000 R11:
0000000000000000 R12: ffff894589365840
> Sep 30 21:19:39 moon kernel: R13: ffff89458c7c20e0 R14:
0000000000000000 R15: ffff89458c7c20e0
> Sep 30 21:19:39 moon kernel: FS: 0000721ca92168c0(0000)
GS:ffff898464f80000(0000) knlGS:0000000000000000
> Sep 30 21:19:39 moon kernel: CS: 0010 DS: 0000 ES: 0000 CR0:
0000000080050033
> Sep 30 21:19:39 moon kernel: CR2: 00005afd46663fc8 CR3:
0000000111bf4000 CR4: 00000000003506f0
> Sep 30 21:19:39 moon kernel: Call Trace:
> Sep 30 21:19:39 moon kernel: <TASK>
> Sep 30 21:19:39 moon kernel: ? kfree+0x2dd/0x360
> Sep 30 21:19:39 moon kernel: kvfree+0x31/0x40
> Sep 30 21:19:39 moon kernel: blk_mq_free_tags+0x4b/0x70
> Sep 30 21:19:39 moon kernel: blk_mq_free_map_and_rqs+0x4d/0x70
> Sep 30 21:19:39 moon kernel: blk_mq_free_sched_tags+0x35/0x90
> Sep 30 21:19:39 moon kernel: elevator_change_done+0x53/0x200
> Sep 30 21:19:39 moon kernel: elevator_change+0xdf/0x190
> Sep 30 21:19:39 moon kernel: elv_iosched_store+0x151/0x190
> Sep 30 21:19:39 moon kernel: queue_attr_store+0xf1/0x120
> Sep 30 21:19:39 moon kernel: ? putname+0x65/0x90
> Sep 30 21:19:39 moon kernel: ? aa_file_perm+0x54/0x2e0
> Sep 30 21:19:39 moon kernel: ? _copy_from_iter+0x9d/0x690
> Sep 30 21:19:39 moon kernel: sysfs_kf_write+0x6f/0x90
> Sep 30 21:19:39 moon kernel: kernfs_fop_write_iter+0x15e/0x210
> Sep 30 21:19:39 moon kernel: vfs_write+0x271/0x490
> Sep 30 21:19:39 moon kernel: ksys_write+0x6f/0xf0
> Sep 30 21:19:39 moon kernel: __x64_sys_write+0x19/0x30
> Sep 30 21:19:39 moon kernel: x64_sys_call+0x79/0x2330
> Sep 30 21:19:39 moon kernel: do_syscall_64+0x80/0xac0
> Sep 30 21:19:39 moon kernel: ?
arch_exit_to_user_mode_prepare.isra.0+0xd/0xe0
> Sep 30 21:19:39 moon kernel: ? do_syscall_64+0xb6/0xac0
> Sep 30 21:19:39 moon kernel: ?
arch_exit_to_user_mode_prepare.isra.0+0xd/0xe0
> Sep 30 21:19:39 moon kernel: ? __seccomp_filter+0x47/0x5d0
> Sep 30 21:19:39 moon kernel: ? __x64_sys_fcntl+0x97/0x130
> Sep 30 21:19:39 moon kernel: ?
arch_exit_to_user_mode_prepare.isra.0+0xd/0xe0
> Sep 30 21:19:39 moon kernel: ? do_syscall_64+0xb6/0xac0
> Sep 30 21:19:39 moon kernel: entry_SYSCALL_64_after_hwframe+0x76/0x7e
> Sep 30 21:19:39 moon kernel: RIP: 0033:0x721ca911c5a4
> Sep 30 21:19:39 moon kernel: Code: c7 00 16 00 00 00 b8 ff ff ff ff
c3 66 2e 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 80 3d a5 ea 0e 00 00 74 13
b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 55 48 89 e5
48 83 ec 20 48 89
> Sep 30 21:19:39 moon kernel: RSP: 002b:00007ffdfffb8b58 EFLAGS:
00000202 ORIG_RAX: 0000000000000001
> Sep 30 21:19:39 moon kernel: RAX: ffffffffffffffda RBX:
0000000000000003 RCX: 0000721ca911c5a4
> Sep 30 21:19:39 moon kernel: RDX: 0000000000000003 RSI:
00007ffdfffb8df0 RDI: 000000000000002a
> Sep 30 21:19:39 moon kernel: RBP: 00007ffdfffb8b80 R08:
0000721ca9202228 R09: 00007ffdfffb8bd0
> Sep 30 21:19:39 moon kernel: R10: 0000000000000000 R11:
0000000000000202 R12: 0000000000000003
> Sep 30 21:19:39 moon kernel: R13: 00007ffdfffb8df0 R14:
00005afd465c5100 R15: 0000000000000003
> Sep 30 21:19:39 moon kernel: </TASK>
> Sep 30 21:19:39 moon kernel: Modules linked in: bfq nfsd tcp_bbr
sch_fq auth_rpcgss nfs_acl lockd grace nvme_fabrics efi_pstore sunrpc
nfnetlink dmi_sysfs ip_tables x_tables autofs4 xfs btrfs blake2b_generic
dm_crypt raid10 raid456 async_raid6_recov async_memcpy async_pq
async_xor asy>
> Sep 30 21:19:39 moon kernel: Oops: general protection fault, probably
for non-canonical address 0x3ce12d676eacb08: 0000 [#2] SMP NOPTI
> Sep 30 21:19:39 moon kernel: ---[ end trace 0000000000000000 ]---
> Sep 30 21:19:39 moon kernel: CPU: 3 UID: 0 PID: 792 Comm:
(udev-worker) Tainted: G D 6.17.0-061700-generic
#202509282239 PREEMPT(voluntary)
> Sep 30 21:19:39 moon kernel: Tainted: [D]=DIE
> Sep 30 21:19:39 moon kernel: Hardware name: Supermicro Super
Server/A2SDi-8C-HLN4F, BIOS 2.0 03/08/2024
> Sep 30 21:19:39 moon kernel: RIP: 0010:kfree+0x6b/0x360
> Sep 30 21:19:40 moon kernel: Code: 80 48 01 d8 0f 82 f6 02 00 00 48
c7 c2 00 00 00 80 48 2b 15 af 3f 61 01 48 01 d0 48 c1 e8 0c 48 c1 e0 06
48 03 05 8d 3f 61 01 <48> 8b 50 08 49 89 c4 f6 c2 01 0f 85 2f 02 00 00
0f 1f 44 00 00 41
> Sep 30 21:19:40 moon kernel: RSP: 0018:ffffc9e80425f990 EFLAGS: 00010207
> Sep 30 21:19:40 moon kernel: RAX: 03ce12d676eacb00 RBX:
f3854f723ab2cae5 RCX: 0000000000000000
> Sep 30 21:19:40 moon kernel: RDX: 000076bb00000000 RSI:
ffffffff983b7c31 RDI: f3854f723ab2cae5
> Sep 30 21:19:40 moon kernel: RBP: ffffc9e80425f9d8 R08:
0000000000000000 R09: 0000000000000000
> Sep 30 21:19:40 moon kernel: R10: 0000000000000000 R11:
0000000000000000 R12: ffff894580056160
> Sep 30 21:19:40 moon kernel: R13: ffff89458c7c20e0 R14:
0000000000000000 R15: ffff89458c7c20e0
> Sep 30 21:19:40 moon kernel: FS: 0000721ca92168c0(0000)
GS:ffff898465000000(0000) knlGS:0000000000000000
> Sep 30 21:19:40 moon kernel: RIP: 0010:kfree+0x6b/0x360
> Sep 30 21:19:40 moon kernel: CS: 0010 DS: 0000 ES: 0000 CR0:
0000000080050033
> Sep 30 21:19:40 moon kernel: Code: 80 48 01 d8 0f 82 f6 02 00 00 48
c7 c2 00 00 00 80 48 2b 15 af 3f 61 01 48 01 d0 48 c1 e8 0c 48 c1 e0 06
48 03 05 8d 3f 61 01 <48> 8b 50 08 49 89 c4 f6 c2 01 0f 85 2f 02 00 00
0f 1f 44 00 00 41
> Sep 30 21:19:40 moon kernel: CR2: 00007ffdfffb5b70 CR3:
000000010c1aa000 CR4: 00000000003506f0
> Sep 30 21:19:40 moon kernel: RSP: 0018:ffffc9e804257930 EFLAGS: 00010207
> Sep 30 21:19:40 moon kernel: Call Trace:
> Sep 30 21:19:40 moon kernel:
> Sep 30 21:19:40 moon kernel: RAX: 02cdf52296eacb00 RBX:
b37de27a3ab2cae5 RCX: 0000000000000000
> Sep 30 21:19:40 moon kernel: <TASK>
> Sep 30 21:19:40 moon kernel: ? kfree+0x2dd/0x360
> Sep 30 21:19:40 moon kernel: RDX: 000076bb00000000 RSI:
ffffffff983b7c31 RDI: b37de27a3ab2cae5
> Sep 30 21:19:40 moon kernel: kvfree+0x31/0x40
> Sep 30 21:19:40 moon kernel: blk_mq_free_tags+0x4b/0x70
> Sep 30 21:19:40 moon kernel: blk_mq_free_map_and_rqs+0x4d/0x70
> Sep 30 21:19:40 moon kernel: RBP: ffffc9e804257978 R08:
0000000000000000 R09: 0000000000000000
> Sep 30 21:19:40 moon kernel: blk_mq_free_sched_tags+0x35/0x90
> Sep 30 21:19:40 moon kernel: R10: 0000000000000000 R11:
0000000000000000 R12: ffff894589365840
> Sep 30 21:19:40 moon kernel: elevator_change_done+0x53/0x200
next parent reply other threads:[~2025-10-01 5:20 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <20250730074614.2537382-1-nilay@linux.ibm.com>
[not found] ` <20250730074614.2537382-3-nilay@linux.ibm.com>
2025-10-01 5:20 ` Kyle Sanderson [this message]
2025-10-01 13:05 ` [6.16.9 / 6.17.0 PANIC REGRESSION] block: fix lockdep warning caused by lock dependency in elv_iosched_store Kyle Sanderson
2025-10-02 15:30 ` Nilay Shroff
2025-10-02 15:58 ` Jens Axboe
2025-10-02 16:49 ` Linus Torvalds
2025-10-02 16:54 ` Jens Axboe
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=25a87311-70fd-4248-86e4-dd5fecf6cc99@gmail.com \
--to=kyle.leet@gmail.com \
--cc=axboe@kernel.dk \
--cc=gjoyce@ibm.com \
--cc=gregkh@linuxfoundation.org \
--cc=hare@suse.de \
--cc=hch@lst.de \
--cc=linux-block@vger.kernel.org \
--cc=linux-fsdevel@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=ming.lei@redhat.com \
--cc=nilay@linux.ibm.com \
--cc=sth@linux.ibm.com \
--cc=torvalds@linux-foundation.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox