[syzbot] [block?] WARNING in blk_mq_start_request

[syzbot] [block?] WARNING in blk_mq_start_request

[syzbot]

Hello,

syzbot found the following issue on:

HEAD commit:    4652b8e4f3ff Merge tag '6.7-rc-ksmbd-server-fixes' of git:..
git tree:       upstream
console output: https://syzkaller.appspot.com/x/log.txt?x=117a6ebb680000
kernel config:  https://syzkaller.appspot.com/x/.config?x=5316ad879647e3c5
dashboard link: https://syzkaller.appspot.com/bug?extid=fcc47ba2476570cbbeb0
compiler:       gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40
syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=13489af3680000
C reproducer:   https://syzkaller.appspot.com/x/repro.c?x=145c7d07680000

Downloadable assets:
disk image (non-bootable): https://storage.googleapis.com/syzbot-assets/7bc7510fe41f/non_bootable_disk-4652b8e4.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/9c5be65b1bc8/vmlinux-4652b8e4.xz
kernel image: https://storage.googleapis.com/syzbot-assets/8799a4da0be5/bzImage-4652b8e4.xz

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+fcc47ba2476570cbbeb0@syzkaller.appspotmail.com

R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffff489c59c
R13: 00007ffff489c5b0 R14: 00007ffff489c5f0 R15: 0000000000000003
 </TASK>
------------[ cut here ]------------
WARNING: CPU: 3 PID: 5178 at block/blk-mq.c:1258 blk_mq_start_request+0x643/0x780 block/blk-mq.c:1258
Modules linked in:
CPU: 3 PID: 5178 Comm: syz-executor268 Not tainted 6.6.0-syzkaller-10396-g4652b8e4f3ff #0
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.2-debian-1.16.2-1 04/01/2014
RIP: 0010:blk_mq_start_request+0x643/0x780 block/blk-mq.c:1258
Code: 00 00 fc ff df 48 8d 7d 10 48 89 fa 48 c1 ea 03 80 3c 02 00 0f 85 2f 01 00 00 48 89 df ff 55 10 e9 64 fc ff ff e8 1d 9c 44 fd <0f> 0b e9 f6 fa ff ff e8 11 9c 44 fd 0f 0b e9 06 fa ff ff e8 05 9c
RSP: 0018:ffffc900035ef298 EFLAGS: 00010293
RAX: 0000000000000000 RBX: ffff888018539e00 RCX: ffffffff8442db37
RDX: ffff888030114f00 RSI: ffffffff8442e043 RDI: 0000000000000005
RBP: ffff888018539e94 R08: 0000000000000005 R09: 0000000000000000
R10: 0000000000000001 R11: 0000000000000000 R12: ffff888106a33350
R13: 0000000000000001 R14: ffff888018539f1d R15: ffff888018539f20
FS:  0000555555f00480(0000) GS:ffff88806b900000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000555555f00788 CR3: 000000002238a000 CR4: 0000000000350ef0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 <TASK>
 null_queue_rq+0x2e2/0x7a0 drivers/block/null_blk/main.c:1727
 __blk_mq_issue_directly+0xe3/0x270 block/blk-mq.c:2578
 blk_mq_request_issue_directly+0x120/0x190 block/blk-mq.c:2663
 blk_mq_plug_issue_direct+0x19e/0x670 block/blk-mq.c:2684
 blk_mq_flush_plug_list.part.0+0x16b6/0x1e90 block/blk-mq.c:2793
 blk_mq_flush_plug_list+0x62/0x80 block/blk-mq.c:2770
 __blk_flush_plug+0x2c0/0x430 block/blk-core.c:1142
 blk_finish_plug block/blk-core.c:1166 [inline]
 blk_finish_plug+0x54/0xa0 block/blk-core.c:1163
 read_pages+0x69b/0xdb0 mm/readahead.c:183
 page_cache_ra_unbounded+0x457/0x5e0 mm/readahead.c:269
 do_page_cache_ra mm/readahead.c:299 [inline]
 page_cache_ra_order+0x72b/0xa80 mm/readahead.c:546
 ondemand_readahead+0x493/0x1130 mm/readahead.c:668
 page_cache_sync_ra+0x174/0x1d0 mm/readahead.c:695
 page_cache_sync_readahead.constprop.0+0xb2/0xf0 include/linux/pagemap.h:1293
 cramfs_blkdev_read fs/cramfs/inode.c:218 [inline]
 cramfs_read+0x33f/0xae0 fs/cramfs/inode.c:278
 cramfs_read_super+0xbd/0x9b0 fs/cramfs/inode.c:522
 cramfs_blkdev_fill_super+0x116/0x2f0 fs/cramfs/inode.c:622
 get_tree_bdev+0x3b5/0x650 fs/super.c:1595
 cramfs_get_tree fs/cramfs/inode.c:958 [inline]
 cramfs_get_tree+0x40/0x50 fs/cramfs/inode.c:948
 vfs_get_tree+0x8c/0x370 fs/super.c:1768
 do_new_mount fs/namespace.c:3337 [inline]
 path_mount+0x1492/0x1ed0 fs/namespace.c:3664
 do_mount fs/namespace.c:3677 [inline]
 __do_sys_mount fs/namespace.c:3886 [inline]
 __se_sys_mount fs/namespace.c:3863 [inline]
 __x64_sys_mount+0x293/0x310 fs/namespace.c:3863
 do_syscall_x64 arch/x86/entry/common.c:51 [inline]
 do_syscall_64+0x3f/0x110 arch/x86/entry/common.c:82
 entry_SYSCALL_64_after_hwframe+0x63/0x6b
RIP: 0033:0x7feca35df9f9
Code: 28 00 00 00 75 05 48 83 c4 28 c3 e8 a1 1a 00 00 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b0 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007ffff489c548 EFLAGS: 00000246 ORIG_RAX: 00000000000000a5
RAX: ffffffffffffffda RBX: 00007ffff489c550 RCX: 00007feca35df9f9
RDX: 0000000020000040 RSI: 00000000200000c0 RDI: 0000000020000000
RBP: 0000000000000002 R08: 0000000000000000 R09: 0000000000003134
R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffff489c59c
R13: 00007ffff489c5b0 R14: 00007ffff489c5f0 R15: 0000000000000003
 </TASK>


---
This report is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzkaller@googlegroups.com.

syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.

If the report is already addressed, let syzbot know by replying with:
#syz fix: exact-commit-title

If you want syzbot to run the reproducer, reply with:
#syz test: git://repo/address.git branch-or-commit-hash
If you attach or paste a git patch, syzbot will apply it before testing.

If you want to overwrite report's subsystems, reply with:
#syz set subsystems: new-subsystem
(See the list of subsystem names on the web dashboard)

If the report is a duplicate of another one, reply with:
#syz dup: exact-subject-of-another-report

If you want to undo deduplication, reply with:
#syz undup

Re: [syzbot] [block?] WARNING in blk_mq_start_request

[syzbot]

Hello,

syzbot has tested the proposed patch and the reproducer did not trigger any issue:

Reported-and-tested-by: syzbot+fcc47ba2476570cbbeb0@syzkaller.appspotmail.com

Tested on:

commit:         4652b8e4 Merge tag '6.7-rc-ksmbd-server-fixes' of git:..
git tree:       https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
console output: https://syzkaller.appspot.com/x/log.txt?x=1629a00f680000
kernel config:  https://syzkaller.appspot.com/x/.config?x=5316ad879647e3c5
dashboard link: https://syzkaller.appspot.com/bug?extid=fcc47ba2476570cbbeb0
compiler:       gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40
patch:          https://syzkaller.appspot.com/x/patch.diff?x=107e3ac0e80000

Note: testing is done by a robot and is best-effort only.

Re: [syzbot] [PATCH] Test for 4652b8e4f3ff

[syzbot]

For archival purposes, forwarding an incoming command email to
linux-kernel@vger.kernel.org.

***

Subject: [PATCH] Test for 4652b8e4f3ff
Author: eadavis@qq.com

please test warning in blk_mq_start_request

#syz test https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git 4652b8e4f3ff

diff --git a/drivers/block/null_blk/main.c b/drivers/block/null_blk/main.c
index 22a3cf7f32e2..0726534a5a24 100644
--- a/drivers/block/null_blk/main.c
+++ b/drivers/block/null_blk/main.c
@@ -1724,6 +1724,8 @@ static blk_status_t null_queue_rq(struct blk_mq_hw_ctx *hctx,
 	cmd->fake_timeout = should_timeout_request(rq) ||
 		blk_should_fake_timeout(rq->q);
 
+	if (READ_ONCE(rq->state))
+		WRITE_ONCE(rq->state, MQ_RQ_IDLE);
 	blk_mq_start_request(rq);
 
 	if (should_requeue_request(rq)) {

Re: [syzbot] [block?] WARNING in blk_mq_start_request

[syzbot]

Hello,

syzbot has tested the proposed patch and the reproducer did not trigger any issue:

Reported-and-tested-by: syzbot+fcc47ba2476570cbbeb0@syzkaller.appspotmail.com

Tested on:

commit:         4652b8e4 Merge tag '6.7-rc-ksmbd-server-fixes' of git:..
git tree:       https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
console output: https://syzkaller.appspot.com/x/log.txt?x=1400e647680000
kernel config:  https://syzkaller.appspot.com/x/.config?x=5316ad879647e3c5
dashboard link: https://syzkaller.appspot.com/bug?extid=fcc47ba2476570cbbeb0
compiler:       gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40
patch:          https://syzkaller.appspot.com/x/patch.diff?x=12a5d81f680000

Note: testing is done by a robot and is best-effort only.

Re: [syzbot] [PATCH] Test for 4652b8e4f3ff

[syzbot]

For archival purposes, forwarding an incoming command email to
linux-kernel@vger.kernel.org.

***

Subject: [PATCH] Test for 4652b8e4f3ff
Author: eadavis@qq.com

please test warning in blk_mq_start_request

#syz test https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git 4652b8e4f3ff

diff --git a/block/blk-mq.c b/block/blk-mq.c
index e2d11183f62e..26f2921cbc50 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c
@@ -2575,6 +2575,8 @@ static blk_status_t __blk_mq_issue_directly(struct blk_mq_hw_ctx *hctx,
 	 * Any other error (busy), just add it to our list as we
 	 * previously would have done.
 	 */
+	if (READ_ONCE(rq->state))
+		WRITE_ONCE(rq->state, MQ_RQ_IDLE);
 	ret = q->mq_ops->queue_rq(hctx, &bd);
 	switch (ret) {
 	case BLK_STS_OK:

Re: [syzbot] [block?] WARNING in blk_mq_start_request

[syzbot]

Hello,

syzbot has tested the proposed patch and the reproducer did not trigger any issue:

Reported-and-tested-by: syzbot+fcc47ba2476570cbbeb0@syzkaller.appspotmail.com

Tested on:

commit:         4652b8e4 Merge tag '6.7-rc-ksmbd-server-fixes' of git:..
git tree:       https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
console output: https://syzkaller.appspot.com/x/log.txt?x=14ca647b680000
kernel config:  https://syzkaller.appspot.com/x/.config?x=5316ad879647e3c5
dashboard link: https://syzkaller.appspot.com/bug?extid=fcc47ba2476570cbbeb0
compiler:       gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40
patch:          https://syzkaller.appspot.com/x/patch.diff?x=126f8204e80000

Note: testing is done by a robot and is best-effort only.

Re: [syzbot] [block?] WARNING in blk_mq_start_request

[syzbot]

syzbot has bisected this issue to:

commit d78bfa1346ab1fe04d20aa45a0678d1fc866f37c
Author: Chengming Zhou <zhouchengming@bytedance.com>
Date:   Wed Sep 13 15:16:16 2023 +0000

    block/null_blk: add queue_rqs() support

bisection log:  https://syzkaller.appspot.com/x/bisect.txt?x=106414a8e80000
start commit:   13d88ac54ddd Merge tag 'vfs-6.7.fsid' of git://git.kernel...
git tree:       upstream
final oops:     https://syzkaller.appspot.com/x/report.txt?x=126414a8e80000
console output: https://syzkaller.appspot.com/x/log.txt?x=146414a8e80000
kernel config:  https://syzkaller.appspot.com/x/.config?x=beb32a598fd79db9
dashboard link: https://syzkaller.appspot.com/bug?extid=fcc47ba2476570cbbeb0
syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=1465bb08e80000
C reproducer:   https://syzkaller.appspot.com/x/repro.c?x=13e7881f680000

Reported-by: syzbot+fcc47ba2476570cbbeb0@syzkaller.appspotmail.com
Fixes: d78bfa1346ab ("block/null_blk: add queue_rqs() support")

For information about bisection process see: https://goo.gl/tpsmEJ#bisection

Re: [syzbot] [block?] WARNING in blk_mq_start_request

[Chengming Zhou]

On 2023/11/9 05:18, syzbot wrote:
> syzbot has bisected this issue to:
> 
> commit d78bfa1346ab1fe04d20aa45a0678d1fc866f37c
> Author: Chengming Zhou <zhouchengming@bytedance.com>
> Date:   Wed Sep 13 15:16:16 2023 +0000
> 
>     block/null_blk: add queue_rqs() support
> 
> bisection log:  https://syzkaller.appspot.com/x/bisect.txt?x=106414a8e80000
> start commit:   13d88ac54ddd Merge tag 'vfs-6.7.fsid' of git://git.kernel...
> git tree:       upstream
> final oops:     https://syzkaller.appspot.com/x/report.txt?x=126414a8e80000
> console output: https://syzkaller.appspot.com/x/log.txt?x=146414a8e80000
> kernel config:  https://syzkaller.appspot.com/x/.config?x=beb32a598fd79db9
> dashboard link: https://syzkaller.appspot.com/bug?extid=fcc47ba2476570cbbeb0
> syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=1465bb08e80000
> C reproducer:   https://syzkaller.appspot.com/x/repro.c?x=13e7881f680000
> 
> Reported-by: syzbot+fcc47ba2476570cbbeb0@syzkaller.appspotmail.com
> Fixes: d78bfa1346ab ("block/null_blk: add queue_rqs() support")
> 
> For information about bisection process see: https://goo.gl/tpsmEJ#bisection

CONFIG_BLK_DEV_NULL_BLK_FAULT_INJECTION is enabled in the kernel config,
so null_queue_rq() will return BLK_STS_RESOURCE or BLK_STS_DEV_RESOURCE
for some requests, which have been marked as IN_FLIGHT status.

Then null_queue_rqs() put these requests in the rqlist and return back,
blk-mq will try to queue them individually once again, caused the warning
"WARN_ON_ONCE(blk_mq_rq_state(rq) != MQ_RQ_IDLE)" in blk_mq_start_request().

So handling of return value of null_queue_rq() in null_queue_rqs() is wrong,
maybe we should __blk_mq_requeue_request() for these requests, before
adding them in the rqlist?

Thanks!

Re: [syzbot] [block?] WARNING in blk_mq_start_request

[Bart Van Assche]

On 11/8/23 17:27, Chengming Zhou wrote:
> CONFIG_BLK_DEV_NULL_BLK_FAULT_INJECTION is enabled in the kernel config,
> so null_queue_rq() will return BLK_STS_RESOURCE or BLK_STS_DEV_RESOURCE
> for some requests, which have been marked as IN_FLIGHT status.
> 
> Then null_queue_rqs() put these requests in the rqlist and return back,
> blk-mq will try to queue them individually once again, caused the warning
> "WARN_ON_ONCE(blk_mq_rq_state(rq) != MQ_RQ_IDLE)" in blk_mq_start_request().
> 
> So handling of return value of null_queue_rq() in null_queue_rqs() is wrong,
> maybe we should __blk_mq_requeue_request() for these requests, before
> adding them in the rqlist?

Please follow the example of virtio_queue_rqs() and send any requests
that need to be requeued back to the block layer core instead of
handling these directly in null_queue_rqs().

Thanks,

Bart.

Re: [External] Re: [syzbot] [block?] WARNING in blk_mq_start_request

[Chengming Zhou]

On 2023/11/10 02:13, Bart Van Assche wrote:
> On 11/8/23 17:27, Chengming Zhou wrote:
>> CONFIG_BLK_DEV_NULL_BLK_FAULT_INJECTION is enabled in the kernel config,
>> so null_queue_rq() will return BLK_STS_RESOURCE or BLK_STS_DEV_RESOURCE
>> for some requests, which have been marked as IN_FLIGHT status.
>>
>> Then null_queue_rqs() put these requests in the rqlist and return back,
>> blk-mq will try to queue them individually once again, caused the warning
>> "WARN_ON_ONCE(blk_mq_rq_state(rq) != MQ_RQ_IDLE)" in blk_mq_start_request().
>>
>> So handling of return value of null_queue_rq() in null_queue_rqs() is wrong,
>> maybe we should __blk_mq_requeue_request() for these requests, before
>> adding them in the rqlist?
> 
> Please follow the example of virtio_queue_rqs() and send any requests
> that need to be requeued back to the block layer core instead of
> handling these directly in null_queue_rqs().
> 

Ok, I reviewed the code of virtio_queue_rqs(), found the main difference
is that request won't fail after blk_mq_start_request().

But in null_blk case, the request will fail after blk_mq_start_request(),
return BLK_STS_RESOURCE or BLK_STS_DEV_RESOURCE. If we return these rqs
back to the block layer core, they will be queued individually once again.
So caused the warning.

Thanks!

Re: [External] Re: [syzbot] [block?] WARNING in blk_mq_start_request

[Bart Van Assche]

On 11/13/23 07:05, Chengming Zhou wrote:
> Ok, I reviewed the code of virtio_queue_rqs(), found the main difference
> is that request won't fail after blk_mq_start_request().
> 
> But in null_blk case, the request will fail after blk_mq_start_request(),
> return BLK_STS_RESOURCE or BLK_STS_DEV_RESOURCE. If we return these rqs
> back to the block layer core, they will be queued individually once again.
> So caused the warning.

I think it is safe to move the blk_mq_start_request() call under the if-block
that decides whether or not to requeue a request in null_queue_rq()

Thanks,

Bart.

Re: [External] Re: [syzbot] [block?] WARNING in blk_mq_start_request

[Chengming Zhou]

On 2023/11/14 07:57, Bart Van Assche wrote:
> On 11/13/23 07:05, Chengming Zhou wrote:
>> Ok, I reviewed the code of virtio_queue_rqs(), found the main difference
>> is that request won't fail after blk_mq_start_request().
>>
>> But in null_blk case, the request will fail after blk_mq_start_request(),
>> return BLK_STS_RESOURCE or BLK_STS_DEV_RESOURCE. If we return these rqs
>> back to the block layer core, they will be queued individually once again.
>> So caused the warning.
> 
> I think it is safe to move the blk_mq_start_request() call under the if-block
> that decides whether or not to requeue a request in null_queue_rq()
> 

Right! And null_handle_throttled() in null_handle_cmd() may also return the
BLK_STS_DEV_RESOURCE, it's also needed to put in null_queue_rq() and before
the blk_mq_start_request().

Then request must return BLK_STS_OK after blk_mq_start_request().

Thanks!