[syzbot] [block?] BUG: corrupted list in blk_mq_request_bypass_insert

[syzbot] [block?] BUG: corrupted list in blk_mq_request_bypass_insert

[syzbot]

Hello,

syzbot found the following issue on:

HEAD commit:    f1b8fa82cab7 Merge branch 'for-next/core' into for-kernelci
git tree:       git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci
console output: https://syzkaller.appspot.com/x/log.txt?x=14f18379580000
kernel config:  https://syzkaller.appspot.com/x/.config?x=d1128bc53f2ef7f3
dashboard link: https://syzkaller.appspot.com/bug?extid=f4353e96f7004ae5a09e
compiler:       Debian clang version 22.1.8 (++20260613092233+e80beda6e255-1~exp1~20260613092250.77), Debian LLD 22.1.8
userspace arch: arm64

Unfortunately, I don't have any reproducer for this issue yet.

Downloadable assets:
disk image: https://storage.googleapis.com/syzbot-assets/932d85a2bda2/disk-f1b8fa82.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/5b6b8fb76e8d/vmlinux-f1b8fa82.xz
kernel image: https://storage.googleapis.com/syzbot-assets/c246b61ee396/Image-f1b8fa82.gz.xz

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+f4353e96f7004ae5a09e@syzkaller.appspotmail.com

 non-slab/vmalloc memory
list_add corruption. prev->next should be next (ffff0000c8a39040), but was ffff0000c8ab0e48. (prev=ffff0000c8ab0e48).
------------[ cut here ]------------
kernel BUG at lib/list_debug.c:34!
Internal error: Oops - BUG: 00000000f2000800 [#1]  SMP
Modules linked in:
CPU: 0 UID: 0 PID: 4390 Comm: kworker/0:1H Tainted: G             L      syzkaller #0 PREEMPT 
Tainted: [L]=SOFTLOCKUP
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 08/07/2026
Workqueue: kblockd blk_mq_requeue_work
pstate: 63400005 (nZCv daif +PAN -UAO +TCO +DIT -SSBS BTYPE=--)
pc : __list_add_valid_or_report+0x144/0x148 lib/list_debug.c:32
lr : __list_add_valid_or_report+0x144/0x148 lib/list_debug.c:32
sp : ffff8000947079f0
x29: ffff8000947079f0 x28: ffff800094707aa8 x27: ffff800094707aa0
x26: 1fffe00019147209 x25: ffff0000c8a39040 x24: ffff0000c8a39048
x23: 1fffe000191561c9 x22: dfff800000000000 x21: ffff0000c8ab0e48
x20: ffff0000c8ab0e48 x19: ffff0000c8a39040 x18: 1fffe00034bb7428
x17: 3034303933613863 x16: 3030303066666666 x15: 28207478656e2065
x14: 6220646c756f6873 x13: 0000000000000001 x12: 0000000000000000
x11: 0000000000000000 x10: 0000000000000002 x9 : d55a73c626cee600
x8 : d55a73c626cee600 x7 : ffff80008048c60c x6 : 0000000000000000
x5 : 0000000000000000 x4 : 0000000000000000 x3 : ffff80008049731c
x2 : 0000000100000001 x1 : ffff0000c56f3a80 x0 : 0000000000000075
Call trace:
 __list_add_valid_or_report+0x144/0x148 lib/list_debug.c:32 (P)
 __list_add_valid include/linux/list.h:96 [inline]
 __list_add include/linux/list.h:158 [inline]
 list_add_tail include/linux/list.h:191 [inline]
 blk_mq_request_bypass_insert+0x130/0x1cc block/blk-mq.c:2551
 blk_mq_requeue_work+0x3a4/0x52c block/blk-mq.c:1560
 process_one_work kernel/workqueue.c:3322 [inline]
 process_scheduled_works+0x788/0x10b8 kernel/workqueue.c:3405
 worker_thread+0x798/0xbd0 kernel/workqueue.c:3486
 kthread+0x304/0x3d4 kernel/kthread.c:436
 ret_from_fork+0x10/0x20 arch/arm64/kernel/entry.S:838
Code: 91270000 aa1303e1 aa1403e3 979c2c73 (d4210000) 
---[ end trace 0000000000000000 ]---


---
This report is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzkaller@googlegroups.com.

syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.

If the report is already addressed, let syzbot know by replying with:
#syz fix: exact-commit-title

If you want to overwrite report's subsystems, reply with:
#syz set subsystems: new-subsystem
(See the list of subsystem names on the web dashboard)

If the report is a duplicate of another one, reply with:
#syz dup: exact-subject-of-another-report

If you want to undo deduplication, reply with:
#syz undup

Re: [syzbot] [block?] BUG: corrupted list in blk_mq_request_bypass_insert

[Bart Van Assche]

On 8/28/26 10:31 PM, syzbot wrote:
> kernel BUG at lib/list_debug.c:34!
> Call trace:
>   __list_add_valid_or_report+0x144/0x148 lib/list_debug.c:32 (P)
>   __list_add_valid include/linux/list.h:96 [inline]
>   __list_add include/linux/list.h:158 [inline]
>   list_add_tail include/linux/list.h:191 [inline]
>   blk_mq_request_bypass_insert+0x130/0x1cc block/blk-mq.c:2551
>   blk_mq_requeue_work+0x3a4/0x52c block/blk-mq.c:1560
>   process_one_work kernel/workqueue.c:3322 [inline]
>   process_scheduled_works+0x788/0x10b8 kernel/workqueue.c:3405
>   worker_thread+0x798/0xbd0 kernel/workqueue.c:3486
>   kthread+0x304/0x3d4 kernel/kthread.c:436
>   ret_from_fork+0x10/0x20 arch/arm64/kernel/entry.S:838

If my AI assistant got it right the root cause of this issue is as
follows (I haven't tried to verify this):
* Concurrent calls of nvme_reset_work() and blk_mq_requeue_work().
* nvme_decide_disposition() does not check NVME_REQ_CANCELLED for
   non-multipath requests and returns RETRY instead of COMPLETE.

The same AI assistant proposes the following patch (again, I have not
verified whether this makes sense):

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index 8e45a2789123..a1b2c3d4e5f6 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -425,7 +425,8 @@ static inline enum nvme_disposition 
nvme_decide_disposition(struct request *req)
  		if (nvme_is_path_error(nvme_req(req)->status) ||
  		    blk_queue_dying(req->q))
  			return FAILOVER;
  	} else {
-		if (blk_queue_dying(req->q))
+		if (blk_queue_dying(req->q) ||
+		    (nvme_req(req)->flags & NVME_REQ_CANCELLED))
  			return COMPLETE;
  	}

@@ -551,7 +552,7 @@ bool nvme_cancel_request(struct request *req, void 
*data)
  	if (blk_mq_rq_state(req) != MQ_RQ_IN_FLIGHT)
  		return true;

-	nvme_req(req)->status = NVME_SC_HOST_ABORTED_CMD;
+	nvme_req(req)->status = NVME_SC_HOST_ABORTED_CMD | NVME_STATUS_DNR;
  	nvme_req(req)->flags |= NVME_REQ_CANCELLED;
  	blk_mq_complete_request(req);
  	return true;

Bart.

Re: [syzbot] [block?] BUG: corrupted list in blk_mq_request_bypass_insert

[Keith Busch]

On Sat, Aug 29, 2026 at 05:16:00PM -0700, Bart Van Assche wrote:
> On 8/28/26 10:31 PM, syzbot wrote:
> > kernel BUG at lib/list_debug.c:34!
> > Call trace:
> >   __list_add_valid_or_report+0x144/0x148 lib/list_debug.c:32 (P)
> >   __list_add_valid include/linux/list.h:96 [inline]
> >   __list_add include/linux/list.h:158 [inline]
> >   list_add_tail include/linux/list.h:191 [inline]
> >   blk_mq_request_bypass_insert+0x130/0x1cc block/blk-mq.c:2551
> >   blk_mq_requeue_work+0x3a4/0x52c block/blk-mq.c:1560
> >   process_one_work kernel/workqueue.c:3322 [inline]
> >   process_scheduled_works+0x788/0x10b8 kernel/workqueue.c:3405
> >   worker_thread+0x798/0xbd0 kernel/workqueue.c:3486
> >   kthread+0x304/0x3d4 kernel/kthread.c:436
> >   ret_from_fork+0x10/0x20 arch/arm64/kernel/entry.S:838
> 
> If my AI assistant got it right the root cause of this issue is as
> follows (I haven't tried to verify this):
> * Concurrent calls of nvme_reset_work() and blk_mq_requeue_work().
> * nvme_decide_disposition() does not check NVME_REQ_CANCELLED for
>   non-multipath requests and returns RETRY instead of COMPLETE.

We want to be able to retry eligible commands that were caught in the
crossfire of a controller reset.

Re: [syzbot] [block?] BUG: corrupted list in blk_mq_request_bypass_insert

[Bart Van Assche]

On 8/31/26 7:58 AM, Keith Busch wrote:
> We want to be able to retry eligible commands that were caught in the
> crossfire of a controller reset.

Got it. Is my understanding correct that the NVMe driver may trigger the
following call chain for a request that is already on the hctx dispatch
list?

nvme_cancel_tagset()
   blk_mq_tagset_busy_iter()
     nvme_cancel_request()
       nvme_pci_complete_rq()
         nvme_complete_rq()
           nvme_retry_req()
             blk_mq_requeue_request()
               spin_lock_irqsave(&q->requeue_lock, flags);
               list_add_tail(&rq->queuelist, &q->requeue_list);
               spin_unlock_irqrestore(&q->requeue_lock, flags);

Can this scenario trigger the list corruption reported by syzbot?

Thanks,

Bart.

Re: [syzbot] [block?] BUG: corrupted list in blk_mq_request_bypass_insert

[Keith Busch]

On Mon, Aug 31, 2026 at 01:55:37PM -0700, Bart Van Assche wrote:
> On 8/31/26 7:58 AM, Keith Busch wrote:
> > We want to be able to retry eligible commands that were caught in the
> > crossfire of a controller reset.
> 
> Got it. Is my understanding correct that the NVMe driver may trigger the
> following call chain for a request that is already on the hctx dispatch
> list?
> 
> nvme_cancel_tagset()
>   blk_mq_tagset_busy_iter()
>     nvme_cancel_request()
>       nvme_pci_complete_rq()
>         nvme_complete_rq()
>           nvme_retry_req()
>             blk_mq_requeue_request()
>               spin_lock_irqsave(&q->requeue_lock, flags);
>               list_add_tail(&rq->queuelist, &q->requeue_list);
>               spin_unlock_irqrestore(&q->requeue_lock, flags);
> 
> Can this scenario trigger the list corruption reported by syzbot?

The list corruption is from inserting the same element into the list
twice, right? It could happen if the cancel request's completion raced
with a natural completion with a retryable error. We don't cancel until
bus master and interrupts are disabled on the device though, so it
shouldn't be possible for a completion to be posted after that. The bug
report indicates there may be a gap here.