From: Jens Axboe <axboe@kernel.dk>
To: Diangang Li <lidiangang@bytedance.com>,
Fengnan Chang <fengnanchang@gmail.com>,
asml.silence@gmail.com, io-uring@vger.kernel.org
Cc: Fengnan Chang <changfengnan@bytedance.com>
Subject: Re: [RFC PATCH 2/2] io_uring: fix io may accumulation in poll mode
Date: Wed, 17 Dec 2025 09:25:59 -0700 [thread overview]
Message-ID: <9a8418d8-439f-4dd2-b3fe-33567129861e@kernel.dk> (raw)
In-Reply-To: <f2836fb8-9ad7-4277-948b-430dcd24d1b6@bytedance.com>
On 12/17/25 5:34 AM, Diangang Li wrote:
> Hi Jens,
>
> We?ve identified one critical panic issue here.
>
> [ 4504.422964] [ T63683] list_del corruption, ff2adc9b51d19a90->next is
> LIST_POISON1 (dead000000000100)
> [ 4504.422994] [ T63683] ------------[ cut here ]------------
> [ 4504.422995] [ T63683] kernel BUG at lib/list_debug.c:56!
> [ 4504.423006] [ T63683] Oops: invalid opcode: 0000 [#1] SMP NOPTI
> [ 4504.423017] [ T63683] CPU: 38 UID: 0 PID: 63683 Comm: io_uring
> Kdump: loaded Tainted: G S E 6.19.0-rc1+ #1
> PREEMPT(voluntary)
> [ 4504.423032] [ T63683] Tainted: [S]=CPU_OUT_OF_SPEC, [E]=UNSIGNED_MODULE
> [ 4504.423040] [ T63683] Hardware name: Inventec S520-A6/Nanping MLB,
> BIOS 01.01.01.06.03 03/03/2023
> [ 4504.423050] [ T63683] RIP:
> 0010:__list_del_entry_valid_or_report+0x94/0x100
> [ 4504.423064] [ T63683] Code: 89 fe 48 c7 c7 f0 78 87 b5 e8 38 07 ae
> ff 0f 0b 48 89 ef e8 6e 40 cd ff 48 89 ea 48 89 de 48 c7 c7 20 79 87 b5
> e8 1c 07 ae ff <0f> 0b 4c 89 e7 e8 52 40 cd ff 4c 89 e2 48 89 de 48 c7
> c7 58 79 87
> [ 4504.423085] [ T63683] RSP: 0018:ff4efd9f3838fdb0 EFLAGS: 00010246
> [ 4504.423093] [ T63683] RAX: 000000000000004e RBX: ff2adc9b51d19a90
> RCX: 0000000000000027
> [ 4504.423103] [ T63683] RDX: 0000000000000000 RSI: 0000000000000001
> RDI: ff2add151cf99580
> [ 4504.423112] [ T63683] RBP: dead000000000100 R08: 0000000000000000
> R09: 0000000000000003
> [ 4504.423120] [ T63683] R10: ff4efd9f3838fc60 R11: ff2add151cdfffe8
> R12: dead000000000122
> [ 4504.423130] [ T63683] R13: ff2adc9b51d19a00 R14: 0000000000000000
> R15: 0000000000000000
> [ 4504.423139] [ T63683] FS: 00007fae4f7ff6c0(0000)
> GS:ff2add15665f5000(0000) knlGS:0000000000000000
> [ 4504.423148] [ T63683] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 4504.423157] [ T63683] CR2: 000055aa8afe5000 CR3: 00000083037ee006
> CR4: 0000000000773ef0
> [ 4504.423166] [ T63683] PKRU: 55555554
> [ 4504.423171] [ T63683] Call Trace:
> [ 4504.423178] [ T63683] <TASK>
> [ 4504.423184] [ T63683] io_do_iopoll+0x298/0x330
> [ 4504.423193] [ T63683] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
> [ 4504.423204] [ T63683] __do_sys_io_uring_enter+0x421/0x770
> [ 4504.423214] [ T63683] do_syscall_64+0x67/0xf00
> [ 4504.423223] [ T63683] entry_SYSCALL_64_after_hwframe+0x76/0x7e
> [ 4504.423232] [ T63683] RIP: 0033:0x55aa707e99c3
>
> It can be reproduced in three ways:
> - Running iopoll tests while switching the block scheduler
> - A split IO scenario in iopoll (e.g., bs=512k with max_sectors_kb=256k)
> - Multi poll queues with multi threads
>
> All cases appear related to IO completions occurring outside the
> io_do_iopoll() loop. The root cause remains unclear.
Ah I see what it is - we can get multiple completions on the iopoll
side, if you have multiple bio's per request. This didn't matter before
the patch that uses a lockless list to collect them, as it just marked
the request completed by writing to ->iopoll_complete and letting the
reaper find them. But it matters with the llist change, as then we're
adding the request to the llist more than once.
--
Jens Axboe
next prev parent reply other threads:[~2025-12-17 16:26 UTC|newest]
Thread overview: 28+ messages / expand[flat|nested] mbox.gz Atom feed top
2025-12-10 8:54 [RFC PATCH 0/2] io_uring: fix io may accumulation in poll mode Fengnan Chang
2025-12-10 8:55 ` [RFC PATCH 1/2] blk-mq: delete task running check in blk_hctx_poll Fengnan Chang
2025-12-10 9:19 ` Jens Axboe
2025-12-10 9:53 ` Jens Axboe
2025-12-10 8:55 ` [RFC PATCH 2/2] io_uring: fix io may accumulation in poll mode Fengnan Chang
2025-12-11 2:15 ` Jens Axboe
2025-12-11 4:10 ` Jens Axboe
2025-12-11 7:38 ` Fengnan
2025-12-11 10:22 ` Jens Axboe
2025-12-11 10:33 ` Jens Axboe
2025-12-11 11:13 ` Fengnan Chang
2025-12-11 11:19 ` Jens Axboe
2025-12-12 1:41 ` Fengnan Chang
2025-12-12 1:53 ` Jens Axboe
2025-12-12 2:12 ` Fengnan Chang
2025-12-12 5:11 ` Jens Axboe
2025-12-12 8:58 ` Jens Axboe
2025-12-12 9:49 ` Fengnan Chang
2025-12-12 20:22 ` Jens Axboe
2025-12-12 13:32 ` Diangang Li
2025-12-12 20:09 ` Jens Axboe
2025-12-15 6:25 ` Diangang Li
2025-12-17 12:34 ` Diangang Li
2025-12-17 16:25 ` Jens Axboe [this message]
2025-12-19 5:43 ` Diangang Li
2026-01-09 8:35 ` Diangang Li
2026-01-09 23:27 ` Jens Axboe
2025-12-10 9:53 ` (subset) [RFC PATCH 0/2] " Jens Axboe
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=9a8418d8-439f-4dd2-b3fe-33567129861e@kernel.dk \
--to=axboe@kernel.dk \
--cc=asml.silence@gmail.com \
--cc=changfengnan@bytedance.com \
--cc=fengnanchang@gmail.com \
--cc=io-uring@vger.kernel.org \
--cc=lidiangang@bytedance.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.