From: "Yu Kuai" <yukuai@fnnas.com>
To: "Josh Hunt" <johunt@akamai.com>, <song@kernel.org>,
<linan122@huawei.com>, <linux-raid@vger.kernel.org>,
<yukuai@fnnas.com>
Cc: <ncroxon@redhat.com>, <stable@vger.kernel.org>
Subject: Re: [PATCH v3] md/raid10: fix deadlock with check operation and nowait requests
Date: Wed, 4 Mar 2026 10:18:40 +0800 [thread overview]
Message-ID: <54af459e-2671-417b-bc9f-2b13f111f749@fnnas.com> (raw)
In-Reply-To: <20260303005619.1352958-1-johunt@akamai.com>
在 2026/3/3 8:56, Josh Hunt 写道:
> When an array check is running it will raise the barrier at which point
> normal requests will become blocked and increment the nr_pending value to
> signal there is work pending inside of wait_barrier(). NOWAIT requests
> do not block and so will return immediately with an error, and additionally
> do not increment nr_pending in wait_barrier(). Upstream change
> 43806c3d5b9b ("raid10: cleanup memleak at raid10_make_request") added a
> call to raid_end_bio_io() to fix a memory leak when NOWAIT requests hit
> this condition. raid_end_bio_io() eventually calls allow_barrier() and
> it will unconditionally do an atomic_dec_and_test(&conf->nr_pending) even
> though the corresponding increment on nr_pending didn't happen in the
> NOWAIT case.
>
> This can be easily seen by starting a check operation while an application is
> doing nowait IO on the same array. This results in a deadlocked state due to
> nr_pending value underflowing and so the md resync thread gets stuck waiting
> for nr_pending to == 0.
>
> Output of r10conf state of the array when we hit this condition:
>
> crash> struct r10conf.barrier,nr_pending,nr_waiting,nr_queued <addr of r10conf>
> barrier = 1,
> nr_pending = {
> counter = -41
> },
> nr_waiting = 15,
> nr_queued = 0,
>
> Example of md_sync thread stuck waiting on raise_barrier() and other requests
> stuck in wait_barrier():
>
> md1_resync
> [<0>] raise_barrier+0xce/0x1c0
> [<0>] raid10_sync_request+0x1ca/0x1ed0
> [<0>] md_do_sync+0x779/0x1110
> [<0>] md_thread+0x90/0x160
> [<0>] kthread+0xbe/0xf0
> [<0>] ret_from_fork+0x34/0x50
> [<0>] ret_from_fork_asm+0x1a/0x30
>
> kworker/u1040:2+flush-253:4
> [<0>] wait_barrier+0x1de/0x220
> [<0>] regular_request_wait+0x30/0x180
> [<0>] raid10_make_request+0x261/0x1000
> [<0>] md_handle_request+0x13b/0x230
> [<0>] __submit_bio+0x107/0x1f0
> [<0>] submit_bio_noacct_nocheck+0x16f/0x390
> [<0>] ext4_io_submit+0x24/0x40
> [<0>] ext4_do_writepages+0x254/0xc80
> [<0>] ext4_writepages+0x84/0x120
> [<0>] do_writepages+0x7a/0x260
> [<0>] __writeback_single_inode+0x3d/0x300
> [<0>] writeback_sb_inodes+0x1dd/0x470
> [<0>] __writeback_inodes_wb+0x4c/0xe0
> [<0>] wb_writeback+0x18b/0x2d0
> [<0>] wb_workfn+0x2a1/0x400
> [<0>] process_one_work+0x149/0x330
> [<0>] worker_thread+0x2d2/0x410
> [<0>] kthread+0xbe/0xf0
> [<0>] ret_from_fork+0x34/0x50
> [<0>] ret_from_fork_asm+0x1a/0x30
>
> Fixes: 43806c3d5b9b ("raid10: cleanup memleak at raid10_make_request")
> Cc: stable@vger.kernel.org
> Signed-off-by: Josh Hunt <johunt@akamai.com>
> ---
> v3:
> * Call free_r10bio() as per Yu Kuai's suggestion
> ---
> drivers/md/raid10.c | 4 ++--
> 1 file changed, 2 insertions(+), 2 deletions(-)
There are some checkpatch errors and warnings, applied to md-7.0 with some
changes to commit message.
> diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c
> index 9debb20cf129..b4892c5d571c 100644
> --- a/drivers/md/raid10.c
> +++ b/drivers/md/raid10.c
> @@ -1184,7 +1184,7 @@ static void raid10_read_request(struct mddev *mddev, struct bio *bio,
> }
>
> if (!regular_request_wait(mddev, conf, bio, r10_bio->sectors)) {
> - raid_end_bio_io(r10_bio);
> + free_r10bio(r10_bio);
> return;
> }
>
> @@ -1372,7 +1372,7 @@ static void raid10_write_request(struct mddev *mddev, struct bio *bio,
>
> sectors = r10_bio->sectors;
> if (!regular_request_wait(mddev, conf, bio, sectors)) {
> - raid_end_bio_io(r10_bio);
> + free_r10bio(r10_bio);
> return;
> }
>
--
Thansk,
Kuai
prev parent reply other threads:[~2026-03-04 2:18 UTC|newest]
Thread overview: 2+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-03-03 0:56 [PATCH v3] md/raid10: fix deadlock with check operation and nowait requests Josh Hunt
2026-03-04 2:18 ` Yu Kuai [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=54af459e-2671-417b-bc9f-2b13f111f749@fnnas.com \
--to=yukuai@fnnas.com \
--cc=johunt@akamai.com \
--cc=linan122@huawei.com \
--cc=linux-raid@vger.kernel.org \
--cc=ncroxon@redhat.com \
--cc=song@kernel.org \
--cc=stable@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox