Linux RAID subsystem development
 help / color / mirror / Atom feed
From: Yunye Zhao <yunye.zhao@linux.alibaba.com>
To: Yu Kuai <yukuai@fygo.io>, Song Liu <song@kernel.org>
Cc: Yunye Zhao <yunye.zhao@linux.alibaba.com>,
	Li Nan <magiclinan@didiglobal.com>, Xiao Ni <xiao@kernel.org>,
	Joseph Qi <joseph.qi@linux.alibaba.com>,
	linux-raid@vger.kernel.org, linux-kernel@vger.kernel.org
Subject: Re: [PATCH] md: fix soft lockup during resync when sync is repeatedly skipped
Date: Mon, 20 Jul 2026 14:20:26 +0800	[thread overview]
Message-ID: <20260720062026.19245-1-yunye.zhao@linux.alibaba.com> (raw)
In-Reply-To: <7abeaff6-1387-4046-bea3-f1b04465addd@fygo.io>

Hi Kuai,

On 2026/7/17 14:27, Yu Kuai wrote:
>> md_do_sync()'s main loop advances io_sectors only when I/O is actually
>> issued (skipped == 0).  When sync_request() keeps returning skipped == 1,
>> io_sectors never increases, [...]
>
> That's not expected, io_sectors should always increase in the skip case.

Sorry, my description was not accurate. The problem is not that io_sectors
stays 0. md_do_sync()'s loop exit condition is j == max_sectors, and in
raid10_sync_request()'s recovery path sectors is always 128, so for a very
large max_sectors the loop iterates a huge number of times.

raid10_sync_request(), recovery branch:

    max_sync = RESYNC_PAGES << (PAGE_SHIFT-9);      /* 128 */
    must_sync = md_bitmap_start_sync(mddev, sect, &sync_blocks, true);
    if (sync_blocks < max_sync)   /* sync_blocks huge, never true */
        max_sync = sync_blocks;   /* so max_sync stays 128 */
    ...
    if (biolist == NULL) {
        *skipped = 1;
        return max_sync;          /* 128; the large span is discarded */
    }

Back in md_do_sync()'s main loop, j then crawls forward 128 sectors per
call until it reaches max_sectors:

    while (j < max_sectors) {
        sectors = mddev->pers->sync_request(mddev, j, max_sectors, &skipped);
        if (!skipped)             /* skipped == 1 -> io_sectors stays 0 */
            io_sectors += sectors;
        j += sectors;             /* j += 128 only */
        if (last_check + window > io_sectors || j == max_sectors)
            continue;
    }

From the vmcore:

    sync_blocks returned = 0x5ED03680 (~1.59e8 sectors), clamped to 128
    recovery max_sectors = dev_sectors = 2^40
    iterations = 2^40 / 128 = 2^33 (~8.6e9)

> What kernel version you're testing?

6.6.102. I also tested mainline and hit the same problem.

> If this is latest kernel, bitmap_start_sync() need to be fixed. It can't
> return skip while setting skipping sectors to 0.

bitmap_start_sync() is actually fine -- it returns a large clean span
(0x5ED03680 above). The recovery path just discards it: max_sync is only
ever clamped down, so it returns 128 regardless.

> And since this is dead loop, a cond_resched() will not fix anything.

Agreed -- cond_resched() only stops the watchdog; the thread still spins
~8.6e9 no-op iterations and pins a CPU for ~313s.

For v2 I'd fix this at the source: make the recovery path honour the clean
span reported by the bitmap instead of capping the skip at 128. Does that
direction look right to you?

Thanks,
Yunye

  reply	other threads:[~2026-07-20  6:20 UTC|newest]

Thread overview: 11+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-07-17  6:27 [PATCH] md: fix soft lockup during resync when sync is repeatedly skipped Yunye Zhao
2026-07-17  6:51 ` sashiko-bot
2026-07-19 10:45 ` yu kuai
2026-07-20  6:20   ` Yunye Zhao [this message]
2026-07-22  4:00     ` yu kuai
2026-07-23 13:55 ` [PATCH v2 0/3] md/raid10: fix recovery corruption and soft lockup on large arrays Yunye Zhao
2026-07-23 13:55   ` [PATCH v2 1/3] md/raid10: fix still_degraded being inverted in raid10_sync_request() Yunye Zhao
2026-07-23 13:55   ` [PATCH v2 2/3] md: add cond_resched() to md_do_sync()'s skip path Yunye Zhao
2026-07-23 14:05     ` sashiko-bot
2026-07-23 13:55   ` [PATCH v2 3/3] md/raid10: skip clean regions in bulk during recovery Yunye Zhao
2026-07-23 14:16     ` sashiko-bot

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20260720062026.19245-1-yunye.zhao@linux.alibaba.com \
    --to=yunye.zhao@linux.alibaba.com \
    --cc=joseph.qi@linux.alibaba.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-raid@vger.kernel.org \
    --cc=magiclinan@didiglobal.com \
    --cc=song@kernel.org \
    --cc=xiao@kernel.org \
    --cc=yukuai@fygo.io \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox