From: Yunye Zhao <yunye.zhao@linux.alibaba.com>
To: Song Liu <song@kernel.org>, Yu Kuai <yukuai@fygo.io>
Cc: Li Nan <magiclinan@didiglobal.com>, Xiao Ni <xiao@kernel.org>,
Joseph Qi <joseph.qi@linux.alibaba.com>,
linux-raid@vger.kernel.org, linux-kernel@vger.kernel.org,
Yunye Zhao <yunye.zhao@linux.alibaba.com>
Subject: [PATCH v2 0/3] md/raid10: fix recovery corruption and soft lockup on large arrays
Date: Thu, 23 Jul 2026 21:55:32 +0800 [thread overview]
Message-ID: <20260723135535.101995-1-yunye.zhao@linux.alibaba.com> (raw)
In-Reply-To: <20260717062743.128189-1-yunye.zhao@linux.alibaba.com>
This series fixes two independent problems in raid10 recovery, plus a
safety net in md core.
Problem 1: silent data corruption when recovering with more than one
mirror missing.
Commit fe6a19d40ceb ("md/md-bitmap: merge md_bitmap_start_sync() into
bitmap_operations") inverted still_degraded in raid10_sync_request()
(int 1 -> bool false; the raid1 side of the same conversion is correct).
Recovering one disk while another is still missing therefore clears
bitmap bits the missing disk still needs; after its re-add the dirty
regions are skipped and the disk is marked In_sync with stale data.
Patch 1 restores the correct value.
Problem 2: soft lockup / CPU burn on recovery of a large sparse array
(the original report behind v1).
Recovering a degraded RAID10 with a mostly-clean bitmap makes
raid10_sync_request() take the "everything skipped" path, which returns
only 128 sectors per call even though the bitmap already reported a much
larger clean span. md_do_sync() then crawls across the whole device in
128-sector steps with no reschedule; on a 2^40-sector array that is 2^33
no-op iterations:
watchdog: BUG: soft lockup - CPU#149 stuck for 313s! [mdX_resync]
md_bitmap_start_sync+0x6f/0xe0
raid10_sync_request+0x2c9/0x1530 [raid10]
md_do_sync+0x810/0x1030
md_thread+0xa7/0x150
Patch 2 adds a cond_resched() to md_do_sync()'s skip path (safety net,
any personality/layout). Patch 3 fixes the root cause for near layouts:
convert the bitmap-reported clean span from array to per-device sectors
and skip it in one step. The conversion cannot be done generically in
the bitmap layer because the recovery cursor is in per-device space
(llbitmap's skip_sync_blocks() likewise returns 0 when degraded), and
resync paths already skip whole clean spans, so the fix lives in raid10.
Tested on QEMU/KVM on the latest mainline (raid10 near=2, 4 disks,
internal bitmap):
- data consistency: write, fail one disk of each pair, write 256MiB
while degraded, re-add both, wait for recovery, run "check":
mismatch_cnt=262272 and changed file hashes before patch 1,
mismatch_cnt=0 and identical hashes after.
- recovery of two re-added disks on a sparse 8T-per-disk array: one
37.2s recovery sweep of 100% CPU spinning before, 6.4 ms and 12.8 ms
for the two recovery passes after patch 3 (dmesg "recover of RAID
array" -> "recover done": 44.886958 -> 44.893378 and
44.923957 -> 44.936716); no soft lockup with softlockup_panic=1.
Changes since v1 (a single cond_resched patch):
- patch 1 (new): fix the still_degraded inversion, a silent data
corruption found while testing.
- patch 2 (was v1): rewrite the changelog and retitle; the v1 claim
that "io_sectors never increases" was wrong (Yu Kuai).
- patch 3 (new): fix the 128-sector recovery crawl at the source
instead of only silencing the watchdog.
v1: https://lore.kernel.org/linux-raid/20260717062743.128189-1-yunye.zhao@linux.alibaba.com/
Yunye Zhao (3):
md/raid10: fix still_degraded being inverted in raid10_sync_request()
md: add cond_resched() to md_do_sync()'s skip path
md/raid10: skip clean regions in bulk during recovery
drivers/md/md.c | 4 ++-
drivers/md/raid10.c | 57 ++++++++++++++++++++++++++++++++++++++++++---
2 files changed, 56 insertions(+), 5 deletions(-)
base-commit: 4539944e515183668109bdf4d0c3d7d228383d88
--
2.19.1.6.gb485710b
next prev parent reply other threads:[~2026-07-23 13:55 UTC|newest]
Thread overview: 11+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-07-17 6:27 [PATCH] md: fix soft lockup during resync when sync is repeatedly skipped Yunye Zhao
2026-07-17 6:51 ` sashiko-bot
2026-07-19 10:45 ` yu kuai
2026-07-20 6:20 ` Yunye Zhao
2026-07-22 4:00 ` yu kuai
2026-07-23 13:55 ` Yunye Zhao [this message]
2026-07-23 13:55 ` [PATCH v2 1/3] md/raid10: fix still_degraded being inverted in raid10_sync_request() Yunye Zhao
2026-07-23 13:55 ` [PATCH v2 2/3] md: add cond_resched() to md_do_sync()'s skip path Yunye Zhao
2026-07-23 14:05 ` sashiko-bot
2026-07-23 13:55 ` [PATCH v2 3/3] md/raid10: skip clean regions in bulk during recovery Yunye Zhao
2026-07-23 14:16 ` sashiko-bot
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20260723135535.101995-1-yunye.zhao@linux.alibaba.com \
--to=yunye.zhao@linux.alibaba.com \
--cc=joseph.qi@linux.alibaba.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-raid@vger.kernel.org \
--cc=magiclinan@didiglobal.com \
--cc=song@kernel.org \
--cc=xiao@kernel.org \
--cc=yukuai@fygo.io \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox