Linux RAID subsystem development
 help / color / mirror / Atom feed
From: Yunye Zhao <yunye.zhao@linux.alibaba.com>
To: Song Liu <song@kernel.org>, Yu Kuai <yukuai@fygo.io>
Cc: Li Nan <magiclinan@didiglobal.com>, Xiao Ni <xiao@kernel.org>,
	Joseph Qi <joseph.qi@linux.alibaba.com>,
	linux-raid@vger.kernel.org, linux-kernel@vger.kernel.org,
	Yunye Zhao <yunye.zhao@linux.alibaba.com>
Subject: [PATCH v2 0/3] md/raid10: fix recovery corruption and soft lockup on large arrays
Date: Thu, 23 Jul 2026 21:55:32 +0800	[thread overview]
Message-ID: <20260723135535.101995-1-yunye.zhao@linux.alibaba.com> (raw)
In-Reply-To: <20260717062743.128189-1-yunye.zhao@linux.alibaba.com>

This series fixes two independent problems in raid10 recovery, plus a
safety net in md core.

Problem 1: silent data corruption when recovering with more than one
mirror missing.

Commit fe6a19d40ceb ("md/md-bitmap: merge md_bitmap_start_sync() into
bitmap_operations") inverted still_degraded in raid10_sync_request()
(int 1 -> bool false; the raid1 side of the same conversion is correct).
Recovering one disk while another is still missing therefore clears
bitmap bits the missing disk still needs; after its re-add the dirty
regions are skipped and the disk is marked In_sync with stale data.
Patch 1 restores the correct value.

Problem 2: soft lockup / CPU burn on recovery of a large sparse array
(the original report behind v1).

Recovering a degraded RAID10 with a mostly-clean bitmap makes
raid10_sync_request() take the "everything skipped" path, which returns
only 128 sectors per call even though the bitmap already reported a much
larger clean span.  md_do_sync() then crawls across the whole device in
128-sector steps with no reschedule; on a 2^40-sector array that is 2^33
no-op iterations:

  watchdog: BUG: soft lockup - CPU#149 stuck for 313s! [mdX_resync]
   md_bitmap_start_sync+0x6f/0xe0
   raid10_sync_request+0x2c9/0x1530 [raid10]
   md_do_sync+0x810/0x1030
   md_thread+0xa7/0x150

Patch 2 adds a cond_resched() to md_do_sync()'s skip path (safety net,
any personality/layout).  Patch 3 fixes the root cause for near layouts:
convert the bitmap-reported clean span from array to per-device sectors
and skip it in one step.  The conversion cannot be done generically in
the bitmap layer because the recovery cursor is in per-device space
(llbitmap's skip_sync_blocks() likewise returns 0 when degraded), and
resync paths already skip whole clean spans, so the fix lives in raid10.

Tested on QEMU/KVM on the latest mainline (raid10 near=2, 4 disks,
internal bitmap):

 - data consistency: write, fail one disk of each pair, write 256MiB
   while degraded, re-add both, wait for recovery, run "check":
   mismatch_cnt=262272 and changed file hashes before patch 1,
   mismatch_cnt=0 and identical hashes after.
 - recovery of two re-added disks on a sparse 8T-per-disk array: one
   37.2s recovery sweep of 100% CPU spinning before, 6.4 ms and 12.8 ms
   for the two recovery passes after patch 3 (dmesg "recover of RAID
   array" -> "recover done": 44.886958 -> 44.893378 and
   44.923957 -> 44.936716); no soft lockup with softlockup_panic=1.

Changes since v1 (a single cond_resched patch):
 - patch 1 (new): fix the still_degraded inversion, a silent data
   corruption found while testing.
 - patch 2 (was v1): rewrite the changelog and retitle; the v1 claim
   that "io_sectors never increases" was wrong (Yu Kuai).
 - patch 3 (new): fix the 128-sector recovery crawl at the source
   instead of only silencing the watchdog.

v1: https://lore.kernel.org/linux-raid/20260717062743.128189-1-yunye.zhao@linux.alibaba.com/

Yunye Zhao (3):
  md/raid10: fix still_degraded being inverted in raid10_sync_request()
  md: add cond_resched() to md_do_sync()'s skip path
  md/raid10: skip clean regions in bulk during recovery

 drivers/md/md.c     |  4 ++-
 drivers/md/raid10.c | 57 ++++++++++++++++++++++++++++++++++++++++++---
 2 files changed, 56 insertions(+), 5 deletions(-)


base-commit: 4539944e515183668109bdf4d0c3d7d228383d88
-- 
2.19.1.6.gb485710b


  parent reply	other threads:[~2026-07-23 13:55 UTC|newest]

Thread overview: 11+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-07-17  6:27 [PATCH] md: fix soft lockup during resync when sync is repeatedly skipped Yunye Zhao
2026-07-17  6:51 ` sashiko-bot
2026-07-19 10:45 ` yu kuai
2026-07-20  6:20   ` Yunye Zhao
2026-07-22  4:00     ` yu kuai
2026-07-23 13:55 ` Yunye Zhao [this message]
2026-07-23 13:55   ` [PATCH v2 1/3] md/raid10: fix still_degraded being inverted in raid10_sync_request() Yunye Zhao
2026-07-23 13:55   ` [PATCH v2 2/3] md: add cond_resched() to md_do_sync()'s skip path Yunye Zhao
2026-07-23 14:05     ` sashiko-bot
2026-07-23 13:55   ` [PATCH v2 3/3] md/raid10: skip clean regions in bulk during recovery Yunye Zhao
2026-07-23 14:16     ` sashiko-bot

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20260723135535.101995-1-yunye.zhao@linux.alibaba.com \
    --to=yunye.zhao@linux.alibaba.com \
    --cc=joseph.qi@linux.alibaba.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-raid@vger.kernel.org \
    --cc=magiclinan@didiglobal.com \
    --cc=song@kernel.org \
    --cc=xiao@kernel.org \
    --cc=yukuai@fygo.io \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox