[REGRESSION] md/raid5: discards on an array of non-TRIM HDDs now fail with BLK_STS_NOTSUPP; bcache counts them as backing-device errors and stops the device

[REGRESSION] md/raid5: discards on an array of non-TRIM HDDs now fail with BLK_STS_NOTSUPP; bcache counts them as backing-device errors and stops the device

[Konstantin Nikiforov]

#regzbot introduced: 90573092673c

Hi,

since upgrading from 7.1.8 to 7.2.11 a machine with root on
LVM -> bcache -> md raid5 (three SATA HDDs without TRIM) loses its root
device a few minutes after boot. The same stack ran on 7.1.x without
any bcache backing-device errors.

Storage stack
-------------
  3x ST1000DM010-2EP102 (no TRIM; discard_max_hw_bytes = 0)
   -> md126: raid5, 3 devices, 512K chunk, metadata 1.2
   -> bcache0: md126 as backing device, cache set detached
      (also reproduced while an NVMe cache was attached)
   -> LVM vgpcrd5
   -> btrfs (/usr, /var, /opt; mounted without any discard option)
      + ext4 (/, /home)

Kernel: 7.2.11-arch1-1 (Arch Linux), mdadm 4.6, bcache-tools 1.1

Symptoms
--------
Every boot, starting ~3 minutes after mount, dmesg fills with:

  bcache: bch_count_backing_io_errors() md126: IO error on backing
device, unrecoverable

in bursts of up to ~35 lines 2 ms apart. Per boot I counted 11, 17, 22
and 63 of them. bcache's io_error_limit is 64, so once a boot crosses it
bch_cached_dev_error() stops bcache0 and root, /usr, /var and /home
disappear. No I/O errors are logged by libata, sd or md; all three
members are healthy (SMART clean, array [UUU], no rebuild).

Analysis
--------
Since 90573092673c ("md/raid5: validate discard support at request
time", v7.2-rc1) raid5_set_limits() unconditionally sets

  lim.max_hw_discard_sectors = UINT_MAX;

and records member support in conf->raid5_discard_unsupported. On this
array sysfs now shows:

  /sys/block/md126/queue/discard_max_hw_bytes: 2199023255040
  /sys/block/md126/queue/discard_max_bytes:    2199023255040
  /sys/block/md126/queue/discard_granularity:  0
  /sys/block/sd[abc]/queue/discard_max_hw_bytes: 0
  (md127, a raid1 on the same disks, still shows 0 for all of these)

Two upper layers react to the advertised discard support:

1. btrfs enables discard=async by default because the device advertises
   discard (mount shows "ssd,discard=async" although nothing in fstab
   asks for it).

2. bcache forwards discards to the backing device whenever
   bdev_max_discard_sectors(dc->bdev) is non-zero (cached_dev_write()
   and detached_dev_do_request()). Before 7.2 that was 0 for this array
   and bcache completed the discards itself.

raid5_discard_limits() then completes every such bio with
BLK_STS_NOTSUPP. bcache's backing_request_endio() treats any non-zero
bi_status as a media error and calls bch_count_backing_io_errors(), so
each discard brings the backing device one step closer to being stopped.
With 64 small deletions on the btrfs volume a root filesystem on
raid5+bcache is gone.

The granularity is also inconsistent: discard_granularity is 0 while
discard_max_hw_bytes is non-zero, so writing any non-zero value to
discard_max_bytes fails with EINVAL (only 0 is accepted).

Related changes I found:
- efdffeb6d491 ("md: skip discard on unsupported member devices", 7.3)
  fixes the same bcache symptom for raid0/raid10 via
  md_submit_discard_bio(), but raid5 does not use that helper and still
  returns BLK_STS_NOTSUPP in current mainline (raid5_discard_limits()).
- 85764f475f3b ("md/raid5: complete discard bios while reshape is
  active") addresses a different fallout of the same commit (hang during
  reshape).

I have not found a report for this raid5 + bcache case on lore.

Possible fixes (either side would do):
- md/raid5: when conf->raid5_discard_unsupported is set, complete the
  bio successfully (discard is advisory) instead of BLK_STS_NOTSUPP, or
  do not advertise max_hw_discard_sectors in that case, as before 7.2.
- bcache: do not count BLK_STS_NOTSUPP (or failed REQ_OP_DISCARD) in
  bch_count_backing_io_errors(), the same way REQ_RAHEAD failures are
  already ignored.

Workaround in use
-----------------
A udev rule writing 0 to /sys/block/md126/queue/discard_max_bytes and
/sys/block/bcache0/queue/discard_max_bytes (also inside the initramfs),
plus "nodiscard" on the btrfs mounts. With that, io_errors stays at 0.

I can test patches on this machine.

Regards,
Konstantin Nikiforov
Claude Fable 5.1

Re: [REGRESSION] md/raid5: discards on an array of non-TRIM HDDs now fail with BLK_STS_NOTSUPP; bcache counts them as backing-device errors and stops the device

[yu kuai]

Hi,

在 2026/10/25 20:38, Konstantin Nikiforov 写道:
> #regzbot introduced: 90573092673c
>
> Hi,
>
> since upgrading from 7.1.8 to 7.2.11 a machine with root on
> LVM -> bcache -> md raid5 (three SATA HDDs without TRIM) loses its root
> device a few minutes after boot. The same stack ran on 7.1.x without
> any bcache backing-device errors.
>
> Storage stack
> -------------
>    3x ST1000DM010-2EP102 (no TRIM; discard_max_hw_bytes = 0)
>     -> md126: raid5, 3 devices, 512K chunk, metadata 1.2
>     -> bcache0: md126 as backing device, cache set detached
>        (also reproduced while an NVMe cache was attached)
>     -> LVM vgpcrd5
>     -> btrfs (/usr, /var, /opt; mounted without any discard option)
>        + ext4 (/, /home)
>
> Kernel: 7.2.11-arch1-1 (Arch Linux), mdadm 4.6, bcache-tools 1.1
>
> Symptoms
> --------
> Every boot, starting ~3 minutes after mount, dmesg fills with:
>
>    bcache: bch_count_backing_io_errors() md126: IO error on backing
> device, unrecoverable
>
> in bursts of up to ~35 lines 2 ms apart. Per boot I counted 11, 17, 22
> and 63 of them. bcache's io_error_limit is 64, so once a boot crosses it
> bch_cached_dev_error() stops bcache0 and root, /usr, /var and /home
> disappear. No I/O errors are logged by libata, sd or md; all three
> members are healthy (SMART clean, array [UUU], no rebuild).
>
> Analysis
> --------
> Since 90573092673c ("md/raid5: validate discard support at request
> time", v7.2-rc1) raid5_set_limits() unconditionally sets
>
>    lim.max_hw_discard_sectors = UINT_MAX;
>
> and records member support in conf->raid5_discard_unsupported. On this
> array sysfs now shows:
>
>    /sys/block/md126/queue/discard_max_hw_bytes: 2199023255040
>    /sys/block/md126/queue/discard_max_bytes:    2199023255040
>    /sys/block/md126/queue/discard_granularity:  0
>    /sys/block/sd[abc]/queue/discard_max_hw_bytes: 0
>    (md127, a raid1 on the same disks, still shows 0 for all of these)
>
> Two upper layers react to the advertised discard support:
>
> 1. btrfs enables discard=async by default because the device advertises
>     discard (mount shows "ssd,discard=async" although nothing in fstab
>     asks for it).
>
> 2. bcache forwards discards to the backing device whenever
>     bdev_max_discard_sectors(dc->bdev) is non-zero (cached_dev_write()
>     and detached_dev_do_request()). Before 7.2 that was 0 for this array
>     and bcache completed the discards itself.
>
> raid5_discard_limits() then completes every such bio with
> BLK_STS_NOTSUPP. bcache's backing_request_endio() treats any non-zero
> bi_status as a media error and calls bch_count_backing_io_errors(), so
> each discard brings the backing device one step closer to being stopped.
> With 64 small deletions on the btrfs volume a root filesystem on
> raid5+bcache is gone.
>
> The granularity is also inconsistent: discard_granularity is 0 while
> discard_max_hw_bytes is non-zero, so writing any non-zero value to
> discard_max_bytes fails with EINVAL (only 0 is accepted).
>
> Related changes I found:
> - efdffeb6d491 ("md: skip discard on unsupported member devices", 7.3)
>    fixes the same bcache symptom for raid0/raid10 via
>    md_submit_discard_bio(), but raid5 does not use that helper and still
>    returns BLK_STS_NOTSUPP in current mainline (raid5_discard_limits()).
> - 85764f475f3b ("md/raid5: complete discard bios while reshape is
>    active") addresses a different fallout of the same commit (hang during
>    reshape).
>
> I have not found a report for this raid5 + bcache case on lore.
>
> Possible fixes (either side would do):
> - md/raid5: when conf->raid5_discard_unsupported is set, complete the
>    bio successfully (discard is advisory) instead of BLK_STS_NOTSUPP, or
>    do not advertise max_hw_discard_sectors in that case, as before 7.2.

I think it's fine to complete the bio successfully in this case, the same
as the old code did. I can cook a patch, however, if you like, feel free to
test and send a patch by yourself :)

> - bcache: do not count BLK_STS_NOTSUPP (or failed REQ_OP_DISCARD) in
>    bch_count_backing_io_errors(), the same way REQ_RAHEAD failures are
>    already ignored.
>
> Workaround in use
> -----------------
> A udev rule writing 0 to /sys/block/md126/queue/discard_max_bytes and
> /sys/block/bcache0/queue/discard_max_bytes (also inside the initramfs),
> plus "nodiscard" on the btrfs mounts. With that, io_errors stays at 0.
>
> I can test patches on this machine.
>
> Regards,
> Konstantin Nikiforov
> Claude Fable 5.1

-- 
Thanks,
Kuai

[PATCH] md/raid5: complete unsupported discards successfully

[Konstantin Nikiforov]

Since commit 90573092673c ("md/raid5: validate discard support at
request time") raid5 advertises max_hw_discard_sectors = UINT_MAX no
matter whether the member devices support discard or
devices_handle_discard_safely is set, and make_discard_request() fails
the discards it cannot handle with BLK_STS_NOTSUPP.

Before that commit such arrays did not advertise discard at all, so
upper layers never sent one. Now they do, and the failure is not
harmless:

 - bcache forwards discards to its backing device whenever
   bdev_max_discard_sectors() is non-zero, and backing_request_endio()
   treats every non-zero bi_status as a media error. After io_error_limit
   (64) of them bch_cached_dev_error() stops the cached device. On a
   machine with root on LVM over bcache over a raid5 of three HDDs the
   root filesystem disappeared a few minutes after every boot.

 - btrfs enables discard=async by default on any device that advertises
   discard, which is what generated the discards in the first place.

Discard is advisory: blkdev_issue_discard() already turns -EOPNOTSUPP
into success for its callers, and md_submit_discard_bio() silently skips
raid0/raid10 members without discard support. Do the same here and
complete the bio successfully when the array cannot pass it down safely,
so that callers submitting their own discard bios, like bcache, see the
same behaviour.

Fixes: 90573092673c ("md/raid5: validate discard support at request time")
Cc: stable@vger.kernel.org # 7.2
Closes: https://lore.kernel.org/all/CAO96y0V6oncv54ihXnHfJMYiG_QAQ-NM8BOdqPDXKL8r8-kR9Q@mail.gmail.com/
Signed-off-by: Konstantin Nikiforov <helllamer@gmail.com>
---
Kuai, this is the raid5-side change you were fine with in the report
thread. Reproducer: raid5 over three loop devices with a bcache backing
device on top, four blkdiscard calls on the bcache device. On 7.2.11
without the patch bcache's io_errors goes to 4 (bch_count_backing_io_errors
in dmesg); with the patch applied to 7.2.11 it stays at 0, and the
affected machine (root on LVM over bcache over raid5 of HDDs) boots and
runs with the patched module without any backing-device errors.
blkdiscard on the md device itself reports success either way because
blkdev_issue_discard() masks EOPNOTSUPP.

One correction to my report: the "discard_granularity is 0" observation
was a side effect of my own workaround (writing 0 to discard_max_bytes
makes the block layer report granularity 0); on a fresh array the
granularity is the stripe size as expected.

 drivers/md/raid5.c | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
index b91545ce090d..b462b18e47ff 100644
--- a/drivers/md/raid5.c
+++ b/drivers/md/raid5.c
@@ -1144,7 +1144,16 @@ static bool raid5_discard_limits(struct mddev *mddev, struct bio *bi)
 	if (!conf->raid5_discard_unsupported)
 		return true;
 
-	bi->bi_status = BLK_STS_NOTSUPP;
+	/*
+	 * Discard is advisory. The array advertises discard regardless of
+	 * member support, so an upper layer may legitimately send discards
+	 * that cannot be forwarded safely. Complete them successfully
+	 * instead of failing them: bcache counts every failed backing
+	 * device bio as an I/O error and stops the device after
+	 * io_error_limit of them, and btrfs enables discard=async on any
+	 * device that advertises discard. This also matches what
+	 * md_submit_discard_bio() does for members without discard support.
+	 */
 	bio_endio(bi);
 	return false;
 }

base-commit: 6b4b543fa18757297ac756a6fe6e64f506b7844a
-- 
2.56.0