public inbox for linux-raid@vger.kernel.org
 help / color / mirror / Atom feed
From: Johannes Truschnigg <johannes@truschnigg.info>
To: eyal@eyal.emu.id.au
Cc: linux-raid@vger.kernel.org
Subject: Re: extremely slow writes to array [now not degraded]
Date: Mon, 13 Nov 2023 10:20:46 +0100	[thread overview]
Message-ID: <ZVHqbnFRU4bXBDNQ@vault.lan> (raw)
In-Reply-To: <182d07c8-a76e-430d-90e8-6df8c1f1c54d@eyal.emu.id.au>

[-- Attachment #1: Type: text/plain, Size: 1822 bytes --]

Interesting data; thanks for providing it. Unfortunately, I am not familiar
with that part of kernel code at all, but there's two observations that I can
contribute:

According to kernel source, `ext4_mb_scan_aligned` is a "special case for
storages like raid5", where "we try to find stripe-aligned chunks for
stripe-size-multiple requests" - and it seems that on your system, it might be
trying a tad too hard. I don't have a kernel source tree handy right now to
take a look at what might have changed in the function and any of its
calle[er]s during recent times, but it's the first place I'd go take a closer
look at.

Also, there's a recent Kernel bugzilla entry[0] that observes a similarly
pathological behavior from ext4 on a single disk of spinning rust where that
particular function appears in the call stack, and which revolves around an
mkfs-time-enabled feature which will, afaik, happen to also be set if
mke2fs(8) detects md RAID in the storage stack beneath the device it is
supposed to format (and which SHOULD get set, esp. for parity-based RAID).

Chances are you may be able to disable this particular optimization by running
`tune2fs -E stride=0` against the filesystem's backing array (be warned that I
did NOT verify if that might screw your data, which it very well could!!) and
remounting it afterwards, to check if that is indeed (part of) the underlying
cause to the poor performance you see. If you choose to try that, make sure to
record the current stride-size, so you may re-apply it at a later time
(`tune2fs -l` should do).

[0]: https://bugzilla.kernel.org/show_bug.cgi?id=217965

-- 
with best regards:
- Johannes Truschnigg ( johannes@truschnigg.info )

www:   https://johannes.truschnigg.info/
phone: +436502133337
xmpp:  johannes@truschnigg.info

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

  reply	other threads:[~2023-11-13  9:20 UTC|newest]

Thread overview: 16+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2023-11-05 23:06 extremely slow writes to degraded array eyal
2023-11-07 21:14 ` Peter Grandi
2023-11-08  1:37   ` eyal
2023-11-08  5:32     ` eyal
2023-11-08 16:50       ` Peter Grandi
2023-11-09  2:59 ` extremely slow writes to array [now not degraded] eyal
2023-11-09 13:16   ` Roger Heflin
2023-11-13  0:53     ` eyal
2023-11-13  6:54       ` Johannes Truschnigg
2023-11-13  8:06         ` eyal
2023-11-13  9:20           ` Johannes Truschnigg [this message]
2023-11-13  9:36             ` eyal
2023-11-13  9:58               ` eyal
2023-11-13 10:09                 ` Johannes Truschnigg
2023-11-13 10:31                   ` eyal
2023-11-13 12:26               ` Roger Heflin

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=ZVHqbnFRU4bXBDNQ@vault.lan \
    --to=johannes@truschnigg.info \
    --cc=eyal@eyal.emu.id.au \
    --cc=linux-raid@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox