From: Dave Chinner <david@fromorbit.com>
To: Anton Gavriliuk <antosha20xx@gmail.com>
Cc: Laurence Oberman <loberman@redhat.com>,
linux-nvme@lists.infradead.org, linux-xfs@vger.kernel.org,
linux-block@vger.kernel.org
Subject: Re: Sequential read from NVMe/XFS twice slower on Fedora 42 than on Rocky 9.5
Date: Thu, 8 May 2025 07:59:28 +1000 [thread overview]
Message-ID: <aBvXwComvV-uAqmb@dread.disaster.area> (raw)
In-Reply-To: <CAAiJnjp6WuVaxXbjndF8dB3fuWCuWz7Nqzpz0uEu2BOqyZUQHg@mail.gmail.com>
On Wed, May 07, 2025 at 03:26:08PM +0300, Anton Gavriliuk wrote:
> > `iostat -dxm 5` output during the fio run on both kernels will give us some indication of the differences in IO patterns, queue depths, etc.
>
> iostat files attached.
Yeah, that definitely looks like MD is the bottleneck. In both
traces the NVMe drives are completing read IOs in about 110-120us.
In fedora 42, the nvme drives are not at 100% utilisation so the md
device is not feeding them fast enough.
That can also be seen in that the rocky 9.5 kernel with a nvme
device queue depth of about 4 IOs, whilst it is only 1.5 for the
fedora 42 kernel.
Given that nobody from the block/MD side of things has responded
with any ideas yet, you might just have to bisect it to find out
where things went wrong...
-Dave.
--
Dave Chinner
david@fromorbit.com
prev parent reply other threads:[~2025-05-07 21:59 UTC|newest]
Thread overview: 14+ messages / expand[flat|nested] mbox.gz Atom feed top
2025-05-03 21:04 Sequential read from NVMe/XFS twice slower on Fedora 42 than on Rocky 9.5 Anton Gavriliuk
2025-05-03 22:16 ` Dave Chinner
2025-05-04 7:22 ` Anton Gavriliuk
2025-05-04 21:50 ` Dave Chinner
2025-05-05 12:29 ` Laurence Oberman
2025-05-05 13:21 ` Laurence Oberman
2025-05-05 17:39 ` Laurence Oberman
2025-05-22 15:07 ` Laurence Oberman
2025-05-23 9:39 ` Anton Gavriliuk
2025-05-05 22:56 ` Dave Chinner
2025-05-06 11:03 ` Anton Gavriliuk
2025-05-06 21:46 ` Dave Chinner
2025-05-07 12:26 ` Anton Gavriliuk
2025-05-07 21:59 ` Dave Chinner [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=aBvXwComvV-uAqmb@dread.disaster.area \
--to=david@fromorbit.com \
--cc=antosha20xx@gmail.com \
--cc=linux-block@vger.kernel.org \
--cc=linux-nvme@lists.infradead.org \
--cc=linux-xfs@vger.kernel.org \
--cc=loberman@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.