From: Keith Busch <kbusch@kernel.org>
To: Mikhail Gavrilov <mikhail.v.gavrilov@gmail.com>
Cc: Christoph Hellwig <hch@lst.de>,
axboe@kernel.dk, linux-nvme@lists.infradead.org,
Linux List Kernel Mailing <linux-kernel@vger.kernel.org>,
Linux regressions mailing list <regressions@lists.linux.dev>
Subject: Re: 6.13/regression/bisected - new nvme timeout errors
Date: Thu, 6 Mar 2025 08:19:29 -0700 [thread overview]
Message-ID: <Z8m9AbD3tjNpBt6p@kbusch-mbp> (raw)
In-Reply-To: <CABXGCsMcxu3pCF8jYPeqF_jN34saBwc8Fci+c-Dg2Lh7rqvuFQ@mail.gmail.com>
On Wed, Jan 15, 2025 at 02:58:04AM +0500, Mikhail Gavrilov wrote:
> Hi,
> During 6.13 development cycle I spotted strange new nvme errors in the
> log which I never seen before.
>
> [87774.010474] nvme nvme1: I/O tag 0 (3000) opcode 0x1 (I/O Cmd) QID 1 timeout, aborting req_op:WRITE(1) size:131072
...
> I still haven't found a stable way to reproduce this.
> But I'm pretty sure that if this error don't appearing within two
> days, then we can assume that the kernel isn't affected by the
> problem.
> So I made bisection with above assumption and found this commit:
>
> beadf0088501d9dcf2454b05d90d5d31ea3ba55f is the first bad commit
> commit beadf0088501d9dcf2454b05d90d5d31ea3ba55f
> Author: Christoph Hellwig <hch@lst.de>
> Date: Wed Nov 13 16:20:41 2024 +0100
>
> nvme-pci: reverse request order in nvme_queue_rqs
The patch here uses the order recieved to dispatch commands in
consequetive submission queue entries, which is supposed to be the
desired behavior for any device. I did some testing on mailine, and it
sure looks like the order the driver does this is optimal, so I'm not
sure what's going on with your observation.
Do you have a scheduler enabled on your device?
How are you generating IO? Is it a pattern I should be able to replicate
with 'fio'?
next prev parent reply other threads:[~2025-03-06 15:19 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2025-01-14 21:58 6.13/regression/bisected - new nvme timeout errors Mikhail Gavrilov
2025-02-10 8:47 ` Mikhail Gavrilov
2025-03-05 22:44 ` Mikhail Gavrilov
2025-03-06 15:19 ` Keith Busch [this message]
2025-03-23 9:49 ` Mikhail Gavrilov
2025-03-24 10:48 ` Maurizio Lombardi
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=Z8m9AbD3tjNpBt6p@kbusch-mbp \
--to=kbusch@kernel.org \
--cc=axboe@kernel.dk \
--cc=hch@lst.de \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-nvme@lists.infradead.org \
--cc=mikhail.v.gavrilov@gmail.com \
--cc=regressions@lists.linux.dev \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox