From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from bsdbackstore.eu (128-116-240-228.dyn.eolo.it [128.116.240.228]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1ADB625DAF2 for ; Mon, 24 Mar 2025 11:15:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=128.116.240.228 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742814930; cv=none; b=aMI+jvXfYqoXFOjvxjbHgkgdRzqalYQL2F3wZdUN/vsU2FvoZ2nEAqO6sIMDVJtcCc/I84SUoHX+QE3FbIm7BXLAj0kOZdx4EYvA2OmpnBmmhK4SBdZn810p3sS+bvG0G3wDatxRRxBHRP8mUcwSenT8X2UDF6RdawAfrPYRSt0= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742814930; c=relaxed/simple; bh=P7kczrFvAEkecebYg+0SctMa4bfEcRb/otiqw2URygw=; h=Mime-Version:Content-Type:Date:Message-Id:Cc:Subject:From:To: References:In-Reply-To; b=ZrLmr4Al61ObqeowhISAeooDUKa5GQxLtBTesRdaZTxdr/B5yMeD2uwNYxia/oasah+nM6+pxwGS7jPb19rAxiRQmTUa3TLpYiU3KOKuZzWGikWg738+dkwLht5c2L6OfnP4in7oXDvf9cgHQWV3LOoSExqlxmG6OBwU1q9Eu6Y= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=bsdbackstore.eu; spf=pass smtp.mailfrom=bsdbackstore.eu; arc=none smtp.client-ip=128.116.240.228 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=bsdbackstore.eu Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bsdbackstore.eu Received: from localhost (25.205.forpsi.net [80.211.205.25]) by bsdbackstore.eu (OpenSMTPD) with ESMTPSA id 59e3081f (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Mon, 24 Mar 2025 11:48:40 +0100 (CET) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=UTF-8 Date: Mon, 24 Mar 2025 11:48:39 +0100 Message-Id: Cc: "Christoph Hellwig" , , , "Linux List Kernel Mailing" , "Linux regressions mailing list" Subject: Re: 6.13/regression/bisected - new nvme timeout errors From: "Maurizio Lombardi" To: "Mikhail Gavrilov" , "Keith Busch" X-Mailer: aerc References: In-Reply-To: On Sun Mar 23, 2025 at 10:49 AM CET, Mikhail Gavrilov wrote: > On Thu, Mar 6, 2025 at 8:19=E2=80=AFPM Keith Busch wr= ote: >> >> > >> > nvme-pci: reverse request order in nvme_queue_rqs >> >> The patch here uses the order recieved to dispatch commands in >> consequetive submission queue entries, which is supposed to be the >> desired behavior for any device. I did some testing on mailine, and it >> sure looks like the order the driver does this is optimal, so I'm not >> sure what's going on with your observation. >> >> Do you have a scheduler enabled on your device? >> >> How are you generating IO? Is it a pattern I should be able to replicate >> with 'fio'? > > Sorry for not answering for a long time, but all because I was testing > the off-list patch that was sent to me by Maurizio Lombardi. > > diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c > index 950289405ef2..990d9ef1eef9 100644 > --- a/drivers/nvme/host/pci.c > +++ b/drivers/nvme/host/pci.c > @@ -989,6 +989,9 @@ static void nvme_submit_cmds(struct nvme_queue > *nvmeq, struct rq_list *rqlist) > { > struct request *req; > > + if (rq_list_empty(rqlist)) > + return; > + > spin_lock(&nvmeq->sq_lock); > while ((req =3D rq_list_pop(rqlist))) { > struct nvme_iod *iod =3D blk_mq_rq_to_pdu(req); > > And I want to say right away that this patch solved my problem. I > double checked, and both times the system worked for 4 days without > errors: > Glad to hear it worked! Keith, I think the problem here is simply that nvme_write_sq_db() ends up being called even when the rqlist is empty and this couldn't happen before the changes introduced by commit "nvme-pci: reverse request order in nvme_queue_rqs". If you say that this change makes sense I can submit a formal patch. Maurizio