Re: [PATCH] nvme-pci: slimmer CQ head update

From: Will Deacon <will@kernel.org>
To: Keith Busch <kbusch@kernel.org>
Cc: sagi@grimberg.me, John Garry <john.garry@huawei.com>,
	linux-nvme@lists.infradead.org, Christoph Hellwig <hch@lst.de>,
	axboe@fb.com, Robin Murphy <robin.murphy@arm.com>,
	Alexey Dobriyan <adobriyan@gmail.com>
Subject: Re: [PATCH] nvme-pci: slimmer CQ head update
Date: Thu, 7 May 2020 18:44:29 +0100	[thread overview]
Message-ID: <20200507174429.GA4466@willie-the-truck> (raw)
In-Reply-To: <20200507173545.GC2621480@dhcp-10-100-145-180.wdl.wdc.com>

On Thu, May 07, 2020 at 10:35:45AM -0700, Keith Busch wrote:
> On Thu, May 07, 2020 at 05:26:58PM +0100, Robin Murphy wrote:
> > On 2020-05-07 4:35 pm, Keith Busch wrote:
> > > On Thu, May 07, 2020 at 04:11:23PM +0100, John Garry wrote:
> > > 
> > > > [  885.344575] WARNING: CPU: 41 PID: 4565 at block/blk-mq.c:665  blk_mq_start_request+0xc4/0xcc
> > > 
> > > This warning appears to support my suspicion: the completion side is
> > > observing a new phase with a stale command id, and that command id was
> > > reallocated as a new request that we're still constructing at the time
> > > the double-completion occured.
> > > 
> > > Host software is supposed to be guaranteed the entire CQE is written
> > > once we see an updated phase, per spec: "If a Completion Queue
> > > Entry is constructed via multiple writes, the Phase Tag bit shall be
> > > updated in the last write of that Completion Queue Entry."
> > 
> > Hmm, that makes me wonder if there might be some interaction with the Arm
> > memory model here - if there are strict ordering requirements for things in
> > memory being observed, could we be missing appropriate barriers between
> > reads/writes of the various fields?
> 
> Yeah, that might be something to consider. If that's the case, then it
> should be reproducible with a different vendor's nvme controller.
> 
> If the behavior is unique to this particular nvme model, then it would
> sound like the controller is creating a CQE in multiple memory writes
> either high-to-low, or with Relaxed Ordering enabled in the TLPs.

[disclaimer: I don't know anything about nvme, so this might be way off! I
know a bit about the Arm memory model though, so I can help with that side.]

Are you sure there's an ordering requirement? My reading of the quote
from the spec is that "last write" just means the final write, not
necessarily the word at the highest address.

> But if some other interaction with the arm memory model is suspect, I'm
> not sure how to confirm or debug.

From what you described, it sounds like the flow should go something like:

  <Read the word of the CQE containing the updated Phase Tag Bit>
  dma_rmb();
  <Read the other words of the CQE>

How easy is it to check if that sort of thing is being followed?

Will

_______________________________________________
linux-nvme mailing list
linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme