Re: [PATCHv2 2/2] nvme-pci: remove cached shadow doorbell offsets

[John Levon <levon@movementarian.org>]

On Thu, Oct 14, 2021 at 10:49:38AM -0700, Keith Busch wrote:

> On Thu, Oct 14, 2021 at 06:09:30PM +0100, John Levon wrote:
> > On Thu, Oct 14, 2021 at 09:45:43AM -0700, Keith Busch wrote:
> > 
> > > And when this feature is in use, the specification requires all queue
> > > updates use this mechanism, so don't don't treat the admin queue
> > > differently.
> > 
> > This would cause implementation difficulties for a controller, right?
> 
> If the controller is not maintaining the admin queue shadow, then the
> event index will always tell the host to ring the doorbell, right? So
> the same host behvaior would ultimately happen.

The eventidx only causes the additional BAR0 writes if the doorbell update
crosses it (for example, a 4->6 transition when host sees eventidx of 5). So we
couldn't just leave eventidx at zero, and neither can we maintain it to force
BAR0 writes, as there are races that means we'd have to look at the shadow
doorbell value - and make assumptions - anyway.

This would all be a lot easier if zero wasn't a valid doorbell value, but we're
stuck now.

> > Currently we presume that the admin queue's shadow doorbells aren't valid, so
> > always look at the BAR0 location. Given we need to support older Linux versions,
> > we don't have any non-hacky way to handle this: we'd need something horrible
> > that presumes those Linux versions - and any other driver implementations -
> > always leave the admin queue shadows at zero, so if we see a non-zero value, it
> > must be doing shadow doorbells.
> > 
> > It's unclear to me what the advantage of fixing this is, given Linux has been
> > out of spec for so long.
> 
> Ah, I skimmed the part that said we are not up to spec compliance, so
> aligning with the spec with any feature usually justifies itself. I see
> your point was the opposite :)

Exactly. The test SPDK client is out of spec in exactly the same way.

I'm trying to follow up on the spec side so it reflects reality (I can't find a
driver that behaves differently here), but that's obviously a slow process.

thanks
john