Re: NVMe PCI driver ignores SQHD from completion entries

[James Smart <james.smart@broadcom.com>]

On 10/8/2019 8:59 AM, Keith Busch wrote:
> On Mon, Oct 07, 2019 at 05:32:55PM -0700, Vaibhav Nagarnaik wrote:
>> On Sat, Oct 5, 2019 at 7:27 AM Keith Busch <kbusch@kernel.org> wrote:
>>> Checking SQ head is required only if the host might submit more commands
>>> than there are entries. The Linux nvme driver allocates enough tags
>>> for the depth of the queue, leaving one entry empty, so having a tag
>>> available means the next sq entry must be available.
>> But the driver does overwrite submission queue entries under process
>> (which are already fetched by the controller). Are there guarantees
>> from controllers out there that once fetched, the SQ entries will not
>> be fetched again for any reason? The spec doesn't prohibit that.
> Every controller I've encountered fetches commands exactly once. I have to
> admit I don't find this spelled out in great detail in the spec, though
> out-of-order fetching doesn't sound like it would satisfy the properties of a
> queue.
>
> Closest thing I can find is in section 4.1:
>
>    The consumer of entries on a queue uses the current Head entry pointer to
>    identify the slot containing the next entry to be consumed.
>
> And later says:
>
>    Once a Submission Queue or Completion Queue entry has been consumed, the slot
>    in which it was placed is free and available for reuse.
>
> If you have to re-fetch an entry, then the controller didn't "consume" it on
> the original fetch. If the controller completes a later entry, it has consumed
> an entry that wasn't the "next" entry in violation of the consumer requirement.
>

Agree - Nearly everything so far works in a fetch-once-in-order manner, 
but there's also nothing quoted that has great conviction. There's a lot 
of implications, especially with the word "next", but no hard 
requirement spelled out.

What I read that is more conclusive is.:

(4.1) "A Submission Queue entry has been consumed by the controller when 
a Completion Queue entry is posted that indicates that the Submission 
Queue Head Pointer has moved past the slot in which that Submission 
Queue entry was placed."
and
(4.1) "The controller uses the SQ Head Pointer (SQHD) field in 
Completion Queue entries to communicate new values of the Submission 
Queue Head Pointer to the host. A new SQHD value indicates that 
Submission Queue entries have been consumed, but does not indicate 
either execution or completion of any command."
and
(CQE SQHD description) "A Submission Queue entry has been consumed by 
the controller when a Completion Queue entry is posted that indicates 
that the Submission Queue Head Pointer has moved past the slot in which 
that Submission Queue entry was placed."

All of which say an SQE slot isn't consumed until the controller says so 
via a CQE with a SQHD pointer that says it's moved past the SQE slot. 
And it becomes absolutely necessary with a fabric and OOO SQE delivery.

Given there's some deviation in interpretation, it's likely a good thing 
to take up in the NVM Express group.

-- james


-- 

_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme