Re: [SPDK] NVMe vendor specific commands

[Walker, Benjamin <benjamin.walker at intel.com>]

[-- Attachment #1: Type: text/plain, Size: 3102 bytes --]

On Fri, 2019-08-30 at 16:11 -0600, Michael Haeuptle wrote:
> Hello,
> 
> I have a couple of observations/questions regarding vendor specific
> commands over NVMf and I was hoping to get your take on this.
> 
> - I can get vendor specific *IO *commands to work in my custom bdev by
> supporting SPDK_BDEV_IO_TYPE_NVME_IO. However, doing the same for
> *Admin *commands
> fails. Is there a reason why we are blocking the vendor specific range of
> C0h-FFh for Admin commands (see spdk_nvmf_ctrlr_process_admin_cmd).

I'm a bit torn on what to do about vendor specific commands.In the NVMe-oF
target, each bdev is treated as a namespace in a virtualized controller. For
physical NVMe devices, each namespace on the device becomes a bdev. The issues
all arise when a vendor specific command is performing some operation that
targets the whole controller instead of just a namespace. For example:

1) It's entirely possible that a device with two namespaces places each of those
namespaces in an entirely different subsystem on the NVMe-oF side. The hosts
then think these namespaces aren't related to one another. If a vendor specific
command that changes some global state on the SSD is allowed through, this can
cause serious problems to the other unwitting subsystem.

2) The NVMe-oF target allows multiple hosts to connect simultaneously, emulating
a separate NVMe controller for each. Even in the case where the physical SSD has
a single namespace, surfaced as a bdev and added to a single subsystem, allowing
a vendor specific command through may have an impact on both controllers
presented to the two hosts, but there's no way for the NVMe-oF code to know
that.

I think the reality is that all vendor-specific commands will likely need to be
handled in the NVMe-oF software emulation instead of simply passing through to
the backing device blindly. The cleanest way to do this is to probably add a
public API function to the nvmf library that allows users to register a handler
for a vendor specific command, and then in spdk_nvmf_ctrlr_process_admin_cmd it
looks up if a handler is present and calls it rather than just failing
everything in the vendor-specific range.

> - The current bdev API to complete a NVMe specific request is
> spdk_bdev_io_complete_nvme_status(struct spdk_bdev_io *bdev_io, int sct,
> int sc) but it only takes in the 2 status code that are written to the
> completion queue status field. I would also like to set the CDW0 of the
> completion queue. Are there any plans to support this or do we want to keep
> the bdev API as front-end protocol agnostic as possible?
> I guess one way to support it would be to add an additional field to
> bdev_io's structure like there is for the NVMe/SCSI specific status code
> handling, but again, do we want to add more protocol specifics to bdev_io?
> Are the any other options to set CDW0?
> 
> Thanks for your help.
> 
> -- Michael
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk