Re: [RFC PATCH 5/5] nvme-vfio: Add a document for the NVMe device

Jonathan Derrick
2022-12-07T22:42:40Z

On 12/5/2022 10:58 PM, Lei Rao wrote:
> The documentation describes the details of the NVMe hardware
> extension to support VFIO live migration.
>
> Signed-off-by: Lei Rao <lei.rao@intel.com>
> Signed-off-by: Yadong Li <yadong.li@intel.com>
> Signed-off-by: Chaitanya Kulkarni <kch@nvidia.com>
> Reviewed-by: Eddie Dong <eddie.dong@intel.com>
> Reviewed-by: Hang Yuan <hang.yuan@intel.com>
> ---
> drivers/vfio/pci/nvme/nvme.txt | 278 +++++++++++++++++++++++++++++++++
> 1 file changed, 278 insertions(+)
> create mode 100644 drivers/vfio/pci/nvme/nvme.txt
>
> diff --git a/drivers/vfio/pci/nvme/nvme.txt b/drivers/vfio/pci/nvme/nvme.txt
> new file mode 100644
> index 000000000000..eadcf2082eed
> --- /dev/null
> +++ b/drivers/vfio/pci/nvme/nvme.txt
> @@ -0,0 +1,278 @@
> +===========================
> +NVMe Live Migration Support
> +===========================
> +
> +Introduction
> +------------
> +To support live migration, NVMe device designs its own implementation,
> +including five new specific admin commands and a capability flag in
> +the vendor-specific field in the identify controller data structure to
> +support VF's live migration usage. Software can use these live migration
> +admin commands to get device migration state data size, save and load the
> +data, suspend and resume the given VF device. They are submitted by software
> +to the NVMe PF device's admin queue and ignored if placed in the VF device's
> +admin queue. This is due to the NVMe VF device being passed to the virtual
> +machine in the virtualization scenario. So VF device's admin queue is not
> +available for the hypervisor to submit VF device live migration commands.
> +The capability flag in the identify controller data structure can be used by
> +software to detect if the NVMe device supports live migration. The following
> +chapters introduce the detailed format of the commands and the capability flag.
> +
> +Definition of opcode for live migration commands
> +------------------------------------------------
> +
> ++---------------------------+-----------+-----------+------------+
> +| | | | |
> +| Opcode by Field | | | |
> +| | | | |
> ++--------+---------+--------+ | | |
> +| | | | Combined | Namespace | |
> +| 07 | 06:02 | 01:00 | Opcode | Identifier| Command |
> +| | | | | used | |
> ++--------+---------+--------+ | | |
> +|Generic | Function| Data | | | |
> +|command | |Transfer| | | |
> ++--------+---------+--------+-----------+-----------+------------+
> +| |
> +| Vendor SpecificOpcode |
> ++--------+---------+--------+-----------+-----------+------------+
> +| | | | | | Query the |
> +| 1b | 10001 | 00 | 0xC4 | | data size |
> ++--------+---------+--------+-----------+-----------+------------+
> +| | | | | | Suspend the|
> +| 1b | 10010 | 00 | 0xC8 | | VF |
> ++--------+---------+--------+-----------+-----------+------------+
> +| | | | | | Resume the |
> +| 1b | 10011 | 00 | 0xCC | | VF |
> ++--------+---------+--------+-----------+-----------+------------+
> +| | | | | | Save the |
> +| 1b | 10100 | 10 | 0xD2 | |device data |
> ++--------+---------+--------+-----------+-----------+------------+
> +| | | | | | Load the |
> +| 1b | 10101 | 01 | 0xD5 | |device data |
> ++--------+---------+--------+-----------+-----------+------------+
> +
I'm assuming by using these vendor specific opcodes and id-ctrl's vu space, that
you should make this code be protected by some kind of vendor/device specific-flag
or preferably something standard.

No way of knowing what 0xC4 will do on a non-lm drive, for instance