[PATCH 0/11] nvmet: FDP bridge for NVMe-oF targets

[Guixin Liu <kanie@linux.alibaba.com>]

Flexible Data Placement (FDP) is defined in the NVMe base specification
and therefore applies to fabrics as well.  An Ethernet Bunch Of
Flash (EBOF) style
storage system that exposes its local FDP-capable NVMe drives to remote
hosts over NVMe-oF currently has no way to pass the capability through:
nvmet neither advertises FDP nor translates or forwards any FDP command,
so a host connected through the target loses the write amplification
benefits that the local drives provide.

This series adds an FDP bridge to the NVMe target.  A namespace backed
by an NVMe namespace opts in through the fdp_enable configfs attribute;
the bridge then advertises the FDPS capability, reports the backing FDP
configuration through the feature and log pages, translates the
placement identifiers of write commands to the backing write streams,
and forwards the FDP management commands and log pages to the backing
controller.  The backing geometry is verified at namespace enable time,
so the bridge only comes up when the backing placement identifiers are
identity mapped, which the translation relies on.

The FDP commands are served by the bridge directly instead of relying
on the existing passthru support, for several reasons:

  - Passthru subsystems do not support reservations; the reservation
    commands are rejected and the controller is not advertised as
    reservation capable, because the underlying device cannot
    differentiate the hosts that connect over fabrics.  A regular
    namespace goes through the target's own reservation implementation,
    which is not available to a passthru subsystem.

  - A passthru subsystem is bound to a single backend controller: the
    whole drive, with all of its namespaces, is passed through as one
    unit, and no regular namespaces can be added to the subsystem.  An
    EBOF system needs to compose a storage pool out of individual
    namespaces from multiple backend drives, which only the regular
    namespace path provides.

  - The regular namespace path already handles the target side of
    reservations, ANA reporting and per-namespace configuration; the
    bridge extends that path rather than replacing it.

  - With passthru, the namespace identity of the backend drive is
    exposed to the host as is, and the host would talk to the backend
    controller's FDP state through the passthrough of every command.
    The bridge instead owns the FDP presentation of the exported
    namespace, while the state that the backing controller owns is
    forwarded to it: the reclaim unit handle status hosts read and the
    reclaim unit handle updates they issue are carried out by the
    controller that actually manages the media.

The Linux host side needs no changes: the existing FDP discovery path
(FDPS bit, endurance group identifier, FDP feature, FDP configurations
log page and reclaim unit handle status) works against the bridge
unmodified.

The series is structured as follows:

  1-2   wire structures and block layer helper
  3     nvme_bdev_ops export, used to recognize an NVMe backing device
  4     per-namespace bridge state and configuration
  5     endurance group identifier in the CS-INDEP identify data
  6     FDPS advertisement
  7     FDP feature (FID 1Dh)
  8     I/O management command forwarding
  9     FDP log pages (LID 20h-23h)
  10    placement identifier translation on the write path
  11    tracepoint for the translation

This was tested with a QEMU FDP backing device through an nvmet-tcp
loopback setup: the host discovers and registers the backing write
streams over the bridge, writes with placement identifiers land in the
reclaim units of the backend device they select (verified through the
reclaim unit handle status of the backend), and the forwarded log pages
match the backend byte for byte.

Guixin Liu (11):
  nvme: add the FDP wire structures for the I/O management send command
  block: add bdev_write_stream_granularity()
  nvme: export nvme_bdev_ops
  nvmet: add per-namespace FDP bridge state and configuration
  nvmet: report the Endurance Group Identifier in the CS-INDEP identify
    data
  nvmet: advertise the FDP capability
  nvmet: implement the FDP feature
  nvmet: forward the FDP I/O management commands to the backing device
  nvmet: serve the FDP log pages
  nvmet: translate placement identifiers of writes to write streams
  nvmet: add an FDP write translation tracepoint

 drivers/nvme/host/core.c          |   1 +
 drivers/nvme/target/Kconfig       |  14 +
 drivers/nvme/target/Makefile      |   1 +
 drivers/nvme/target/admin-cmd.c   |  33 +++
 drivers/nvme/target/configfs.c    |  31 +++
 drivers/nvme/target/core.c        |  38 +++-
 drivers/nvme/target/fdp.c         | 431 ++++++++++++++++++++++++++++++++++++++
 drivers/nvme/target/io-cmd-bdev.c |   8 +
 drivers/nvme/target/nvmet.h       |  93 ++++++++
 drivers/nvme/target/trace.h       |  23 ++
 include/linux/blkdev.h            |   8 +
 include/linux/nvme.h                          |  24 +
 12 files changed, 703 insertions(+), 2 deletions(-)

-- 
2.50.0