[SPDK] NVMe-oF and multiple targets

[SPDK] NVMe-oF and multiple targets

[Walker, Benjamin]

[-- Attachment #1: Type: text/plain, Size: 1890 bytes --]

Hi all,

I've got an open design question that I wanted to run by the community regarding
the NVMe-oF target. There are a few primitives that the library currently
exposes:

1) NVMe-oF transports are a networking transport abstraction. These have memory
and request object pools and can create and destroy connections.
2) NVMe-oF poll groups are sets of NVMe-oF queue pairs (that aren't necessarily
related to one another). A single poll group can aggregate connections from
multiple different transports.
3) NVMe-oF subsystems are sets of related namespaces. A subsystem is effectively
an access control list.
4) NVMe-oF controllers are network sessions. It's a set of NVMe-oF queue pairs
(connections) that are all connected to the same subsystem and accessing the
same namespaces.
6) NVMe-oF targets are a set of NVMe-oF subsystems, controllers, and poll
groups. The target object really defines the NVMe-oF discovery service.

Historically, the SPDK NVMe-oF target application created a single, global NVMe-
oF target object. But we're trying to generalize the underlying nvmf library to
support multiple targets so that a single application can support multiple
discovery services for more complex use cases.

My question is around how to map transports (#1) to targets (#6). There are two
paths we could go down. I'll use the RDMA transport as an example.

A) Each target gets its own transport. That means there would be two RDMA
transports for two targets, and each transport would allocate it's own pool of
independent resources.
B) transports can be shared across targets. Two different targets can share a
single RDMA transport. They can't share connections or listen on the same
addresses, of course, but they share the same request/buffer pools internally.

Does anyone have a strong opinion on what the right choice is here?

Thanks,
Ben