[PATCH 13/14] megaraid_sas: NVME passthru command support

[dgilbert@interlog.com (Douglas Gilbert)]

On 2018-01-10 11:22 AM, Bart Van Assche wrote:
> On Tue, 2018-01-09@22:07 +0530, Kashyap Desai wrote:
>> Overall NVME support behind MR controller is really a SCSI device. On top
>> of that, for MegaRaid, NVME device can be part of Virtual Disk and those
>> drive will not be exposed to the driver. User application may like to talk
>> to hidden NVME devices (part of VDs). This patch will extend the existing
>> interface for megaraid product in the same way it is currently supported
>> for other protocols like SMP, SATA pass-through.
> 
> It seems to me like there is a contradiction in the above paragraph: if some
> NVMe devices are not exposed to the driver, how can a user space application
> ever send NVMe commands to it?

I think that he meant that the NVMe physical devices (e.g. SSDs) are not
exposed to the upper layers (e.g. the SCSI mid-layer and above). The
SCSI subsystem has a no_uld_attach device flag that lets a LLD attach
physical devices but the sd driver and hence the block layer do not
"see" them. The idea is that maintenance programs like smartmontools
can use them via the bsg or sg drivers. The Megaraid driver code does
not seem to use no_uld_attach. Does the NVMe subsystem have similar
"generic" (i.e. non-block) devices accessible to the user space?

> Anyway, has it been considered to implement the NVMe support as an NVMe
> transport driver? The upstream kernel already supports NVMe communication
> with NVMe PCI devices, NVMe over RDMA and NVMe over FC. If communication to
> the NVMe devices behind the MegaRaid controller would be implemented as an
> NVMe transport driver then all functionality of the Linux NVMe driver could
> be reused, including its sysfs entries.

Broadcom already sell "SAS" HBAs that have "tri-mode" phys. That is a phy
that can connect to a SAS device (e.g. a SAS expander), a SATA device or a
NVMe device. Now if I was Broadcom designing a 24 Gbps SAS-4 next generation
expander I would be thinking of using those tri-mode phys on it. But then
there is a problem, SAS currently supports 3 protocols: SSP (for SCSI
storage and enclosure management (SES)), STP (for SATA storage ) and SMP
(for expander management). The problem is how those NVMe commands, status
and data cross the wire between the OS HBA (or MegaRaid type controller) and
an expander. Solving that might need some lateral thinking.

On one hand the NVM Express folks seem to have shelved the idea of a SCSI
to NVMe Translation Layer (SNTL) and have not updated an old white paper
on the subject. Currently there is no SNTL on Linux (there was but it was
removed) or FreeBSD but there is one on Windows.

On the other hand I'm informed that recently the same body accepted the
SES-3 standard pretty much as-is. That is done with the addition of SES
Send and SES Receive commands to NVME-MI. The library under sg_ses has
already been modified to use them (by implementing a specialized SNTL).

Doug Gilbert