[PATCH 0/5] nvmet/nvmet_fc: add events for discovery controller rescan

[PATCH 0/5] nvmet/nvmet_fc: add events for discovery controller rescan

[James Smart]

A transport may have a transport-specific mechanism that can signal
when discovery controller content has changed and request a host
to reconnect to the discovery controller.

FC is such a transport. RSCNs may be generated by the FC port with
the discovery server, with the RSCNs then broadcast to the FC-NVME
hosts. A host, upon receiving the RSCN, would validate connectivity
then initiate a discovery controller rescan, allowing new subsystems
to be connected to or updating subsystem connectivity tables.

These patches:
- Modify the nvmet core layer to call a transport callback on every
  subsystem add or remove from a transport port.
- Modify the nvmet-fc transport to support the callback, and add its
  own internal lldd api to generate RSCN's via the lldd.
- Modify the lpfc driver to send/receive RSCNs for FC-NVME: transmit
  the changed attribute RSCN on the target, receiving the RSCN on
  the initiator and invoking the nvmet-fc transport rescan api.
  Also adds manual sysfs mechanism to generate the RSCN on the target.


Dick Kennedy (1):
  lpfc: Add sysfs interface to post NVME RSCN

James Smart (4):
  nvmet: call transport on subsystem add and delete
  nvmet_fc: support transport subsystem events
  lpfc: Add support to generate RSCN events for nport
  lpfc: Add NVME rescan support via RSCNs

 drivers/nvme/target/configfs.c   |   2 +
 drivers/nvme/target/core.c       |  10 ++++
 drivers/nvme/target/fc.c         |  10 ++++
 drivers/nvme/target/nvmet.h      |   2 +
 drivers/scsi/lpfc/lpfc.h         |   2 +
 drivers/scsi/lpfc/lpfc_attr.c    |  62 ++++++++++++++++++++
 drivers/scsi/lpfc/lpfc_crtn.h    |   4 ++
 drivers/scsi/lpfc/lpfc_els.c     | 118 +++++++++++++++++++++++++++++++++++++++
 drivers/scsi/lpfc/lpfc_hbadisc.c |  35 ++++++++++++
 drivers/scsi/lpfc/lpfc_hw.h      |   9 +++
 drivers/scsi/lpfc/lpfc_nvme.c    |  42 ++++++++++++++
 drivers/scsi/lpfc/lpfc_nvmet.c   |  18 ++++++
 drivers/scsi/lpfc/lpfc_sli.c     |   1 +
 include/linux/nvme-fc-driver.h   |   6 ++
 14 files changed, 321 insertions(+)

-- 
2.13.1

[PATCH 0/5] nvmet/nvmet_fc: add events for discovery controller rescan

[Sagi Grimberg]

James,

> A transport may have a transport-specific mechanism that can signal
> when discovery controller content has changed and request a host
> to reconnect to the discovery controller.
> 
> FC is such a transport. RSCNs may be generated by the FC port with
> the discovery server, with the RSCNs then broadcast to the FC-NVME
> hosts. A host, upon receiving the RSCN, would validate connectivity
> then initiate a discovery controller rescan, allowing new subsystems
> to be connected to or updating subsystem connectivity tables.

How does this fit with possible future discovery enhancements discussed
in nvme TWG right now? The notification for discovery records change
will not be transport specific in the future.

[PATCH 0/5] nvmet/nvmet_fc: add events for discovery controller rescan

[James Smart]

On 10/29/2017 9:11 AM, Sagi Grimberg wrote:
> James,
>
>> A transport may have a transport-specific mechanism that can signal
>> when discovery controller content has changed and request a host
>> to reconnect to the discovery controller.
>>
>> FC is such a transport. RSCNs may be generated by the FC port with
>> the discovery server, with the RSCNs then broadcast to the FC-NVME
>> hosts. A host, upon receiving the RSCN, would validate connectivity
>> then initiate a discovery controller rescan, allowing new subsystems
>> to be connected to or updating subsystem connectivity tables.
>
> How does this fit with possible future discovery enhancements discussed
> in nvme TWG right now? The notification for discovery records change
> will not be transport specific in the future.

Its independent yet can coexist very well.

Currently, the FC-NVME standard strongly suggests a discovery controller 
on each target. Having this simple notification avoids lots of 
long-lived connections and rescans are done only when/where needed - 
with no change in any standard. It works very well on current products.? 
There's nothing preventing it coexisting with long-lived discovery 
controller connections and other configurations of discovery servers.

I don't buy into your last statement yet. Long lived discovery 
connections are fine and an admin can already configure a set of systems 
as they describe. I also expect any attempt to mandate centralized 
discovery controllers to have a fair number of issues to be worked out 
not just with hosts but with targets and cross-fabric issues and I 
believe there will be pushback to give complete control of presentation 
to third party entity.? In some respects, the history of iSNS/SLP for 
iSCSI gave a brief example of trying to go down this path.

-- james

[PATCH 0/5] nvmet/nvmet_fc: add events for discovery controller rescan

[Martin K. Petersen]

James,

> Note: These patches were posted to the nvme list this morning. I should
> have cc'd the scsi list as well - so I'm not posting to the scsi list.
> I fully expect all patches to be pulled via the nvme-4.15 tree then the
> block tree.

The SCSI pieces look OK to me.

Acked-by: Martin K. Petersen <martin.petersen at oracle.com>

-- 
Martin K. Petersen	Oracle Linux Engineering

[PATCH 0/5] nvmet/nvmet_fc: add events for discovery controller rescan

[Christoph Hellwig]

On Sat, Oct 28, 2017@10:21:09AM -0700, James Smart wrote:
> A transport may have a transport-specific mechanism that can signal
> when discovery controller content has changed and request a host
> to reconnect to the discovery controller.

Can you point to the part of the NVMe spec that allows this?
While the FC transport obviously could do it there is nothing in the
NVMe architecture model documenting this behavior.

[PATCH 0/5] nvmet/nvmet_fc: add events for discovery controller rescan

[Christoph Hellwig]

On Sun, Oct 29, 2017@09:43:14PM -0700, James Smart wrote:
> Currently, the FC-NVME standard strongly suggests a discovery controller on
> each target.

Yikes.  This is a new requirement not backed by anything in NVMe or
NVMeoF itself.  Please have discussion in the technical working group
on this behavior first.

It's not that I'm against it - but I really want this sort of optional
transport behavior clearly documented in the actual NVMe spec first.

[PATCH 0/5] nvmet/nvmet_fc: add events for discovery controller rescan

[James Smart]

On 11/1/2017 8:36 AM, Christoph Hellwig wrote:
> On Sat, Oct 28, 2017@10:21:09AM -0700, James Smart wrote:
>> A transport may have a transport-specific mechanism that can signal
>> when discovery controller content has changed and request a host
>> to reconnect to the discovery controller.
> 
> Can you point to the part of the NVMe spec that allows this?
> While the FC transport obviously could do it there is nothing in the
> NVMe architecture model documenting this behavior.

There will not be anything in the NVMe base spec about such a thing as 
it's pci, nor will there be anything in the NVMe Fabrics spec as nothing 
about this changes or alters nvmf discovery. It can and is fully 
documented in the transport spec which has been visible in drafts for 
almost a year. It's not disallowed simply because an arch section 
doesn't mention it. It also falls under the "method that a host uses to 
obtain the information necessary to connect to the initial Discovery 
Service is implementation specific" - but in this case, the transport, 
in a published way, is making that implementation easier. I don't know 
what you're trying to say.

-- james

[PATCH 0/5] nvmet/nvmet_fc: add events for discovery controller rescan

[Christoph Hellwig]

On Wed, Nov 01, 2017@08:55:36AM -0700, James Smart wrote:
> There will not be anything in the NVMe base spec about such a thing as it's
> pci, nor will there be anything in the NVMe Fabrics spec as nothing about
> this changes or alters nvmf discovery. It can and is fully documented in the
> transport spec which has been visible in drafts for almost a year. It's not
> disallowed simply because an arch section doesn't mention it. It also falls
> under the "method that a host uses to obtain the information necessary to
> connect to the initial Discovery Service is implementation specific" - but
> in this case, the transport, in a published way, is making that
> implementation easier. I don't know what you're trying to say.

You'll need to discuss this with the technical working group or we won't
support it in Linux.  I'm getting sick and tired of the FC smokey
backroom secret sauce.

[PATCH 0/5] nvmet/nvmet_fc: add events for discovery controller rescan

[James Smart]

On 11/1/2017 8:38 AM, Christoph Hellwig wrote:
> On Sun, Oct 29, 2017@09:43:14PM -0700, James Smart wrote:
>> Currently, the FC-NVME standard strongly suggests a discovery controller on
>> each target.
> 
> Yikes.  This is a new requirement not backed by anything in NVMe or
> NVMeoF itself.  Please have discussion in the technical working group
> on this behavior first.
> 
> It's not that I'm against it - but I really want this sort of optional
> transport behavior clearly documented in the actual NVMe spec first.
> 

See my last email.

The "strongly suggest" is laymans language for the word "should" that 
was used in the standard. Should was used so that discovery of the 
available discovery controllers on FC could be done in an automatic and 
dynamic way without apriori knowledge. Any FC target device is free to 
choose if and where they implement discovery controllers, just as on any 
other transport in nvmf.

We have had discussions in the technical working group and with nvme 
leadership on this and as I said, its well within what a transport can 
do. It's doing nothing illegal or odd. It doesn't require explicit 
documentation in a NVMe spec. It is in a NVME fabrics-based transport 
specification.

-- james

[PATCH 0/5] nvmet/nvmet_fc: add events for discovery controller rescan

[James Smart]

On 11/1/2017 8:57 AM, Christoph Hellwig wrote:
> On Wed, Nov 01, 2017@08:55:36AM -0700, James Smart wrote:
>> There will not be anything in the NVMe base spec about such a thing as it's
>> pci, nor will there be anything in the NVMe Fabrics spec as nothing about
>> this changes or alters nvmf discovery. It can and is fully documented in the
>> transport spec which has been visible in drafts for almost a year. It's not
>> disallowed simply because an arch section doesn't mention it. It also falls
>> under the "method that a host uses to obtain the information necessary to
>> connect to the initial Discovery Service is implementation specific" - but
>> in this case, the transport, in a published way, is making that
>> implementation easier. I don't know what you're trying to say.
> 
> You'll need to discuss this with the technical working group or we won't
> support it in Linux.  I'm getting sick and tired of the FC smokey
> backroom secret sauce.
> 

Seriously ?  It is in a published transport spec that was reviewed by 
working group leadership and members.

[PATCH 0/5] nvmet/nvmet_fc: add events for discovery controller rescan

[Christoph Hellwig]

On Wed, Nov 01, 2017@09:03:37AM -0700, James Smart wrote:
> The "strongly suggest" is laymans language for the word "should" that was
> used in the standard. Should was used so that discovery of the available
> discovery controllers on FC could be done in an automatic and dynamic way
> without apriori knowledge. Any FC target device is free to choose if and
> where they implement discovery controllers, just as on any other transport
> in nvmf.
> 
> We have had discussions in the technical working group and with nvme
> leadership on this and as I said, its well within what a transport can do.
> It's doing nothing illegal or odd. It doesn't require explicit documentation
> in a NVMe spec. It is in a NVME fabrics-based transport specification.

Please send a TP against Section 5 of the NVMeoF spec to add your
transport specific re-discovery notifications.

[PATCH 0/5] nvmet/nvmet_fc: add events for discovery controller rescan

[Christoph Hellwig]

On Wed, Nov 01, 2017@09:12:00AM -0700, James Smart wrote:
> Seriously ?  It is in a published transport spec that was reviewed by
> working group leadership and members.

NVMe-FC has _never_ been reviewed or even posted to the NVMe technical
working group list.  Which is a big part of all the problems it has been
causing.  The whole idea of doing a NVMe transport outside of NVMe has
been nothing but a nightmare.