[GIT PULL] nvme fixes for 5.2

[GIT PULL] nvme fixes for 5.2

[Christoph Hellwig]

The following changes since commit 936b33f7243fa1e54c8f4f2febd3472cc00e66fd:

  brd: add cond_resched to brd_free_pages (2019-05-09 12:51:27 -0600)

are available in the Git repository at:

  git://git.infradead.org/nvme.git nvme-5.2

for you to fetch changes up to 1b1031ca63b2ce1d3b664b35b77ec94e458693e9:

  nvme: validate cntlid during controller initialisation (2019-05-14 17:19:50 +0200)

----------------------------------------------------------------
Chaitanya Kulkarni (1):
      nvme: trace all async notice events

Christoph Hellwig (2):
      nvme: change locking for the per-subsystem controller list
      nvme: validate cntlid during controller initialisation

Gustavo A. R. Silva (1):
      nvme-pci: mark expected switch fall-through

Hannes Reinecke (2):
      nvme-fc: use separate work queue to avoid warning
      nvme-multipath: avoid crash on invalid subsystem cntlid enumeration

Max Gurtovoy (1):
      nvme-rdma: remove redundant reference between ib_device and tagset

Maxim Levitsky (2):
      nvme-pci: init shadow doorbell after each reset
      nvme-pci: add known admin effects to augument admin effects log page

Minwoo Im (2):
      nvme-fabrics: remove unused argument
      nvme: fix typos in nvme status code values

 drivers/nvme/host/core.c      | 79 ++++++++++++++++++++++---------------------
 drivers/nvme/host/fabrics.c   |  4 +--
 drivers/nvme/host/fc.c        | 14 ++++++--
 drivers/nvme/host/multipath.c |  2 +-
 drivers/nvme/host/pci.c       |  4 +--
 drivers/nvme/host/rdma.c      | 34 +++----------------
 drivers/nvme/host/trace.h     |  1 +
 include/linux/nvme.h          |  4 +--
 8 files changed, 64 insertions(+), 78 deletions(-)

[GIT PULL] nvme fixes for 5.2

[Jens Axboe]

On 5/16/19 2:25 AM, Christoph Hellwig wrote:
> The following changes since commit 936b33f7243fa1e54c8f4f2febd3472cc00e66fd:
> 
>   brd: add cond_resched to brd_free_pages (2019-05-09 12:51:27 -0600)
> 
> are available in the Git repository at:
> 
>   git://git.infradead.org/nvme.git nvme-5.2
> 
> for you to fetch changes up to 1b1031ca63b2ce1d3b664b35b77ec94e458693e9:
> 
>   nvme: validate cntlid during controller initialisation (2019-05-14 17:19:50 +0200)
> 
> ----------------------------------------------------------------
> Chaitanya Kulkarni (1):
>       nvme: trace all async notice events
> 
> Christoph Hellwig (2):
>       nvme: change locking for the per-subsystem controller list
>       nvme: validate cntlid during controller initialisation
> 
> Gustavo A. R. Silva (1):
>       nvme-pci: mark expected switch fall-through
> 
> Hannes Reinecke (2):
>       nvme-fc: use separate work queue to avoid warning
>       nvme-multipath: avoid crash on invalid subsystem cntlid enumeration
> 
> Max Gurtovoy (1):
>       nvme-rdma: remove redundant reference between ib_device and tagset
> 
> Maxim Levitsky (2):
>       nvme-pci: init shadow doorbell after each reset
>       nvme-pci: add known admin effects to augument admin effects log page
> 
> Minwoo Im (2):
>       nvme-fabrics: remove unused argument
>       nvme: fix typos in nvme status code values
> 
>  drivers/nvme/host/core.c      | 79 ++++++++++++++++++++++---------------------
>  drivers/nvme/host/fabrics.c   |  4 +--
>  drivers/nvme/host/fc.c        | 14 ++++++--
>  drivers/nvme/host/multipath.c |  2 +-
>  drivers/nvme/host/pci.c       |  4 +--
>  drivers/nvme/host/rdma.c      | 34 +++----------------
>  drivers/nvme/host/trace.h     |  1 +
>  include/linux/nvme.h          |  4 +--
>  8 files changed, 64 insertions(+), 78 deletions(-)

Pulled, thanks.

-- 
Jens Axboe

[GIT PULL] nvme fixes for 5.2

[John Donnelly]

On 5/16/19 3:25 AM, Christoph Hellwig wrote:
> The following changes since commit 936b33f7243fa1e54c8f4f2febd3472cc00e66fd:
> 
>    brd: add cond_resched to brd_free_pages (2019-05-09 12:51:27 -0600)
> 
> are available in the Git repository at:
> 
>    git://git.infradead.org/nvme.git nvme-5.2
> 
> for you to fetch changes up to 1b1031ca63b2ce1d3b664b35b77ec94e458693e9:
> 
>    nvme: validate cntlid during controller initialisation (2019-05-14 17:19:50 +0200)
> 
> ----------------------------------------------------------------
> Chaitanya Kulkarni (1):
>        nvme: trace all async notice events
> 
> Christoph Hellwig (2):
>        nvme: change locking for the per-subsystem controller list
>    **     nvme: validate cntlid during controller initialisation
> 
> Gustavo A. R. Silva (1):
>        nvme-pci: mark expected switch fall-through
> 
> Hannes Reinecke (2):
>        nvme-fc: use separate work queue to avoid warning
>     **   nvme-multipath: avoid crash on invalid subsystem cntlid enumeration



Hi
   In testing these commits denoted by "**"  I encountered another 
failure during fail-over of two name-spaces:


May 24 21:09:31 interop-57-161 kernel: RIP 0010:kernfs_remove_by_name_ns 
+ 0x7e/0x87
May 24 21:09:31 interop-57-161 kernel: RSP: 0018:ffffb7528937fd70 EFLAGS:
00010246
May 24 21:09:31 interop-57-161 kernel: RAX: 000000000000002c RBX:
0000000000000000 RCX: 0000000000000006
May 24 21:09:31 interop-57-161 kernel: RDX: 0000000000000000 RSI:
0000000000000086 RDI: ffff984560396930
May 24 21:09:31 interop-57-161 kernel: RBP: ffffb7528937fd88 R08:
ffffffff95972a84 R09: 00000000000013b1
May 24 21:09:31 interop-57-161 kernel: R10: 0000000000000000 R11:
0000000000000004 R12: ffff98455e3aa648
May 24 21:09:31 interop-57-161 kernel: R13: ffff9845605be310 R14:
ffff98455f03bc00 R15: ffff9845605be710
May 24 21:09:31 interop-57-161 kernel: FS:  0000000000000000(0000)
GS:ffff984560380000(0000) knlGS:0000000000000000
May 24 21:09:31 interop-57-161 kernel: CS:  0010 DS: 0000 ES: 0000 CR0:
0000000080050033
May 24 21:09:31 interop-57-161 kernel: CR2: 00007f44fa788000 CR3:
000000104940a005 CR4: 00000000007606e0
May 24 21:09:31 interop-57-161 kernel: DR0: 0000000000000000 DR1:
0000000000000000 DR2: 0000000000000000
May 24 21:09:31 interop-57-161 kernel: DR3: 0000000000000000 DR6:
00000000fffe0ff0 DR7: 0000000000000400
May 24 21:09:31 interop-57-161 kernel: PKRU: 55555554
May 24 21:09:31 interop-57-161 kernel: Call Trace:
May 24 21:09:31 interop-57-161 kernel: sysfs_remove_link+0x19/0x2b
May 24 21:09:31 interop-57-161 kernel: nvme_free_ctrl+0xaf/0x120 [nvme_core]
May 24 21:09:31 interop-57-161 kernel: device_release+0x35/0x8b
May 24 21:09:31 interop-57-161 kernel: kobject_cleanup+0x66/0x172
May 24 21:09:31 interop-57-161 kernel: kobject_put+0x2a/0x45
May 24 21:09:31 interop-57-161 kernel: put_device+0x17/0x1a
May 24 21:09:31 interop-57-161 kernel: nvme_delete_ctrl_work+0x86/0x90
[nvme_core]
May 24 21:09:31 interop-57-161 kernel: process_one_work+0x169/0x3a6
May 24 21:09:31 interop-57-161 kernel: worker_thread+0x4d/0x3e5
May 24 21:09:31 interop-57-161 kernel: kthread+0x105/0x138
May 24 21:09:31 interop-57-161 kernel: ? rescuer_thread+0x380/0x375
May 24 21:09:31 interop-57-161 kernel: ? kthread_bind+0x20/0x15
May 24 21:09:31 interop-57-161 kernel: ret_from_fork+0x24/0x49


Does this look familiar ?

Thank you,

John.

[GIT PULL] nvme fixes for 5.2

[John Donnelly]

> On May 28, 2019,@11:28 AM, John Donnelly <john.p.donnelly@oracle.com> wrote:
> 
> On 5/16/19 3:25 AM, Christoph Hellwig wrote:
>> The following changes since commit 936b33f7243fa1e54c8f4f2febd3472cc00e66fd:
>>   brd: add cond_resched to brd_free_pages (2019-05-09 12:51:27 -0600)
>> are available in the Git repository at:
>>   git://git.infradead.org/nvme.git nvme-5.2
>> for you to fetch changes up to 1b1031ca63b2ce1d3b664b35b77ec94e458693e9:
>>   nvme: validate cntlid during controller initialisation (2019-05-14 17:19:50 +0200)
>> ----------------------------------------------------------------
>> Chaitanya Kulkarni (1):
>>       nvme: trace all async notice events
>> Christoph Hellwig (2):
>>       nvme: change locking for the per-subsystem controller list
>>   **     nvme: validate cntlid during controller initialisation
>> Gustavo A. R. Silva (1):
>>       nvme-pci: mark expected switch fall-through
>> Hannes Reinecke (2):
>>       nvme-fc: use separate work queue to avoid warning
>>    **   nvme-multipath: avoid crash on invalid subsystem cntlid enumeration
> 
> 
> 
> Hi
>  In testing these commits denoted by "**"  I encountered another failure during fail-over of two name-spaces:
> 
> 
> May 24 21:09:31 interop-57-161 kernel: RIP 0010:kernfs_remove_by_name_ns + 0x7e/0x87
> May 24 21:09:31 interop-57-161 kernel: RSP: 0018:ffffb7528937fd70 EFLAGS:
> 00010246
> May 24 21:09:31 interop-57-161 kernel: RAX: 000000000000002c RBX:
> 0000000000000000 RCX: 0000000000000006
> May 24 21:09:31 interop-57-161 kernel: RDX: 0000000000000000 RSI:
> 0000000000000086 RDI: ffff984560396930
> May 24 21:09:31 interop-57-161 kernel: RBP: ffffb7528937fd88 R08:
> ffffffff95972a84 R09: 00000000000013b1
> May 24 21:09:31 interop-57-161 kernel: R10: 0000000000000000 R11:
> 0000000000000004 R12: ffff98455e3aa648
> May 24 21:09:31 interop-57-161 kernel: R13: ffff9845605be310 R14:
> ffff98455f03bc00 R15: ffff9845605be710
> May 24 21:09:31 interop-57-161 kernel: FS:  0000000000000000(0000)
> GS:ffff984560380000(0000) knlGS:0000000000000000
> May 24 21:09:31 interop-57-161 kernel: CS:  0010 DS: 0000 ES: 0000 CR0:
> 0000000080050033
> May 24 21:09:31 interop-57-161 kernel: CR2: 00007f44fa788000 CR3:
> 000000104940a005 CR4: 00000000007606e0
> May 24 21:09:31 interop-57-161 kernel: DR0: 0000000000000000 DR1:
> 0000000000000000 DR2: 0000000000000000
> May 24 21:09:31 interop-57-161 kernel: DR3: 0000000000000000 DR6:
> 00000000fffe0ff0 DR7: 0000000000000400
> May 24 21:09:31 interop-57-161 kernel: PKRU: 55555554
> May 24 21:09:31 interop-57-161 kernel: Call Trace:
> May 24 21:09:31 interop-57-161 kernel: sysfs_remove_link+0x19/0x2b
> May 24 21:09:31 interop-57-161 kernel: nvme_free_ctrl+0xaf/0x120 [nvme_core]
> May 24 21:09:31 interop-57-161 kernel: device_release+0x35/0x8b
> May 24 21:09:31 interop-57-161 kernel: kobject_cleanup+0x66/0x172
> May 24 21:09:31 interop-57-161 kernel: kobject_put+0x2a/0x45
> May 24 21:09:31 interop-57-161 kernel: put_device+0x17/0x1a
> May 24 21:09:31 interop-57-161 kernel: nvme_delete_ctrl_work+0x86/0x90
> [nvme_core]
> May 24 21:09:31 interop-57-161 kernel: process_one_work+0x169/0x3a6
> May 24 21:09:31 interop-57-161 kernel: worker_thread+0x4d/0x3e5
> May 24 21:09:31 interop-57-161 kernel: kthread+0x105/0x138
> May 24 21:09:31 interop-57-161 kernel: ? rescuer_thread+0x380/0x375
> May 24 21:09:31 interop-57-161 kernel: ? kthread_bind+0x20/0x15
> May 24 21:09:31 interop-57-161 kernel: ret_from_fork+0x24/0x49
> 
> 
> Does this look familiar ?
> 
> Thank you,
> 
> John.
> 
> 
> 


  It appears to me the caller nvme_free_ctrl()  is using a device name that has been cleared :

    sysfs_remove_link(&subsys->dev.kobj, dev_name(ctrl->device));


The name passed in is null in kernfs_remove_by_name_ns() :


 if (!parent) {
   WARN(1, KERN_WARNING "kernfs: can not remove '%s', no directory\n",   name);

I don?t have a actual crash dump to examine though.




> 
> _______________________________________________
> Linux-nvme mailing list
> Linux-nvme at lists.infradead.org
> https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.infradead.org_mailman_listinfo_linux-2Dnvme&d=DwICAg&c=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=t2fPg9D87F7D8jm0_3CG9yoiIKdRg4qc_thBw4bzMhc&m=9Gz0dhjfej8TkwR2-NUdzOLHHJWOAGSMbWtk2hhJAf4&s=2eY268xQ95WW-f4xD3y7ceYbvLnvdw-TH1vwTc9bxec&e=

[GIT PULL] nvme fixes for 5.2

[Sagi Grimberg]

>    It appears to me the caller nvme_free_ctrl()  is using a device name that has been cleared :
> 
>      sysfs_remove_link(&subsys->dev.kobj, dev_name(ctrl->device));
> 
> 
> The name passed in is null in kernfs_remove_by_name_ns() :
> 
> 
>   if (!parent) {
>     WARN(1, KERN_WARNING "kernfs: can not remove '%s', no directory\n",   name);

Where do you see this warning in the log?

> I don?t have a actual crash dump to examine though.

Is this a regression then? or did this not work before the above
patches?

Would it be possible to get some more of the log?

Also, I see that this is the removal flow, was that intentional
removal during the reconnect? or did ctrl_loss_tmo expired?

[GIT PULL] nvme fixes for 5.2

[John Donnelly]

> On May 29, 2019,@1:22 AM, Sagi Grimberg <sagi@grimberg.me> wrote:
> 
> 
>>   It appears to me the caller nvme_free_ctrl()  is using a device name that has been cleared :
>>     sysfs_remove_link(&subsys->dev.kobj, dev_name(ctrl->device));
>> The name passed in is null in kernfs_remove_by_name_ns() :
>>  if (!parent) {
>>    WARN(1, KERN_WARNING "kernfs: can not remove '%s', no directory\n",   name);
> 
> Where do you see this warning in the log?
> 
>> I don?t have a actual crash dump to examine though.
> 
> Is this a regression then? or did this not work before the above
> patches?
> 
> Would it be possible to get some more of the log?
> 
> Also, I see that this is the removal flow, was that intentional
> removal during the reconnect? or did ctrl_loss_tmo expired?


HI Sagi;

    Prior to implementing the noted commits we never gotten this far before in recovery because the system encountered the OOPS as noted in those commits and fell apart with just one namespace. Now this  occurs during path failover testing while trying to reconnect two name spaces,  and it  ended up with max reconnect attempts of 60 that produced a  different stack trace.

A message did appear in the log with the name ?nvme0? , but I can?t tell if that is the device name that was  being deleted , or the next one :  nvme1;  I have no farther explanation why the  OOPS would occur in the print statement unless ?name? was null,   do you ?   


Here is the preamble to the previous stack trace. 

May 24 21:09:31 interop-57-161 kernel: nvme nvme0: NVME-FC{0}: reset: Reconnect
attempt failed (-22)
May 24 21:09:31 interop-57-161 kernel: nvme nvme0: NVME-FC{0}: Max reconnect
attempts (60) reached.
May 24 21:09:31 interop-57-161 kernel: nvme nvme0: Removing ctrl: NQN
"nqn.1992-08.com.netapp:sn.3d9037c53fed11e9922200a0986a8b60:subsystem.ol_nvme1_ss1"
May 24 21:09:31 interop-57-161 kernel: kernfs: can not remove 'nvme0', no directory
May 24 21:09:31 interop-57-161 kernel: ------------[ cut here ]------------
May 24 21:09:31 interop-57-161 kernel: WARNING: CPU: 6 PID: 14206 at
fs/kernfs/dir.c:1481 kernfs_remove_by_name_ns+0x7e/0x87

We are not seeing failover work at all using the lpfc driver.   The connections are established on boot, but when one of the paths are disabled on the NetApp target the system never is able to restores the paths. 

[GIT PULL] nvme fixes for 5.2

[Sagi Grimberg]

> HI Sagi;
> 
>      Prior to implementing the noted commits we never gotten this far before in recovery because the system encountered the OOPS as noted in those commits and fell apart with just one namespace. Now this  occurs during path failover testing while trying to reconnect two name spaces,  and it  ended up with max reconnect attempts of 60 that produced a  different stack trace.

I see.

> A message did appear in the log with the name ?nvme0? , but I can?t tell if that is the device name that was  being deleted , or the next one :  nvme1;  I have no farther explanation why the  OOPS would occur in the print statement unless ?name? was null,   do you ?

This does not realte to the device name, but rather to the device parent
that seems to be NULL (we have a valid kobj but it points to a NULL
parent. This usually happens in a use-after-free conditions.

What is the FC device parent? the FC adapter device? or the 
/dev/nvme-fabrics device?

> Here is the preamble to the previous stack trace.
> 
> May 24 21:09:31 interop-57-161 kernel: nvme nvme0: NVME-FC{0}: reset: Reconnect
> attempt failed (-22)
> May 24 21:09:31 interop-57-161 kernel: nvme nvme0: NVME-FC{0}: Max reconnect
> attempts (60) reached.
> May 24 21:09:31 interop-57-161 kernel: nvme nvme0: Removing ctrl: NQN
> "nqn.1992-08.com.netapp:sn.3d9037c53fed11e9922200a0986a8b60:subsystem.ol_nvme1_ss1"
> May 24 21:09:31 interop-57-161 kernel: kernfs: can not remove 'nvme0', no directory
> May 24 21:09:31 interop-57-161 kernel: ------------[ cut here ]------------
> May 24 21:09:31 interop-57-161 kernel: WARNING: CPU: 6 PID: 14206 at
> fs/kernfs/dir.c:1481 kernfs_remove_by_name_ns+0x7e/0x87
> 
> We are not seeing failover work at all using the lpfc driver.   The connections are established on boot, but when one of the paths are disabled on the NetApp target the system never is able to restores the paths.

So I/O failover is not working with lpfc? that seems to be a different
issue to me. What do you mean that the NetApp target never is able to
restore the paths? That is just an explanation to what triggers the
bug or is that somehow related?

[GIT PULL] nvme fixes for 5.2

[James Smart]

On 5/30/2019 11:35 AM, Sagi Grimberg wrote:
>
>> HI Sagi;
>>
>> ???? Prior to implementing the noted commits we never gotten this far 
>> before in recovery because the system encountered the OOPS as noted 
>> in those commits and fell apart with just one namespace. Now this? 
>> occurs during path failover testing while trying to reconnect two 
>> name spaces,? and it? ended up with max reconnect attempts of 60 that 
>> produced a? different stack trace.
>
> I see.
>
>> A message did appear in the log with the name ?nvme0? , but I can?t 
>> tell if that is the device name that was? being deleted , or the next 
>> one :? nvme1;? I have no farther explanation why the? OOPS would 
>> occur in the print statement unless ?name? was null,?? do you ?
>
> This does not realte to the device name, but rather to the device parent
> that seems to be NULL (we have a valid kobj but it points to a NULL
> parent. This usually happens in a use-after-free conditions.
>
> What is the FC device parent? the FC adapter device? or the 
> /dev/nvme-fabrics device?

Warning:?? tracking down an issue based on 2 individual commits that 
were pulled into a kernel base that does not contain the other commits 
found in the upstream tree, is dangerous.

It's recommended that the issue be reported when running what is in the 
upstream tree (at least the infradead branch).

Also, relative to:
 >? It appears to me the caller nvme_free_ctrl()? is using a device name 
that has been cleared :
 >? ? ?? sysfs_remove_link(&subsys->dev.kobj, dev_name(ctrl->device));
 >? The name passed in is null in kernfs_remove_by_name_ns() :
 >
 >? if (!parent) {
 >? ? ? WARN(1, KERN_WARNING "kernfs: can not remove '%s', no 
directory\n",?? name);

This is the nvme controller device, referencing it's own device struct 
which it created it's own name for, which has its parent set to the the 
nvmf_device passed through nvmf_create_ctrl() to the transport 
ops->create_ctrl() to nvme_init_ctrl().


>
>> Here is the preamble to the previous stack trace.
>>
>> May 24 21:09:31 interop-57-161 kernel: nvme nvme0: NVME-FC{0}: reset: 
>> Reconnect
>> attempt failed (-22)
>> May 24 21:09:31 interop-57-161 kernel: nvme nvme0: NVME-FC{0}: Max 
>> reconnect
>> attempts (60) reached.
>> May 24 21:09:31 interop-57-161 kernel: nvme nvme0: Removing ctrl: NQN
>> "nqn.1992-08.com.netapp:sn.3d9037c53fed11e9922200a0986a8b60:subsystem.ol_nvme1_ss1" 
>>
>> May 24 21:09:31 interop-57-161 kernel: kernfs: can not remove 
>> 'nvme0', no directory
>> May 24 21:09:31 interop-57-161 kernel: ------------[ cut here 
>> ]------------
>> May 24 21:09:31 interop-57-161 kernel: WARNING: CPU: 6 PID: 14206 at
>> fs/kernfs/dir.c:1481 kernfs_remove_by_name_ns+0x7e/0x87
>>
>> We are not seeing failover work at all using the lpfc driver. The 
>> connections are established on boot, but when one of the paths are 
>> disabled on the NetApp target the system never is able to restores 
>> the paths.
>
> So I/O failover is not working with lpfc? that seems to be a different
> issue to me. What do you mean that the NetApp target never is able to
> restore the paths? That is just an explanation to what triggers the
> bug or is that somehow related?

too many leaps and bounds are being stated when the base sources aren't 
consistent.?? Things work when all the dependencies are put together.? 
When bits are cherry picked - it's a b*#%@ to figure out which piece is 
missing to get everything working together.

-- james

[GIT PULL] nvme fixes for 5.2

[John Donnelly]

> On May 30, 2019,@1:35 PM, Sagi Grimberg <sagi@grimberg.me> wrote:
> 
> 
>> HI Sagi;
>>     Prior to implementing the noted commits we never gotten this far before in recovery because the system encountered the OOPS as noted in those commits and fell apart with just one namespace. Now this  occurs during path failover testing while trying to reconnect two name spaces,  and it  ended up with max reconnect attempts of 60 that produced a  different stack trace.
> 
> I see.
> 
>> A message did appear in the log with the name ?nvme0? , but I can?t tell if that is the device name that was  being deleted , or the next one :  nvme1;  I have no farther explanation why the  OOPS would occur in the print statement unless ?name? was null,   do you ?
> 
> This does not realte to the device name, but rather to the device parent
> that seems to be NULL (we have a valid kobj but it points to a NULL
> parent. This usually happens in a use-after-free conditions.
> 
> What is the FC device parent? the FC adapter device? or the /dev/nvme-fabrics device?

   I don?t know yet.  This  event appears to  only happen with two name-spaces configured. 
   This particular event is not seen with a single name-space. 

> 
>> Here is the preamble to the previous stack trace.
>> May 24 21:09:31 interop-57-161 kernel: nvme nvme0: NVME-FC{0}: reset: Reconnect
>> attempt failed (-22)
>> May 24 21:09:31 interop-57-161 kernel: nvme nvme0: NVME-FC{0}: Max reconnect
>> attempts (60) reached.
>> May 24 21:09:31 interop-57-161 kernel: nvme nvme0: Removing ctrl: NQN
>> "nqn.1992-08.com.netapp:sn.3d9037c53fed11e9922200a0986a8b60:subsystem.ol_nvme1_ss1"
>> May 24 21:09:31 interop-57-161 kernel: kernfs: can not remove 'nvme0', no directory
>> May 24 21:09:31 interop-57-161 kernel: ------------[ cut here ]------------
>> May 24 21:09:31 interop-57-161 kernel: WARNING: CPU: 6 PID: 14206 at
>> fs/kernfs/dir.c:1481 kernfs_remove_by_name_ns+0x7e/0x87
>> We are not seeing failover work at all using the lpfc driver.   The connections are established on boot, but when one of the paths are disabled on the NetApp target the system never is able to restores the paths.
> 
> So I/O failover is not working with lpfc? that seems to be a different
> issue to me.
  
That is correct. Had the FO worked  it would not have gone through the recovery attempts and gotten into this situation.  What makes matters worse is the use of ?WARNING? in the nvme subsystem  ( see my previous email of the use of WARNINGS ) ; It launches  abrt (sos)  handling which does inspections of various filesystems using tools like pvscan that causes hung processes and hung dead-man timer events.    

> What do you mean that the NetApp target never is able to
> restore the paths? 

   That was mis-wording on my part; The NetApp target did present the paths again, but  client never recovered and those nvme devices stayed in ?live?  ,  ? inaccessible ? state.    

>  That is just an explanation to what triggers the
> bug or is that somehow related?

[GIT PULL] nvme fixes for 5.2

[John Donnelly]

> On May 30, 2019,@2:21 PM, James Smart <james.smart@broadcom.com> wrote:
> 
> 
> 
> On 5/30/2019 11:35 AM, Sagi Grimberg wrote:
>> 
>>> HI Sagi;
>>> 
>>>      Prior to implementing the noted commits we never gotten this far before in recovery because the system encountered the OOPS as noted in those commits and fell apart with just one namespace. Now this  occurs during path failover testing while trying to reconnect two name spaces,  and it  ended up with max reconnect attempts of 60 that produced a  different stack trace.
>> 
>> I see.
>> 
>>> A message did appear in the log with the name ?nvme0? , but I can?t tell if that is the device name that was  being deleted , or the next one :  nvme1;  I have no farther explanation why the  OOPS would occur in the print statement unless ?name? was null,   do you ?
>> 
>> This does not realte to the device name, but rather to the device parent
>> that seems to be NULL (we have a valid kobj but it points to a NULL
>> parent. This usually happens in a use-after-free conditions.
>> 
>> What is the FC device parent? the FC adapter device? or the /dev/nvme-fabrics device?
> 
> Warning:   tracking down an issue based on 2 individual commits that were pulled into a kernel base that does not contain the other commits found in the upstream tree, is dangerous.

 Yes. I can see that ; 
> 
> It's recommended that the issue be reported when running what is in the upstream tree (at least the infradead branch).
> 
> Also, relative to:
> >  It appears to me the caller nvme_free_ctrl()  is using a device name that has been cleared :
> >       sysfs_remove_link(&subsys->dev.kobj, dev_name(ctrl->device));
> >  The name passed in is null in kernfs_remove_by_name_ns() :
> >
> >  if (!parent) {
> >      WARN(1, KERN_WARNING "kernfs: can not remove '%s', no directory\n",   name);
> 
> This is the nvme controller device, referencing it's own device struct which it created it's own name for, which has its parent set to the the nvmf_device passed through nvmf_create_ctrl() to the transport ops->create_ctrl() to nvme_init_ctrl().
> 
> 
>> 
>>> Here is the preamble to the previous stack trace.
>>> 
>>> May 24 21:09:31 interop-57-161 kernel: nvme nvme0: NVME-FC{0}: reset: Reconnect
>>> attempt failed (-22)
>>> May 24 21:09:31 interop-57-161 kernel: nvme nvme0: NVME-FC{0}: Max reconnect
>>> attempts (60) reached.
>>> May 24 21:09:31 interop-57-161 kernel: nvme nvme0: Removing ctrl: NQN
>>> "nqn.1992-08.com.netapp:sn.3d9037c53fed11e9922200a0986a8b60:subsystem.ol_nvme1_ss1" 
>>> May 24 21:09:31 interop-57-161 kernel: kernfs: can not remove 'nvme0', no directory
>>> May 24 21:09:31 interop-57-161 kernel: ------------[ cut here ]------------
>>> May 24 21:09:31 interop-57-161 kernel: WARNING: CPU: 6 PID: 14206 at
>>> fs/kernfs/dir.c:1481 kernfs_remove_by_name_ns+0x7e/0x87
>>> 
>>> We are not seeing failover work at all using the lpfc driver. The connections are established on boot, but when one of the paths are disabled on the NetApp target the system never is able to restores the paths.
>> 
>> So I/O failover is not working with lpfc? that seems to be a different
>> issue to me. What do you mean that the NetApp target never is able to
>> restore the paths? That is just an explanation to what triggers the
>> bug or is that somehow related?
> 
> too many leaps and bounds are being stated when the base sources aren't consistent.   Things work when all the dependencies are put together.  When bits are cherry picked - it's a b*#%@ to figure out which piece is missing to get everything working together.
> 
> -- james
> 
> 
> 
> 
> _______________________________________________
> Linux-nvme mailing list
> Linux-nvme at lists.infradead.org
> https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.infradead.org_mailman_listinfo_linux-2Dnvme&d=DwIGaQ&c=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=t2fPg9D87F7D8jm0_3CG9yoiIKdRg4qc_thBw4bzMhc&m=EJDfq5Vcx2ufCv4SsZ1s5hnMGt-2lrtfPwCuuS-dbhI&s=vJVjop4Ccn8M-VxkQ2WwRE4nHdxS39qjgARDpnrOWTI&e=

[GIT PULL] nvme fixes for 5.2

[Sagi Grimberg]

> too many leaps and bounds are being stated when the base sources aren't 
> consistent.?? Things work when all the dependencies are put together. 
> When bits are cherry picked - it's a b*#%@ to figure out which piece is 
> missing to get everything working together.


Didn't realize that this was not upstream tested.. So in upstream we are
good?

[GIT PULL] nvme fixes for 5.2

[James Smart]

On 5/31/2019 9:47 AM, Sagi Grimberg wrote:
>
>> too many leaps and bounds are being stated when the base sources 
>> aren't consistent.?? Things work when all the dependencies are put 
>> together. When bits are cherry picked - it's a b*#%@ to figure out 
>> which piece is missing to get everything working together.
>
>
> Didn't realize that this was not upstream tested.. So in upstream we are
> good?

AFAIK - there is plenty of success elsewhere.

-- james