[bug report] nvme authentication setup failed observed during blktests nvme/041 nvme/042 nvme/043

[bug report] nvme authentication setup failed observed during blktests nvme/041 nvme/042 nvme/043

[Yi Zhang]

Hello
Just found the blktests nvme/041 nvme/042 nvme/043[2] failed on the
latest linux-block/for-next[1],
from the log I can see it was due to authentication setup failed,
please help check it, thanks.

[1]
https://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux-block.git/log/?h=for-next
e3db512c4ab6 (HEAD -> for-next, origin/for-next) Merge branch
'for-6.7/block' into for-next

[2]
# ./check nvme/041
nvme/041 (Create authenticated connections)                  [failed]
    runtime  3.274s  ...  3.980s
    --- tests/nvme/041.out      2023-10-17 08:02:17.046653814 -0400
    +++ /root/blktests/results/nodev/nvme/041.out.bad   2023-10-18
02:50:03.496539083 -0400
    @@ -2,5 +2,5 @@
     Test unauthenticated connection (should fail)
     NQN:blktests-subsystem-1 disconnected 0 controller(s)
     Test authenticated connection
    -NQN:blktests-subsystem-1 disconnected 1 controller(s)
    +NQN:blktests-subsystem-1 disconnected 0 controller(s)
     Test complete

# dmesg
[ 2701.636964] loop: module loaded
[ 2702.074262] run blktests nvme/041 at 2023-10-18 02:49:59
[ 2702.302067] nvmet: adding nsid 1 to subsystem blktests-subsystem-1
[ 2702.447496] nvmet: creating nvm controller 1 for subsystem
blktests-subsystem-1 for NQN
nqn.2014-08.org.nvmexpress:uuid:0f01fb42-9f7f-4856-b0b3-51e60b8de349
with DH-HMAC-CHAP.
[ 2702.447707] nvme nvme0: qid 0: authentication setup failed
[ 2704.099618] nvmet: creating nvm controller 1 for subsystem
blktests-subsystem-1 for NQN
nqn.2014-08.org.nvmexpress:uuid:0f01fb42-9f7f-4856-b0b3-51e60b8de349
with DH-HMAC-CHAP.
[ 2704.099688] nvme nvme0: qid 0: authentication setup failed


-- 
Best Regards,
  Yi Zhang

Re: [bug report][bisected] nvme authentication setup failed observed during blktests nvme/041 nvme/042 nvme/043

[Yi Zhang]

Hi Hanns

Bisect shows it was introduced with this commit.

commit d680063482885c15d68e958212c3d6ad40a510dd (HEAD)
Author: Hannes Reinecke <hare@suse.de>
Date:   Thu Oct 12 14:22:48 2023 +0200

    nvme: rework NVME_AUTH Kconfig selection

    Having a single Kconfig symbol NVME_AUTH conflates the selection
    of the authentication functions from nvme/common and nvme/host,
    causing kbuild robot to complain when building the nvme target
    only. So introduce a Kconfig symbol NVME_HOST_AUTH for the nvme
    host bits and use NVME_AUTH for the common functions only.
    And move the CRYPTO selection into nvme/common to make it
    easier to read.

On Wed, Oct 18, 2023 at 2:57 PM Yi Zhang <yi.zhang@redhat.com> wrote:
>
> Hello
> Just found the blktests nvme/041 nvme/042 nvme/043[2] failed on the
> latest linux-block/for-next[1],
> from the log I can see it was due to authentication setup failed,
> please help check it, thanks.
>
> [1]
> https://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux-block.git/log/?h=for-next
> e3db512c4ab6 (HEAD -> for-next, origin/for-next) Merge branch
> 'for-6.7/block' into for-next
>
> [2]
> # ./check nvme/041
> nvme/041 (Create authenticated connections)                  [failed]
>     runtime  3.274s  ...  3.980s
>     --- tests/nvme/041.out      2023-10-17 08:02:17.046653814 -0400
>     +++ /root/blktests/results/nodev/nvme/041.out.bad   2023-10-18
> 02:50:03.496539083 -0400
>     @@ -2,5 +2,5 @@
>      Test unauthenticated connection (should fail)
>      NQN:blktests-subsystem-1 disconnected 0 controller(s)
>      Test authenticated connection
>     -NQN:blktests-subsystem-1 disconnected 1 controller(s)
>     +NQN:blktests-subsystem-1 disconnected 0 controller(s)
>      Test complete
>
> # dmesg
> [ 2701.636964] loop: module loaded
> [ 2702.074262] run blktests nvme/041 at 2023-10-18 02:49:59
> [ 2702.302067] nvmet: adding nsid 1 to subsystem blktests-subsystem-1
> [ 2702.447496] nvmet: creating nvm controller 1 for subsystem
> blktests-subsystem-1 for NQN
> nqn.2014-08.org.nvmexpress:uuid:0f01fb42-9f7f-4856-b0b3-51e60b8de349
> with DH-HMAC-CHAP.
> [ 2702.447707] nvme nvme0: qid 0: authentication setup failed
> [ 2704.099618] nvmet: creating nvm controller 1 for subsystem
> blktests-subsystem-1 for NQN
> nqn.2014-08.org.nvmexpress:uuid:0f01fb42-9f7f-4856-b0b3-51e60b8de349
> with DH-HMAC-CHAP.
> [ 2704.099688] nvme nvme0: qid 0: authentication setup failed
>
>
> --
> Best Regards,
>   Yi Zhang



-- 
Best Regards,
  Yi Zhang

Re: [bug report][bisected] nvme authentication setup failed observed during blktests nvme/041 nvme/042 nvme/043

[Yi Zhang]

Hi Hannes

The issue still can be reproduced on the latest linux-block/for-next,
do you have a chance to check it, thanks.

On Thu, Oct 19, 2023 at 3:16 PM Yi Zhang <yi.zhang@redhat.com> wrote:
>
> Hi Hanns
>
> Bisect shows it was introduced with this commit.
>
> commit d680063482885c15d68e958212c3d6ad40a510dd (HEAD)
> Author: Hannes Reinecke <hare@suse.de>
> Date:   Thu Oct 12 14:22:48 2023 +0200
>
>     nvme: rework NVME_AUTH Kconfig selection
>
>     Having a single Kconfig symbol NVME_AUTH conflates the selection
>     of the authentication functions from nvme/common and nvme/host,
>     causing kbuild robot to complain when building the nvme target
>     only. So introduce a Kconfig symbol NVME_HOST_AUTH for the nvme
>     host bits and use NVME_AUTH for the common functions only.
>     And move the CRYPTO selection into nvme/common to make it
>     easier to read.
>
> On Wed, Oct 18, 2023 at 2:57 PM Yi Zhang <yi.zhang@redhat.com> wrote:
> >
> > Hello
> > Just found the blktests nvme/041 nvme/042 nvme/043[2] failed on the
> > latest linux-block/for-next[1],
> > from the log I can see it was due to authentication setup failed,
> > please help check it, thanks.
> >
> > [1]
> > https://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux-block.git/log/?h=for-next
> > e3db512c4ab6 (HEAD -> for-next, origin/for-next) Merge branch
> > 'for-6.7/block' into for-next
> >
> > [2]
> > # ./check nvme/041
> > nvme/041 (Create authenticated connections)                  [failed]
> >     runtime  3.274s  ...  3.980s
> >     --- tests/nvme/041.out      2023-10-17 08:02:17.046653814 -0400
> >     +++ /root/blktests/results/nodev/nvme/041.out.bad   2023-10-18
> > 02:50:03.496539083 -0400
> >     @@ -2,5 +2,5 @@
> >      Test unauthenticated connection (should fail)
> >      NQN:blktests-subsystem-1 disconnected 0 controller(s)
> >      Test authenticated connection
> >     -NQN:blktests-subsystem-1 disconnected 1 controller(s)
> >     +NQN:blktests-subsystem-1 disconnected 0 controller(s)
> >      Test complete
> >
> > # dmesg
> > [ 2701.636964] loop: module loaded
> > [ 2702.074262] run blktests nvme/041 at 2023-10-18 02:49:59
> > [ 2702.302067] nvmet: adding nsid 1 to subsystem blktests-subsystem-1
> > [ 2702.447496] nvmet: creating nvm controller 1 for subsystem
> > blktests-subsystem-1 for NQN
> > nqn.2014-08.org.nvmexpress:uuid:0f01fb42-9f7f-4856-b0b3-51e60b8de349
> > with DH-HMAC-CHAP.
> > [ 2702.447707] nvme nvme0: qid 0: authentication setup failed
> > [ 2704.099618] nvmet: creating nvm controller 1 for subsystem
> > blktests-subsystem-1 for NQN
> > nqn.2014-08.org.nvmexpress:uuid:0f01fb42-9f7f-4856-b0b3-51e60b8de349
> > with DH-HMAC-CHAP.
> > [ 2704.099688] nvme nvme0: qid 0: authentication setup failed
> >
> >
> > --
> > Best Regards,
> >   Yi Zhang
>
>
>
> --
> Best Regards,
>   Yi Zhang



-- 
Best Regards,
  Yi Zhang

Re: [bug report][bisected] nvme authentication setup failed observed during blktests nvme/041 nvme/042 nvme/043

[Chris Leech]

Hi Yi Zhang,

Where is your kernel configuration coming from?  If it's carried forward
from an older kernel, it may have the NVME_AUTH symbol set but not
NVME_HOST_AUTH.  That would now just enabled the shared host/target core
auth code, but not the host support.  I think updating your kernel
config to include NVME_HOST_AUTH will fix this.

- Chris

On Thu, Oct 19, 2023 at 03:16:13PM +0800, Yi Zhang wrote:
> Hi Hanns
> 
> Bisect shows it was introduced with this commit.
> 
> commit d680063482885c15d68e958212c3d6ad40a510dd (HEAD)
> Author: Hannes Reinecke <hare@suse.de>
> Date:   Thu Oct 12 14:22:48 2023 +0200
> 
>     nvme: rework NVME_AUTH Kconfig selection
> 
>     Having a single Kconfig symbol NVME_AUTH conflates the selection
>     of the authentication functions from nvme/common and nvme/host,
>     causing kbuild robot to complain when building the nvme target
>     only. So introduce a Kconfig symbol NVME_HOST_AUTH for the nvme
>     host bits and use NVME_AUTH for the common functions only.
>     And move the CRYPTO selection into nvme/common to make it
>     easier to read.
> 
> On Wed, Oct 18, 2023 at 2:57 PM Yi Zhang <yi.zhang@redhat.com> wrote:
> >
> > Hello
> > Just found the blktests nvme/041 nvme/042 nvme/043[2] failed on the
> > latest linux-block/for-next[1],
> > from the log I can see it was due to authentication setup failed,
> > please help check it, thanks.
> >
> > [1]
> > https://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux-block.git/log/?h=for-next
> > e3db512c4ab6 (HEAD -> for-next, origin/for-next) Merge branch
> > 'for-6.7/block' into for-next
> >
> > [2]
> > # ./check nvme/041
> > nvme/041 (Create authenticated connections)                  [failed]
> >     runtime  3.274s  ...  3.980s
> >     --- tests/nvme/041.out      2023-10-17 08:02:17.046653814 -0400
> >     +++ /root/blktests/results/nodev/nvme/041.out.bad   2023-10-18
> > 02:50:03.496539083 -0400
> >     @@ -2,5 +2,5 @@
> >      Test unauthenticated connection (should fail)
> >      NQN:blktests-subsystem-1 disconnected 0 controller(s)
> >      Test authenticated connection
> >     -NQN:blktests-subsystem-1 disconnected 1 controller(s)
> >     +NQN:blktests-subsystem-1 disconnected 0 controller(s)
> >      Test complete
> >
> > # dmesg
> > [ 2701.636964] loop: module loaded
> > [ 2702.074262] run blktests nvme/041 at 2023-10-18 02:49:59
> > [ 2702.302067] nvmet: adding nsid 1 to subsystem blktests-subsystem-1
> > [ 2702.447496] nvmet: creating nvm controller 1 for subsystem
> > blktests-subsystem-1 for NQN
> > nqn.2014-08.org.nvmexpress:uuid:0f01fb42-9f7f-4856-b0b3-51e60b8de349
> > with DH-HMAC-CHAP.
> > [ 2702.447707] nvme nvme0: qid 0: authentication setup failed
> > [ 2704.099618] nvmet: creating nvm controller 1 for subsystem
> > blktests-subsystem-1 for NQN
> > nqn.2014-08.org.nvmexpress:uuid:0f01fb42-9f7f-4856-b0b3-51e60b8de349
> > with DH-HMAC-CHAP.
> > [ 2704.099688] nvme nvme0: qid 0: authentication setup failed
> >
> >
> > --
> > Best Regards,
> >   Yi Zhang
> 
> 
> 
> -- 
> Best Regards,
>   Yi Zhang
> 

Re: [bug report][bisected] nvme authentication setup failed observed during blktests nvme/041 nvme/042 nvme/043

[Shinichiro Kawasaki]

On Nov 13, 2023 / 09:58, Chris Leech wrote:
> Hi Yi Zhang,
> 
> Where is your kernel configuration coming from?  If it's carried forward
> from an older kernel, it may have the NVME_AUTH symbol set but not
> NVME_HOST_AUTH.  That would now just enabled the shared host/target core
> auth code, but not the host support.  I think updating your kernel
> config to include NVME_HOST_AUTH will fix this.

Hi, Yi, Chris,

I observed the same failure using kernel version v6.7-rc1. In my environment,
five test cases failed from nvme/041 to nvme/045. I enabled NVME_HOST_AUTH and
confirmed the failures go away.

My understanding is that the test cases newly require NVME_HOST_AUTH since
kernel v6.7. Now I'm preparing blktests patches to check that requirement [1].
Will post them after some more confirmations.

[1] https://github.com/kawasaki/blktests/commit/e0d9d4c8b981e291baf25c6179802278cd726521