TLS over TCP connection failure

TLS over TCP connection failure

[Nir Rigai]

Hi all,

We failed to enable TLS over TCP with the latest 6.8-rc3 from upstream.
We have followed the guidelines outlined in https://lwn.net/Articles/941139/.
The NVMe connection failed post-authentication

The steps and the outputs from the process:  
# Target creation
modprobe nvmet-tcp
mkdir /sys/kernel/config/nvmet/ports/10
echo -n "127.0.0.1" > /sys/kernel/config/nvmet/ports/10/addr_traddr
echo -n ipv4 > /sys/kernel/config/nvmet/ports/10/addr_adrfam
echo -n tcp > /sys/kernel/config/nvmet/ports/10/addr_trtype
echo -n 4420 > /sys/kernel/config/nvmet/ports/10/addr_trsvcid
echo tls1.3 > /sys/kernel/config/nvmet/ports/10/addr_tsas
mkdir /sys/kernel/config/nvmet/subsystems/nqn.test
echo 1 > /sys/kernel/config/nvmet/subsystems/nqn.test/attr_allow_any_host
mkdir /sys/kernel/config/nvmet/subsystems/nqn.test/namespaces/1
echo "/dev/nvme1n1" > /sys/kernel/config/nvmet/subsystems/nqn.test/namespaces/1/device_path
echo 1 > /sys/kernel/config/nvmet/subsystems/nqn.test/namespaces/1/enable
ln -s /sys/kernel/config/nvmet/subsystems/nqn.test /sys/kernel/config/nvmet/ports/10/subsystems/
  # Create keys and start the tlshd service:
modprobe nvme-tcp
nvme gen-tls-key --subsysnqn=nqn.test -i
nvme gen-tls-key --subsysnqn=nqn.2014-08.org.nvmexpress.discovery -i
#tlshd -c /etc/tlshd.conf
systemctl start tlshd.service
nvme connect -t tcp -a 127.0.0.1 -s 4420 -n nqn.test --tls

------------------------

# Outcome of enabling TLS over TCP – While authentication succeeded as expected,
we encountered an issue where the NVMe connection failed:

root@ip-10-0-40-151:~/work# nvme list-subsys
nvme-subsys2 - NQN=nqn.test
               hostnqn=nqn.2014-08.org.nvmexpress:uuid:ec20ce15-ac61-d1d3-0b2e-dcf390b05133
               iopolicy=numa
\
+- nvme2 tcp traddr=127.0.0.1,trsvcid=4420 connecting  nvme-subsys1 - NQN=nqn:2008-08.com.amazon.aws:ebs:vol0ada4ab89cbeec6a8
               hostnqn=nqn.2014-08.org.nvmexpress:uuid:ec20ce15-ac61-d1d3-0b2e-dcf390b05133
               iopolicy=numa
\
+- nvme1 pcie 0000:00:1f.0 live
nvme-subsys0 - NQN=nqn:2008-08.com.amazon.aws:ebs:vol0867e65caaa7cb8ca
               hostnqn=nqn.2014-08.org.nvmexpress:uuid:ec20ce15-ac61-d1d3-0b2e-dcf390b05133
               iopolicy=numa
\
+- nvme0 pcie 0000:00:04.0 live

# The initiator syslog:
Feb  5 16:17:36 ip-10-0-40-151 kernel: [ 2067.496761] nvmet: creating nvm controller 1 for subsystem nqn.test for NQN nqn.2014-08.org.nvmexpress:uuid:ec20ce15-ac61-d1d3-0b2e-dcf390b05133.
Feb  5 16:17:36 ip-10-0-40-151 kernel: [ 2067.497751] nvme nvme2: creating 32 I/O queues.
Feb  5 16:17:36 ip-10-0-40-151 kernel: [ 2067.646851] nvme nvme2: mapped 32/0/0 default/read/poll queues.
Feb  5 16:17:36 ip-10-0-40-151 kernel: [ 2067.647105] nvme nvme2: Successfully reconnected (1 attempt)
Feb  5 16:17:43 ip-10-0-40-151 kernel: [ 2075.076927] nvme nvme2: I/O tag 0 (a000) type 4 opcode 0x18 (Admin Cmd) QID 0 timeout
Feb  5 16:17:43 ip-10-0-40-151 kernel: [ 2075.076940] nvme nvme2: starting error recovery
Feb  5 16:17:43 ip-10-0-40-151 kernel: [ 2075.078899] nvme nvme2: failed nvme_keep_alive_end_io error=10
Feb  5 16:17:43 ip-10-0-40-151 kernel: [ 2075.080682] nvme nvme2: reading non-mdts-limits failed: -4
Feb  5 16:17:43 ip-10-0-40-151 kernel: [ 2075.092748] nvme nvme2: Reconnecting in 10 seconds…
  # tlshd configuration:
[authentication]
keyrings = .nvme
[debug]
loglevel=255
tls=255

We would greatly appreciate your assistance with this matter. Your response
would be highly valued.

Thanks,
Nir Rigai     

Re: TLS over TCP connection failure

[Chaitanya Kulkarni]

On 2/11/24 07:05, Nir Rigai wrote:
> Hi all,
>
> We failed to enable TLS over TCP with the latest 6.8-rc3 from upstream.
> We have followed the guidelines outlined in https://lwn.net/Articles/941139/.
> The NVMe connection failed post-authentication
>
> The steps and the outputs from the process:
> # Target creation
> modprobe nvmet-tcp
> mkdir /sys/kernel/config/nvmet/ports/10
> echo -n "127.0.0.1" > /sys/kernel/config/nvmet/ports/10/addr_traddr
> echo -n ipv4 > /sys/kernel/config/nvmet/ports/10/addr_adrfam
> echo -n tcp > /sys/kernel/config/nvmet/ports/10/addr_trtype
> echo -n 4420 > /sys/kernel/config/nvmet/ports/10/addr_trsvcid
> echo tls1.3 > /sys/kernel/config/nvmet/ports/10/addr_tsas
> mkdir /sys/kernel/config/nvmet/subsystems/nqn.test
> echo 1 > /sys/kernel/config/nvmet/subsystems/nqn.test/attr_allow_any_host
> mkdir /sys/kernel/config/nvmet/subsystems/nqn.test/namespaces/1
> echo "/dev/nvme1n1" > /sys/kernel/config/nvmet/subsystems/nqn.test/namespaces/1/device_path
> echo 1 > /sys/kernel/config/nvmet/subsystems/nqn.test/namespaces/1/enable
> ln -s /sys/kernel/config/nvmet/subsystems/nqn.test /sys/kernel/config/nvmet/ports/10/subsystems/
>    # Create keys and start the tlshd service:
> modprobe nvme-tcp
> nvme gen-tls-key --subsysnqn=nqn.test -i
> nvme gen-tls-key --subsysnqn=nqn.2014-08.org.nvmexpress.discovery -i
> #tlshd -c /etc/tlshd.conf
> systemctl start tlshd.service
> nvme connect -t tcp -a 127.0.0.1 -s 4420 -n nqn.test --tls
>

can you please submit the blktests for this if it is not there in nvme 
category ?
it will help this to get regularly tested and establish stability ...

please CC Shinichiro (CC'd here) and me on that

https://github.com/osandov/blktests

-ck

Re: TLS over TCP connection failure

[Nir Rigai]

Thanks for the quick response.
The issue is related to an unsuccessful NVMe connection to the target. In this case there isn't a block device available to test with the blktests framework.

Nir Rigai

> On 12 Feb 2024, at 1:45, Chaitanya Kulkarni <chaitanyak@nvidia.com> wrote:
> 
> On 2/11/24 07:05, Nir Rigai wrote:
>> Hi all,
>> 
>> We failed to enable TLS over TCP with the latest 6.8-rc3 from upstream.
>> We have followed the guidelines outlined in https://lwn.net/Articles/941139/.
>> The NVMe connection failed post-authentication
>> 
>> The steps and the outputs from the process:
>> # Targetreation
>> modprobe nvmet-tcp
>> mkdir /sys/kernel/config/nvmet/ports/10
>> echo -n "127.0.0.1" > /sys/kernel/config/nvmet/cports/10/addr_traddr
>> echo -n ipv4 > /sys/kernel/config/nvmet/ports/10/addr_adrfam
>> echo -n tcp > /sys/kernel/config/nvmet/ports/10/addr_trtype
>> echo -n 4420 > /sys/kernel/config/nvmet/ports/10/addr_trsvcid
>> echo tls1.3 > /sys/kernel/config/nvmet/ports/10/addr_tsas
>> mkdir /sys/kernel/config/nvmet/subsystems/nqn.test
>> echo 1 > /sys/kernel/config/nvmet/subsystems/nqn.test/attr_allow_any_host
>> mkdir /sys/kernel/config/nvmet/subsystems/nqn.test/namespaces/1
>> echo "/dev/nvme1n1" > /sys/kernel/config/nvmet/subsystems/nqn.test/namespaces/1/device_path
>> echo 1 > /sys/kernel/config/nvmet/subsystems/nqn.test/namespaces/1/enable
>> ln -s /sys/kernel/config/nvmet/subsystems/nqn.test /sys/kernel/config/nvmet/ports/10/subsystems/
>>   # Create keys and start the tlshd service:
>> modprobe nvme-tcp
>> nvme gen-tls-key --subsysnqn=nqn.test -i
>> nvme gen-tls-key --subsysnqn=nqn.2014-08.org.nvmexpress.discovery -i
>> #tlshd -c /etc/tlshd.conf
>> systemctl start tlshd.service
>> nvme connect -t tcp -a 127.0.0.1 -s 4420 -n nqn.test --tls
>> 
> 
> can you please submit the blktests for this if it is not there in nvme 
> category ?
> it will help this to get regularly tested and establish stability ...
> 
> please CC Shinichiro (CC'd here) and me on that
> 
> https://github.com/osandov/blktests
> 
> -ck