[PATCHv3 0/2] add nvme test for creating sleep while atomic kernel BUG

[PATCHv3 0/2] add nvme test for creating sleep while atomic kernel BUG

[Nilay Shroff]

Hi,

There're two patches in this series. The first patch is a preparation patch
for reusing a common function nvmf_wait_for_ns from multiple nvme test scripts.
The second patch adds a new nvme regression[1] test for commit 505363957fad
("nvmet: fix nvme status code when namespace is disabled").

For fixing regression, as suggested by Christoph[2], I would send a separate
patch upstream.

[1] https://lore.kernel.org/linux-nvme/tqcy3sveity7p56v7ywp7ssyviwcb3w4623cnxj3knoobfcanq@yxgt2mjkbkam/
[2] https://lore.kernel.org/linux-nvme/20241022070252.GA11389@lst.de/

Changes from v2:
    - Update the comment in second pacth to highlight the fact that this 
      test doesn't depend on udev rules to recreate the regression
      (Shinichiro Kawasaki)
    - Instead of creating source/data file using mktemp, use /dev/urandom
      as data file to nvme write command (Shinichiro Kawasaki)

Changes from v1:
    - Update the second patch in the series so that we can trigger the
      regression 100% of time when we run the test (Chaitanya Kulkarni)


Nilay Shroff (2):
  nvme/052: move nvmf_wait_for_ns() to common/nvme
  nvme: add test for writing to file-ns just after disabling it

 common/nvme        |  26 ++++++++++
 tests/nvme/052     |  30 +-----------
 tests/nvme/055     | 117 +++++++++++++++++++++++++++++++++++++++++++++
 tests/nvme/055.out |   2 +
 4 files changed, 147 insertions(+), 28 deletions(-)
 create mode 100755 tests/nvme/055
 create mode 100644 tests/nvme/055.out

-- 
2.45.2

[PATCHv3 1/2] nvme/052: move nvmf_wait_for_ns() to common/nvme

[Nilay Shroff]

In a preparation for the use of nvmf_wait_for_ns function from
other nvme test script, move nvmf_wait_for_ns from nvme/052 to
common/nvme file so that we can reuse the nvmf_wait_for_ns from
multiple test scripts.

Reviewed-by: Chaitanya Kulkarni <kch@nvidia.com>
Signed-off-by: Nilay Shroff <nilay@linux.ibm.com>
---
 common/nvme    | 26 ++++++++++++++++++++++++++
 tests/nvme/052 | 30 ++----------------------------
 2 files changed, 28 insertions(+), 28 deletions(-)

diff --git a/common/nvme b/common/nvme
index fd472fe..6b8a57a 100644
--- a/common/nvme
+++ b/common/nvme
@@ -483,6 +483,32 @@ _remove_nvmet_port() {
 	rmdir "${NVMET_CFS}/ports/${port}"
 }
 
+# Wait for the namespace with specified uuid to fulfill the specified condtion,
+# "created" or "removed".
+_nvmf_wait_for_ns() {
+    local ns
+    local timeout="5"
+    local uuid="$1"
+    local condition="$2"
+
+    ns=$(_find_nvme_ns "${uuid}" 2>> "$FULL")
+
+    start_time=$(date +%s)
+    while [[ -z "$ns" && "$condition" == created ]] ||
+              [[ -n "$ns" && "$condition" == removed ]]; do
+        sleep .1
+        end_time=$(date +%s)
+        if (( end_time - start_time > timeout )); then
+            echo "namespace with uuid \"${uuid}\" not " \
+                "${condition} within ${timeout} seconds"
+            return 1
+        fi
+        ns=$(_find_nvme_ns "${uuid}" 2>> "$FULL")
+    done
+
+    return 0
+}
+
 _create_nvmet_ns() {
 	local subsysnqn="${def_subsysnqn}"
 	local nsid="${def_nsid}"
diff --git a/tests/nvme/052 b/tests/nvme/052
index 8443c90..22d8a9e 100755
--- a/tests/nvme/052
+++ b/tests/nvme/052
@@ -20,32 +20,6 @@ set_conditions() {
 	_set_nvme_trtype "$@"
 }
 
-# Wait for the namespace with specified uuid to fulfill the specified condtion,
-# "created" or "removed".
-nvmf_wait_for_ns() {
-	local ns
-	local timeout="5"
-	local uuid="$1"
-	local condition="$2"
-
-	ns=$(_find_nvme_ns "${uuid}" 2>> "$FULL")
-
-	start_time=$(date +%s)
-	while [[ -z "$ns" && "$condition" == created ]] ||
-		      [[ -n "$ns" && "$condition" == removed ]]; do
-		sleep .1
-		end_time=$(date +%s)
-		if (( end_time - start_time > timeout )); then
-			echo "namespace with uuid \"${uuid}\" not " \
-				"${condition} within ${timeout} seconds"
-			return 1
-		fi
-		ns=$(_find_nvme_ns "${uuid}" 2>> "$FULL")
-	done
-
-	return 0
-}
-
 test() {
 	echo "Running ${TEST_NAME}"
 
@@ -65,7 +39,7 @@ test() {
 		uuid=$(_create_nvmet_ns --blkdev "$filepath" --nsid "${nsid}")
 
 		# wait until async request is processed and ns is created
-		if ! nvmf_wait_for_ns "${uuid}" created; then
+		if ! _nvmf_wait_for_ns "${uuid}" created; then
 			echo "FAIL"
 			rm "$filepath"
 			break
@@ -74,7 +48,7 @@ test() {
 		_remove_nvmet_ns "${def_subsysnqn}" "${nsid}"
 
 		# wait until async request is processed and ns is removed
-		if ! nvmf_wait_for_ns "${uuid}" removed; then
+		if ! _nvmf_wait_for_ns "${uuid}" removed; then
 			echo "FAIL"
 			rm "$filepath"
 			break
-- 
2.45.2

[PATCHv3 2/2] nvme: add test for writing to file-ns just after disabling it

[Nilay Shroff]

This is a regression test for commit 505363957fad ("nvmet: fix nvme
status code when namespace is disabled")[1].

This test creates a regular file backed loop target namespace, disables
the asynchronous event notification for ns-changed events and then write
to the namespace just after disabling it.

[1] https://lore.kernel.org/linux-nvme/tqcy3sveity7p56v7ywp7ssyviwcb3w4623cnxj3knoobfcanq@yxgt2mjkbkam/

Signed-off-by: Nilay Shroff <nilay@linux.ibm.com>
---
 tests/nvme/055     | 117 +++++++++++++++++++++++++++++++++++++++++++++
 tests/nvme/055.out |   2 +
 2 files changed, 119 insertions(+)
 create mode 100755 tests/nvme/055
 create mode 100644 tests/nvme/055.out

diff --git a/tests/nvme/055 b/tests/nvme/055
new file mode 100755
index 0000000..e53a759
--- /dev/null
+++ b/tests/nvme/055
@@ -0,0 +1,117 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-3.0+
+# Copyright (C) 2024 Nilay Shroff
+#
+# Regression test for commit 505363957fad ("nvmet: fix nvme
+# status code when namespace is disabled"). We may hit this
+# regression using nvme/052 however that depends on the udev
+# rules which triggers the IO to a ns after it's enabled.
+# The udev rules could change from one distro to another and
+# it's not in our control. So we implemented this test where
+# we have full control of the test.
+
+. tests/nvme/rc
+
+DESCRIPTION="Test nvme write to a loop target ns just after ns is disabled"
+
+QUICK=1
+
+requires() {
+	_nvme_requires
+	_have_loop
+	_require_nvme_trtype_is_loop
+	_have_kernel_option DEBUG_ATOMIC_SLEEP
+}
+
+set_conditions() {
+	_set_nvme_trtype "$@"
+}
+
+nvmf_disable_ns_change_aen() {
+
+	local disk="$1"
+	local timeout="5"
+
+	# get async event configuration value
+	aen_conf=$(nvme get-feature "$disk" --feature-id=0xB | cut -d':' -f3)
+
+	# mask async ns change event notfication
+	mask_ns_change=$(( aen_conf & 0xFEFF ))
+
+	# stop receiving aen event for ns change from target
+	nvme set-feature "$disk" --feature-id=0xB \
+			--value="$mask_ns_change" 2>>"$FULL" 1>&2
+
+	start_time=$(date +%s)
+
+	while true; do
+		aen_conf=$(nvme get-feature "$disk" \
+			--feature-id=0xB | cut -d':' -f3)
+
+		# Validate whether ns-changed notification is masked or not;
+		# If it's already masked then break and return success
+		if ! (( aen_conf & 0x100 )); then
+			break
+		fi
+
+		sleep 1
+
+		end_time=$(date +%s)
+
+		if (( end_time - start_time > timeout )); then
+			echo "can't mask ns-changed async event "\
+				"notification within $timeout seconds"
+			return 1
+		fi
+	done
+
+	return 0
+}
+
+test() {
+	echo "Running ${TEST_NAME}"
+
+	local iteration=1 i=0
+
+	_setup_nvmet
+
+	_nvmet_target_setup
+
+	_nvme_connect_subsys
+
+	subsys_path="${NVMET_CFS}/subsystems/${def_subsysnqn}"
+	ns_path="${subsys_path}/namespaces/${def_nsid}"
+
+	while (( i < iteration )); do
+		# Wait until async request is processed and default ns
+		# is created
+		if ! _nvmf_wait_for_ns "${def_subsys_uuid}" created; then
+			echo "FAIL"
+			break
+		fi
+
+		disk="/dev/$(_find_nvme_ns "${def_subsys_uuid}")"
+
+		# Mask ns change async event notification from target. It
+		# would ensure that when we disable the target ns, the host
+		# would not receive ns removal notification from target and
+		# so from host we can attempt writing to a disabled ns.
+		if ! nvmf_disable_ns_change_aen "${disk}"; then
+			echo "FAIL"
+			break
+		fi
+
+		# disable target namespace and write to it
+		echo 0 > ${ns_path}/enable
+		nvme write --start-block=1 --block-count=0 \
+			--data-size=512 --data="/dev/urandom" "$disk" 2>>"$FULL"
+
+		((i++))
+	done
+
+	_nvme_disconnect_subsys >> "$FULL" 2>&1
+
+	_nvmet_target_cleanup
+
+	echo "Test complete"
+}
diff --git a/tests/nvme/055.out b/tests/nvme/055.out
new file mode 100644
index 0000000..bbefc28
--- /dev/null
+++ b/tests/nvme/055.out
@@ -0,0 +1,2 @@
+Running nvme/055
+Test complete
-- 
2.45.2

Re: [PATCHv3 1/2] nvme/052: move nvmf_wait_for_ns() to common/nvme

[Daniel Wagner]

On Fri, Dec 06, 2024 at 04:48:07PM +0530, Nilay Shroff wrote:
> In a preparation for the use of nvmf_wait_for_ns function from
> other nvme test script, move nvmf_wait_for_ns from nvme/052 to
> common/nvme file so that we can reuse the nvmf_wait_for_ns from
> multiple test scripts.
> 
> Reviewed-by: Chaitanya Kulkarni <kch@nvidia.com>
> Signed-off-by: Nilay Shroff <nilay@linux.ibm.com>

Reviewed-by: Daniel Wagner <dwagner@suse.de>

Re: [PATCHv3 2/2] nvme: add test for writing to file-ns just after disabling it

[Daniel Wagner]

On Fri, Dec 06, 2024 at 04:48:08PM +0530, Nilay Shroff wrote:
> +		# disable target namespace and write to it
> +		echo 0 > ${ns_path}/enable
> +		nvme write --start-block=1 --block-count=0 \
> +			--data-size=512 --data="/dev/urandom" "$disk" 2>>"$FULL"
>
I haven't checked if this was already discussed. Just wondering what
--block-count=0 is supposed to. Is it the 0 based thing?

Rest looks good. So feel free to add

Reviewed-by: Daniel Wagner <dwagner@suse.de>

Re: [PATCHv3 2/2] nvme: add test for writing to file-ns just after disabling it

[Nilay Shroff]

On 12/6/24 18:02, Daniel Wagner wrote:
> On Fri, Dec 06, 2024 at 04:48:08PM +0530, Nilay Shroff wrote:
>> +		# disable target namespace and write to it
>> +		echo 0 > ${ns_path}/enable
>> +		nvme write --start-block=1 --block-count=0 \
>> +			--data-size=512 --data="/dev/urandom" "$disk" 2>>"$FULL"
>>
> I haven't checked if this was already discussed. Just wondering what
> --block-count=0 is supposed to. Is it the 0 based thing?
> 
Yes it's zero based field.

# nvme write --help 
Usage: nvme write <device> [OPTIONS]

Copy from provided data buffer (default
buffer is stdin) to specified logical blocks on the given device.

Options:
  [  --verbose, -v ]                    --- Increase output verbosity
  <snip>
  [  --block-count=<NUM>, -c <NUM> ]    --- number of blocks (zeroes based)
                                            on device to access

> Rest looks good. So feel free to add
> 
> Reviewed-by: Daniel Wagner <dwagner@suse.de>
> 

Re: [PATCHv3 0/2] add nvme test for creating sleep while atomic kernel BUG

[Chaitanya Kulkarni]

On 12/6/24 03:18, Nilay Shroff wrote:
> Hi,
>
> There're two patches in this series. The first patch is a preparation patch
> for reusing a common function nvmf_wait_for_ns from multiple nvme test scripts.
> The second patch adds a new nvme regression[1] test for commit 505363957fad
> ("nvmet: fix nvme status code when namespace is disabled").
>
> For fixing regression, as suggested by Christoph[2], I would send a separate
> patch upstream.
>
> [1]https://lore.kernel.org/linux-nvme/tqcy3sveity7p56v7ywp7ssyviwcb3w4623cnxj3knoobfcanq@yxgt2mjkbkam/
> [2]https://lore.kernel.org/linux-nvme/20241022070252.GA11389@lst.de/


Looks good.

Reviewed-by: Chaitanya Kulkarni <kch@nvidia.com>

-ck

Re: [PATCHv3 0/2] add nvme test for creating sleep while atomic kernel BUG

[Shinichiro Kawasaki]

On Dec 06, 2024 / 16:48, Nilay Shroff wrote:
> Hi,
> 
> There're two patches in this series. The first patch is a preparation patch
> for reusing a common function nvmf_wait_for_ns from multiple nvme test scripts.
> The second patch adds a new nvme regression[1] test for commit 505363957fad
> ("nvmet: fix nvme status code when namespace is disabled").

I have applied them with a couple of minor amendments: fixed indents in the 1st
patch, and added local variable declarations to the 2nd patch. Thanks!

Re: [PATCHv3 1/2] nvme/052: move nvmf_wait_for_ns() to common/nvme

[Sagi Grimberg]

Reviewed-by: Sagi Grimberg <sagi@grimberg.me>

Re: [PATCHv3 2/2] nvme: add test for writing to file-ns just after disabling it

[Sagi Grimberg]

Reviewed-by: Sagi Grimberg <sagi@grimberg.me>